# Improving RL Training with Token ID Recording Enhancing AI model performance through token recording system. - **Source:** [Amazon Science](https://www.amazon.science) - **Published:** 2026-07-09 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/785530a6-ccb6-42f5-81b0-3563c9aef82d ## Reinforcement Learning and Token Recording Reinforcement learning (RL) improves the performance of language models in multi-step tasks. However, accurate recording of tokens generated by the model during training is crucial. These records are essential for correctly updating the model's parameters. ## Tokenizer and Token IDs A tokenizer converts text into token IDs for a specific model. This process is deterministic, and even minor format changes can affect token IDs. Therefore, complete token recording during training is of great importance. ## Rollout and Behavior Policy A rollout is a recorded attempt at a task, and the version of the model that produced it is called the behavior policy. RL training should be optimized against the context seen by the behavior policy. ## The Problem of Retokenization Drift Retokenization drift is a mismatch that occurs when a tokenizer is rerun. This can degrade the training signal and cause the model to be trained against a context different from what it actually experienced. ## The Role of Turnstile Proxy Turnstile is a proxy that ensures complete token recording during RL training. It records the token IDs, log probabilities, and loss masks generated by the model, which are used during training. ## How Turnstile Works Turnstile supports the OpenAI Chat Completions API used by modern agent harnesses. The harness is directed to Turnstile, and the proxy records token information while forwarding each request to the backend. ## The Importance of Token Recording Token recording enhances the accuracy of RL training. These records represent the tokens actually generated by the model, and this information is used to optimize the model's parameters during training. ## RL Training with Turnstile Turnstile improves model performance by providing token records during RL training. The proxy accurately records the tokens used during training and transfers this information to the training stack. ## Benefits of Turnstile Turnstile enhances model performance by providing token records during RL training. The proxy accurately records the tokens used during training and transfers this information to the training stack. ## Accuracy in RL Training Accuracy in RL training is achieved by recording the tokens actually generated by the model. Turnstile provides these records, ensuring that the model's parameters are optimized correctly during training. ## Conclusion and Significance Turnstile enhances model performance by providing token records during RL training. The proxy accurately records the tokens used during training and transfers this information to the training stack. This ensures that the model's parameters are optimized more accurately. ## Sources covering this story - https://www.amazon.science/blog/capturing-token-ids-during-agentic-interactions-for-better-reinforcement-learning — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._