RL Environment API Design: Tokens-In-Tokens-Out
Source: https://x.com (tweet) Author: unknown (RL infrastructure practitioner) Date: 2025-2026
Summary
Common bug in LLM-RL codebases: OpenAI-style messages parsed and retokenized at env boundaries causes token sequence inconsistencies. Solution: env API should use tokens-in-tokens-out signature with a Trajectory class ensuring context consistency between inference and training.
Key Claims
- Common bug: OpenAI message format → parsing → retokenizing → inconsistent token sequences
- The bug: inference-time context ≠ training-time context (silent failure)
- Fix: env API with tokens-in-tokens-out signature
- Trajectory class: ensures consistency between inference and training contexts
- This is not a subtle bug — it caused major issues in multi-turn, multi-env RL
Connection to Other Sources
Direct implementation detail for RL Infrastructure. Relates to RL Collapse (training-inference gap).
Concepts
- RL Infrastructure — API design as correctness concern