TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning
Organizations: University of Washington, Seattle, Washington, USA · Brown University, Providence, Rhode Island, USA · UC Berkeley, Berkeley, California, USA · New York University, New York, New York, USA
Abstract
Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes. To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques: (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete and (2) dynamically migrating rollouts between inference engines to boost decoding throughput. TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperforming all evaluated on-policy baselines and achieving up to 97% of optimal throughput.
Figures & tables
| Idle | Ineff. | Misc. | Of | ||
|---|---|---|---|---|---|
| Application | GPUs | loss | loss | idle | optimal |
| DAPO Math | 8 H200 | 15.2 | 8.4 | 1.3 | 75.1 |
| R1-Distill-8B | 4 H100 | 12.7 | 3.9 | 0.2 | 83.2 |
| Text2SQL | 8 H200 | 30.0 | 9.8 | 3.6 | 56.6 |
| Qwen2.5-Coder-7B | 4 L40S | 18.5 | 2.5 | 0.6 | 78.4 |
| Application | GPUs | Colocated | TIDE |
|---|---|---|---|
| DAPO Math | 8 H200 | 75.1% | 87.0% |
| R1-Distill-8B | 4 H100 | 83.2% | 97.4% |
| Text2SQL | 8 H200 | 56.6% | 77.2% |
| Qwen2.5-Coder-7B | 4 L40S | 78.4% | 95.1% |
| 8 H200 | 4 H100 | |
|---|---|---|
| System | ( TIDE-Megatron ) | ( TIDE-FSDP ) |
| Colocated | 689.1 (1.00 ) | 1142.6 (1.00 ) |
| + Repurposing | 666.8 (1.03 ) | 1025.5 (1.11 ) |
| + Load balancing | 617.9 (1.12 ) | N/A † |
| + Migration | 572.5 (1.20 ) | 1052.4 (1.09 ) |
| + KV veto | 576.9 (1.19 ) | 1002.9 (1.14 ) |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
RL Post-Training
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
See all 460 papers