TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning
Authors: Mathew Jacob, Morgan Borjigin-Wang, Kan Zhu, Po Hao Chen, Yi Pan, Andy Ruan, Nayani Modugula, Ethan Lavi, Luis Ceze, Malte Schwarzkopf, Akshay Narayan, Deepti Raghavan, Stephanie Wang
Organizations: University of Washington, Seattle, Washington, USA · Brown University, Providence, Rhode Island, USA · UC Berkeley, Berkeley, California, USA · New York University, New York, New York, USA
Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes. To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques: (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete and (2) dynamically migrating rollouts between inference engines to boost decoding throughput. TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperforming all evaluated on-policy baselines and achieving up to 97% of optimal throughput.
Figures & tables
Figure 1: DAPO Math on 4 × H100. Repurposing fills the idle time of the colocated baseline, and migration raises decode throughput through larger batches, together reducing the GPU-hours spent in the rollout phase.
Figure 2: Rollout lengths vary across workloads (a, c), and completion times vary across hardware (b, d).
Figure 3: Skew in inference-engine finish times across a 20-step colocated DAPO-Math run (DeepSeek-R1-Distill-8B, 8 × H200). In every step, most engines finish well before the last one, leaving their GPUs idle.
Figure 4: As rollouts complete, the batch shrinks and throughput falls far below peak while engines decode the last few stragglers.
Idle
Ineff.
Misc.
Of
Application
GPUs
loss
loss
idle
optimal
DAPO Math
8 × H200
15.2
8.4
1.3
75.1
R1-Distill-8B
4 × H100
12.7
3.9
0.2
83.2
Text2SQL
8 × H200
30.0
9.8
3.6
56.6
Qwen2.5-Coder-7B
4 × L40S
18.5
2.5
0.6
78.4
Table 1: Breakdown of each step’s GPU hours under colocated execution (%). The three losses and the fraction of optimal throughput sum to 100%. Misc. idle is GPU time outside both phases, spent offloading and onloading engine and trainer state at each phase switch.
Figure 5: TIDE overview mid-step. Each GPU hosts an inference and a training engine (one live, one asleep); routers run on the host. 1 Prompts arrive at the Rollout Router, which 2 dispatches them to inference GPUs that 3 report load. When a GPU’s live rollouts fall below B decided by the migration policy (§ 4.1 ), its requests are m re-dispatched and the GPU is 4 repurposed for training. 5 Completed rollouts stream to the Train Router (§ 4.2 ), which computes GRPO advantages and 6 dispatches batches. 7 Gradients accumulate; 8 after the last rollout, the optimizer steps and weights sync via host memory.
Figure 6: Simulated (§ 6.4 ) performance loss from the utilization-optimal at different values of B for a single step of DAPO-Math. Migrating too early (higher B ) and migrating too late (lower B ) both result in underutilization.
Figure 7: End-to-end speedup over fully synchronous colocated execution (baseline step time divided by system step time). † TLT’s speedup is relative to its own run without speculative decoding.
Application
GPUs
Colocated
TIDE
DAPO Math
8 × H200
75.1%
87.0%
R1-Distill-8B
4 × H100
83.2%
97.4%
Text2SQL
8 × H200
56.6%
77.2%
Qwen2.5-Coder-7B
4 × L40S
78.4%
95.1%
Table 2: Fraction of utilization-optimal throughput (optimal / system step time).
Figure 8: Per-step speedup of TIDE over colocated (points: steps; diamonds: overall; dashed: 1 × ). The median step is faster in every configuration, with overall speedups of 1.16–1.36 × .
8 × H200
4 × H100
System
( TIDE-Megatron )
( TIDE-FSDP )
Colocated
689.1 (1.00 × )
1142.6 (1.00 × )
+ Repurposing
666.8 (1.03 × )
1025.5 (1.11 × )
+ Load balancing
617.9 (1.12 × )
N/A †
+ Migration
572.5 (1.20 × )
1052.4 (1.09 × )
+ KV veto
576.9 (1.19 × )
1002.9 (1.14 × )
Table 3: Ablation on DAPO Math: seconds per step, with speedup over colocated in parentheses. Each row adds one mechanism to the row above; bold marks the best per column.
Figure 9: Simulated performance of fixed- B policies and TIDE ’s adaptive policy, relative to an oracle that picks the best B every step (higher is better). Fixed policies set B to the given fraction of the initial per-GPU batch size. The best fixed B differs across workloads, and a poor choice costs much more on some (DAPO Math) than others (Text2SQL); TIDE is comparable to the best fixed policy without manual tuning.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Per-GPU timeline of a sampled rollout step under the colocated baseline (top) and TIDE (bottom). As batch sizes shrink during the rollout tail, TIDE consolidates in-flight requests onto fewer engines and switches the freed GPUs to training. This removes most of the idle time that colocation leaves and shortens the step.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, RA2P selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6× over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58×, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.
Speculative decoding can substantially accelerate LLM inference, but realizing its benefits in practice is challenging due to evolving workloads. We present TIDE (Temporal Incremental Draft Engine), a serving-engine-native framework that integrates online draft adaptation directly into high-performance LLM inference systems. TIDE reuses target model's intermediate hidden states generated during inference as training signals for draft adaptation, thereby avoiding additional target model computation and serving-time overhead. It employs adaptive runtime control to activate speculation and draft model training only when beneficial. TIDE exploits heterogeneous clusters by mapping inference and training to appropriate GPU classes. Across diverse real-world workloads, TIDE achieves up to 1.66× throughput over no-speculation baselines while recovering performance on misaligned workloads where static draft models degrade throughput. TIDE also reduces training time by up to 3.02× and storage requirements by 24× compared to existing draft training approaches, and improves system throughput by up to 1.22× on heterogeneous GPU clusters.
Large language models run every token through every layer, regardless of difficulty. We present TIDE, a post-training system that attaches tiny learned routers at periodic checkpoint layers and, at inference time, selects the earliest layer whose hidden state has converged for each token. TIDE requires no model retraining, works with any HuggingFace causal LM, auto-detects GPU architecture, and supports float32, float16, and bfloat16 through fused CUDA kernels. On an NVIDIA A100 with DeepSeek R1 Distill 8B, TIDE achieves 100% prefill exit rate (5% of tokens exit at layer 11, the remaining at layer 31), reduces prefill latency by 7.2%, and increases single-batch throughput by 6.6%. During autoregressive decoding, 98-99% of tokens exit early while the model correctly solves a multi-step math problem with 95 unique output tokens. On Qwen3 8B (36 layers), throughput improves by 8.1% at batch size 8. Calibration on 2,000 WikiText samples takes under 3 minutes and produces a ~4 MB router checkpoint. The system comprises 1,308 lines of Python and 1,081 lines of CUDA/C++ with 74 passing tests. Code: https://github.com/RightNow-AI/TIDE