Reinforcement learning (RL) post-training with generative reward models (GenRM) has become a central recipe for improving large language models (LLMs), and it is also increasingly resource-intensive: policy rollout, GenRM-based reward evaluation, and policy training all compete for GPU resources. However, existing colocated and disaggregated systems rely on static resource allocation for these roles. Such allocation cannot adapt as each role's workload changes during RL and therefore leaves a role's resources underutilized. We present PivotRL, an RL post-training system that dynamically shifts resources among different roles according to their runtime demands. PivotRL introduces a resource-shiftable runtime built on a Global Parameter Storage, together with a unified mechanism that efficiently shifts role-specific context. A shift-aware scheduler enables sequence migration upon a shift, while a planner considers the aggregate throughput benefit and the shift overhead to determine when and how to shift. We evaluate PivotRL on a 128-GPU cluster using four model groups spanning dense and mixture-of-experts models. PivotRL enables resource shifting within seconds and improves end-to-end throughput by 1.81-2.05 times over colocated deployment and 1.50-1.95 times over disaggregated deployment, while preserving RL convergence.
Figures & tables
Figure 1. (a) Three stages of a typical RL post-training step. (b) Generative reward models (GenRMs) provide more fine-grained feedback by reasoning over the responses of the policy model.
Figure 2. Comparison of deployment paradigms. Both (a) colocated and (b) disaggregated deployments statically allocate resources, while (c) resource-shiftable deployment dynamically reallocates GPUs to match workload demand. Different colors indicate instantaneous stage load, and shaded regions indicate throughput headroom, where workload is substantially below the processing capacity of allocated GPUs.
Figure 3. The per-GPU throughput under different deployment paradigms on 32 GPUs, using Qwen2.5-7B as the policy model and Qwen3-30B-A3B as the GenRM. Shaded regions indicate the fraction of execution time spent at low per-GPU throughput.
Figure 4. Overview of PivotRL , which dynamically shifts resources across RL stages via a resource-shiftable runtime, shift-aware schedulers, and a planner with monitor and executor.
Figure 5. The GPS interface. PM: policy model. DP/PP/TP: data/pipeline/tensor parallelism. (a) Read : after a resource shift, newly added resources fetch the required model parameters from GPS and partition them according to the parallelism strategy. (b) Write : after each RL step, training GPUs update GPS with the latest policy model parameters. (c) Both Read and Write leverage all-to-all collectives.
Figure 6. The role-shifting mechanism. Role context is categorized into different types and handled accordingly.
Figure 7. Leveraging the shared waiting queue, sequence migration for both shift down (left) and shift up (right) requires only two operations: Abort and Dispatch .
Figure 8. The five-step procedure of the planner. In this example, each rollout replica holds 1 GPU, while each reward replica holds 4 GPUs. NRoleSeq denotes the total number of sequences processed by the role, TRole denotes its current throughput, and ϕ denotes the penalty.
Figure 9. The end-to-end throughput of each deployment baseline and PivotRL for different model groups. The values in parentheses indicate the speedup of PivotRL over the corresponding baseline, while ± indicates the variation.
Policy Model
GenRM
Qwen2.5-1.5B (TP=1)
GLM-Z1-9B (TP=1)
Qwen2.5-7B (TP=1)
Qwen3-30B-A3B (TP, EP=4)
Qwen2.5-32B (TP=4)
Qwen3-30B-A3B (TP, EP=4)
Qwen2.5-72B (TP=8)
Qwen3.5-122B-A10B (TP, EP=8)
Table 1. The settings of each model group. TP: tensor parallelism, EP: expert parallelism.
Figure 10. Throughput comparison between the disaggregated deployment and PivotRL in Qwen2.5-7B & Qwen3-30B-A3B. Top to bottom: rollout, reward, and overall throughput (Equation 1 ). Resource shifts triggered by PivotRL are marked, with the GPU allocations annotated for each role (RO: Rollout, RM: Reward, TR: Training). The pink interval indicates that PivotRL is in training (i.e., TR is not 0).
Figure 11. Ablation study of shift strategies and system components across two model groups.
Qwen2.5-1.5B & GLM-Z1-9B
GPU count
16
32
64
128
Pre-processing (s)
0.06±0.19
0.18±0.41
0.47±0.90
0.39±0.71
Construction (s)
0.15±0.14
0.32±0.28
0.68±0.53
1.34±1.21
Simulation (s)
0.10±0.13
0.24±0.17
0.56±0.52
1.22±0.91
Total (s)
0.31±0.27
0.74±0.52
1.71±1.17
2.95±1.67
Table 2. The time breakdown of the planner for Qwen2.5-1.5B & GLM-Z1-9B and Qwen2.5-7B & Qwen3-30B-A3B.
Figure 12. Overhead of shift up and shift down for inference and training across different model sizes.
Figure 13. Overhead of shift up and shift down for Qwen2.5-7B and Qwen3-30B-A3B under different GPU counts.
Qwen2.5-1.5B & GLM-Z1-9B
GPU count
16
32
64
128
Abort (s)
0.07±0.13
0.07±0.10
0.10±0.07
0.12±0.08
Dispatch (s)
0.72±2.99
1.03±2.75
0.53±1.33
0.55±1.00
Re-prefill (s)
0.15±0.24
0.09±0.08
0.12±0.10
0.09±0.05
Total (s)
0.94±3.06
1.18±2.79
0.75±1.36
0.76±1.01
Table 3. The per-sequence time breakdown during migration for Qwen2.5-1.5B & GLM-Z1-9B and Qwen2.5-7B & Qwen3-30B-A3B.
Figure 14. The throughput and average number of shifts taken per step with different H .
Figure 15. RL post-training convergence of PivotRL , colocated, and disaggregated deployments. Reward and AIME@1 are plotted against wall-clock time (left) and RL steps (right).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 16. The all-to-all collective underlying Read ( Write is symmetric). In this example, GPU0 and GPU1 reside on CPU2, which can reach both GPUs over PCIe, while other CPUs must reach them via RDMA. ❶ The GPS layout and the caller’s target layout are refined to their common granularity, yielding atomic slices S1 – S4 . ❷ We sweep the (slice, receiver) demands from top to bottom, assigning each to the candidate holder with the smallest accumulated estimated latency at that point (shaded). The cells show the accumulated latency at the time of each assignment. ❸ The resulting plan is balanced and cached for reuse.
Policy Model
GenRM
Deployment
GPUs for Each Role
Rollout
Reward
Training
Qwen2.5-1.5B (TP=1)
GLM-Z1-9B (TP=1)
Colocated
32
Disaggregated
8
8
16
Rollout-Reward Colocated
16
16
PivotRL
0–32
0–32
0 or 16
Qwen2.5-7B (TP=1)
Qwen3-30B-A3B (TP, EP=4)
Colocated
32
Appendix
Table 4. Resource configurations for different model groups under various deployment paradigms. In the colocated paradigm, all three roles (rollout, reward, and training) share the same GPUs, so the total number is shown once across the three role columns. In the disaggregated paradigm, the numbers in the Rollout, Reward, and Training columns indicate the GPUs allocated to each role. In the rollout-reward colocated mode, the number spanning the Rollout and Reward columns denotes the GPUs shared by these two roles. For PivotRL , since resources can be dynamically shifted across roles, each column shows the range of GPUs that can be allocated to the corresponding role. Since training is shifted as a whole, its allocation can only be zero or the full training workload.
Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.
Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time. However, adapting a job whose models share GPUs entails significant challenges: deciding whether a new plan is worth the transition cost, reusing the job's distributed state, and coordinating GPU transfers across models and stages. Nereus targets these challenges as a cost-aware runtime that adapts RL post-training jobs into efficient execution plans. Its low-overhead controller selects a memory-feasible global plan and admits the transition using a cost model calibrated against the running job. To estimate and execute a transition, Nereus represents the distributed state of each replica of a model-stage (one model in one stage) as an Elastic Model Unit. It then employs a global transition graph to order the transformations and GPU transfers of these units. In a trace built from real data, online TP/PP adaptation reduces average step latency by 27.7% relative to the initial fixed TP/PP layout with DP scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions consume 0.079% of total run time. Nereus improves end-to-end 8B PPO throughput by 2.14--7.27× over OpenRLHF and by 1.10--1.47× over Verl across diverse clusters.
Songlin Jiang, Tuo Shi, Sitong Zhang +3
Aalto University · Shenzhen University of Advanced Technology · Zhejiang University
Reinforcement learning (RL) has emerged as a standard post-training paradigm for shaping large language models (LLMs) into capable agents. In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads that expose two fundamental challenges. First, due to the long-tailed response distribution, a small fraction of trajectories dominates rollout makespan.Second, rollout and training differ in their compute patterns, memory demands, and sensitivity to sequence length. As the policy evolves, shifts in the workload distribution further change their relative resource demands, making it difficult to maintain balanced execution across the two stages. We present Libra, an adaptive runtime for agentic RL post-training with two complementary components: (1) intra-stage scheduling via a Causality-Guided Bucket Scheduler that routes requests across execution buckets with different parallelism configurations, reducing delays from rollout stragglers; and (2) cross-stage coordination that dynamically reallocates workers between rollout and training as the workload changes. It moves workers between the two stages through a non-blocking protocol without interrupting ongoing training. Evaluated on a 48x NVIDIA A800 GPU cluster and a 160x Ascend 910B3 NPU cluster across three agentic benchmarks, Libra achieves up to 4.2x higher throughput and up to 2.7x faster reward convergence
Kaiwen Chen, Xin Tan, Jingzong Li +6
The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher
Reinforcement learning (RL) has been a critical paradigm for improving large language model (LLM) reasoning and agentic capabilities, but its efficiency remains a major bottleneck. Existing RL systems commonly rely on asynchronous execution to overlap rollout generation with policy optimization for enhanced efficiency. However, this approach faces inherent limitations: mild policy staleness offers only limited training efficiency gains, whereas aggressive asynchrony can degrade model quality and yields diminishing efficiency gains. The dynamic nature of LLM RL and multi-turn agentic workloads further challenges RL system design, as resource demands shift over time, making static resource allocation and generation strategy inefficient. This paper presents ReFlow, a system-algorithm co-designed RL framework for efficient LLM post-training. Its key insight is that, unlike existing RL frameworks which consume each trajectory only once, experience replay of recent trajectories enables effective sample reuse, thereby amortizing the high cost of rollout generation without incurring excessive policy staleness. To fully exploit this opportunity, ReFlow adopts a decoupled and adaptive architecture with three key designs: a dual replay data cycle supports efficient experience reuse while balancing training throughput and optimization stability; an elastic resource pool enables dynamic resource placement to accommodate evolving RL workloads; an adaptive scheduler coordinates generation strategies and resource allocation to improve resource utilization and boost generation throughput. Experiments across representative single-turn reasoning and multi-turn agentic RL tasks show that ReFlow improves training throughput by up to 2.06× over state-of-the-art synchronous and asynchronous RL systems, while maintaining comparable or better model performance.
Yukun Tian, Junwei Su, Haofeng Xu +6
The University of Hong Kong · Alibaba Group · University of Science and Technology of China +1