Efficient Resource-Shiftable RL Post-training with Generative Rewards

Yanfeng Zhao; Haoyang Li; Sheng Lin; Yuming Zhou; Jiawei Jiang; Bin Cui; Fangcheng Fu

cs.LGOct 12, 2026

Efficient Resource-Shiftable RL Post-training with Generative Rewards

Authors: Yanfeng Zhao, Haoyang Li, Sheng Lin, Yuming Zhou, Jiawei Jiang, Bin Cui, Fangcheng Fu

Organizations: Shanghai Jiao Tong University · Peking University · Central China Normal University

Abstract

Reinforcement learning (RL) post-training with generative reward models (GenRM) has become a central recipe for improving large language models (LLMs), and it is also increasingly resource-intensive: policy rollout, GenRM-based reward evaluation, and policy training all compete for GPU resources. However, existing colocated and disaggregated systems rely on static resource allocation for these roles. Such allocation cannot adapt as each role's workload changes during RL and therefore leaves a role's resources underutilized. We present PivotRL, an RL post-training system that dynamically shifts resources among different roles according to their runtime demands. PivotRL introduces a resource-shiftable runtime built on a Global Parameter Storage, together with a unified mechanism that efficiently shifts role-specific context. A shift-aware scheduler enables sequence migration upon a shift, while a planner considers the aggregate throughput benefit and the shift overhead to determine when and how to shift. We evaluate PivotRL on a 128-GPU cluster using four model groups spanning dense and mixture-of-experts models. PivotRL enables resource shifting within seconds and improves end-to-end throughput by 1.81-2.05 times over colocated deployment and 1.50-1.95 times over disaggregated deployment, while preserving RL convergence.

Figures & tables

Figure 1. (a) Three stages of a typical RL post-training step. (b) Generative reward models (GenRMs) provide more fine-grained feedback by reasoning over the responses of the policy model.

Figure 2. Comparison of deployment paradigms. Both (a) colocated and (b) disaggregated deployments statically allocate resources, while (c) resource-shiftable deployment dynamically reallocates GPUs to match workload demand. Different colors indicate instantaneous stage load, and shaded regions indicate throughput headroom, where workload is substantially below the processing capacity of allocated GPUs.

Figure 3. The per-GPU throughput under different deployment paradigms on 32 GPUs, using Qwen2.5-7B as the policy model and Qwen3-30B-A3B as the GenRM. Shaded regions indicate the fraction of execution time spent at low per-GPU throughput.

Figure 4. Overview of PivotRL , which dynamically shifts resources across RL stages via a resource-shiftable runtime, shift-aware schedulers, and a planner with monitor and executor.

Figure 5. The GPS interface. PM: policy model. DP/PP/TP: data/pipeline/tensor parallelism. (a) Read : after a resource shift, newly added resources fetch the required model parameters from GPS and partition them according to the parallelism strategy. (b) Write : after each RL step, training GPUs update GPS with the latest policy model parameters. (c) Both Read and Write leverage all-to-all collectives.

Figure 6. The role-shifting mechanism. Role context is categorized into different types and handled accordingly.

Figure 7. Leveraging the shared waiting queue, sequence migration for both shift down (left) and shift up (right) requires only two operations: Abort and Dispatch .

Figure 8. The five-step procedure of the planner. In this example, each rollout replica holds 1 GPU, while each reward replica holds 4 GPUs. $N^{\text{Seq}}_{\text{Role}}$ denotes the total number of sequences processed by the role, $T_{\text{Role}}$ denotes its current throughput, and $\phi$ denotes the penalty.

Figure 9. The end-to-end throughput of each deployment baseline and PivotRL for different model groups. The values in parentheses indicate the speedup of PivotRL over the corresponding baseline, while

\pm

indicates the variation.

Policy Model	GenRM
Qwen2.5-1.5B (TP=1)	GLM-Z1-9B (TP=1)
Qwen2.5-7B (TP=1)	Qwen3-30B-A3B (TP, EP=4)
Qwen2.5-32B (TP=4)	Qwen3-30B-A3B (TP, EP=4)
Qwen2.5-72B (TP=8)	Qwen3.5-122B-A10B (TP, EP=8)

Table 1. The settings of each model group. TP: tensor parallelism, EP: expert parallelism.

Figure 10. Throughput comparison between the disaggregated deployment and PivotRL in Qwen2.5-7B & Qwen3-30B-A3B. Top to bottom: rollout, reward, and overall throughput (Equation 1 ). Resource shifts triggered by PivotRL are marked, with the GPU allocations annotated for each role (RO: Rollout, RM: Reward, TR: Training). The pink interval indicates that PivotRL is in training (i.e., TR is not 0).

Figure 11. Ablation study of shift strategies and system components across two model groups.

Qwen2.5-1.5B & GLM-Z1-9B
GPU count	16	32	64	128
Pre-processing (s)	$0.06\,{\scriptstyle\pm 0.19}$	$0.18\,{\scriptstyle\pm 0.41}$	$0.47\,{\scriptstyle\pm 0.90}$	$0.39\,{\scriptstyle\pm 0.71}$
Construction (s)	$0.15\,{\scriptstyle\pm 0.14}$	$0.32\,{\scriptstyle\pm 0.28}$	$0.68\,{\scriptstyle\pm 0.53}$	$1.34\,{\scriptstyle\pm 1.21}$
Simulation (s)	$0.10\,{\scriptstyle\pm 0.13}$	$0.24\,{\scriptstyle\pm 0.17}$	$0.56\,{\scriptstyle\pm 0.52}$	$1.22\,{\scriptstyle\pm 0.91}$
Total (s)	$0.31\,{\scriptstyle\pm 0.27}$	$0.74\,{\scriptstyle\pm 0.52}$	$1.71\,{\scriptstyle\pm 1.17}$	$2.95\,{\scriptstyle\pm 1.67}$

Table 2. The time breakdown of the planner for Qwen2.5-1.5B & GLM-Z1-9B and Qwen2.5-7B & Qwen3-30B-A3B.

Figure 12. Overhead of shift up and shift down for inference and training across different model sizes.

Figure 13. Overhead of shift up and shift down for Qwen2.5-7B and Qwen3-30B-A3B under different GPU counts.

Qwen2.5-1.5B & GLM-Z1-9B
GPU count	16	32	64	128
Abort (s)	$0.07\,{\scriptstyle\pm 0.13}$	$0.07\,{\scriptstyle\pm 0.10}$	$0.10\,{\scriptstyle\pm 0.07}$	$0.12\,{\scriptstyle\pm 0.08}$
Dispatch (s)	$0.72\,{\scriptstyle\pm 2.99}$	$1.03\,{\scriptstyle\pm 2.75}$	$0.53\,{\scriptstyle\pm 1.33}$	$0.55\,{\scriptstyle\pm 1.00}$
Re-prefill (s)	$0.15\,{\scriptstyle\pm 0.24}$	$0.09\,{\scriptstyle\pm 0.08}$	$0.12\,{\scriptstyle\pm 0.10}$	$0.09\,{\scriptstyle\pm 0.05}$
Total (s)	$0.94\,{\scriptstyle\pm 3.06}$	$1.18\,{\scriptstyle\pm 2.79}$	$0.75\,{\scriptstyle\pm 1.36}$	$0.76\,{\scriptstyle\pm 1.01}$

Table 3. The per-sequence time breakdown during migration for Qwen2.5-1.5B & GLM-Z1-9B and Qwen2.5-7B & Qwen3-30B-A3B.

Figure 14. The throughput and average number of shifts taken per step with different

\mathcal{H}

Figure 15. RL post-training convergence of PivotRL , colocated, and disaggregated deployments. Reward and AIME@1 are plotted against wall-clock time (left) and RL steps (right).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 16. The all-to-all collective underlying Read ( Write is symmetric). In this example, GPU0 and GPU1 reside on CPU2, which can reach both GPUs over PCIe, while other CPUs must reach them via RDMA. ❶ The GPS layout and the caller’s target layout are refined to their common granularity, yielding atomic slices

S_{1}

–

S_{4}

. ❷ We sweep the (slice, receiver) demands from top to bottom, assigning each to the candidate holder with the smallest accumulated estimated latency at that point (shaded). The cells show the accumulated latency at the time of each assignment. ❸ The resulting plan is balanced and cached for reuse.

Policy Model	GenRM	Deployment	GPUs for Each Role
			Rollout	Reward	Training
Qwen2.5-1.5B (TP=1)	GLM-Z1-9B (TP=1)	Colocated	32
		Disaggregated	8	8	16
		Rollout-Reward Colocated	16		16
		PivotRL	0–32	0–32	0 or 16
Qwen2.5-7B (TP=1)	Qwen3-30B-A3B (TP, EP=4)	Colocated	32

Appendix

Table 4. Resource configurations for different model groups under various deployment paradigms. In the colocated paradigm, all three roles (rollout, reward, and training) share the same GPUs, so the total number is shown once across the three role columns. In the disaggregated paradigm, the numbers in the Rollout, Reward, and Training columns indicate the GPUs allocated to each role. In the rollout-reward colocated mode, the number spanning the Rollout and Reward columns denotes the GPUs shared by these two roles. For PivotRL , since resources can be dynamically shifted across roles, each column shows the range of GPUs that can be allocated to the corresponding role. Since training is shifted as a whole, its allocation can only be zero or the full training workload.

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 426 papers

Nereus: Adaptive Parallelism for LLM Post-Training
Sep 28, 2026Songlin Jiang, Tuo Shi, Sitong Zhang +3LLM Inference Scheduling RL Post-Training
Aalto University · Shenzhen University of Advanced Technology · Zhejiang University
Libra: Efficient Resource Management for Agentic RL Post-Training
Jun 2, 2026Kaiwen Chen, Xin Tan, Jingzong Li +6Agentic RL Reinforcement Learning
The Chinese University of Hong Kong · The Hang Seng University of Hong Kong · Independent Researcher
ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
Oct 15, 2026Yukun Tian, Junwei Su, Haofeng Xu +6RL Post-Training RL for Language Model Reasoning
The University of Hong Kong · Alibaba Group · University of Science and Technology of China +1

Sep 28, 2026J/K move · Enter open · S save

Nereus: Adaptive Parallelism for LLM Post-Training

Songlin Jiang, Tuo Shi, Sitong Zhang, Zeke Wang+2

Aalto University · Shenzhen University of Advanced Technology · Zhejiang University

LLM Inference Scheduling RL Post-Training LLM Training GPU-Accelerated RL

Open paper AskPDF arXiv

Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time. However, adapting a job whose models share GPUs entails significant challenges: deciding whether a new plan is worth the transition cost, reusing the job's distributed state, and coordinating GPU transfers across models and stages. Nereus targets these challenges as a cost-aware runtime that adapts RL post-training jobs into efficient execution plans. Its low-overhead controller selects a memory-feasible global plan and admits the transition using a cost model calibrated against the running job. To estimate and execute a transition, Nereus represents the distributed state of each replica of a model-stage (one model in one stage) as an Elastic Model Unit. It then employs a global transition graph to order the transformations and GPU transfers of these units. In a trace built from real data, online TP/PP adaptation reduces average step latency by 27.7% relative to the initial fixed TP/PP layout with DP scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions consume 0.079% of total run time. Nereus improves end-to-end 8B PPO throughput by 2.14--7.27

\times

over OpenRLHF and by 1.10--1.47

\times

over Verl across diverse clusters.

Abstract

RL Post-Training

Explore similar work

Nereus: Adaptive Parallelism for LLM Post-Training

Libra: Efficient Resource Management for Agentic RL Post-Training

ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training