Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

Niloofar Gholipour; Marcos Assuncao; Gursimran Singh; Timothy Yu; Rajkumar Buyya; Julien Gascon-Samson; Zhenan Fan; Yong Zhang

cs.AISep 21, 2026

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

Authors: Niloofar Gholipour, Marcos Assuncao, Gursimran Singh, Timothy Yu, Rajkumar Buyya, Julien Gascon-Samson, Zhenan Fan, Yong Zhang, +3 more

Organizations: École de technologie supérieure, Univ. of Québec, Canada · Huawei Technologies, Canada · The Univ. of Melbourne, Australia · Huawei Technologies, China

Abstract

Reasoning-oriented reinforcement learning enables large language models to solve mathematical, coding, and other multi-step tasks, but shifts a substantial portion of the training cost to rollout, where trajectories are generated for policy updates. Efficient rollout mechanisms are therefore essential to reduce this cost while maintaining the freshness, consistency, and statistical validity of training data. This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives. Based on this taxonomy, we analyze how different technique families address distinct sources of rollout inefficiency, examine opportunities and potential conflicts for combining them, identify gaps in the evaluation and reporting of efficiency gains, and discuss open challenges and future research directions.

Figures & tables

Figure 1. Phase-time breakdown of a synchronous RL training step.

Survey	Year	Scope	Rollout-efficiency coverage			Empirical analysis
Survey	Year	Scope	System lever	Algorithmic lever	Bottleneck mapping	Reported gains	Compos- ability	Metric standardization
Bai et al. ( bai2401beyond )	2024	Training (+inference)
Duan et al. ( Duan et al., 2024 )	2024	Training
Tie et al. ( Tie et al., 2025 )	2025	Training
Kumar et al. ( Kumar et al., 2025 )	2025	Training
Qu et al. ( Qu et al., 2025 )	2025	Inference

Table 1. Comparison of representative surveys and this work. Scope states each survey’s primary subject; parentheses mark a secondary subject treated in passing. full coverage; partial coverage; limited or no coverage.

Figure 2. Survey methodology. The taxonomy is constructed through four stages: search, screen, classify, and group.

Figure 3. Conceptual workflow of reasoning RL.

Method	System gain	Algorithmic gain	Scale	Task	Hardware	Q	Cost / requirement
A.1 Pipeline Decoupling (Problem A: idle resources from stage coupling — 2 of 11 shown)
AReaL ( Fu et al., 2026 )	tput $2.77\times$	—	1.5B–32B	math, code	—	$=$ / $+$	stale trajectories within lag bound
LlamaRL ( Wu et al., 2025b )	step $10.7\times$ (405B)	—	8B–405B	math	1,024 H100	$=$	1-step policy lag; AIPO correction
A.2 Resource-Aware Execution (Problem A: idle, mismatched, or over-provisioned hardware — 2 of 16 shown)
RLBoost ( Wu et al., 2025a )	tput $1.51$ – $1.97\times$	—	—	—	preempt. GPUs	—	needs preemptible capacity; interruption recovery
AReaL-DTE ( Peng et al., 2026 )	weight sync $6.8$ – $7.6\times$ ( $19.9\times$ cross-cluster)	—	8B; 30B-A3B	math, code, logic	8–16 H200	✓	$<2\%$ weights change/step; AdamW inversion; periodic full anchors

Table 2. Representative methods per family (2 of each; full 80-method comparison in Table 9 , Appendix F ). Q:

=

preserved,

+

improved,

\approx

comparable, ✓ exact/lossless; — not reported.

Figure 4. Mechanism taxonomy of rollout-efficiency methods.

Figure 5. Bottleneck taxonomy of rollout-efficiency methods.

	Decoup- ling	Resource- aware	Sched- uling	Partial rollout	Specu- lative	Rollout selection
Resource-aware	$(\times/\bullet^{\mathrm{m}}/\circ)^{{\dagger}}$	—	$\bullet$	$\bullet$	$(\bullet/\circ)^{{\dagger}}$	$\bullet$
Scheduling	$\times^{\mathrm{s}}$	$\bullet$	—	$\times$	$\times^{\mathrm{m}}$	$\bullet$
Partial rollout	$\circ^{\mathrm{s}}$	$\bullet$	$\times$	—	$\times$	$\circ$
Speculative	$\times$	$(\bullet/\circ)^{{\dagger}}$	$\times^{\mathrm{m}}$	$\times$	—	$\bullet$
Rollout selection	$\circ$	$\bullet$	$\bullet$	$\circ$	$\bullet$	—
Prompt filtering	$\bullet$	$\bullet$	$\circ$	$\bullet^{\mathrm{s}}$	$\times$	$\times$

Table 3. Pairwise composability of technique families, with evidence status.

Mechanism family	Methods	System gain	Algorithmic gain
A.1 Pipeline decoupling	17	(15/17)	(2/17)
A.2 Resource-aware execution	15
B.1 Scheduling and load balancing	7		(1/7)
B.2 Partial and early-stop rollout	7		(5/7)
B.3 Speculative decoding	12
C. 1 Rollout selection	5	(1/5)	(4/5)

Table 4. Which efficiency lever each mechanism family reports , from Table 9 . every method; some (fraction shown); none. I-PPO ( Shu et al., 2026 ) reports an unquantified claim and is counted on neither lever.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Deployment inference	RL rollout generation
Request arrival	Requests arrive online according to an externally determined and potentially bursty workload. Future load is generally not known exactly.	The trainer explicitly launches a rollout workload, typically a batch of $B$ tasks available at the beginning of the rollout phase.
Workload control	The serving system controls scheduling after requests arrive, but generally cannot choose which requests arrive or reorder them arbitrarily across long time scales.	The trainer controls task sampling, batching, ordering, and often the number of trajectories generated for each task.
Request recurrence	Requests from different users are generally treated as unrelated, and the system cannot assume that a particular request will appear again.	Tasks are sampled repeatedly from a training distribution or finite dataset across optimization iterations or epochs, allowing information from previous rollouts to guide later scheduling and allocation decisions.
Primary objective	Per-request latency, throughput, and service-level objectives are central; requests should generally begin and complete promptly.	Individual request latency is usually secondary. In synchronous training, the primary systems objective is reducing the makespan of the complete rollout workload that blocks the next policy update.
Scheduling freedom	Deliberately delaying an admitted request typically worsens user-visible latency or fairness.	Tasks may be reordered, delayed, assigned different resources, truncated, or occasionally dropped if doing so reduces overall rollout cost without harming the learning objective.
Workload predictability	Prompt lengths, output lengths, and future arrivals are only partially known before execution.	The batch composition is known before generation, and previous executions of the same or similar tasks can provide estimates of trajectory length, reward, or computational cost.

Appendix

Table 5. Deployment inference versus RL rollout generation.

Framework	Control	Placement	Main idea	Reported gain
HybridFlow / veRL ( Sheng et al., 2025 )	One controller between nodes, local controllers inside each node	Shared devices, hybrid engine	3D-HybridEngine: move the actor between training and generation layouts without a second copy of the weights	$1.53$ – $20.57\times$ throughput (range across configurations)
DISTFLOW ( Wang et al., 2025a )	Local controllers only; no central node	Either	Planner plus local data coordinator (cache and double buffer)	Up to $2.63\times$ ; near-linear to 512 GPUs
StreamRL ( Zhong et al., 2025 )	One controller over two services	Separate devices; can span datacenters	Streaming generation; separates pipeline idle time from length idle time	Up to $2.66\times$ (best configuration); $1.33\times$ cost-effectiveness
AsyncFlow ( Han et al., 2025 )	Service interfaces; engines can be swapped	Separate devices	TransferQueue: hand out samples as soon as they are ready	$1.59\times$ avg., up to $2.03\times$ (256 NPUs)

Appendix

Table 6. Training frameworks for RL post-training.

Method	Draft source	Drafter cost	Guarantee
BubbleSpec ( Xu et al., 2026c )	The policy itself, run in idle data-parallel gaps	No additional resources; uses idle GPU capacity	Exact match; fully synchronous
SPEC-RL ( Liu et al., 2025a )	Previous step’s trajectory for the same prompt	Checking only ( $\approx\!20$ s/step)	Suffix re-checked under the current policy
DAS ( Shao et al., 2026 )	Suffix tree over past completions	No training; runs on CPU	Exact match (verified); identical training curves
RhymeRL ( He et al., 2025 )	Suffix tree (HistoSpec)	No training; runs on CPU	RL procedure unchanged
WAR ( Xu et al., 2026d )	Suffix tree, used only at low load	No training; turned off when batching fills the GPUs	Synchronous execution maintained
Seer ( Qin et al., 2026 )	Tokens generated by sibling rollouts in the same group (grouped SD)	No model; online bookkeeping only	Synchronous and on-policy execution maintained

Appendix

Table 7. Speculative decoding for RL rollout.

Method	Signal	Cost of the signal	Tracking policy change	Decision
GRESO ( Zheng et al., 2026b )	Per-prompt reward history	Reuses training rewards; no extra model	Repeated re-estimation	Skip
MoPPS ( Qu et al., 2026 )	Bayesian estimate of prompt difficulty	Lightweight posterior updates	Posterior updated during training	Prioritize / skip
HIVE ( Wu et al., 2026 )	Informativeness estimates carried across the run	Bookkeeping only	Repeated re-estimation	Skip
AERO ( Zhang et al., 2026b )	Statistics already collected in earlier rollouts	No separate predictor	Statistics refresh with the run	Skip
VCRL ( Jiang et al., 2025 )	Group reward variance	Free from current rollouts	Replay revisits filtered prompts	Filter + replay
VIP ( Nguyen et al., 2026 )	Gaussian-process prediction of success probability	Predictor fit plus convex allocation solve	Budget reduced rather than removed	Allocate

Appendix

Table 8. Prompt filtering and selection methods.

Method	System gain	Algorithmic gain	Scale	Task	Hardware	Q	Cost / requirement
A.1 Pipeline Decoupling (Problem A: idle resources from stage coupling)
AReaL ( Fu et al., 2026 )	tput $2.77\times$	—	1.5B–32B	math, code	—	$=$ / $+$	stale trajectories within lag bound
LlamaRL ( Wu et al., 2025b )	step $10.7\times$ (405B)	—	8B–405B	math	1,024 H100	$=$	1-step policy lag; AIPO correction
DORA ( Hu et al., 2026a )	E2E $2.12\times$ ; rollout $8.2\times$	—	560B MoE	reasoning	4,096 acc.	—	trajectories span multiple policy versions
Laminar ( Sheng et al., 2026 )	train tput $5.48\times$	—	—	—	1,024 GPU	—	lag $s\leq 4$ ; relay-worker weight service
FlexMARL ( Jiang et al., 2026b )	speedup $7.3\times$ ; util. $5.6\times$	—	32B+14B	multi-agent	prod. cluster	$=$	async micro-batch coordination

Appendix

Table 9. Master comparison of surveyed rollout-efficiency methods. Q (quality outcome):

=

preserved,

+

improved,

\approx

comparable, ✓ exact/lossless guarantee; — not reported. Cost / requirement: what the method spends or assumes in exchange for its gain. Abbreviations: tput = throughput; E2E = end-to-end; util. = utilization; acc. = accelerators; disagg. = disaggregated; prod. = production; het. = heterogeneous; preempt. = preemptible; conv. = convergence; iter. = iteration; MoE = mixture-of-experts; SWE = software-engineering tasks; KV = key–value cache; OPD = on-policy distillation; ovh. = overhead; SD = speculative decoding; gen. = generation; quant. = quantization; summ. = summarization; pct. = percentile; sync = synchronous; ESS = effective sample size; LR = learning rate; IS = importance sampling; TIS = truncated importance sampling; W4A16/W8A8 = weight/activation bit widths; MTP = multi-token prediction; CP/PP = context/pipeline parallelism; TP = tensor parallelism; HBM = high-bandwidth memory; AR = autoregressive; VLM = vision–language model.

Explore similar work

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning
Jul 24, 2026Heyang Jiang, Henry Liu, Baharan MirzasoleimanReinforcement Learning With Verifiable Reward Reasoning Skills
University of California, Los Angeles
R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Jan 17, 2026Jingchu Wang, Bingbing Xu, Yige Yuan +4LLM Reasoning Strategies Frictive Policy Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore
Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts
Aug 13, 2025Maxime Heuillet, Yufei Cui, Boxing Chen +2Mathematical Reasoning Benchmarks Large Language Model Fine-Tuning
Universit´e Laval (IID), Canada · Mila - Qu´ebec AI Institute, Canada · Huawei Noah’s Ark Lab (Montreal Research Center), Canada +1

Jul 24, 2026, cs.AIJ/K move · Enter open · S save

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

Heyang Jiang, Henry Liu, Baharan Mirzasoleiman

University of California, Los Angeles

Reinforcement Learning With Verifiable Reward Reasoning Skills

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has emerged as a highly effective framework for improving LLM reasoning, with methods such as GRPO among its most successful instantiations. However, GRPO relies on repeated generation of long chain-of-thought rollouts. Training time scales with the number of rollouts, a large fraction of which are uninformative. Thus, GRPO is computationally expensive and unstable. To mitigate this, existing approaches either generate a larger pool of rollouts and filter the most informative prompts, or leverage historical signals for filtering at later stages of training. These strategies offer modest performance gains, but slow down the overall process. To address this, we propose VarIance Guided Online Rollout allocation (VIGOR) which instead of allocating a fixed rollout budget per example, begins with a small number of rollouts for all examples in a batch and iteratively allocates additional rollouts to those with the highest group reward variance until a fixed total rollout budget is reached. Theoretically, we show that under RLVR, reward variance controls the gradient magnitude, and derive VIGOR's closed-form speedup ratio over GRPO, which grows with refinement rounds under Pareto-distributed reward variance. Experiments on mathematical reasoning and coding tasks show that VIGOR reaches target accuracy with up to 2.3

\times

fewer rollouts on math, reaches GRPO's final coding full pass rate with 1.49

\times

fewer rollouts, and improves the coding average test pass rate by 3.4 points.

Abstract

Explore similar work

Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning

R2^22PO: Decoupling Rollout and Inference Policies for LLM Reasoning

Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning