Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Rohan Surana; Gagan Mundada; Xunyi Jiang; Chuhan Wang; Zhenwei Tang; Difan Jiao; Zihan Huang; Yuxin Xiong

cs.LGApr 8, 2026

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Authors: Rohan Surana, Gagan Mundada, Xunyi Jiang, Chuhan Wang, Zhenwei Tang, Difan Jiao, Zihan Huang, Yuxin Xiong,

Organizations: University of California, San Diego · University of Toronto · UIUC · University of Virginia ·

Abstract

Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajectory sampled from a prompt to termination, including intermediate reasoning steps and optional tool or environment interactions, determines the data the optimizer learns from, yet rollout design is often underreported. This survey provides an optimizer-agnostic view of rollout strategies for RL-based post-training of reasoning LLMs. We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR), a lifecycle taxonomy that decomposes rollout pipelines into four modular stages: Generate proposes candidate trajectories and topologies; Filter constructs intermediate signals via verifiers, judges, critics; Control allocates compute and makes continuation/branching/stopping decisions under budgets; and Replay retains and reuses artifacts across rollouts without weight updates, including self-evolving curricula that autonomously generate new training tasks. We complement GFCR with a criterion taxonomy of reliability, coverage, and cost sensitivity that characterizes rollout trade-offs. Using this framework, we synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement. We ground the framework with case studies in math, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks that evaluate skill induction, reuse, and cross-task transfer. Finally, we provide a diagnostic index that maps common rollout pathologies to GFCR modules and mitigation levers, alongside open challenges for building reproducible, compute-efficient, and trustworthy rollout pipelines.

Figures & tables

Figure 1: An overview of the rollout lifecycle and the Generate–Filter–Control–Replay (GFCR) decomposition. Rollout pipelines can be understood as modular choices about how trajectories are proposed, how intermediate signals are constructed, how compute is allocated under budgets, and what artifacts are retained and reused across rollouts.

Figure 2: GFCR as an end-to-end rollout system. Given prompts $x\sim\mathcal{D}$ and a compute budget $B$ , Generate samples a rollout group $\mathcal{T}(x)$ ; Filter maps each rollout to intermediate signals $\phi$ and training-facing supervision $S$ ; Control uses costs and signals to adapt continuation/pruning/resampling and decide what to store; and Replay retrieves/stores artifacts that condition future generation. The overall objective is to maximize expected utility $\mathbb{E}[S(\tau)]$ under compute constraints.

Module	Subcomponent	What it includes
Generate (G): How trajectories are proposed. Defines the proposal distribution + interaction structure.
§ 4.2	Topology & Interaction	Rollout structure and interaction patterns.
§ 4.3	Guidance & scaffolding	Prompt structure, rubrics, and auxiliary traces.
§ 4.4	Sampling & exploration	Decoding diversity and exploration signals.
Filter (F): Triage, scoring, and signal construction. Maps rollouts/prefixes to training-facing signals.
§ 5.2	Structural validity	Schema compliance, executability, and safety gates.

Table 1: GFCR taxonomy at a glance. Rollout behavior is assembled from four composable lifecycle modules.

Symbol	Meaning
$\mathcal{D}$	Distribution (dataset) over prompts/tasks.
$x$	Prompt/task instance sampled from $\mathcal{D}$ .
$\theta$	Parameters of the language model / policy.
$\pi_{\theta}(\cdot\mid s)$	Model policy over next output/action given state $s$ .
$u_{t}$	Model output/action at step $t$ (tokens and/or tool/action call).
$o_{t}$	Observation at step $t$ (tool output/environment response; empty for text-only).

Table 2: Global notation used throughout the paper. Module-specific notation is introduced in the corresponding GFCR sections.

Figure 3: Rollout criterion taxonomy (at a glance). We evaluate rollout strategies along three cross-cutting desiderata: reliability (trustworthy signals via verifiers or robust judges), coverage & informativeness (diverse candidates and disagreement/uncertainty), and cost sensitivity (value under compute budgets via value-per-cost and early stopping).

Reliability
Criterion	Description and Examples
Verifiable Outcomes	Executable / checkable rollouts yielding trustworthy supervision when verification exists (e.g., math or code): structural validity/executability gates, unit tests, symbolic checks, exact-match constraints.
Robust Judge Scoring	Stable evaluation when direct verification is unavailable (e.g., open-ended tasks): calibrated judging, consistency across prompt variants, and resistance to reward hacking.
Coverage & Informativeness
Diverse Candidates	Broad exploration over plausible solution strategies (structured, creative, iterative), avoiding near-duplicate rollouts and encouraging novelty.
Disagreement Signals	Informative uncertainty revealing hard-but-solvable cases: disagreement across rollouts or critics, conflicting rationales, and high-variance outcomes.

Table 3: Rollout criterion taxonomy (reference style). We summarize three cross-cutting desiderata for rollout strategies— reliability , coverage & informativeness , and cost sensitivity —and list concrete signals and mechanisms that operationalize each criterion.

Figure 4: Generate module design space. Rollout proposal mechanisms can be organized along three axes: (left) topology & interaction (single, group, tree/graph, and multi-turn/tool rollouts), (middle) guidance/scaffolding $z$ (ICL/rubrics, plans, reflection, adaptive guidance, tool augmentation), and (right) sampling & exploration configuration $\kappa_{G}$ (decoding knobs, uncertainty-aware allocation, partial rollouts with resumption, and sampling-only inference enhancements). Group candidates are typically scored downstream by Filter, and compute allocation may be coupled to Control.

Subcomponent	Design choice	Primary trade-off	Representative works
Topology & interaction ( $\mathsf{Topo}$ )
Topology	Linear (single rollout)	Simplicity $\uparrow$ ; within-prompt baseline $\times$ ; gradient variance $\uparrow$ .	( Le et al., 2022 ; Liu et al., 2023 ) .
Topology	Group ( $K$ rollouts)	Variance $\downarrow$ via within-prompt baselines; compute $\uparrow$ with $K$ .	( Shao et al., 2024 ; Guo et al., 2025b ; Yu et al., 2025a ) .
Topology	Tree/graph (branching prefixes)	Prefix reuse $\uparrow$ and targeted exploration; control/bookkeeping $\uparrow$ .	( Hou et al., 2025 ; Bamba et al., 2025 ) .
Interaction	Tool / environment loop	Grounded feedback $\uparrow$ ; non-stationarity/latency $\uparrow$ (env-dependent).	( Liu et al., 2023 ; Li et al., 2023 ; Jimenez et al., 2023 ) .
Guidance & scaffolding ( $z$ )

Table 4: Generate design space and primary trade-offs. Rows summarize rollout-construction choices used in representative methods, emphasizing the main efficiency/quality trade-off each choice introduces.

Figure 5: Filter module design space. Filter maps sampled rollouts (or rollout prefixes) into intermediate signals and optimizer-facing supervision. It can be organized along three components: (left) gating & verification (structural validity, schema/executability checks, deterministic correctness tests), (middle) process & comparative scoring (dense step-level process signals and relative judgments via ranking/judging), and (right) signal construction (learning-value diagnostics such as uncertainty/variance/entropy, and mapping raw outcomes into optimizer-ready targets such as masks/weights, advantages, or preference labels). Filter outputs are consumed by the optimizer and may also inform Control decisions.

Stage	Output	Typical mechanisms	Representative works
Gate	$m_{i}\in\{0,1\}$	Pre-semantic validity checks: parseability/normalization, executability, safety constraints; can act on rollouts or prefixes.	( Chen et al., 2025a ; Hugging Face, 2025 ; Liu et al., 2023 ; Le et al., 2022 ; Li et al., 2023 ) .
Eval	$r_{i}^{\mathrm{raw}}\in\mathbb{R}$	Hard semantic verification (unit tests, execution, exact-answer equivalence), process/step scoring (PRMs/step verifiers), or comparative ranking (LLM judge).	( Liu et al., 2023 ; Le et al., 2022 ; Yao et al., 2026 ; Ali et al., 2025 ; Zhang et al., 2025g ; Hendrycks et al., 2021 ; He et al., 2024 ; Lightman et al., 2023 ; Zheng et al., 2025b ; Song et al., 2025 ; Zhang et al., 2025m ; Zheng et al., 2023 ; Shi et al., 2025 ) .
Map	$(w_{i},\tilde{r}_{i},\tilde{A}_{i},\ell_{i})$	Convert raw signals into optimizer-facing supervision: masks/weights, shaped rewards, group-relative advantages, pairwise/listwise labels; often group-dependent.	( Mroueh, 2025 ; Zhang & Zuo, 2025 ; Shrivastava et al., 2025 ; Becker et al., 2025 ) .

Table 5: Filter as a signal pipeline (Gate

\rightarrow

Eval

\rightarrow

Map). This table instantiates Eqs. ( 8 )–( 10 ) with common rollout filters and the optimizer-facing artifacts they produce.

Figure 6: Control module design space. The control layer of rollout pipelines can be organized into four orthogonal components: (left) Instance & Budget Selection (prompt prioritization by difficulty/novelty/uncertainty; adaptive allocation of rollout slots, group size, depth/width, and token budgets); (middle) Rollout Dynamics Control (decoding configuration, partial rollouts with early exit, and branching–pruning policies for quality–efficiency trade-offs); (right-top) Data & Policy Control (on-/off-policy mixing, replay, and correctness-aware filtering); and (right-bottom) Execution & Systems Control (scheduling, speculative decoding, load balancing, and statistics reuse). Together, these mechanisms manage compute allocation and execution under fixed budgets, defining the trade-off between accuracy, efficiency, and cost independently of the underlying optimizer.

Control slice	Decision variable	Primary signals	Representative methods	Operational objective and trade-off
Prompt/task selection	Which prompts are rolled out	Group reward variance, posterior success uncertainty, semantic entropy, category-level learning gain	GRESO ( Zheng et al., 2025c ) ; VCRL ( Jiang et al., 2025b ) ; VADE ( Hu et al., 2025b ) ; SEED-GRPO ( Chen et al., 2025b ) ; SEC ( Chen et al., 2025e )	Avoid low-information prompts and concentrate computation where policy updates are expected to be strongest.
Budgeting and scheduling	How many rollouts (width/depth) per prompt	Fixed- $K$ groups vs. variance-aware and uncertainty-aware allocation, response reuse statistics	DeepSeekMath ( Shao et al., 2024 ) ; 1-shot RLVR ( Wang et al., 2025g ) ; VIP ( Nguyen et al., 2026 ) ; MMR1 ( Leng et al., 2025 ) ; XRPO ( Bamba et al., 2025 ) ; AR3PO ( Zhang et al., 2025l )	Trade static stability for adaptive compute efficiency, often reallocating budget toward uncertain or harder prompts.
Rollout configuration control	Length, think/no-think mode, and token-efficiency of kept responses	SOL targets, decoupled token rewards, hybrid-mode gating, reward-per-token filters, length curricula	ShorterBetter ( Yi et al., 2025 ) ; DECS ( Jiang et al., 2025e ) ; AdaptThink ( Zhang et al., 2025e ) ; LHRMs ( Jiang et al., 2025c ) ; GFPO ( Shrivastava et al., 2025 ) ; Train Long, Think Short ( Hammoud et al., 2025 )	Reduce overthinking and token cost while preserving correctness through reward shaping and curriculum schedules.
Partial rollouts and early exit	Stop, continue, resume, or downsample ongoing rollouts	Prefix-level progress signals, completion counts, diversity/variance among sampled rollouts	S-GRPO ( Dai et al., 2025 ) ; APRIL ( Zhou et al., 2025b ) ; PODS ( Xu et al., 2025 )	Cut long-tail rollout waste by terminating unproductive generation early and prioritizing informative trajectories for updates.
Branching and pruning (tree rollouts)	Where to branch and which subtrees to prune	Local uncertainty, search value estimates, and step-level reward propagation	TreeRL ( Hou et al., 2025 ) ; TreePO ( Li et al., 2025d ) ; DeepSearch ( Wu et al., 2025a ) ; TreeRPO ( Yang et al., 2025c )	Increase search efficiency and credit density by allocating compute to promising reasoning branches under fixed budgets.
On-/off-policy control	Mix ratio of fresh rollouts vs replayed data	Replay-buffer provenance/recency and off-policy correction constraints	RePO ( Li et al., 2025c ) ; ReMix ( Liang et al., 2025 )	Improve sample efficiency using replay while constraining off-policy drift and optimization instability.

Table 6: Taxonomy of control mechanisms in rollout-based RL for reasoning LLMs. Rows correspond to control loci in § 6 ; columns specify the controlled decision variable, the signals that drive that decision, representative methods, and the operational objective/trade-off targeted by each family.

Figure 7: This figure summarizes the Replay layer of rollout pipelines, which governs how past rollouts are retained and reused without updating model parameters. Section 7 includes three components: (1) Response Resampling & Retention, which reuses full trajectories to stabilize learning; (2) Recomposition & Segmentation Reuse, which reuses segments or shared prefixes for modular efficiency; and (3) Self-Evolving Curricula & Intrinsic Feedback, which generates new tasks and data through intrinsic feedback. Together, these mechanisms improve data efficiency and training stability.

Verifiable Language
Interface family	Case study	Interface and feedback	Representative benchmarks / resources
Verifiable language	Math	Text-only rollout ( $o_{1:T}=\emptyset$ ); deterministic final-answer verification $V(x,y)$ after normalization.	MATH ( Hendrycks et al., 2021 ) ; OlympiadBench ( He et al., 2024 ) ; TreeRL/TreeRPO ( Hou et al., 2025 ; Yang et al., 2025c ) .
Verifiable language	Code	Program/patch/query outputs; verification via compilation, execution, and unit tests.	LiveCodeBench ( Jain et al., 2024 ) ; CodeRL/RLTF ( Le et al., 2022 ; Liu et al., 2023 ) .
Verifiable language	SQL	Query execution with a database engine; correctness by execution result.	BIRD ( Li et al., 2023 ) ; execution-rewarded Text-to-SQL RL ( Kulkarni & Srikumar, 2025 ) ; Arctic-Text2SQL-R1 ( Yao et al., 2026 ) .
Multimodal Reasoning
Multimodal reasoning	General VLM post-training	Modality-aware verification via structured answer extraction + label/rule checks.	Multimodal RLVR resources (R1-VL; MMR1) ( Zhang et al., 2025f ; Leng et al., 2025 ) .

Table 7: Benchmark families viewed as rollout interfaces, with representative case studies and verifiers.

Issue	Primary module	Diagnosis and mitigation pointers
Filter: scoring, verification, supervision
Spurious signals and bias	Filter	Diagnosis. Scores correlate with superficial properties (e.g., length, style, formatting); models exploit judge/verifier artifacts. Mitigation. Strengthen structural validity and safety gates (§ 5.2 ); use calibrated comparative assessment with abstention where appropriate (§ 5.5 ); design supervision to reduce proxy optimization (§ 5.7 ).
Verifier brittleness and metric mismatch	Filter	Diagnosis. High sensitivity to normalization and parsing; false positives/negatives under small output perturbations. Mitigation. Prefer deterministic, execution-based verification when available (§ 5.3 ); add redundant checks or consistency constraints, and incorporate process-level signals when terminal verification is noisy (§ 5.4 ).
Reward hacking and over-optimization	Filter	Diagnosis. Policies exploit proxy rewards (including judge idiosyncrasies) rather than improving task success; gains may not transfer and can regress under distribution shift. Mitigation. Prefer verifiable checks when possible (§ 5.3 ); improve reward/judge robustness and calibration (§ 5.5 ); use supervision mappings that reduce proxy exploitation and length/style bias (§ 5.7 ).
Generate: exploration and diversity
Coverage collapse and near-duplicate rollouts	Generate	Diagnosis. Candidate sets have low diversity, high duplication, and limited disagreement, reducing effective supervision. Mitigation. Increase exploration and diversity controls (§ 4.4 ); diversify guidance and scaffolding (§ 4.3 ); allocate budget explicitly to exploration when needed (§ 6.3 ).

Table 8: Troubleshooting index for rollout pathologies. Each row maps an observed failure mode to the GFCR module that is typically the best first place to intervene, and provides diagnosis/mitigation pointers.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Sep 21, 2026

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

Reasoning-oriented reinforcement learning enables large language models to solve mathematical, coding, and other multi-step tasks, but shifts a substantial portion of the training cost to rollout, where trajectories are generated for policy updates. Efficient rollout mechanisms are therefore essential to reduce this cost while maintaining the freshness, consistency, and statistical validity of training data. This survey provides a systematic taxonomy of recent research on rollout efficiency for reasoning-oriented reinforcement learning, classifying existing approaches from both mechanism and bottleneck perspectives. Based on this taxonomy, we analyze how different technique families address distinct sources of rollout inefficiency, examine opportunities and potential conflicts for combining them, identify gaps in the evaluation and reporting of efficiency gains, and discuss open challenges and future research directions.

Niloofar Gholipour, Marcos Assuncao, Gursimran Singh +8

École de technologie supérieure, Univ. of Québec, Canada · Huawei Technologies, Canada · The Univ. of Melbourne, Australia +1

LLM Inference Efficiency Reinforcement Learning RL for Language Model Reasoning Efficient Language Model Reasoning

Ask PDF

Jan 17, 2026

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning

Existing reinforcement learning methods for LLM reasoning implicitly assume that the policy generating training trajectories should coincide with the one producing inference responses. We argue that this is a misleading inductive bias: the optimization-optimal trajectory distribution favors informative gradients, whereas the inference-optimal response distribution emphasizes accuracy and consistency. Forcing both into a single policy entangles their gradients and suppresses exploration. We propose R

^2

PO (Residual Rollout Policy Optimization), which attaches a lightweight Residual Rollout-Head atop the policy to decouple training trajectories from inference responses, diversifying rollouts during training while keeping inference generation intact. Experiments show that R

^2

PO consistently outperforms baselines, with average accuracy gains of 3.4% on MATH-500 and 1.3% on APPS, alongside more diverse rollouts and reduced length bias. Our code is available at https://github.com/RRPO-ARR/Code.

Jingchu Wang, Bingbing Xu, Yige Yuan +4

State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore

RL for Language Model Reasoning Policy Optimization RL Exploration RL Post-Training Policy Learning

Ask PDF

Jun 25, 2026

RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning

Large language model (LLM) post-training for reasoning increasingly relies on reinforcement learning with verifiable rewards (RLVR), where models learn from ground-truth feedback on mathematical, logical, and scientific tasks. To enable flexible resource allocation and support heterogeneous training setups, modern RLVR systems adopt disaggregated architectures that decouple rollout generation and policy training across independent GPU pools. However, existing synchronous on-policy GRPO (Group Relative Policy Optimization) RLVR systems finish an entire rollout before starting training, leaving the trainer GPU pool idle while rollout is still ongoing. Asynchronous RL pipelines overlap the two stages, but at the cost of training on stale data. To address these challenges, we propose RolloutPipe, a post-training framework for disaggregated RLVR systems, which turns the fixed-weight rollout into a complete-group pipeline where trainable groups move to the trainer while later groups are still being generated. RolloutPipe achieves this through two techniques including complete-group pipelining (CGP) and frontier-group dispatch (FGD). CGP dispatches each trainable complete group to the trainer FIFO as soon as group materialization finishes, and FGD is an admission policy on the Rollout node that first admits requests for the frontier groups needed to form the next training batch, so that trainer-ready groups arrive earlier and more steadily. The design starts training before the rollout completes while maintaining on-policy correctness. Evaluated on Qwen3-1.7B across four reasoning and science benchmarks and twelve rollout settings, RolloutPipe shortens the rollout-to-train-end time by 30.7%-42.3%, and lowers the trainer waiting ratio by 37%-76% compared to Slime, a state-of-the-art rollout and training system.

Rongjian Chen, Jianmin Hu, Kejiang Ye +1

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China · University of Chinese Academy of Sciences, Beijing, China · Southern University of Science and Technology, Shenzhen, China

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Post-Training

Ask PDF

Abstract

RL for Language Model Reasoning

Explore similar work

Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directions

R2^22PO: Decoupling Rollout and Inference Policies for LLM Reasoning

RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning

R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning