cs.LGApr 8, 2026
SaveGenerate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
Organizations: University of California, San Diego · University of Toronto · UIUC · University of Virginia ·
Abstract
Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajectory sampled from a prompt to termination, including intermediate reasoning steps and optional tool or environment interactions, determines the data the optimizer learns from, yet rollout design is often underreported. This survey provides an optimizer-agnostic view of rollout strategies for RL-based post-training of reasoning LLMs. We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR), a lifecycle taxonomy that decomposes rollout pipelines into four modular stages: Generate proposes candidate trajectories and topologies; Filter constructs intermediate signals via verifiers, judges, critics; Control allocates compute and makes continuation/branching/stopping decisions under budgets; and Replay retains and reuses artifacts across rollouts without weight updates, including self-evolving curricula that autonomously generate new training tasks. We complement GFCR with a criterion taxonomy of reliability, coverage, and cost sensitivity that characterizes rollout trade-offs. Using this framework, we synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement. We ground the framework with case studies in math, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks that evaluate skill induction, reuse, and cross-task transfer. Finally, we provide a diagnostic index that maps common rollout pathologies to GFCR modules and mitigation levers, alongside open challenges for building reproducible, compute-efficient, and trustworthy rollout pipelines.
Figures & tables
Figure 1: An overview of the rollout lifecycle and the Generate–Filter–Control–Replay (GFCR) decomposition. Rollout pipelines can be understood as modular choices about how trajectories are proposed, how intermediate signals are constructed, how compute is allocated under budgets, and what artifacts are retained and reused across rollouts.
Figure 2: GFCR as an end-to-end rollout system. Given prompts and a compute budget , Generate samples a rollout group ; Filter maps each rollout to intermediate signals and training-facing supervision ; Control uses costs and signals to adapt continuation/pruning/resampling and decide what to store; and Replay retrieves/stores artifacts that condition future generation. The overall objective is to maximize expected utility under compute constraints.
| Module | Subcomponent | What it includes |
|---|---|---|
| Generate (G): How trajectories are proposed. Defines the proposal distribution + interaction structure. | ||
| § 4.2 | Topology & Interaction | Rollout structure and interaction patterns. |
| § 4.3 | Guidance & scaffolding | Prompt structure, rubrics, and auxiliary traces. |
| § 4.4 | Sampling & exploration | Decoding diversity and exploration signals. |
| Filter (F): Triage, scoring, and signal construction. Maps rollouts/prefixes to training-facing signals. | ||
| § 5.2 | Structural validity | Schema compliance, executability, and safety gates. |
Table 1: GFCR taxonomy at a glance. Rollout behavior is assembled from four composable lifecycle modules.
| Symbol | Meaning |
|---|---|
| Distribution (dataset) over prompts/tasks. | |
| Prompt/task instance sampled from . | |
| Parameters of the language model / policy. | |
| Model policy over next output/action given state . | |
| Model output/action at step (tokens and/or tool/action call). | |
| Observation at step (tool output/environment response; empty for text-only). |
Table 2: Global notation used throughout the paper. Module-specific notation is introduced in the corresponding GFCR sections.
Figure 3: Rollout criterion taxonomy (at a glance). We evaluate rollout strategies along three cross-cutting desiderata: reliability (trustworthy signals via verifiers or robust judges), coverage & informativeness (diverse candidates and disagreement/uncertainty), and cost sensitivity (value under compute budgets via value-per-cost and early stopping).
| Criterion | Description and Examples |
|---|---|
| Reliability | |
| Verifiable Outcomes | Executable / checkable rollouts yielding trustworthy supervision when verification exists (e.g., math or code): structural validity/executability gates, unit tests, symbolic checks, exact-match constraints. |
| Robust Judge Scoring | Stable evaluation when direct verification is unavailable (e.g., open-ended tasks): calibrated judging, consistency across prompt variants, and resistance to reward hacking. |
| Coverage & Informativeness | |
| Diverse Candidates | Broad exploration over plausible solution strategies (structured, creative, iterative), avoiding near-duplicate rollouts and encouraging novelty. |
| Disagreement Signals | Informative uncertainty revealing hard-but-solvable cases: disagreement across rollouts or critics, conflicting rationales, and high-variance outcomes. |
Table 3: Rollout criterion taxonomy (reference style). We summarize three cross-cutting desiderata for rollout strategies— reliability , coverage & informativeness , and cost sensitivity —and list concrete signals and mechanisms that operationalize each criterion.
Figure 4: Generate module design space. Rollout proposal mechanisms can be organized along three axes: (left) topology & interaction (single, group, tree/graph, and multi-turn/tool rollouts), (middle) guidance/scaffolding (ICL/rubrics, plans, reflection, adaptive guidance, tool augmentation), and (right) sampling & exploration configuration (decoding knobs, uncertainty-aware allocation, partial rollouts with resumption, and sampling-only inference enhancements). Group candidates are typically scored downstream by Filter, and compute allocation may be coupled to Control.
| Subcomponent | Design choice | Primary trade-off | Representative works |
| Topology & interaction ( ) | |||
| Topology | Linear (single rollout) | Simplicity ; within-prompt baseline ; gradient variance . | ( Le et al., 2022 ; Liu et al., 2023 ) . |
| Topology | Group ( rollouts) | Variance via within-prompt baselines; compute with . | ( Shao et al., 2024 ; Guo et al., 2025b ; Yu et al., 2025a ) . |
| Topology | Tree/graph (branching prefixes) | Prefix reuse and targeted exploration; control/bookkeeping . | ( Hou et al., 2025 ; Bamba et al., 2025 ) . |
| Interaction | Tool / environment loop | Grounded feedback ; non-stationarity/latency (env-dependent). | ( Liu et al., 2023 ; Li et al., 2023 ; Jimenez et al., 2023 ) . |
| Guidance & scaffolding ( ) | |||
Table 4: Generate design space and primary trade-offs. Rows summarize rollout-construction choices used in representative methods, emphasizing the main efficiency/quality trade-off each choice introduces.
Figure 5: Filter module design space. Filter maps sampled rollouts (or rollout prefixes) into intermediate signals and optimizer-facing supervision. It can be organized along three components: (left) gating & verification (structural validity, schema/executability checks, deterministic correctness tests), (middle) process & comparative scoring (dense step-level process signals and relative judgments via ranking/judging), and (right) signal construction (learning-value diagnostics such as uncertainty/variance/entropy, and mapping raw outcomes into optimizer-ready targets such as masks/weights, advantages, or preference labels). Filter outputs are consumed by the optimizer and may also inform Control decisions.
| Stage | Output | Typical mechanisms | Representative works |
|---|---|---|---|
| Gate | Pre-semantic validity checks: parseability/normalization, executability, safety constraints; can act on rollouts or prefixes. | ( Chen et al., 2025a ; Hugging Face, 2025 ; Liu et al., 2023 ; Le et al., 2022 ; Li et al., 2023 ) . | |
| Eval | Hard semantic verification (unit tests, execution, exact-answer equivalence), process/step scoring (PRMs/step verifiers), or comparative ranking (LLM judge). | ( Liu et al., 2023 ; Le et al., 2022 ; Yao et al., 2026 ; Ali et al., 2025 ; Zhang et al., 2025g ; Hendrycks et al., 2021 ; He et al., 2024 ; Lightman et al., 2023 ; Zheng et al., 2025b ; Song et al., 2025 ; Zhang et al., 2025m ; Zheng et al., 2023 ; Shi et al., 2025 ) . | |
| Map | Convert raw signals into optimizer-facing supervision: masks/weights, shaped rewards, group-relative advantages, pairwise/listwise labels; often group-dependent. | ( Mroueh, 2025 ; Zhang & Zuo, 2025 ; Shrivastava et al., 2025 ; Becker et al., 2025 ) . |
Table 5: Filter as a signal pipeline (Gate Eval Map). This table instantiates Eqs. ( 8 )–( 10 ) with common rollout filters and the optimizer-facing artifacts they produce.
Figure 6: Control module design space. The control layer of rollout pipelines can be organized into four orthogonal components: (left) Instance & Budget Selection (prompt prioritization by difficulty/novelty/uncertainty; adaptive allocation of rollout slots, group size, depth/width, and token budgets); (middle) Rollout Dynamics Control (decoding configuration, partial rollouts with early exit, and branching–pruning policies for quality–efficiency trade-offs); (right-top) Data & Policy Control (on-/off-policy mixing, replay, and correctness-aware filtering); and (right-bottom) Execution & Systems Control (scheduling, speculative decoding, load balancing, and statistics reuse). Together, these mechanisms manage compute allocation and execution under fixed budgets, defining the trade-off between accuracy, efficiency, and cost independently of the underlying optimizer.
| Control slice | Decision variable | Primary signals | Representative methods | Operational objective and trade-off |
|---|---|---|---|---|
| Prompt/task selection | Which prompts are rolled out | Group reward variance, posterior success uncertainty, semantic entropy, category-level learning gain | GRESO ( Zheng et al., 2025c ) ; VCRL ( Jiang et al., 2025b ) ; VADE ( Hu et al., 2025b ) ; SEED-GRPO ( Chen et al., 2025b ) ; SEC ( Chen et al., 2025e ) | Avoid low-information prompts and concentrate computation where policy updates are expected to be strongest. |
| Budgeting and scheduling | How many rollouts (width/depth) per prompt | Fixed- groups vs. variance-aware and uncertainty-aware allocation, response reuse statistics | DeepSeekMath ( Shao et al., 2024 ) ; 1-shot RLVR ( Wang et al., 2025g ) ; VIP ( Nguyen et al., 2026 ) ; MMR1 ( Leng et al., 2025 ) ; XRPO ( Bamba et al., 2025 ) ; AR3PO ( Zhang et al., 2025l ) | Trade static stability for adaptive compute efficiency, often reallocating budget toward uncertain or harder prompts. |
| Rollout configuration control | Length, think/no-think mode, and token-efficiency of kept responses | SOL targets, decoupled token rewards, hybrid-mode gating, reward-per-token filters, length curricula | ShorterBetter ( Yi et al., 2025 ) ; DECS ( Jiang et al., 2025e ) ; AdaptThink ( Zhang et al., 2025e ) ; LHRMs ( Jiang et al., 2025c ) ; GFPO ( Shrivastava et al., 2025 ) ; Train Long, Think Short ( Hammoud et al., 2025 ) | Reduce overthinking and token cost while preserving correctness through reward shaping and curriculum schedules. |
| Partial rollouts and early exit | Stop, continue, resume, or downsample ongoing rollouts | Prefix-level progress signals, completion counts, diversity/variance among sampled rollouts | S-GRPO ( Dai et al., 2025 ) ; APRIL ( Zhou et al., 2025b ) ; PODS ( Xu et al., 2025 ) | Cut long-tail rollout waste by terminating unproductive generation early and prioritizing informative trajectories for updates. |
| Branching and pruning (tree rollouts) | Where to branch and which subtrees to prune | Local uncertainty, search value estimates, and step-level reward propagation | TreeRL ( Hou et al., 2025 ) ; TreePO ( Li et al., 2025d ) ; DeepSearch ( Wu et al., 2025a ) ; TreeRPO ( Yang et al., 2025c ) | Increase search efficiency and credit density by allocating compute to promising reasoning branches under fixed budgets. |
| On-/off-policy control | Mix ratio of fresh rollouts vs replayed data | Replay-buffer provenance/recency and off-policy correction constraints | RePO ( Li et al., 2025c ) ; ReMix ( Liang et al., 2025 ) | Improve sample efficiency using replay while constraining off-policy drift and optimization instability. |
Table 6: Taxonomy of control mechanisms in rollout-based RL for reasoning LLMs. Rows correspond to control loci in § 6 ; columns specify the controlled decision variable, the signals that drive that decision, representative methods, and the operational objective/trade-off targeted by each family.
Figure 7: This figure summarizes the Replay layer of rollout pipelines, which governs how past rollouts are retained and reused without updating model parameters. Section 7 includes three components: (1) Response Resampling & Retention, which reuses full trajectories to stabilize learning; (2) Recomposition & Segmentation Reuse, which reuses segments or shared prefixes for modular efficiency; and (3) Self-Evolving Curricula & Intrinsic Feedback, which generates new tasks and data through intrinsic feedback. Together, these mechanisms improve data efficiency and training stability.
| Interface family | Case study | Interface and feedback | Representative benchmarks / resources |
|---|---|---|---|
| Verifiable Language | |||
| Verifiable language | Math | Text-only rollout ( ); deterministic final-answer verification after normalization. | MATH ( Hendrycks et al., 2021 ) ; OlympiadBench ( He et al., 2024 ) ; TreeRL/TreeRPO ( Hou et al., 2025 ; Yang et al., 2025c ) . |
| Verifiable language | Code | Program/patch/query outputs; verification via compilation, execution, and unit tests. | LiveCodeBench ( Jain et al., 2024 ) ; CodeRL/RLTF ( Le et al., 2022 ; Liu et al., 2023 ) . |
| Verifiable language | SQL | Query execution with a database engine; correctness by execution result. | BIRD ( Li et al., 2023 ) ; execution-rewarded Text-to-SQL RL ( Kulkarni & Srikumar, 2025 ) ; Arctic-Text2SQL-R1 ( Yao et al., 2026 ) . |
| Multimodal Reasoning | |||
| Multimodal reasoning | General VLM post-training | Modality-aware verification via structured answer extraction + label/rule checks. | Multimodal RLVR resources (R1-VL; MMR1) ( Zhang et al., 2025f ; Leng et al., 2025 ) . |
Table 7: Benchmark families viewed as rollout interfaces, with representative case studies and verifiers.
| Issue | Primary module | Diagnosis and mitigation pointers |
| Filter: scoring, verification, supervision | ||
| Spurious signals and bias | Filter | Diagnosis. Scores correlate with superficial properties (e.g., length, style, formatting); models exploit judge/verifier artifacts. Mitigation. Strengthen structural validity and safety gates (§ 5.2 ); use calibrated comparative assessment with abstention where appropriate (§ 5.5 ); design supervision to reduce proxy optimization (§ 5.7 ). |
| Verifier brittleness and metric mismatch | Filter | Diagnosis. High sensitivity to normalization and parsing; false positives/negatives under small output perturbations. Mitigation. Prefer deterministic, execution-based verification when available (§ 5.3 ); add redundant checks or consistency constraints, and incorporate process-level signals when terminal verification is noisy (§ 5.4 ). |
| Reward hacking and over-optimization | Filter | Diagnosis. Policies exploit proxy rewards (including judge idiosyncrasies) rather than improving task success; gains may not transfer and can regress under distribution shift. Mitigation. Prefer verifiable checks when possible (§ 5.3 ); improve reward/judge robustness and calibration (§ 5.5 ); use supervision mappings that reduce proxy exploitation and length/style bias (§ 5.7 ). |
| Generate: exploration and diversity | ||
| Coverage collapse and near-duplicate rollouts | Generate | Diagnosis. Candidate sets have low diversity, high duplication, and limited disagreement, reducing effective supervision. Mitigation. Increase exploration and diversity controls (§ 4.4 ); diversify guidance and scaffolding (§ 4.3 ); allocate budget explicitly to exploration when needed (§ 6.3 ). |
Table 8: Troubleshooting index for rollout pathologies. Each row maps an observed failure mode to the GFCR module that is typically the best first place to intervene, and provides diagnosis/mitigation pointers.
More in this topic
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,094 papers