Towards Better RL Training Data Utilization via Second-Order Rollout

Zhe Yang; Yudong Wang; Rang Li; Zhifang Sui

cs.CLFeb 26, 2026

Towards Better RL Training Data Utilization via Second-Order Rollout

Authors: Zhe Yang, Yudong Wang, Rang Li, Zhifang Sui

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · ByteDance BandAI

Abstract

Reinforcement Learning (RL) has empowered Large Language Models (LLMs) with strong reasoning capabilities, but vanilla RL mainly focuses on generation capability improvement by training with only first-order rollout (generating multiple responses for a question), and we argue that this approach fails to fully exploit the potential of training data because of the neglect of critique capability training. To tackle this problem, we further introduce the concept of second-order rollout (generating multiple critiques for a response) and propose a unified framework for jointly training generation and critique capabilities. Extensive experiments across various models and datasets demonstrate that our approach can utilize training data more effectively than vanilla RL and achieve better performance under the same training data. Additionally, we uncover several insightful findings regarding second-order rollout and critique training, such as the importance of label balance in critique training and the noise problem of outcome-based rewards, which can be mitigated through sampling techniques. Our work offers a preliminary exploration of dynamic data augmentation and joint generation-critique training in RL, providing meaningful inspiration for the further advancement of RL training

Figures & tables

Figure 1: A demonstration of first/second-order rollout. The policy model generates multiple responses for a question in first-order rollout, and generates multiple critiques for a response in second-order rollout.

Figure 2: Flowchart of a single training step in GC-RL. First, a batch of questions is sampled from the training data, and multiple responses are generated (first-order rollout). Then, without replacement, a batch of <question, response> pairs is sampled from the Question-Response Data Cache , and multiple critiques are generated (second-order rollout). These rollouts are combined and utilized jointly to update the policy model. In addition, the Question-Response Data Cache is maintained by processing the first-order rollout through a Data Filter and adding the filtered data into the cache.

Models	Methods	Math-500	GSM8k	Minerva	AMC23	Olympiad	Avg
Models	Methods	Generation Accuracy (%)					Avg
Qwen2.5-7B	w/o RL	55.6	77.9	16.9	35.0	22.8	41.6
	C-RL	65.1	83.7	19.2	47.5	26.1	48.3
	G-RL	75.4	89.7	24.6	60.0	33.7	56.7
	GC-RL	77.6	92.0	24.6	62.5	39.8	59.3
Qwen2.5-3B	w/o RL	20.4	29.4	4.0	5.0	7.6	13.3

Table 1: Generation and critique capabilities evaluation results on Qwen-2.5-(1.5B,3B,7B). GC-RL outperforms all other RL training methods in both generation and critique capabilities.

		Math-500	GSM8k	Minerva	AMC23	Olympiad	Avg
Genration	Random Sampling	75.0	90.9	22.9	60.0	37.5	57.3
	Random Sampling + Reweight	77.2	91.5	23.2	60.0	38.2	58.0
	Data Filter	77.6	92.0	24.6	62.5	39.8	59.3
Critique	Random Sampling	82.3	84.0	63.3	73.5	71.6	74.9
	Random Sampling + Reweight	83.7	86.7	65.8	77.9	72.8	77.4
	Data Filter	84.6	88.3	67.1	79.4	73.8	78.6

Table 2: A comparison of model performance on Qwen2.5-7B when sampling responses with/without a data filter. Random sampling leads to the worst performance, though adding reward reweighting can alleviate this issue. Utilizing the data filter achieves the best performance.

Figure 3: A comparison of model performance of Qwen2.5-7B with/without reward denoising strategy on Math-500. In both GC-RL and C-RL settings, reward denoising can improve model performance on both generation and critique capabilities.

Figure 4: Performance of Qwen2.5-7B on Math-500 under GC-RL and C-RL settings with static and dynamic critique training data. Dynamic data outperforms static data in the GC-RL setting, while the opposite holds for C-RL.

Figure 5: A comparison of critique performance of Qwen2.5-7B with different reward function on Math-500. Compared to baseline

R(c)

R_{w}(c)

leads to a higher precision while

R_{r}(c)

generates a higher recall.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Our prompt fed to GPT-5 for critique data generation, which is comprised of a question, a solution, and critique instruction.

train bath size	512
ppo mini batch size	128
rollout n	5
adv estimator	grpo
kl loss coef	1e-3
learning rate	1e-6
max prompt length	4096

Appendix

Table 3: RL training hyper-parameters.

Models	Methods	Math-500	GSM8k	Minerva	AMC23	Olympiad	Avg
Models	Methods	Generation Accuracy (%)					Avg
Mistral-7B-Instruct-v0.3	w/o RL	10.8	46.7	7.4	5.0	1.8	14.3
	C-RL	19.8	54.2	9.8	12.5	4.2	20.1
	G-RL	47.6	77.8	18.6	30.0	15.6	37.9
	GC-RL	52.1	81.2	19.4	32.5	17.9	40.6
Llama-3.1-8B-Instruct	w/o RL	48.2	84.2	19.1	25.0	15.0	38.3

Appendix

Table 4: Generation and critique capabilities evaluation results on Mistral-7B and Llama3.1-8B-Instruct. GC-RL outperforms all other RL training methods in both generation and critique capabilities.

Models	Methods	Math-500	GSM8k	Minerva	AMC23	Olympiad	Avg
Models	Methods	Denoised Reward					Avg
Qwen2.5-7B	C-RL	0.901	0.912	0.715	0.797	0.812	0.827
Qwen2.5-7B	GC-RL	0.945	0.986	0.774	0.852	0.836	0.879
Qwen2.5-3B	C-RL	0.752	0.754	0.652	0.699	0.691	0.710
Qwen2.5-3B	GC-RL	0.774	0.802	0.657	0.712	0.698	0.729
Qwen2.5-1.5B	C-RL	0.651	0.638	0.607	0.648	0.612	0.631

Appendix

Table 5: Supplemental critique capability evaluation results on Qwen-2.5-(1.5B,3B,7B). The average denoised reward (§ 5.2 ) is also reported to reflect the critique capability.

Figure 7: Performance of Qwen2.5-7B on 4 datasets under GC-RL and C-RL settings with static and dynamic critique training data. Dynamic data outperforms static data in the GC-RL setting, while the opposite holds for C-RL.

Figure 8: A comparison of critique performance of Qwen2.5-7B with different reward functions on 4 datasets. Compared to baseline

R(c)

R_{w}(c)

leads to a higher precision while

R_{r}(c)

generates a higher recall.

Figure 9: A comparison of model performance of Qwen2.5-7B with/without reward denoising strategy on 4 datasets. In both GC-RL and C-RL settings, reward denoising can improve model performance on both generation and critique capabilities.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
Apr 8, 2026Rohan Surana, Gagan Mundada, Xunyi Jiang +19RL for Language Model Reasoning RL for Language Models
University of California, San Diego · University of Toronto · UIUC +3
Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
May 8, 2026Aswin RRV, Jacob Dineen, Divij Handa +4Synthetic Data Generation RL for Language Model Reasoning
Arizona State University · Google Cloud AI Research · Google DeepMind
A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula
Mar 25, 2026Cansu Sancaktar, David Zhang, Gabriel Synnaeve +1RL for Code Generation Synthetic Data Augmentation
University of Tübingen · Max Planck Institute for Intelligent Systems · Meta FAIR

Apr 8, 2026J/K move · Enter open · S save

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Rohan Surana, Gagan Mundada, Xunyi Jiang, Chuhan Wang+18

University of California, San Diego · University of Toronto · UIUC+3

RL for Language Model Reasoning RL for Language Models LLM Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajectory sampled from a prompt to termination, including intermediate reasoning steps and optional tool or environment interactions, determines the data the optimizer learns from, yet rollout design is often underreported. This survey provides an optimizer-agnostic view of rollout strategies for RL-based post-training of reasoning LLMs. We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR), a lifecycle taxonomy that decomposes rollout pipelines into four modular stages: Generate proposes candidate trajectories and topologies; Filter constructs intermediate signals via verifiers, judges, critics; Control allocates compute and makes continuation/branching/stopping decisions under budgets; and Replay retains and reuses artifacts across rollouts without weight updates, including self-evolving curricula that autonomously generate new training tasks. We complement GFCR with a criterion taxonomy of reliability, coverage, and cost sensitivity that characterizes rollout trade-offs. Using this framework, we synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement. We ground the framework with case studies in math, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks that evaluate skill induction, reuse, and cross-task transfer. Finally, we provide a diagnostic index that maps common rollout pathologies to GFCR modules and mitigation levers, alongside open challenges for building reproducible, compute-efficient, and trustworthy rollout pipelines.

Abstract

RL for Language Model Reasoning

Explore similar work

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models

A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curricula