Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards

Zhen-Yu Zhang; Jiandong Zhang; Huaxiu Yao; Gang Niu; Masashi Sugiyama

cs.LGOct 31, 2026

Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards

Authors: Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao, Gang Niu, Masashi Sugiyama

Organizations: RIKEN-AIP · Northeastern University · UNC-Chapel Hill · The University of Tokyo

Abstract

Recently, reinforcement learning from verifiable rewards (RLVR) is practically very important, where the leading approach is group relative policy optimization (GRPO) that typically faces a trade-off between rollout efficiency and downstream performance. To improve efficiency while preserving performance, existing selective rollout methods mainly exploit only source-side training dynamics to optimize the discrete distribution over source prompts. However, in practice, the source training dataset is rarely identically distributed to the target test dataset. When such a mismatch exists, due to legal or privacy constraints, it is usually not acceptable to share target prompts themselves with the RLVR trainer. Nevertheless, it is still possible for the target data owner to provide limited aggregated feedback from a small held-out target set without revealing target prompts or rollout traces. In this paper, we propose learning-to-reweight rollout (L2RR) that explicitly incorporates target-side performance feedback into GRPO. Specifically, L2RR learns a discrete sampling distribution over source prompts by optimizing a bi-level objective that maximizes accuracy on a held-out target dataset, while simultaneously penalizing rollouts that provide little incremental learning information. Empirical studies on six math reasoning benchmarks and three model scales show that L2RR improves target-domain performance and increases rollout efficiency.

Figures & tables

Qwen2.5-Math-1.5B
Dataset	Method	Math500	AIME24	AMC	Gaokao	Miner.	Olymp.	Avg.	# Rollout	RO ratio
DM	DAPO	76.3 $\pm$ 1.8	15.2 $\pm$ 3.1	60.9 $\pm$ 2.7	63.8 $\pm$ 2.3	31.6 $\pm$ 2.5	38.2 $\pm$ 1.2	47.7 $\pm$ 1.5	7.6M	1.0 $\times$
	GRESO	75.8 $\pm$ 1.8	14.1 $\pm$ 3.0	60.8 $\pm$ 2.8	65.5 $\pm$ 2.1	32.8 $\pm$ 2.8	38.0 $\pm$ 1.5	47.8 $\pm$ 1.6	3.3M	2.3 $\times$
	AR3PO	77.6 $\pm$ 1.3	15.7 $\pm$ 3.1	60.8 $\pm$ 2.5	65.3 $\pm$ 2.3	35.4 $\pm$ 2.5	38.6 $\pm$ 1.9	48.9 $\pm$ 1.5	2.5M	3.0 $\times$
	L2RR	79.4 $\pm$ 1.7	17.2 $\pm$ 2.4	61.8 $\pm$ 2.1	66.8 $\pm$ 2.1	35.2 $\pm$ 2.1	41.4 $\pm$ 0.9	50.3 $\pm$ 1.3	3.0M	2.5 $\times$
OR1	DAPO	76.5 $\pm$ 1.9	15.7 $\pm$ 2.7	49.5 $\pm$ 2.8	65.1 $\pm$ 2.3	30.2 $\pm$ 2.6	39.2 $\pm$ 1.7	46.0 $\pm$ 1.6	3.8M	1.0 $\times$

Table 1: Performance (%) comparison across six math reasoning benchmarks. We train three models on source datasets DAPO+MATH (DM) and the Open R1 subset (OR1) and test on six benchmarks. # Rollout denotes cumulative source responses, in millions. RO ratio is the DAPO count divided by the method count. Accuracies are mean

\pm

standard deviation over five training seeds.

Method	Math500	AIME24	AMC	Gaokao	Miner.	Olymp.	Avg.	# Rollout	RO ratio
L2RR full	79.9	18.5	62.2	67.3	35.7	41.5	50.9	3.0M	2.5 $\times$
L2RR w/o ws	78.4	16.7	61.8	66.0	34.0	40.0	49.5	2.7M	2.8 $\times$
L2RR w/o rollout allo	78.2	16.7	61.5	65.7	33.8	39.8	49.3	4.3M	1.8 $\times$
L2RR uniform $q$	77.9	15.7	61.8	65.2	33.0	39.5	48.9	3.2M	2.4 $\times$

Table 2: Ablation study on key components of L2RR. We report performance (%) on six math reasoning benchmarks for Qwen2.5-Math-1.5B trained on the DAPO + MATH dataset (DM). # Rollout denotes cumulative source responses, in millions. RO ratio is the DAPO count divided by the method count.

# Target (held-out)	Math500	AIME24	AMC	Gaokao	Miner.	Olymp.	Avg.
4%	78.2	16.7	61.5	65.7	34.4	39.5	49.3
6%	78.5	17.6	61.8	66.0	34.7	39.8	49.7
8%	78.9	17.6	61.5	66.3	34.9	40.1	49.9
10%	79.9	18.5	62.2	67.3	35.7	41.5	50.9

Table 3: Effect of held-out target set size for sampler learning. Accuracy (%) on six reasoning benchmarks for Qwen2.5-Math-1.5B trained on DAPO + MATH. We vary the size of held-out target dataset to update

\mathbf{q}_{t}

Panel of Figure 1 — Figure 1: Training-dynamics illustration for L2RR. During training, L2RR increases sampling on prompts that contribute more to improving held-out target accuracy and suppresses repeatedly wasteful prompts as indicated by the history-based counter. Qwen2.5-Math-1.5B trained on DM. DS denotes DAPO with dynamic sampling; DS + ABS adds the adaptive sampling batch size of GRESO [ 48 ] .

Figure 2: Effect of the rollout-waste weight

\lambda

on target accuracy (%). Overall, moderate

\lambda

yields the best trade-off, while overly small or large

\lambda

can under-regularize wasted rollouts or over-penalize informative prompts and thus reduce accuracy.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Training
Framework	verl (FSDP) with vLLM rollouts
Optimizer	AdamW, $\beta=(0.9,0.999)$ , weight decay $0.01$
Learning rate $\eta_{\theta}$	$1\times 10^{-6}$ , constant
Training steps	1000
Batch size $B$	256 (DeepSeek-1.5B: 128)
Prompts per sampling round	384 (DeepSeek-1.5B: 192)

Appendix

Table 4: Hyperparameters. Batch sizes count prompts.

Model	Source	Target-only	Best Target-Mix	L2RR
Qwen2.5-Math-1.5B	DM	39.3	50.4	50.9
Qwen2.5-Math-1.5B	OR1	39.3	47.7	48.7
DeepSeek-R1-Distill-Qwen-1.5B	DM	50.1	61.6	62.4
DeepSeek-R1-Distill-Qwen-1.5B	OR1	50.1	58.1	59.1
Qwen2.5-Math-7B	DM	47.4	58.7	59.3
Qwen2.5-Math-7B	OR1	47.4	56.0	56.8

Appendix

Table 5: Diagnostic comparison with direct target training. We report average test accuracy (%) across the six math reasoning benchmarks. Best Target-Mix is the highest average accuracy among the DAPO, GRESO, and AR3PO Target-Mix variants in Table 12 . Target-only uses no source data, so its result is repeated across the DM and OR1 rows for each model.

Source	Method	Avg. (%)	Total rollouts
DM	Gradient-Similarity Thresholding	49.9	3.0M
	Target-Loss Reweighting	50.0	3.0M
	L2RR	50.9	3.0M
OR1	Gradient-Similarity Thresholding	47.7	1.4M
	Target-Loss Reweighting	47.9	1.4M
	L2RR	48.7	1.4M

Appendix

Table 6: Comparison with simple target-guided baselines. We report average test accuracy (%) across the six math reasoning benchmarks and cumulative source-rollout counts for Qwen2.5-Math-1.5B. All methods use the same target-feedback interface.

Source	Target	s-OTDD	$1-\mathrm{MAUVE}$	$\Delta$ Qwen-1.5B	$\Delta$ DeepSeek-1.5B	$\Delta$ Qwen-7B
DM	Math500	0.235	0.194	$+1.8$	$+1.8$	$+1.2$
	AIME24	0.369	0.303	$+1.5$	$+1.9$	$-0.9$
	AMC	0.204	0.175	$+0.9$	$+1.3$	$+0.9$
	Gaokao	0.290	0.238	$+1.3$	$+1.4$	$+0.5$
	Minerva	0.302	0.288	$-0.2$	$+0.9$	$+1.2$
	OlympiadBench	0.422	0.379	$+2.8$	$+2.5$	$+2.1$

Appendix

Table 7: Source–target mismatch and L2RR gains. Distances are averaged over five source subsamples. Gains are computed from the five-run mean test accuracies in Table 1 . Qwen-1.5B, DeepSeek-1.5B, and Qwen-7B denote Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, respectively.

Benchmark / $\lambda$	0.01	0.10	0.15	0.20	0.25	0.30	0.40	0.50	1.00
Math500	78.8	79.0	79.1	79.3	79.9	79.5	78.7	78.6	78.3
AIME24	16.7	17.6	17.6	18.5	18.5	18.5	17.6	17.6	16.7
AMC	61.1	61.5	61.5	61.8	62.2	62.2	61.8	61.8	61.5
Gaokao	66.1	66.3	66.3	66.3	67.3	66.5	66.1	66.2	65.9

Appendix

Table 8: Fine-grained sensitivity analysis of the rollout-waste weight

\lambda

. We report accuracy (%) for Qwen2.5-Math-1.5B trained on DM. Bold indicates the highest reported accuracy for each benchmark, including ties.

Source	Validation set	Math500	Gaokao	$\Delta$	Total rollouts
DM	Uniform	79.9	67.3	–	3.0M
	Easy-biased	80.0	67.6	$+0.2$	2.8M
	Hard-biased	79.8	66.7	$-0.4$	3.3M
OR1	Uniform	78.8	66.7	–	1.4M
	Easy-biased	78.9	65.8	$-0.4$	1.3M
	Hard-biased	78.1	66.6	$-0.4$	1.6M

Appendix

Table 9: Sensitivity to difficulty-biased target validation data. Accuracy is reported in percent.

\Delta

is the change in the average accuracy over Math500 and Gaokao relative to Uniform for the same source dataset, in percentage points.

Source	Validation set	OlympiadBench	$\Delta$	Total rollouts
DM	Uniform	41.5	–	3.0M
	Algebra-biased	41.0	$-0.5$	3.1M
	Geometry-biased	41.8	$+0.3$	2.9M
	Number-Theory-biased	41.2	$-0.3$	3.1M
	Combinatorics-biased	41.0	$-0.5$	3.2M
OR1	Uniform	41.7	–	1.4M

Appendix

Table 10: Sensitivity to subdomain-biased target validation data. We report OlympiadBench accuracy (%), its change

\Delta

relative to Uniform for the same source dataset in percentage points, and cumulative rollout counts.

Verifier	Method	HumanEval+	MBPP+	Avg.	Total rollouts
Full unit tests	AR3PO	63.3	58.5	60.9	2.5M
Full unit tests	L2RR	64.9	59.4	62.2	2.7M
Random 50% of tests	AR3PO	59.8	55.6	57.7	2.5M
Random 50% of tests	L2RR	61.9	56.9	59.4	2.6M

Appendix

Table 11: Code generation with full and partial verifiers. We report test scores (%) on HumanEval+ and MBPP+, their average, and cumulative source-rollout counts for Qwen2.5-Coder-1.5B-Instruct trained on CodeContests. The verifier condition applies to training and target feedback, while final evaluation always uses the full test suites.

Qwen2.5-Math-1.5B
Dataset	Method	Math500	AIME24	AMC	Gaokao	Miner.	Olymp.	Avg.	# Rollout	RO ratio
DM	DAPO	77.3	16.7	61.8	64.2	31.8	38.7	48.4	7.6M	1.0 $\times$
	DAPO + Target-Mix	77.9	17.6	62.2	65.1	32.6	39.4	49.1	8.2M	0.9 $\times$
	DAPO - TargetVal	77.1	15.7	61.5	63.8	31.2	38.1	47.9	7.6M	1.0 $\times$
	GRESO	76.6	14.8	61.8	66.2	33.3	38.5	48.5	3.3M	2.3 $\times$
	GRESO + Target-Mix	77.4	15.7	62.2	67.3	34.5	39.4	49.4	3.8M	2.0 $\times$

Appendix

Table 12: Performance (%) comparison across six math reasoning benchmarks. Three models are trained on DM or OR1. Base rows use target-validation checkpoint selection; +Target-Mix also trains on target validation examples; -TargetVal omits target-validation selection. Total RO denotes cumulative source responses in millions; RO ratio compares counts with DAPO, not wall-clock time. All entries are single runs; five-seed means of the base methods and L2RR are in Table 1 .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,774 papers

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
Oct 26, 2026Sohyun Son, Yerim Hong, Jaeyun Park +1Reinforcement Learning RL for Language Model Reasoning
Kyung Hee University, Yongin, Republic of Korea
F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
Feb 6, 2026Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4Reinforcement Learning RL for Language Model Reasoning
T-Tech · Saint Petersburg Electrotechnical University “LETI”
DataFlex-RL: An Evaluation Platform for RLVR Data Policies
Sep 5, 2026Hao Liang, Mingrui Chen, Hengyi Feng +2Reinforcement Learning RL Post-Training
Peking University · Zhongguancun Academy · UCAS +1

Oct 26, 2026J/K move · Enter open · S save

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung

Kyung Hee University, Yongin, Republic of Korea

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Reward Design for RL Group Relative Policy Optimization Mathematical Reasoning

Open paper AskPDF arXiv

Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficiency. Despite these gains, the extent to which differences among correct-answer rollouts reflect useful alternative reasoning or unresolved errors remains poorly understood. We investigate this question through a blind paired analysis of a model trained with Group Relative Policy Optimization (GRPO), comparing the highest- and lowest-agreement correct-answer traces for the same problem. Across 126 pairs, low-agreement traces have more spurious reasoning than high-agreement traces (35% vs. 22%), including internal inconsistencies and misuse of problem conditions. Based on these findings, we propose RoAR (Rollout-Agreement Reward), which adds reasoning-level agreement to GRPO's rewards for verified-correct rollouts. RoAR computes this signal from existing on-policy rollout groups, requiring no human-written rationales, additional sampling, or separately trained process evaluator. On five mathematical reasoning benchmarks, the 1.5B GRPO-RoAR model reaches 55.5% mean pass@1 under zero-shot evaluation using 7K training examples, exceeding GRPO and GRPO-CARE by 6.0 and 5.5 percentage points. A separate blind audit of 100 problems answered correctly by all four 1.5B models finds spurious rates of 18% for GRPO-RoAR, 32% for GRPO, 27% for GRPO-CARE, and 37% for DRA-GRPO. Our findings identify reasoning validity as an important consideration when using rollout differences to guide learning. Our code is available at https://github.com/ldi-kyunghee/RoAR.

Abstract

Reinforcement Learning

Explore similar work

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

DataFlex-RL: An Evaluation Platform for RLVR Data Policies