Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.
Figures & tables
Figure 1: The shared GRPO workflow and the intervention point of each data-policy family. Mixture adaptation changes the domains supplying future prompts; selection and reweighting act after rollout and verification by modifying the current update. Uniform sampling with unit mask and weight is the common baseline.
Log Name
Family
Signal
Implemented Intervention
Level
Control
baseline
baseline
none
wiℓ=1 for every valid response token
control
Selection: fixed uniform domain mixture
difffilter
selection
group solve rate qg
retain the whole group iff 0.2<qg<0.8 ; a stricter DAPO-style filter [ 31 ]
retain the top three of five responses within each prompt group [ 21 ]
group
Table 1: The 13 logged configurations, separated into selection, reweighting, and mixture adaptation. Reweighting formulas are mean-normalized, and all mixture probabilities use T=1 and a 0.05 floor.
MATH
LOGIC
SCIENCE
SUMMARY
Method
M500
A24
Oly
Min
GSM8K
K&K
LD
Track
Zebra
GPQA
Chem
Phys
Overall
95% CI
baseline
84.45
14.44
38.06
34.93
91.42
20.46
65.77
78.75
29.78
33.71
53.17
57.05
49.77
[49.29,50.26]
difffilter
84.23
15.01
37.78
35.06
91.10
22.38
65.76
76.97
32.28
32.58
53.75
56.42
49.85
[49.10,50.61]
maxvar
84.18
14.16
37.77
35.81
90.91
18.38
65.13
78.13
29.47
31.65
53.13
56.93
49.19
[48.57,49.82]
gfpo
83.78
13.88
37.88
34.94
91.31
17.62
64.80
76.16
30.53
31.73
53.23
56.03
48.88
[48.03,49.73]
topk
84.30
14.17
37.82
34.69
91.42
20.12
66.31
76.69
30.13
32.79
53.02
56.33
49.39
[48.67,50.11]
Table 3: Twelve-benchmark means for the complete Qwen2.5-7B-base primary experiment. Each row averages 12 matched seeds; Overall gives equal weight to the MATH, LOGIC, and SCIENCE domain means. The final column is the 95% t interval for the seed-level Overall mean; paired method comparisons are reported in Table 4 .
Evaluation Summary
Rank ρ vs DB-12
Method Spread
DB-12 (domain-balanced)
1.00
3.31
Macro-12
0.88
2.79
Item-12
0.88
2.17
Math-Heavy-6
−0.33
0.90
Table 8: Sensitivity of the nine Qwen2.5-7B-Instruct configuration means to evaluation-domain coverage and aggregation. Rank correlation is Spearman’s ρ relative to DB-12; spread is the highest minus lowest configuration mean in percentage points. Each configuration mean uses the same three trained seeds.
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Bauer et al. (2026) showed that in low-data reinforcement learning with verifiable rewards, training GRPO on mixed-difficulty prompts gives up to 5x the sample efficiency of easy-only prompts. They did not test whether this benefit is specific to GRPO, or whether another post-training method given the same data would obtain it too. We test this on their procedural counting task with Qwen3-4B-Base and LoRA, comparing GRPO to rejection-sampling fine-tuning (RFT) on the same prompts, verifier and token cap. Six primary arms cross the two methods with three prompt sets (Easy, Mixed, and a Curated subset of prompts the base model sometimes solves, which makes GRPO's implicit filter explicit). We use three seeds per arm. Hypotheses and falsifiers were registered before training. To rule out a poorly tuned baseline, RFT receives a 27-run hyperparameter sweep, while GRPO uses Bauer et al.'s recipe unchanged. GRPO outperforms RFT at every prompt set: by 10.9 points on held-out test problems with curated prompts, and, in post hoc comparisons with matched rollout budgets, by 8.4 and 9.2 points on the easy and mixed prompts. On test the advantage survives the least favorable treatment of truncated completions; out of distribution, truncation prevents a conclusion. Curating prompts does not help RFT (-2.7 points). Mixed-difficulty data helps RFT (+3.7 points), while for GRPO the evidence is inconclusive. A secondary iterated-RFT arm, which resamples from the current policy, recovers 3.2 of the 10.9-point gap (about 30%). These are controlled measurements on one synthetic task family and one base model with three seeds per arm.
Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations, achieving performance comparable to learning from clean data. In this work, we show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data. After rectifying the dataset with a rigorous re-verification pipeline, we demonstrate that noise is destructive to RLVR. We show that existing RLVR algorithm improvements fail to mitigate the impact of noise, achieving similar performance to that of the basic GRPO. Furthermore, we find that the model trained on truly incorrect annotations performs 8-10% worse than the model trained on clean data across mathematical reasoning benchmarks. Finally, we show that these findings hold for real-world noise in Text2SQL tasks, where training on real-world, human annotation errors cause 5-12% lower accuracy than clean data. Our results show that current RLVR methods cannot yet compensate for poor data quality. High-quality data remains essential.
Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only ≤1%, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.
Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng +11
1National University of Singapore, HPC-AI Lab · 2DAMO Academy, Alibaba Group · 3Hupan Lab +7