Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards

Yuxuan Zhu; Daniel Kang

cs.LGMar 17, 2026

Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewards

Authors: Yuxuan Zhu, Daniel Kang

Organizations: UIUC · Bridgewater AIA Labs

Abstract

Reinforcement learning with verifiable rewards (RLVR) has driven recent capability advances of large language models across various domains. Recent studies suggest that improved RLVR algorithms allow models to learn effectively from incorrect annotations, achieving performance comparable to learning from clean data. In this work, we show that these findings are invalid because the claimed 100% noisy training data is "contaminated" with clean data. After rectifying the dataset with a rigorous re-verification pipeline, we demonstrate that noise is destructive to RLVR. We show that existing RLVR algorithm improvements fail to mitigate the impact of noise, achieving similar performance to that of the basic GRPO. Furthermore, we find that the model trained on truly incorrect annotations performs 8-10% worse than the model trained on clean data across mathematical reasoning benchmarks. Finally, we show that these findings hold for real-world noise in Text2SQL tasks, where training on real-world, human annotation errors cause 5-12% lower accuracy than clean data. Our results show that current RLVR methods cannot yet compensate for poor data quality. High-quality data remains essential.

Figures & tables

Figure 1: Noisy data significantly degrades RLVR performance, while existing algorithms fail to mitigate it. Training Qwen2.5-Math-7B on 100% incorrect annotations performs similarly to format-only rewards and 9% worse than clean data. Even the best-performing RLVR algorithmic improvements [ 16 , 37 , 3 , 7 , 39 ] trained on 50% noise only match baseline GRPO [ 30 ] and underperform the model trained on clean data.

Figure 2: Data re-verification pipeline for synthesizing truly noisy dataset. As shown in the two examples, we identify the issues of insufficient ground-truth annotations and inadequate equivalence checking, and address them via a rigorous pipeline combining LLMs and manual verification.

Panel of Figure 3 — Figure 3: Both incorrect and random annotations significantly decrease the performance of RLVR: RLVR with incorrect annotations achieves comparable or lower accuracy (by 1.2–5.6%) than RLVR with format rewards, and lower accuracy (by 8.5–10.0%) than RLVR with clean data; RLVR with random annotations achieves significantly lower accuracy (by 6.7–27.8%) than the base model.

Panel of Figure 7 — Figure 7: Existing algorithmic improvements fail to mitigate the impact of noisy data. Under 50% noise, none of the improved algorithms achieved a $>2$ % accuracy improvement on all benchmarks simultaneously compared to GRPO with 50% noise, while all algorithms underperform the vanilla GRPO with clean data by 3.1–7.3%.

Figure 8: Evaluated algorithmic improvements fail to achieve a higher pass@

k

than GRPO under 50% noise.

Figure 9: Real-world noise significantly degrades RLVR performance. Models trained on the noisy BIRD dataset (BIRD-Original-600) via GRPO results in 5.7–12.1% lower accuracy than BIRD-Corrected-600. PGFC [ 3 ] , an algorithm designed for noisy data, fails to consistently improve over GRPO and remains 5.1–16.3% behind the clean data.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Training dynamics of RLVR under clean versus 100% noisy data. (a) The model achieves a 149% higher average reward when trained on noisy data, indicating that it easily exploits and satisfies its own flawed priors. (b) Training on noise induces a more rapid decline in policy entropy (9% lower on average), reflecting premature convergence and diminished exploration.

Panel of Figure 11 — Figure 11: Noise leads to lower pass@ $k$ than the base model when $k>1$ , showing that noise does not improve capability boundary.

Panel of Figure 12 — Figure 12: Training on noisy data results in increasing performance degradation (by up to 11.4%) as the noise proportion increases.

Panel of Figure 13 — Figure 13: Training on noisy data results in shorter responses, indicating that noise induces weaker reasoning.

Panel of Figure 14 — Figure 14: Existing algorithm improvements fail to mitigate the impact of noisy data. Under 50% noise, none of the improved algorithm achieved $>2$ % accuracy improvement on all benchmarks simultaneously compared to GRPO with 50% noise, while all algorithms underperform the vanilla GRPO with clean data by 3.1-7.3%

Panel of Figure 15 — Figure 15: None of the evaluated algorithmic improvements achieve higher pass@ $k$ score than GRPO under 50% noise.

Panel of Figure 16 — Figure 16: Training on noisy data with improved algorithms still results in shorter responses.

Panel of Figure 17 — Figure 17: After three epochs (600 steps), no model shows improvements of average pass@1 of all benchmarks in the subsequent 50 steps.

Panel of Figure 18 — Figure 18: Starting the tenth epoch, no model shows improvements in terms of reward.

Category	Hyperparameter	Value
Model	Base model	Qwen/Qwen2.5-Math-7B
	Training strategy	FSDP2 (colocated train + generate)
	GPUs per node	4
	Gradient checkpointing	enabled
Optimization	Train batch size (prompts)	64
	Policy mini-batch size	64

Appendix

Table 1: Hyperparameters shared by every math RLVR run. All algorithms use the GRPO advantage estimator on top of these settings; algorithm-specific overrides are listed in Table 2 .

Algorithm	Loss aggregation	Other
GRPO	token mean	—
PGFC	token mean	$\rho$ matched to dataset noise
Dr. GRPO	sequence mean	grpo_norm_by_std=false
DAPO	token mean	see below
TIS	token mean	DAPO + IS-ratio cap $2.0$
SAPO	token mean	$\tau_{+}{=}1.0$ , $\tau_{-}{=}1.05$

Appendix

Table 2: Algorithm-specific hyperparameters. Entries marked “–” use the shared default from Table 1 , including the learning rate of

5\times 10^{-6}

and the zero KL coefficient. For algorithms inherited from prior work we follow the hyperparameters recommended in the original papers.

Category	Hyperparameter	Value
Optimization	Batch size (prompts)	64
	Group size (rollouts per prompt)	16
	Learning rate	$5\times 10^{-5}$
	LoRA rank	32
	Number of epochs	10
	Loss function	importance sampling

Appendix

Table 3: Hyperparameters shared by every Text2SQL RLVR run. Runs with PGFC additionally set the noise_rate hyperparameter to the value in Table 4 ; all other runs leave it unset.

Model	PGFC noise rate $\rho$
Qwen3-235B-A22B-Instruct-2507	0.62
DeepSeek-V3.1	0.62
Qwen3-32B	0.62
GPT-OSS-120B-A5	0.62
Llama-3.3-70B-Instruct	0.62

Appendix

Table 4: Per-run differences for the Text2SQL experiments. The clean condition trains on bird-plat-588.parquet (BIRD-Corrected) and the noisy condition trains on noisy-600.parquet (BIRD-Original). PGFC uses the noisy training file together with the listed annotation-error-rate estimate.

Explore similar work

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeRL for Code Generation Reinforcement Learning
Handshake AI · ETH Zurich
Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
Apr 6, 2026Kazuki Egashira, Mark Vero, Jasper Dekoninck +3Reinforcement Learning with Verifiable Rewards RL for Language Model Reasoning
ETH Zurich · Max Planck Institute for Intelligent Systems, Tübingen
Can LLMs Learn to Reason Robustly under Noisy Supervision?
Apr 5, 2026Shenzhi Yang, Guangcheng Zhu, Bowen Song +7LLM Reasoning Noisy-Label Learning
Zhejiang University · Ant Group · University of Wisconsin-Madison

Apr 9, 2026J/K move · Enter open · S save

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Andreas Plesner, Francisco Guzmán, Anish Athalye

Handshake AI · ETH Zurich

RL for Code Generation Reinforcement Learning Reinforcement Learning with Verifiable Rewards RL Post-Training

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used for post-training Large Language Models, but practical verifiers can make errors. We study how the rate and structure of reward noise affect RLVR in code generation, with a preliminary scientific-reasoning check. In multi-seed Qwen3 8B experiments on MBPP, mean validation reward over two fixed late evaluations is within 1 percentage point of the clean baseline at the tested resampled group-rollout noise rates through 20%, and within about 2 points at 30%. Confidence intervals allow larger losses; these point estimates do not establish a general tolerance threshold. We also examine full-program pass@k, four controlled noise structures, two model-based verifiers, and policy models from three families spanning 4B-9B parameters. We derive conditional advantage distributions for symmetric and asymmetric group noise, including retained format penalties, and show why clipping limits simple gradient-scaling arguments. The analysis identifies information preserved by whole-group corruption and limits on interpreting our asymmetric sweep as a precision-recall comparison. Overall, the results indicate that imperfect verification can support effective RLVR in the tested settings, while aggregate error rates alone do not characterize the learning signal.

Abstract

Explore similar work

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

Can LLMs Learn to Reason Robustly under Noisy Supervision?