An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Andreas Plesner; Francisco Guzmán; Anish Athalye

cs.LGApr 9, 2026

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Authors: Andreas Plesner, Francisco Guzmán, Anish Athalye

Organizations: Handshake AI · ETH Zurich

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used for post-training Large Language Models, but practical verifiers can make errors. We study how the rate and structure of reward noise affect RLVR in code generation, with a preliminary scientific-reasoning check. In multi-seed Qwen3 8B experiments on MBPP, mean validation reward over two fixed late evaluations is within 1 percentage point of the clean baseline at the tested resampled group-rollout noise rates through 20%, and within about 2 points at 30%. Confidence intervals allow larger losses; these point estimates do not establish a general tolerance threshold. We also examine full-program pass@k, four controlled noise structures, two model-based verifiers, and policy models from three families spanning 4B-9B parameters. We derive conditional advantage distributions for symmetric and asymmetric group noise, including retained format penalties, and show why clipping limits simple gradient-scaling arguments. The analysis identifies information preserved by whole-group corruption and limits on interpreting our asymmetric sweep as a precision-recall comparison. Overall, the results indicate that imperfect verification can support effective RLVR in the tested settings, while aggregate error rates alone do not characterize the learning signal.

Figures & tables

Model	Setup	Seeds	Best	Final	Steps-to-best
Base model (before training)
GLM4 9B	Base model	$3$	$0.609\pm 0.010$	$0.609\pm 0.010$	$\mathrm{NA}$
Qwen3 8B	Base model	$3$	$0.487\pm 0.011$	$0.487\pm 0.011$	$\mathrm{NA}$
Baselines (ground-truth unit tests)
GLM4 9B	Baseline	$3$	$0.905\pm 0.002$	$0.884\pm 0.006$	$159\pm 34.6$
Qwen3 8B	Baseline	$4$	$0.900\pm 0.010$	$0.884\pm 0.013$	$209\pm 60.0$

Table 1 : Noise-type comparison runs on MBPP, evaluated against ground-truth unit tests. Best and final denote the highest and last checkpoint validation reward; for the recovered Qwen3 8B and GLM4 9B baselines and group-rollout runs, both are restricted to evaluation steps through 259, and final is step 259. The Qwen3 8B baseline and group-rollout rows use the expanded seed cohort of Table 8 .

\pm

values are standard deviations across seeds when available.

Setup	Best
Base model	$0.540$
No noise	$0.600$
Noise $p{=}0.05$	$0.604$
Noise $p{=}0.30$	$0.603$

Table 2 : Best validation reward on GPQA Diamond for Qwen3 8B trained with GSPO under group rollout noise (one seed per setting). Both noisy settings are close to the clean baseline.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Optimization
Optimizer	Adam
Learning rate	$1\times 10^{-6}$
LR schedule	Constant
Weight decay	$0.10$
$\beta_{1}$ , $\beta_{2}$	$0.90$ , $0.98$

Appendix

Table 3 : Hyperparameters used for all experiments. MBPP experiments use GRPO; GPQA experiments use GSPO.

Model	Hyperparameters	Seeds	Best	Final
GLM4 9B
	top- $p{=}1.00$ , $T{=}1.00$ , $B{=}96$ (main)	$3$	$0.905\pm 0.002$	$0.884\pm 0.006$
	top- $p{=}0.95$ , $T{=}1.00$ , $B{=}96$	$2$	$0.817\pm 0.005$	$0.764\pm 0.002$
	top- $p{=}1.00$ , $T{=}0.70$ , $B{=}96$	$1$	$0.895$	$0.881$
	top- $p{=}1.00$ , $T{=}1.00$ , $B{=}768$	$1$	$0.888$	$0.888$
Qwen3 8B

Appendix

Table 4 : Sensitivity to sampling hyperparameters for baseline (no noise) training on MBPP.

T

denotes temperature,

B

the global batch size. All runs use 16 rollouts per prompt. The main setting rows are reproduced from Table 1 .

Noise rate	$n$	Peak score (mean $\pm$ SD)	$\Delta$ (pp)	95% CI (pp)	$p$ -value
0	4	$0.9003\pm 0.0104$	—	—	—
0.05	3	$0.8879\pm 0.0220$	-1.24	$[-5.9,+3.5]$	0.3601
0.10	6	$0.8939\pm 0.0168$	-0.63	$[-2.6,+1.4]$	0.5246
0.15	4	$0.8922\pm 0.0102$	-0.81	$[-2.6,+1.0]$	0.3085
0.20	4	$0.8960\pm 0.0071$	-0.42	$[-2.0,+1.2]$	0.5269
0.25	4	$0.8837\pm 0.0051$	-1.66	$[-3.2,-0.1]$	0.0284

Appendix

Table 5: Expanded peak mean unit-test rewards for Qwen3 8B on MBPP with group-rollout noise.

n

is the number of seeds;

\Delta

is the difference from the clean baseline in percentage points, with a Welch 95% confidence interval. Each run contributes its best recorded score, including later checkpoints where available.

Noise rate	$n$	pass@1	pass@4	pass@8	pass@16
0	2	$0.814\pm 0.005$	$0.860\pm 0.017$	$0.875\pm 0.018$	$0.894\pm 0.008$
0.10	2	$0.841\pm 0.001$	$0.891\pm 0.004$	$0.907\pm 0.006$	$0.922\pm 0.000$
0.15	3	$0.816\pm 0.012$	$0.876\pm 0.011$	$0.895\pm 0.012$	$0.915\pm 0.006$
0.20	2	$0.827\pm 0.001$	$0.884\pm 0.001$	$0.911\pm 0.003$	$0.933\pm 0.000$
0.25	2	$0.818\pm 0.004$	$0.868\pm 0.000$	$0.882\pm 0.001$	$0.894\pm 0.008$
0.30	2	$0.810\pm 0.005$	$0.872\pm 0.013$	$0.898\pm 0.013$	$0.922\pm 0.016$

Appendix

Table 6: Peak full-program pass@k for Qwen3 8B on MBPP with group-rollout noise. Values are mean

\pm

standard deviation across

n

seeds. An SD displayed as

0.000

is rounded. The metric-specific seed cohort is smaller than that of Table 5 .

FNR $\backslash$ FPR	0.05	0.15	0.30	0.40	0.50
0.05	0.892	0.906	0.905	0.887	0.878
0.15	0.900	0.899	0.893	0.896	0.886
0.30	0.900	0.893	0.869	0.875	0.864
0.40	0.891	0.901	0.895	0.851	0.823 †
0.50	0.896	0.860	0.888	0.843	0.781 ‡

Appendix

Table 7: Peak mean unit-test rewards in the asymmetric-noise sweep. Rows are false-negative rates and columns are false-positive rates. Runs use seed 12 and evaluate every 10 rollout steps through 259, except

\dagger

, interrupted after step 229.

\ddagger

averages peaks from two records sharing seed 12, as in the original summary.

Noise rate	$n$	Best through 259	At step 259	Mean of 239, 259	$\Delta$ late (pp) [95% CI]
0	4	$0.900\pm 0.010$	$0.884\pm 0.013$	$0.890\pm 0.014$	—
0.05	3	$0.888\pm 0.022$	$0.885\pm 0.023$	$0.882\pm 0.018$	$-0.8$ $[-4.3,+2.7]$
0.10	6	$0.891\pm 0.015$	$0.883\pm 0.019$	$0.884\pm 0.018$	$-0.6$ $[-3.0,+1.8]$
0.15	4	$0.891\pm 0.009$	$0.884\pm 0.009$	$0.885\pm 0.008$	$-0.5$ $[-2.7,+1.7]$
0.20	4	$0.893\pm 0.012$	$0.885\pm 0.007$	$0.884\pm 0.012$	$-0.6$ $[-2.9,+1.8]$
0.25	4	$0.884\pm 0.005$	$0.878\pm 0.010$	$0.876\pm 0.007$	$-1.4$ $[-3.5,+0.8]$

Appendix

Table 8: Qwen3 8B mean unit-test rewards for the expanded cohort at fixed evaluation steps. The late-training metric averages steps 239 and 259 within each seed before aggregation;

\pm

denotes sample SD across seeds.

\Delta

late is its difference from the clean baseline with a Welch 95% confidence interval; at

p{=}0.50

the normal-theory interval extends beyond the attainable range because of the large variation across seeds. The best column uses available evaluations through 259 and still has unequal checkpoint-selection opportunities.

	Qwen2.5-Math-1.5B		Qwen2.5-Math-7B		DeepSeek-R1-Distill-Qwen-1.5B
	Reference	Range	Reference	Range	Reference	Range
Math	$69.4$	$[48,70]$	$75.1$	$[52,80]$	$83.9$	$[60,78]$
MinervaMath	$29.4$	$[6,18]$	$34.6$	$[10,25]$
Olympiad Bench	$31.3$	$[24,32]$	$38.2$	$[26,38]$
AIME 2024	$3.3$	$[7,16]$	$13.3$	$[12,30]$	$28.9$	$[10,20]$
AMC 2023	$45.0$	$[34,50]$	$62.5$	$[45,63]$

Appendix

Table 9: Previously tabulated pass@1 comparisons for Qwen2.5-Math-1.5B, Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B. References are the corresponding technical reports ( Yang et al., 2024 ; DeepSeek-AI, 2025 ) ; ranges span the base and clean-reward post-training scores in Figure 2 of Cai et al. (2025) . Evaluation protocols and benchmark variants have not been matched, so these numbers do not establish inconsistent evaluation or negative training effects.

Explore similar work

Quantifying Empirical Compute-Supervision Tradeoffs in RLVR
May 24, 2026Ryo Mitsuhashi, Patrick Chen, Isabelle Tseng +2Reinforcement Learning With Verifiable Reward Process-Level Supervision
1Princeton University
Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
Sep 28, 2026Christian Moya, Elliott Thornley, Guang LinReinforcement Learning With Verifiable Reward Verifiable Rewards
Purdue University · National University of Singapore
Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards
May 27, 2026Saurabh Dash, Pierre Clavier, John Dang +4Reinforcement Learning With Verifiable Reward Verifiable Rewards
Cohere Labs · Cohere · Adaption Labs +1

May 24, 2026, cs.LGJ/K move · Enter open · S save

Quantifying Empirical Compute-Supervision Tradeoffs in RLVR

Ryo Mitsuhashi, Patrick Chen, Isabelle Tseng, Jasin Cekinmez+1

1Princeton University

Reinforcement Learning With Verifiable Reward Process-Level Supervision

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training language models, but in practice, verifiers are rarely perfect. Recent theoretical work predicts that verifier noise affects the rate of learning but not its final outcome, implying that sufficient compute should close any gap induced by imperfect supervision. We test this prediction empirically by post-training Qwen2.5 (0.5B, 1.5B) with GRPO on GSM8K while injecting controlled false-positive and false-negative noise into the binary correctness signal, and varying rollouts per prompt as a compute axis. In practice, the gap in validation accuracy persists under substantial compute scaling, with returns to compute that are sharply diminishing. We further find a structural asymmetry where false negatives monotonically degrade performance more quickly than false positives. These findings suggest verifier quality and training compute are not interchangeable, and that reducing false negatives is a more effective lever than scaling compute alone.

Abstract

Explore similar work

Quantifying Empirical Compute-Supervision Tradeoffs in RLVR

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

Soft-SVeRL: Self-Verified Reinforcement Learning with Soft Rewards