Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR

Savyasachi Gupta; Aniruddha Srinivas Joshi

cs.LGOct 13, 2026

Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR

Authors: Savyasachi Gupta, Aniruddha Srinivas Joshi

Organizations: Independent Researcher TX, USA · Independent Researcher CA, USA

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a principal recipe for improving the reasoning of large language models (LLMs). It rests on the assumption that the checker scoring each response is itself correct. In practice, rule-based checkers reject correct answers in unexpected formats, unit tests accept incorrect programs, and LLM judges can be led to approve empty or adversarial responses. This work surveys 58 studies that measure, analyze, or mitigate such verifier errors. We ask what errors verifiers make, how they affect training, which mitigations work against which errors, and how the findings extend to self-generated, rubric- and judge-based, and verifier-free rewards. The evidence suggests that the structure of an error matters more than its rate. Random errors mostly slow learning, though under a fixed budget or KL regularization they also lower final accuracy. False positives that the policy can learn to trigger can cause plateaus or collapse, sometimes even with a low initial error rate. Whether a verifier wrongly rejects or accepts depends on what it checks and who built it, not on its family. Its errors shift as the policy improves, and most mitigations have been tested against only one kind of error. We compile the findings into comparison tables, reconcile apparent conflicts in the literature, and propose a reporting protocol to make future results comparable.

Figures & tables

Panel of Figure 1 — Fig. 1: Taxonomy of the surveyed literature on verifier noise in RLVR, organized by research question. Each leaf lists representative studies.

Panel of Figure 2 — Fig. 2: Where errors enter the RLVR loop: (1) wrong or incomplete reference labels, (2) answer extraction and verifier configuration, (3) the checker, test suite, or judge itself, and (4) access that lets the policy alter the verifier or its environment.

Study	Reward signal and error	Outcome	Key result
Egashira [ 14 ]	Rule checker; injected random FP/FN, systematic FN, exploitable FP	Random: delay (lower endpoint for OLMo at 50 % FNR); exploitable FP: plateau or collapse	Pass@1 0.6 with a “python” trigger vs. 85.6 clean
Rad [ 17 ]	Unit tests; random flips across $J$	$J>0$ : delay or lower endpoint; $J=0$ : no learning; $J<0$ : collapse	At $J=0.3$ : 16.0 FN-heavy vs. 14.6 FP-heavy (clean 20.8)
El Mansouri [ 19 ]	Rule and reward models; i.i.d. flips; no ratio clipping	Lower endpoint, strongly model-dependent	Losses of 0.2–30.7 points
Plesner [ 11 ]	Unit tests; symmetric flips; small model verifier	Group-level ${\leq}15\,\%$ : graceful; sample-level: lower endpoint; 4B verifier: plateau	Best reward 0.704 vs. 0.871 with a 30B verifier
Lv [ 12 ]	Rule; question-wise flips; PPO	${\leq}40\,\%$ : graceful or delay; 50 %: no learning; Llama-3.1-8B: collapse at 30 %	72.02 at 40 % flips vs. 75.85 clean
Cai [ 18 ]	Rule; class-conditional and format-dependent noise	Uncorrected: lower endpoint and loss of general capability	ARC-C 73.9 $\rightarrow$ 43.9 (Llama-3.2-3B)

TABLE I: Training Outcomes Under Reference-Based Verifier Errors

Family	Tested against	Cost or gap
Noise-aware gradients [ 18 , 19 ]	Class-conditional noise; format FN; real reward models	Needs error rates; fails on model-aligned wrong labels [ 13 ]
Noise-robust regularization [ 70 ]	Random label corruption	Backbone-dependent; extra hyperparameters
Stronger verifiers [ 21 , 61 , 36 , 20 ]	Natural rule FN; judge hacking	Extra inference; small RL gains; own FP rarely reported
Judge hardening [ 25 , 8 ]	Discovered and master-key FP	One policy–judge pair; offline only in [ 8 ]
Structural checks [ 23 , 37 ]	Enumeration shortcut; PRM hacking	Task-specific; outcome-verifier errors untouched
Data audits [ 13 , 22 , 31 ]	Wrong labels; leaky tests; inherited hacking	Covers only what is audited

TABLE II: Mitigation Families and the Errors They Were Tested On

Theme	Report	Motivation
Verifier	Identity, version, extraction configuration and coverage; judging granularity and output cap; the evaluation grader, separately	[ 6 , 67 , 10 , 52 ]
Verifier error	FPR and FNR, or $\hat{J}$ with an interval, on the policy’s own outputs at the start and end of training; noise structure, with FP and FN tested separately; adversarial probes of model verifiers; invalid-response rate	[ 17 , 14 , 20 , 61 ]
Training	An oracle reward beside the training reward; final and best checkpoints; pass@ $k$ with pass@1; seeds or variance; no reliance on entropy as a health signal	[ 14 , 13 , 35 ]
Evaluation	More than one model family; random- and format-reward baselines; post-cutoff benchmarks and a contamination probe; per-problem and per-category breakdowns	[ 33 , 34 , 43 , 75 ]
Label-free rewards	Reward accuracy against gold, not only pseudo-label accuracy	[ 41 , 42 ]
Data and release	Audited reference labels; distillation data filtered on true reward; released verdicts and code	[ 13 , 31 , 27 ]

TABLE III: Recommended Reporting Protocol

Explore similar work

Apr 6, 2026

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.

Kazuki Egashira, Mark Vero, Jasper Dekoninck +3

ETH Zurich · Max Planck Institute for Intelligent Systems, Tübingen

Reinforcement Learning with Verifiable Rewards RL for Language Model Reasoning

Ask PDF

Oct 1, 2025

Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers

Reinforcement Learning with Verifiable Rewards (RLVR) replaces costly human labeling with automated verifiers. To reduce verifier hacking, many RLVR systems binarize rewards to

\{0,1\}

, but imperfect verifiers inevitably introduce false negatives (rejecting correct answers) and false positives (accepting incorrect ones). We formalize verifier unreliability as a stochastic reward channel with asymmetric noise rates

ρ_0

and

ρ_1

-- the FP rate and the FN rate, respectively. From this abstraction we derive two lightweight corrections: (i) a backward correction that yields an unbiased surrogate reward and thus an unbiased policy-gradient estimator in expectation, and (ii) a forward correction that reweights score-function terms so the expected update aligns with the clean gradient direction and requires only the FN rate. We implement both as lightweight hooks in a group relative policy optimization pipeline, both corrections improve RLVR for math reasoning under synthetic and real verifier noise, with the forward variant being more stable under heavier noise. Finally, an appeals mechanism with a lightweight LLM verifier estimates the FN rate online and further improves performance.

Xin-Qiang Cai, Wei Wang, Feng Liu +3

RIKEN AIP, Tokyo, Japan · The University of Tokyo, Tokyo, Japan · The University of Melbourne, Melbourne, Australia

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards RL Post-Training

Ask PDF

Apr 9, 2026

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used for post-training Large Language Models, but practical verifiers can make errors. We study how the rate and structure of reward noise affect RLVR in code generation, with a preliminary scientific-reasoning check. In multi-seed Qwen3 8B experiments on MBPP, mean validation reward over two fixed late evaluations is within 1 percentage point of the clean baseline at the tested resampled group-rollout noise rates through 20%, and within about 2 points at 30%. Confidence intervals allow larger losses; these point estimates do not establish a general tolerance threshold. We also examine full-program pass@k, four controlled noise structures, two model-based verifiers, and policy models from three families spanning 4B-9B parameters. We derive conditional advantage distributions for symmetric and asymmetric group noise, including retained format penalties, and show why clipping limits simple gradient-scaling arguments. The analysis identifies information preserved by whole-group corruption and limits on interpreting our asymmetric sweep as a precision-recall comparison. Overall, the results indicate that imperfect verification can support effective RLVR in the tested settings, while aggregate error rates alone do not characterize the learning signal.

Andreas Plesner, Francisco Guzmán, Anish Athalye

Handshake AI · ETH Zurich

RL for Code Generation Reinforcement Learning Reinforcement Learning with Verifiable Rewards RL Post-Training

Ask PDF