Reinforcement learning with verifiable rewards (RLVR) has become a principal recipe for improving the reasoning of large language models (LLMs). It rests on the assumption that the checker scoring each response is itself correct. In practice, rule-based checkers reject correct answers in unexpected formats, unit tests accept incorrect programs, and LLM judges can be led to approve empty or adversarial responses. This work surveys 58 studies that measure, analyze, or mitigate such verifier errors. We ask what errors verifiers make, how they affect training, which mitigations work against which errors, and how the findings extend to self-generated, rubric- and judge-based, and verifier-free rewards. The evidence suggests that the structure of an error matters more than its rate. Random errors mostly slow learning, though under a fixed budget or KL regularization they also lower final accuracy. False positives that the policy can learn to trigger can cause plateaus or collapse, sometimes even with a low initial error rate. Whether a verifier wrongly rejects or accepts depends on what it checks and who built it, not on its family. Its errors shift as the policy improves, and most mitigations have been tested against only one kind of error. We compile the findings into comparison tables, reconcile apparent conflicts in the literature, and propose a reporting protocol to make future results comparable.
Figures & tables
Fig. 1: Taxonomy of the surveyed literature on verifier noise in RLVR, organized by research question. Each leaf lists representative studies.
Fig. 2: Where errors enter the RLVR loop: (1) wrong or incomplete reference labels, (2) answer extraction and verifier configuration, (3) the checker, test suite, or judge itself, and (4) access that lets the policy alter the verifier or its environment.
Study
Reward signal and error
Outcome
Key result
Egashira [ 14 ]
Rule checker; injected random FP/FN, systematic FN, exploitable FP
Random: delay (lower endpoint for OLMo at 50 % FNR); exploitable FP: plateau or collapse
Pass@1 0.6 with a “python” trigger vs. 85.6 clean
Rad [ 17 ]
Unit tests; random flips across J
J>0 : delay or lower endpoint; J=0 : no learning; J<0 : collapse
At J=0.3 : 16.0 FN-heavy vs. 14.6 FP-heavy (clean 20.8)
El Mansouri [ 19 ]
Rule and reward models; i.i.d. flips; no ratio clipping
Extra inference; small RL gains; own FP rarely reported
Judge hardening [ 25 , 8 ]
Discovered and master-key FP
One policy–judge pair; offline only in [ 8 ]
Structural checks [ 23 , 37 ]
Enumeration shortcut; PRM hacking
Task-specific; outcome-verifier errors untouched
Data audits [ 13 , 22 , 31 ]
Wrong labels; leaky tests; inherited hacking
Covers only what is audited
TABLE II: Mitigation Families and the Errors They Were Tested On
Theme
Report
Motivation
Verifier
Identity, version, extraction configuration and coverage; judging granularity and output cap; the evaluation grader, separately
[ 6 , 67 , 10 , 52 ]
Verifier error
FPR and FNR, or J^ with an interval, on the policy’s own outputs at the start and end of training; noise structure, with FP and FN tested separately; adversarial probes of model verifiers; invalid-response rate
[ 17 , 14 , 20 , 61 ]
Training
An oracle reward beside the training reward; final and best checkpoints; pass@ k with pass@1; seeds or variance; no reliance on entropy as a health signal
[ 14 , 13 , 35 ]
Evaluation
More than one model family; random- and format-reward baselines; post-cutoff benchmarks and a contamination probe; per-problem and per-category breakdowns
[ 33 , 34 , 43 , 75 ]
Label-free rewards
Reward accuracy against gold, not only pseudo-label accuracy
[ 41 , 42 ]
Data and release
Audited reference labels; distillation data filtered on true reward; released verdicts and code