Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts

Esther Xin

cs.LGSep 6, 2026

Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts

Authors: Esther Xin

Organizations: Independent Researcher

Abstract

Group-based reinforcement learning with verifiable rewards (RLVR) scoresmultiple completions per prompt using automatic verifiers. Analysesbased on independent verifier errors may overlook dependence associatedwith shared answer formats. We investigate this dependence in24,998 groups of eight completions generated by Qwen2.5-1.5B onMATH, GSM8K, and DeepMath-103K. We estimate a pooled within-groupverifier-error correlation of 0.530 (95% confidence interval:0.500--0.560). Under an exchangeable-error model, this correspondsto a design-effect-adjusted effective sample size of 1.70 for aneight-completion group. Dependence varies substantially across answerforms: fractions, radicals, symbolic expressions, and intervals exhibitstronger clustering than unit annotations and percent signs. Replayinggroup-relative advantages across four rule-based verifier configurationsidentifies at least one advantage-sign disagreement in up to 0.83% ofgroups. Because a group is repeated sampling for one prompt, thiswithin-group clustering may reflect shared prompt difficulty as well asshared answer form, and we do not attempt to separate the two here.Unlike studies of correlated judgments across multiple evaluators, ouranalysis examines dependence across completions scored by the sameverifier. These findings motivate prompt- and answer-form-aware analysesof verifier noise rather than characterizations based solely onaggregate error rates.

Figures & tables

Model	$n$ (groups)	$\rho_{\mathrm{cat}}$	95% CI	$n_{\mathrm{eff}}$ of $k{=}8$
Qwen2.5-1.5B	24,998	0.530	[0.500, 0.560]	1.70
Qwen2.5-7B-AWQ †	6,500	0.657	not computed	1.43

Table 1: Pooled within-group verifier-error correlation. † Partial corpus (13/50 planned shards); Gate 1’s mixed-group threshold, calibrated on the 1.5B pilot, is narrowly missed (0.298 vs. 0.30) because the stronger policy solves more problems outright and produces fewer partially-correct groups. Reported as an exploratory comparison; no bootstrap CI was computed for this partial run. The 1.5B estimate was stable across corpus scale during collection (0.523 at

n{=}2

k; 0.523 at

n{=}4

k; 0.530 at full

n{=}24{,}998

Category	$n$ (1.5B)	$\rho_{\mathrm{cat}}$ (1.5B)	$\rho_{\mathrm{cat}}$ (7B) †
degree_percent	109	$-0.001$	n/a
latex_text_unit	2,444	$-0.001$	0.000
trailing_whitespace	1,723	0.093	0.094
small_numeric	8,648	0.381	0.524
large_numeric	243	0.660	0.429
latex_frac	3,826	0.469	0.608

Table 2: Within-group ICC by primary answer category. † 7B column from the partial 6,500-group corpus; category

n

omitted for space (range 41–2,128, see released data). The formatting/whitespace split observed on the 1.5B corpus replicates on the partial 7B corpus for every category with

n\geq 100

: near-zero for latex_text_unit and trailing_whitespace , moderate to strong for every structurally complex category, higher on 7B than 1.5B in 5 of 6 comparable categories.

Verifier	Mean reward $p$	Observed degen.	Homog. reference	Gap
strict	0.247	0.670	0.104	+0.566
loose	0.260	0.648	0.090	+0.558
numeric	0.262	0.648	0.088	+0.560
flex	0.263	0.648	0.088	+0.560

Table 3: Group-level degeneracy (zero group-relative advantage for every member) on the full 1.5B corpus (

n{=}24{,}998

), against the homogeneous-difficulty i.i.d. reference

p^{k}+(1-p)^{k}

of Nie et al. (2026) . Every verifier shows a degenerate-group rate roughly

6.3

–

6.5\times

this reference. We do not attribute this gap to correlated verifier error alone : the reference assumes a single global pass rate, so ordinary between-prompt difficulty heterogeneity independently inflates degeneracy above it even with a perfect, uncorrelated verifier (§ 4.5 ).

Verifier pair	Sign-flip rate	Group corruption
strict vs. loose	0.0010	0.0053
strict vs. numeric	0.0017	0.0079
strict vs. flex	0.0018	0.0083
loose vs. numeric	0.0008	0.0029
loose vs. flex	0.0009	0.0034
numeric vs. flex	0.0001	0.0005

Table 4: Pairwise advantage-sign disagreement between rule-verifier configurations. Sign-flip rate is per-trace; group corruption is the fraction of groups with at least one sign flip. Rates are small because these four configurations agree on the large majority of clean, boxed answers; the consequential disagreement is between rule verifiers and human or model judgment (Table 6 ), not among rule configurations themselves.

Category	$n$	strict	loose	numeric/flex	compass
small_numeric	900	0.763	0.770	0.992	0.979
latex_text_unit	340	0.603	0.603	1.000	0.946
symbolic_other	900	0.696	0.940	0.949 / 0.913	0.960
latex_frac	900	0.720	0.953	0.987	0.950
large_numeric	648	0.869	0.912	0.976	0.949
trailing_whitespace	324	0.944	0.944	0.968	0.891

Table 5: Per-category verifier accuracy against

5{,}556

human-adjudicated labels (see § 3.6 for the annotation protocol). Inter-annotator agreement on the 200-trace double-annotated overlap (raw agreement 1.00, Cohen’s

\kappa=1.00

, Krippendorff’s

\alpha=1.00

) is higher than we would expect for a task including genuinely ambiguous equivalence judgments; see § 6 for a recommendation to replicate with a blinded, pre-registered protocol at larger scale. numeric and flex coincide except where noted (interval/percent/set forms). No single rule verifier or CompassVerifier-3B is most accurate across every category; the best verifier differs by category (Table 6 ).

Verifier	Accuracy	$n$ covered	Coverage
strict	0.759	5,556	100.0%
loose	0.895	5,556	100.0%
numeric	0.982	1,717	30.9%
flex	0.974	2,025	36.4%
CompassVerifier-3B (alone)	0.949	2,367	42.6%
Category router	0.971	3,088	55.6%

Table 6: Accuracy against human-adjudicated labels, reported alongside coverage because these verifiers are not comparable on accuracy alone. We do not claim the router is globally superior to strict or loose at full coverage : no verifier here is evaluated at matched coverage, and the router’s category-selection rule was chosen using the same labeled set on which it is evaluated, with no held-out split. The router sends 1,716 of its 3,088 covered traces to CompassVerifier, a 1.29

\times

query-cost ratio against an appeals-style baseline that queries CompassVerifier on every one of the 1,328 rule-disagreement cases regardless of category.

Figure 1: Within-group error ICC by primary answer category, sorted, with group counts annotated.

Figure 2:

\rho_{\mathrm{cat}}

by policy strength. The 1.5B estimate has a 95% bootstrap CI (Table 1 ); no CI was computed for the partial 7B run. The 7B corpus is partial (13/50 planned shards) and AWQ-quantized; see § 4.4 for the resulting confound.

Figure 3: Observed group degeneracy against the homogeneous-difficulty i.i.d. reference of Nie et al. (2026) for every verifier configuration. This gap is not attributed solely to correlated verifier error; see § 4.5 for the between-prompt-difficulty confound.

Figure 4: Category-conditioned router vs. single verifiers: accuracy against human-adjudicated labels, plotted against coverage. Not a matched-coverage comparison; see § 4.6 .

LLM Answer Verification

Large language model answer verification checks whether generated answers are correct, supported by evidence, and safe to accept.

See all 151 papers

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeRL for Code Generation Reinforcement Learning
Handshake AI · ETH Zurich
Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR
Sep 1, 2026Esther XinLLM Answer Verification Reinforcement Learning with Verifiable Rewards
Independent Researcher
Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR
Oct 13, 2026Savyasachi Gupta, Aniruddha Srinivas JoshiReinforcement Learning with Verifiable Rewards RL for Language Model Reasoning
Independent Researcher TX, USA · Independent Researcher CA, USA

Apr 9, 2026J/K move · Enter open · S save

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Andreas Plesner, Francisco Guzmán, Anish Athalye

Handshake AI · ETH Zurich

RL for Code Generation Reinforcement Learning Reinforcement Learning with Verifiable Rewards RL Post-Training

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used for post-training Large Language Models, but practical verifiers can make errors. We study how the rate and structure of reward noise affect RLVR in code generation, with a preliminary scientific-reasoning check. In multi-seed Qwen3 8B experiments on MBPP, mean validation reward over two fixed late evaluations is within 1 percentage point of the clean baseline at the tested resampled group-rollout noise rates through 20%, and within about 2 points at 30%. Confidence intervals allow larger losses; these point estimates do not establish a general tolerance threshold. We also examine full-program pass@k, four controlled noise structures, two model-based verifiers, and policy models from three families spanning 4B-9B parameters. We derive conditional advantage distributions for symmetric and asymmetric group noise, including retained format penalties, and show why clipping limits simple gradient-scaling arguments. The analysis identifies information preserved by whole-group corruption and limits on interpreting our asymmetric sweep as a precision-recall comparison. Overall, the results indicate that imperfect verification can support effective RLVR in the tested settings, while aggregate error rates alone do not characterize the learning signal.

Abstract

LLM Answer Verification

Explore similar work

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR