A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards

Andreas Plesner; Curtis Northcutt; Francisco Guzmán; Anish Athalye

cs.LGSep 27, 2026

A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewards

Authors: Andreas Plesner, Curtis Northcutt, Francisco Guzmán, Anish Athalye

Organizations: ETH Zurich · Handshake AI

Abstract

When post-training large language models on tasks with semi-verifiable rewards, there are many factors (training steps, base model size, training order, data quality, verifier accuracy, etc.) that practitioners must contend with to maximize model performance. Yet, it remains unclear how well verifier agreement predicts post-training performance on such tasks. In this paper, we explore this question with over 11k H100 GPU-hours, across HealthBench and PRBench tasks in medical, legal, and finance domains. Across the tested domains, Qwen3 trainees (1.7B-8B on HealthBench; 8B on PRBench), evaluation splits, and frontier LLM reference judges (which we call golden verifiers), higher verifier agreement does not consistently identify the best training verifier. Expensive verifiers need not outperform inexpensive ones, and open-weight Gemma verifiers produce strong training outcomes. We compare two low-cost choices retrospectively -- a cost-reducing choice and a balanced choice -- with estimated grading cost reductions of 98.8%-99.7% relative to the golden grading protocols and average post-training score gaps of 1-3 points from the best evaluated training verifier. These averages include larger losses in individual settings; they do not establish that verifier choices are interchangeable.

Figures & tables

Domain	Train ( $n$ )	Hard eval-only ( $n$ )	Total eval ( $n$ )
HealthBench	$3600$	$1000$	$1400$
PRBench legal	$225$	$249$	$274$
PRBench finance	$269$	$299$	$329$

Table 1: Available dataset pools for training and evaluation. “Hard eval-only” contains only hard tasks; “Total eval” also includes held-out non-hard tasks. Policy scoring uses subsets of these pools.

Verifier	Calls/task	Tokens/task	USD/1k tasks
Claude Opus 4.6 (golden)	11.6	21,321	133.20
GPT-5.5 (golden)	11.6	17,093	99.70
Gemma 4 31B	4.3	5,498	0.77
Gemma 4 26B	4.3	5,514	0.45
GPT oss 20B	4.3	6,157	0.08

Table 2: Grading resources and cost on a fixed HealthBench probe. Calls and total input-plus-output tokens are per task; USD is per

1{,}000

tasks at the recorded prices. Golden protocols use one criterion per call and medium thinking; training protocols use up to three criteria and low thinking. Probe sampling and pricing qualifications are in Appendix M .

		Overall		HealthBench		PR-finance		PR-legal
Variant	Metric	all	hard	all	hard	all	hard	all	hard
Cheapest verifier (no quality floor)	Score gap	+0.030	+0.026	+0.062	+0.054	+0.011	+0.011	+0.017	+0.013
	Uplift	77%	86%	46%	70%	95%	95%	89%	93%
	Cost reduction	99.7%	99.7%	99.4%	99.5%	99.8%	99.8%	99.8%	99.8%
Cost-reducing Pareto verifier	Score gap	+0.030	+0.023	+0.062	+0.046	+0.011	+0.011	+0.017	+0.013
	Uplift	77%	88%	46%	76%	95%	95%	89%	93%

Table 3: Verifier selection across

18

evaluation cells (

3

datasets

\times

2

splits

\times

3

goldens; trainee Qwen3 8B). Splits : all = hard

+

non-hard combined with the weights in Appendix I ; hard = the hard subset only. The cost-reducing rule, fixed balanced choice, and score gap and cost reduction metrics are defined in Section 4 . Uplift

=(\text{chosen}-\text{baseline})/(\text{max}-\text{baseline})

is the fraction of available post-training improvement captured (

100\%

matches the best verifier per cell;

0\%

matches no training). Bold marks the better value per column between the cost-reducing and balanced picks.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Optimizer	Adam
Learning rate	$5\times 10^{-6}$ (constant)
Weight decay	$0.10$
Effective KL penalty	None
Algorithm	GRPO (group-std normalized advantages)
Rollouts per prompt	$8$

Appendix

Table 4: Final hyperparameters for HealthBench experiments. Trainees are Qwen3 1.7B, 4B, 8B; the same hyperparameter set is used for all three sizes, modulo per-size GPU/TP layout ( Table 5 ).

Trainee	GPUs/node	Tensor parallel	Approx. GPU-hours per cell
Qwen3 1.7B	4	1	12
Qwen3 4B	4	2	18
Qwen3 8B	6	2	24

Appendix

Table 5: GPU and tensor-parallel configuration per trainee size.

	Agreement	Cohen’s $\boldsymbol{\kappa}$
Pairwise
Claude Opus 4.6 vs. Gemini 3.1 Pro	$90.8\%$	$0.81$
Claude Opus 4.6 vs. GPT-5.5	$87.9\%$	$0.75$
GPT-5.5 vs. Gemini 3.1 Pro	$84.4\%$	$0.67$
vs. $2$ -of- $3$ majority vote
Claude Opus 4.6	$97.1\%$	$0.94$

Appendix

Table 8: Alignment between the three golden verifiers on the

454

rubric criteria of a

40

-task HealthBench slice (

20

hard

+

20

non-hard ), graded on Qwen3 8B generations. Verdicts are within-judge majorities over

3

generations. Top block: pairwise agreement and Cohen’s

\kappa

. Bottom block: agreement of each judge with the per-criterion

2

-of-

3

majority vote over all three judges (the judge itself is part of the vote). All three judges are unanimous on

81.5\%

of criteria.

							HealthBench score
Training verifier	Accuracy	Precision	Recall	F1	Spearman	Intelligence	8B	4B	1.7B
Gemma 4 31B	0.85	0.80	0.87	0.83	0.78	1115	0.34	0.31	0.26
GPT oss 120B	0.82	0.80	0.78	0.79	0.72	832	0.31	–	0.20
GPT oss 20B	0.81	0.77	0.78	0.78	0.71	549	0.28	0.26	0.21
Gemini 3 flash preview	0.81	0.75	0.84	0.79	0.75	1205	0.32	0.32	0.24
Gemma 4 26B	0.79	0.74	0.78	0.76	0.68	1013	0.34	0.31	0.27

Appendix

Table 9: Results for 12 training verifiers aggregated over the three golden verifiers (

\textrm{thinking}{=}\texttt{medium}

\textrm{batch}{=}1

), sorted by precision. Verifier agreement uses a

20

-task HealthBench screening slice; policy-evaluation sampling is described in Appendix I . The intelligence is the model’s GDPval score ( Patwardhan et al., 2025 ) . The three rightmost columns are the late-training (rollout step

99

) HealthBench score on the hard split for the Qwen3 8B, 4B, and 1.7B policies under that training verifier. The Qwen3 8B verifier is a self-judge only for the 8B trainee.

						HealthBench score
Training verifier	Accuracy	Precision	Recall	F1	Spearman	8B	4B	1.7B
Gemma 4 31B	0.86	0.77	0.90	0.83	0.83	0.42	0.41	0.33
GPT oss 120B	0.83	0.75	0.83	0.79	0.73	0.43	–	0.28
GPT oss 20B	0.84	0.74	0.90	0.81	0.87	0.37	0.36	0.26
Grok 4.20	0.78	0.72	0.70	0.71	0.76	0.46	–	–
Gemini 3 flash preview	0.82	0.71	0.89	0.79	0.82	0.44	0.41	0.32

Appendix

Table 10: Results for 12 training verifiers aggregated over the three golden verifiers (

\textrm{thinking}{=}\texttt{medium}

\textrm{batch}{=}1

), sorted by precision. Verifier agreement uses a

40

-task HealthBench screening slice; policy-evaluation sampling is described in Appendix I . The three rightmost columns are the late-training (rollout step

99

) HealthBench score on the “all” split for the Qwen3 8B, 4B, and 1.7B policies under that training verifier. The Qwen3 8B verifier is a self-judge only for the 8B trainee.

Training verifier	Accuracy	Precision	Recall	F1	Spearman	PRBench finance score
Gemma 4 31B	0.91	0.82	0.78	0.80	0.79	0.46
GPT oss 120B	0.90	0.82	0.71	0.76	0.82	0.38
Gemma 4 26B	0.89	0.77	0.73	0.75	0.71	0.45
Gemini 3 flash preview	0.88	0.72	0.82	0.77	0.79	0.45
GPT oss 20B	0.86	0.71	0.65	0.68	0.70	0.45
Qwen3 235B	0.83	0.59	0.80	0.68	0.63	0.45

Appendix

Table 11: Results for 8 training verifiers aggregated over the three golden verifiers (

\textrm{thinking}{=}\texttt{medium}

\textrm{batch}{=}1

), sorted by precision. Verifier agreement uses a

40

-task PRBench-finance screening slice; policy-evaluation sampling is described in Appendix I . The rightmost column is the late-training (rollout step

49

) PRBench-finance score (normalized aggregate; Appendix J ) on the “all” (hard

+

non-hard) split for the Qwen3 8B trainee under that training verifier.

Training verifier	Accuracy	Precision	Recall	F1	Spearman	PRBench finance score
Gemma 4 31B	0.93	0.83	0.85	0.84	0.89	0.42
GPT oss 120B	0.91	0.78	0.71	0.74	0.88	0.35
Gemma 4 26B	0.91	0.77	0.78	0.78	0.70	0.42
GPT oss 20B	0.89	0.74	0.66	0.70	0.70	0.42
Gemini 3 flash preview	0.89	0.68	0.80	0.74	0.85	0.42
Qwen3 Next 80B	0.85	0.59	0.85	0.70	0.78	0.42

Appendix

Table 12: Results for 8 training verifiers aggregated over the three golden verifiers (

\textrm{thinking}{=}\texttt{medium}

\textrm{batch}{=}1

), sorted by precision. Verifier agreement uses a

20

-task PRBench-finance screening slice; policy-evaluation sampling is described in Appendix I . The rightmost column is the late-training (rollout step

49

) PRBench-finance score (normalized aggregate; Appendix J ) on the “hard” split for the Qwen3 8B trainee under that training verifier.

Training verifier	Accuracy	Precision	Recall	F1	Spearman	PRBench legal score
Gemma 4 26B	0.90	0.82	0.81	0.81	0.81	0.41
Gemma 4 31B	0.90	0.78	0.90	0.83	0.85	0.42
GPT oss 120B	0.89	0.77	0.76	0.76	0.79	0.37
GPT oss 20B	0.85	0.76	0.70	0.73	0.76	0.41
Gemini 3 flash preview	0.87	0.72	0.89	0.79	0.83	0.43
Qwen3 235B	0.84	0.66	0.88	0.75	0.80	0.40

Appendix

Table 13: Results for 7 training verifiers aggregated over the three golden verifiers (

\textrm{thinking}{=}\texttt{medium}

\textrm{batch}{=}1

), sorted by precision. Verifier agreement uses a

40

-task PRBench-legal screening slice; policy-evaluation sampling is described in Appendix I . The rightmost column is the late-training (rollout step

49

) PRBench-legal score (normalized aggregate; Appendix J ) on the “all” (hard

+

non-hard) split for the Qwen3 8B trainee under that training verifier.

Training verifier	Accuracy	Precision	Recall	F1	Spearman	PRBench legal score
GPT oss 120B	0.93	0.83	0.69	0.75	0.60	0.35
Gemma 4 26B	0.91	0.83	0.65	0.73	0.40	0.39
GPT oss 20B	0.91	0.81	0.66	0.73	0.49	0.39
Gemma 4 31B	0.94	0.81	0.87	0.84	0.76	0.39
Gemini 3 flash preview	0.91	0.73	0.84	0.78	0.69	0.40
Qwen3 235B	0.88	0.64	0.82	0.72	0.64	0.37

Appendix

Table 14: Results for 7 training verifiers aggregated over the three golden verifiers (

\textrm{thinking}{=}\texttt{medium}

\textrm{batch}{=}1

), sorted by precision. Verifier agreement uses a

20

-task PRBench-legal screening slice; policy-evaluation sampling is described in Appendix I . The rightmost column is the late-training (rollout step

49

) PRBench-legal score (normalized aggregate; Appendix J ) on the “hard” split for the Qwen3 8B trainee under that training verifier.

Split	$n$	Median	$p90$	$p95$	Max
non-hard	$225$	$\phantom{0,0}865$	$34{,}291$	$46{,}456$	$87{,}430$
non-hard-eval	$\phantom{0}25$	$\phantom{0,0}554$	$36{,}978$	$48{,}405$	$58{,}958$
hard	$249$	$1{,}195$	$36{,}141$	$51{,}043$	$84{,}357$

Appendix

Table 15: Character counts of PRBench-legal user/assistant turn content (no system prompt, no rubric), post-

90{,}000

-char filter. Token estimate: divide by

3.4

Split	$n$	Median	$p90$	$p95$	Max
non-hard	$269$	$\phantom{0,}734$	$35{,}565$	$45{,}489$	$88{,}209$
non-hard-eval	$\phantom{0}30$	$\phantom{0,}859$	$22{,}644$	$35{,}598$	$59{,}523$
hard	$299$	$1{,}470$	$37{,}163$	$44{,}230$	$86{,}338$

Appendix

Table 16: Char counts of PRBench-finance user/assistant turn content, post-

90{,}000

-char filter. Token estimate: divide by

3.4

		Claude		Gemini		GPT
Variant	Metric	all	hard	all	hard	all	hard
Cheapest verifier (no quality floor)	Score gap	+0.050	+0.039	+0.021	+0.027	+0.019	+0.012
	Uplift	65%	82%	82%	85%	84%	92%
	Cost reduction	99.8%	99.9%	99.6%	99.7%	99.7%	99.6%
Cost-reducing Pareto verifier	Score gap	+0.050	+0.039	+0.021	+0.027	+0.019	+0.004
	Uplift	65%	82%	82%	85%	84%	97%

Appendix

Table 17: Cost savings hold across all three golden verifiers.

Verifier	Calls	Input tok.	Output tok.	USD
Claude Opus 4.6 (golden)	$11.6$	$19{,}991$	$1{,}330$	$0.1332$
GPT-5.5 (golden)	$11.6$	$16{,}022$	$1{,}071$	$0.0997$
Gemma 4 31B	$4.3$	$5{,}423$	$75$	$0.00077$
Gemma 4 26B (balanced)	$4.3$	$5{,}423$	$91$	$0.00045$
GPT oss 20B (cost-reducing)	$4.3$	$5{,}522$	$635$	$0.00008$

Appendix

Table 18: Per-task HealthBench grading units under the recorded grading protocols. Golden verifiers grade one criterion per call with

\textrm{thinking}{=}\texttt{medium}

; training verifiers grade up to three criteria per call with

\textrm{thinking}{=}\texttt{low}

Strategy	Family	Score gap	Uplift	Cost reduction	Pareto
Always Gemma 4 26B (balanced Pareto, paper pick)	quality-first	$+0.010$	95%	98.8%	13/18
Always Gemma 4 31B	quality-first	$+0.013$	91%	98.0%	3/18
Pareto frontier $+$ argmax $F_{1}\!\cdot\!\rho\!\cdot\!\kappa$	balanced	$+0.017$	88%	98.1%	4/18
Cheapest in top-3 by $F_{1}\,\cap\,$ top-3 by $\rho$	balanced	$+0.019$	86%	98.5%	8/18
Argmax $z(F_{1})+z(\rho)+z(\kappa)-z(\text{cost})$	balanced	$+0.020$	86%	98.5%	7/18
Argmax Spearman $\rho$	quality-first	$+0.023$	85%	97.6%	6/18

Appendix

Table 19: Verifier-selection strategies for the bivariate (cost, post-training score) Pareto optimization task. Sorted by mean Uplift descending. Family : quality-first / cost-first / balanced. Metrics aggregate over

18

cells with the same definitions as Table 3 . The two rows marked “(paper pick)” are the recommendations highlighted in Tables 3 and 17 and Figure 4 .

Explore similar work

LLM-as-a-Verifier: A General-Purpose Verification Framework
Jul 6, 2026Jacky Kwok, Shulu Li, Pranav Atreya +6Large Language Models(Llms Feedback
Stanford University · UC Berkeley · NVIDIA Research
An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
Apr 9, 2026Andreas Plesner, Francisco Guzmán, Anish AthalyeReinforcement Learning With Verifiable Reward Code Generation
Handshake AI · ETH Zurich
Program Learning with Verifiable Rewards: Symbolic Backpropagation for Post-Training LLMs
Aug 28, 2026Vishvesh BhatReinforcement Learning With Verifiable Reward Verifiable Rewards
CoreThink AI

Jul 6, 2026, cs.AIJ/K move · Enter open · S save

LLM-as-a-Verifier: A General-Purpose Verification Framework

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu+5

Stanford University · UC Berkeley · NVIDIA Research

Large Language Models(Llms Feedback Mathematical Reasoning Benchmarks

Open paper AskPDF arXiv

Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this work, we identify verification, the ability to determine the correctness of a solution, as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring token logits to generate continuous scores. This probabilistic formulation enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. In particular, we show that scaling the scoring granularity leads to better separation between positive and negative solutions, resulting in more calibrated comparisons. Moreover, scaling repeated evaluation and criteria decomposition consistently lead to additional gains in verification accuracy through variance and complexity reduction. We further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals from LLM-as-a-Verifier can also serve as a proxy for estimating task progress. We build an extension for Claude Code, enabling developers to monitor and improve their own agentic systems. Finally, we show that LLM-as-a-Verifier can provide dense feedback for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks.

Abstract

Explore similar work

LLM-as-a-Verifier: A General-Purpose Verification Framework

An Imperfect Verifier is Good Enough: Learning with Noisy Rewards

Program Learning with Verifiable Rewards: Symbolic Backpropagation for Post-Training LLMs