Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

Chenle Chen; Yangbo Wei; Chao Yao; Shaoqiang Lu; Junhong Qian; Chen Wu; Lei He

cs.AIAug 19, 2026

Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization

Authors: Chenle Chen, Yangbo Wei, Chao Yao, Shaoqiang Lu, Junhong Qian, Chen Wu, Lei He

Organizations: University of California, Los Angeles · Shanghai Jiao Tong University · Arizona State University · Eastern Institute of Technology, Ningbo

Abstract

Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate. Existing gates rely on verifiable rewards, confining these methods to tasks with an automatic verifier. Replacing the verifier with an LLM-judge gate would lift that restriction, but whether such a gate carries usable signal is untested. We ask a prior question: can we tell, before placing a judge in the loop, whether its scores separate correct from incorrect answers at all? We formalize a reference-free judge as a latent solver -- its verdict rests on agreement with whatever it would itself conclude, so its capacity to evaluate is bounded by its capacity to solve. The model yields a closed-form bound on discriminability (ROC-AUC) in the judge's competence $c$ and answer-space size $k$ , a necessary condition $c > 1/k$ , and the result that the marginal AUC is confounded by item difficulty while a within-question estimator is not. A non-intervening probe records judge scores on genuine optimization runs without altering any decision. We find discriminability at chance where competence sits near the floor and usable above it; that a judge's benchmark accuracy overstates the competence that matters; and, in a closed-loop study, that the screen predicts which kind of gating error occurs. The result is a cheap pre-deployment diagnostic for judge gates.

Figures & tables

Figure 1: (1) A verifier gate has gold answers, so only correct candidates pass. (2) A reference-free judge has none and must in effect re-solve each item to grade it, so once its competence $c$ nears the chance floor $1/k$ its scores are near-random. (3) We therefore measure competence and discriminability before deployment, screening out judges that do not clear the floor—necessary, though not sufficient.

Figure 2: Overview. Skill optimization (2) remains verifier-controlled: $\pi^{\prime}$ is accepted only when its verifier score improves, while target weights stay frozen. A non-intervening probe (3) records $(s_{i},y_{i})$ per episode without affecting acceptance, yielding $\mathcal{D}=\{(s_{i},y_{i})\}$ . Offline diagnostics (4) estimate within-question AUC with a question-clustered bootstrap, closed-book competence, and the effects of censoring, exposure, and the solve/grade gap. The deployment screen (5) rules out judge gating unless competence exceeds the chance floor—a necessary but not sufficient condition.

Task	$k$	$\widehat{\mathrm{AUC}}$	$\widehat{\mathrm{AUC}}_{w}$	confound
Research math	5	0.457	0.489	$-0.03$
Factual QA	$\infty$	0.855	0.735	$+0.12$
GPQA-Diamond	4	0.735	n/a	—

Table 1: Marginal vs. within-question discriminability (Sonnet judge), with the difficulty-confound share

\widehat{\mathrm{AUC}}-\widehat{\mathrm{AUC}}_{w}

. 0.5 is chance. GPQA has no within-question strata.

Task	Judge	$k$	$\hat{c}_{\mathrm{solve}}$	pred.	obs.	obs $\leq$ pred
Math	Sonnet	5	0.267	0.542	0.489 w	✓
Factual	Sonnet	$\infty$	0.567	0.783	0.735 w	✓
Factual	Haiku	$\infty$	0.633	0.817	0.634 w	✓
GPQA	Sonnet	4	0.626	0.751	0.735 m	✓
GPQA	Haiku	4	0.540	0.694	0.794 m	$\times$

Table 2: Closed-form bound (Proposition 1) against observation. w within-question

\widehat{\mathrm{AUC}}_{w}

; m marginal

\widehat{\mathrm{AUC}}

(GPQA admits no strata). The single violation falls in the only cell lacking both a stratified estimate and a contamination-free competence measurement; see text.

Judge	Answered-only	Empty-incl.	Proxy $B^{\prime}$	Inversion
Sonnet	0.840	0.626	0.547	0.602
Haiku	—	0.540	0.353	0.691

Table 3: Competence estimates on GPQA-Diamond. The first two differ only in counting non-convergent episodes; the last two are independent estimators with orthogonal errors.

Figure 3: Competence-gap decomposition for Sonnet on GPQA-Diamond.

	Math		Factual		GPQA
Judge	comp.	AUC	comp.	AUC	comp.	AUC
DeepSeek-8B	14%	0.45	35%	0.75	37%	0.48
Qwen-7B	21%	0.32	33%	0.77	26%	0.55
MiMo-7B	11%	0.57	42%	0.67	42%	0.47
V4-flash	8% ∗	0.31	80%	0.80	73%	0.76

Table 4: Cross-provider competence (empty-inclusive) and discriminability. ∗ high non-convergence (75% empty); competence is a bound.

Figure 4: Discriminability versus genuine competence across four judges and three tasks. It approaches chance near the competence floor and becomes usable only clearly above it. The solid line connects V4-flash across tasks, giving within-judge evidence.

Task	Gate	$\widehat{\mathrm{AUC}}_{w}$	Final	FA	FR
Factual	Verifier	—	0.908	0.00	0.00
	Judge (passes)	0.735	0.825	0.00	0.44
	Judge (fails)	0.620	0.842	0.33	0.33
	Random	—	0.875	0.33	0.50
Math	Verifier	—	0.363	0.00	0.00
	Judge (fails)	0.425	0.238	0.50	0.50

Table 5: Diagnostic discriminability against closed-loop outcome. “Final” is the held-out verifier score of the final skill, averaged over seeds (3 for factual QA, 2 for math); FA/FR are false-accept and false-reject rates. “Passes” denotes a judge whose

\widehat{\mathrm{AUC}}_{w}

interval excludes

\tfrac{1}{2}

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
Aug 7, 2026Yiyao Zhang, Diksha Goel, Hussain AhmadLLM-as-a-Judge LLM Reliability
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia · CSIRO’s Data61, Clayton, VIC, Australia · School of Computer Science and Information Technology, Adelaide University, Adelaide, SA, Australia
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Jul 7, 2026Chenyu ZhouReward Hacking RL for Language Models
School of Engineering, Institute of Science Tokyo
Self-Supervised Skill Optimization
Jul 30, 2026Siran Peng, Cuiyu Yang, Tianyu Fu +9Large Language Model-Guided Optimization LLM Agent Self-Improvement
1MAIS, CASIA · 3Mininglamp Technology · 2SAI, UCAS +2

Aug 7, 2026J/K move · Enter open · S save

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines

Yiyao Zhang, Diksha Goel, Hussain Ahmad

School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia · CSIRO’s Data61, Clayton, VIC, Australia · School of Computer Science and Information Technology, Adelaide University, Adelaide, SA, Australia

LLM-as-a-Judge LLM Reliability LLM Decision-Making LLM Reasoning

Open paper AskPDF arXiv

An LLM judge deployed inside a reasoning pipeline does not merely measure quality, it decides which answer ships. We show that the cost of that decision depends less on judge accuracy than on the decision rule the judge is embedded in. On frozen candidate pools from four GRPO policies, an unconstrained scalar DeepSeek-R1-7B judge buys almost nothing over answer-level majority vote (+1.0 pp on 500 GSM8K questions, +0.34 EM on 300 HotpotQA questions), and on a frozen-rule 30-question confirmation split it is 10 points worse than majority, a judge that destroys accuracy while scoring candidates confidently. We then subordinate the same judge to Evidence-Locked Derive-Gate-Repair (EL-DGR), a task-adaptive non-compensatory rule under which a judge preference may override evidence-supported consensus only with an extractive evidence certificate, and a repair only when neither alternative is certified and the repair is. With no change to the judge, the candidates, or the budget, EL-DGR reaches 58.2% on GSM8K (vs. 56.8% judge, 55.8% majority, 55.4% first candidate) and 17.33 EM / 25.46 F1 on HotpotQA (vs. 15.67/23.49, 15.33/23.19, 15.33/22.97), improving on first-candidate GRPO by +2.8 pp (exact McNemar p=0.0026) and +2.00 EM (p=0.070, borderline). A decision audit shows why: EL-DGR overturns consensus on only 8 of 30 pilot questions and never converts a correct consensus into an incorrect answer. We also report what did not work: the same seven-channel decomposition used as a step-level gated training reward is null, and corrected channel-drop ablations show no channel is individually necessary (p=1.0 throughout). The practitioner-facing finding is negative about judges and positive about admissibility, bound the judge's blast radius rather than trying to make it accurate.

Abstract

LLM-as-a-Judge

Explore similar work

When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines

More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

Self-Supervised Skill Optimization