The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express

Lovedeep Gondara; Jekaterina Novikova; Sagarika Suresh; Jack Ji; Jithin Pradeep

cs.AIOct 12, 2026

The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express

Authors: Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji, Jithin Pradeep

Organizations: The Vanguard Group, Inc.

Abstract

When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty, and a binary verdict leaves the top score tied in 77--78% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall's $τ_b$ hides them: it ranks binary best while $τ_a$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report.

Figures & tables

	DeepSeek-V4-Flash							Qwen3.5-9B							Nemotron-30B-A3B
Format	Tie	$D_{2}$	TT	Spr	$\tau_{b}$	$\tau_{a}$	T1	Tie	$D_{2}$	TT	Spr	$\tau_{b}$	$\tau_{a}$	T1	Tie	$D_{2}$	TT	Spr	$\tau_{b}$	$\tau_{a}$	T1
Binary	.501	2.0	.78	.636	.548	.222	.505	.511	2.0	.77	.603	.517	.210	.505	.578	1.7	.78	.517	.462	.156	.481
Likert-5	.240	4.2	.60	.685	.527	.344	.582	.304	3.3	.64	.680	.533	.337	.572	.261	3.8	.66	.427	.357	.224	.483
Numeric-100	.143	7.0	.45	.722	.515	.384	.607	.101	9.9	.40	.708	.502	.398	.616	.121	8.2	.47	.532	.401	.297	.537
Anchored	.365	2.7	.23	.412	.401	.315	.526	.259	3.9	.36	.397	.390	.308	.526	.317	3.2	.67	.207	.125	.086	.396
Tournament	.072	13.8	.05	.282	.336	.304	.544	.036	27.7	.10	.299	.335	.292	.536	.051	19.7	.04	.060	.036	.035	.378

Table 1: Scoring-stage diagnostics on the identical 15,406-candidate pool. Tie : probability two random candidates receive the same score;

D_{2}=1/\text{Tie}

: effective number of equally used levels. TT : fraction of prompts whose top score is tied, so that the selected response is drawn at random. Spr : global Spearman with the GPT-4 reference;

\tau_{b}

\tau_{a}

: mean within-prompt Kendall; T1 : rate at which the top-scored response (one seeded tie-break; the exact expectation differs by at most 0.006) is a reference-top response; 28% of prompts have a tied reference top, so uniform selection scores 0.357. Pairs tied in the reference count in no

\tau

numerator. Bold: best format per judge. Distinct-value counts and Somers’

D

are in Appendix C , Table 5 .

	All three judges			w/o Nem.
Diagnostic	Fmt %	Judge %	Res. %	Judge %
Tie probability	97.0	0.4	2.6	0.5
Distinct values	99.1	0.2	0.6	0.0
$D_{2}$ /nominal levels †	96.0	0.3	3.8	0.0
Global Spearman	76.4	21.9	1.7	0.1
Within-prompt $\tau_{b}$	55.6	36.0	8.5	0.4

Table 2: Variance shares over the format

\times

judge grid. The last column drops Nemotron (DeepSeek and Qwen only); dropping either other judge leaves the judge share at 0.0–0.5% (structural) and 23–50% (reference agreement). † Pointwise and anchored formats only. Full leave-one-judge-out results in Appendix C , Table 6 .

	$r$	Contrast	DS-Pro	Gemini	Grok	Olmo
4/4	50	anch. $>$ rand.	$+.093^{*}$	$+.086^{*}$	$+.084^{*}$	$+.049^{*}$
	50	Lik. $>$ rand.	$+.099^{*}$	$+.087^{*}$	$+.110^{*}$	$+.054^{*}$
	50	num. $>$ rand.	$+.080^{*}$	$+.088^{*}$	$+.091^{*}$	$+.038^{*}$
3/4	25	Lik. $>$ rand.	$+.065^{*}$	$+.077^{*}$	$+.068^{*}$	$+.027$
	25	num. $>$ rand.	$+.061^{*}$	$+.094^{*}$	$+.072^{*}$	$+.009$
	50	bin. $>$ rand.	$+.048^{*}$	$+.086^{*}$	$+.082^{*}$	$+.026$

Table 3: Downstream contrasts by the number of evaluation judges concurring (all 3/4 and 4/4 contrasts; two of the seven 1/4 contrasts). Cells are mean win-rate differences; ∗ :

p_{\text{Holm}}<0.05

for that judge (seed-clustered bootstrap).

Figure 1: Numeric-100 score distributions, one panel per curation judge; blue bars are multiples of five. Nemotron departs most from the round-number grid (74.7% on multiples of five) yet has the lowest reference agreement.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	$q$	$b$	$\rho(q_{i},\Delta)$	Self-TIE
DeepSeek	$-0.52$	$-0.33$	0.44	95.8%
	[ $-.65,-.39$ ]	[ $-.42,-.24$ ]	[.37,.51]
Qwen	$-0.16$	$-0.33$	0.37	99.6%
	[ $-.26,-.07$ ]	[ $-.42,-.24$ ]	[.30,.44]
Nemotron	$-0.07$	$+0.89$	0.10	42.3%
	[ $-.14,+.00$ ]	[ $+.79,+.99$ ]	[.01,.19]

Appendix

Table 4: Order-swap decomposition of anchored verdicts (

n=500

per judge; 95% CIs below).

q

: preference for the candidate over the anchor net of position (

<0

: anchor preferred, as expected).

b

: slot bias (

>0

: favors Response A).

\rho(q_{i},\Delta)

: per-item correlation of

q_{i}

with the GPT-4 score gap, i.e. whether preferences track quality. Self-TIE: TIE rate when the anchor is compared with itself (

n=4{,}000

). DeepSeek and Qwen share a small slot-B bias but track quality; Nemotron is dominated by a slot-A bias.

Judge	Format	Vals	Somers’ $D$
DeepSeek	Binary	2	.261
	Likert-5	5	.406
	Numeric-100	37	.457
	Anchored	5	.369
	Tournament	65	.364
Qwen	Binary	2	.247

Appendix

Table 5: Distinct values emitted and mean within-prompt Somers’

D

Judge share (%)	All	$-$ DS	$-$ Qwen	$-$ Nem.
Tie probability	0.4	0.4	0.0	0.5
$D_{2}$ /nominal †	0.3	0.4	0.2	0.0
Distinct values	0.2	0.2	0.3	0.0
Global Spearman	21.9	22.8	23.7	0.1
$\tau_{b}$	36.0	32.4	34.4	0.4
$\tau_{a}$	49.8	47.0	50.4	0.2

Appendix

Table 6: Leave-one-judge-out judge share of the sum of squares, computed from unrounded values; top-1 uses the exact expectation under random tie-breaking. Column headers name the judge removed. † Binary, Likert-5, numeric-100, anchored.

	All three judges		DeepSeek + Qwen
Diagnostic	Fmt %	Judge %	Fmt %	Judge %
Tie probability	97.5	0.4	98.0	0.1
Global Spearman	9.9	80.2	93.4	4.6
$\tau_{b}$	7.7	80.8	52.9	19.8
$\tau_{a}$	70.3	27.5	99.4	0.0
Somers’ $D$	71.5	26.3	99.5	0.0

Appendix

Table 7: Variance shares over the three pointwise formats only (binary, Likert-5, numeric-100), which have no slot-assignment confound.

$r=50\%$	DS-Pro	Gemini	Grok	Olmo
Binary	.048/.047	.086/.083	.082/.080	.026/.025
Likert-5	.099/.094	.087/.083	.110/.106	.054/.052
Numeric-100	.080/.081	.088/.086	.091/.090	.038/.038
Anchored	.093/.080	.086/.083	.084/.078	.049/.044
Tournament	.064/.051	.053/.049	.058/.052	.047/.042

Appendix

Table 8: Curated minus Random win rate at

r=50\%

, unadjusted / adjusted for log response length relative to the reference and a markdown indicator (OLS with prompt fixed effects over seed-level outcomes). The markdown indicator is 1 if a response contains a heading, a bullet or numbered list item, bold text, or a code fence. The largest change is 0.013 (at

r=25\%

it is smaller); dropping numbered lists, adding tables, using headings and bold only, or using a log count of markdown elements gives 0.009–0.014.

Judge	Format	Nom.	Dist.	$D_{1}$	$D_{2}$
DeepSeek	Binary	2	2	2.00	1.99
	Likert-5	5	5	4.44	4.17
	Anchored	5	5	3.17	2.74
	Numeric-100	101	37	9.07	6.98
Qwen	Binary	2	2	1.98	1.96
	Likert-5	5	5	3.76	3.28

Appendix

Table 9: Nominal levels, distinct values emitted, and effective levels.

	DeepSeek	Qwen	Nemotron
Distinct	$28.0\pm 2.0$	$27.8\pm 1.5$	$27.8\pm 1.4$
shift	$-9.0$	$-5.2$	$-5.2$
$D_{1}$	$9.05\pm.09$	$12.36\pm.12$	$10.52\pm.11$
shift	$-.02$	$-.03$	$-.03$
$D_{2}$	$6.98\pm.08$	$9.93\pm.11$	$8.22\pm.10$
shift	$-.00$	$-.01$	$-.01$

Appendix

Table 10: Numeric-100 measures at

n/3

(mean

\pm

s.d., 2,000 resamples without replacement) and their shift from the full-pool value.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 529 papers

When LLM Judge Scores Look Good but Best-of-N Decisions Fail
Mar 12, 2026Eddie LandesbergLLM Evaluation LLM-as-a-Judge
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict
Sep 26, 2026Sourabrata Mukherjee, Sunayana SitaramLLM-as-a-Judge Bias in LLM-as-a-Judge
Microsoft Research

Mar 12, 2026J/K move · Enter open · S save

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

Eddie Landesberg

LLM Evaluation LLM-as-a-Judge Best-of-N Sampling LLM Reranking

Open paper AskPDF arXiv

Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.

Abstract

LLM-as-a-Judge

Explore similar work

When LLM Judge Scores Look Good but Best-of-N Decisions Fail

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict