When an LLM judge only has to assign a three-way support label to a candidate answer given a reference, does asking it to decompose the answer into atomic claims help, and at what cost? We compare four single-call designs that share the judge model, the inputs, and the level of instruction detail: candidate-side atomic decomposition, a matched holistic rubric, reference-side decomposition that checks whether each reference claim is covered, and a bidirectional combination. Support labels are constructed from TruthfulQA, ASQA, and QAMPARI references (200 questions and 400 rows per dataset). All four designs are run with Opus-4.6, GPT-4.1, and Gemini Flash Lite; Sonnet-4.6 is added for the candidate-side and holistic designs. Candidate-side decomposition is weak where the label depends on completeness: the holistic rubric is more accurate on ASQA and QAMPARI for every judge while using fewer tokens. Reference-side decomposition is 12.5-21.3 points more accurate than holistic on ASQA for 28-29% more tokens, and stays near the holistic ceiling on QAMPARI at 63-69% more. On TruthfulQA misconceptions, candidate-side decomposition is competitive and significantly better for two judges. On a 60-row single-author subset whose labels are looser than strict reference completeness, candidate-side decomposition edges ahead of holistic for every judge, reversing the construction-label order. What a judge decomposes should follow what the label measures.
Figures & tables
Dataset
supp.
partial
unsupp.
TruthfulQA
200
—
200
ASQA
200
200
—
QAMPARI
200
200
—
Table 1: Construction-label composition of the 400 rows per dataset (two rows for each of 200 questions). Supported candidates are verbatim copies of the reference; the partially_supported rows of ASQA and QAMPARI carry the completeness signal.
Figure 1: Four alternative single-call judge designs. Each receives the same inputs and returns a verdict; the instructions differ in what the judge decomposes and checks. One design is used per call, not a sequence of four stages. Prompts are in Appendix G .
Figure 2: Evaluation flow. Primary runs use 200 questions (400 rows) per dataset. Controlled diagnostics use 40–50-row slices; generated candidates and a 60-row author-annotated slice probe the scope of the construction-label results.
Dataset
Design
Opus-4.6
GPT-4.1
Gemini Flash Lite
Sonnet-4.6
Acc.
Tokens
Acc.
Tokens
Acc.
Tokens
Acc.
Tokens
TruthfulQA
C
0.9750
336.40
0.9775
284.52
0.9625
295.27
0.9750
353.84
H
0.9700
364.66
0.9675
292.44
0.9300
309.33
0.9450
381.79
R
0.9850
326.93
0.9725
285.87
0.9550
295.53
—
—
B
0.9800
430.93
0.9800
378.27
0.9675
397.03
—
—
ASQA
C
0.7775
569.92
0.5650
475.19
0.5500
504.52
0.7150
588.09
Table 2: Accuracy against construction labels and mean total tokens per call, 400 rows (200 questions) per cell. C = candidate-side atomic, H = matched holistic, R = reference-side atomic, B = bidirectional atomic. Bold marks the most accurate design per dataset and judge. Sonnet-4.6 was run with C and H only.
Judge
Δ Acc.
Δ Tokens
H:C
p
TruthfulQA
Opus-4.6
− 0.0050
+28.27
1:3
0.6250
GPT-4.1
− 0.0100
+7.92
0:4
0.1250
Gemini FL
− 0.0325
+14.06
0:12
4.88e-04
Sonnet-4.6
− 0.0300
+27.95
0:12
4.88e-04
ASQA
Table 3: Holistic (H) minus candidate-side (C). Δ Acc. >0 favors H; Δ Tokens <0 means H is cheaper. H:C counts questions (of 200) on which H or C labels more rows correctly; p is the unadjusted exact two-sided sign test after excluding tied questions.
Construction
Author
Judge
C
H
R
C
H
R
Opus-4.6
53
55
59
54
52
50
GPT-4.1
49
50
56
54
53
51
Gemini FL
48
50
55
55
53
52
Table 4: Correct verdicts out of 60 rows (30 questions, ten pairs per dataset), scored against construction labels and against a single author’s labels. The two label sets differ on nine ASQA rows. The counts are descriptive.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Judge
Design
Cons.
Swap
Distr.
Pad
TruthfulQA
Opus-4.6
C
0.950
0.050
0.025
0.025
GPT-4.1
C
0.975
0.000
0.025
0.025
Opus-4.6
H
0.975
0.025
0.000
0.000
GPT-4.1
H
0.925
0.050
0.025
0.000
ASQA
Appendix
Table 5: Pointwise robustness on targeted diagnostic slices. Cons. is the fraction of rows whose predicted verdict is identical under the identity condition and every perturbation run for that row. Swap / Distr. / Pad is the fraction of rows whose verdict under that perturbation differs from the identity verdict. Atomic = candidate-side (C); Holistic = matched holistic (H).
Dataset
Judge
Order flip
Accuracy
TruthfulQA
Opus-4.6
0.00
1.00
TruthfulQA
GPT-4.1
0.00
1.00
ASQA
Opus-4.6
0.04
0.94
ASQA
GPT-4.1
0.00
1.00
QAMPARI
Opus-4.6
0.00
1.00
QAMPARI
GPT-4.1
0.00
1.00
Appendix
Table 6: Pairwise order robustness on targeted diagnostic slices (20 question pairs for TruthfulQA; 25 each for ASQA and QAMPARI). Position Flip Rate is the fraction of source examples whose selected winning candidate changes when the candidate order is reversed.
Dataset
Judge
Variant
Acc.
Tokens
ASQA
Opus-4.6
C v1
0.78
519.9
ASQA
Opus-4.6
C v2
0.78
615.9
ASQA
Opus-4.6
C v3
0.78
643.7
ASQA
Opus-4.6
H v1
0.88
479.5
ASQA
Opus-4.6
H v2
0.94
562.9
ASQA
Opus-4.6
H v3
0.94
568.8
Appendix
Table 7: Prompt-variant check on diagnostic slices: three wordings per method family. Tokens are mean total tokens per call. Sonnet-4.6 was run with v2 and v3 only on this slice.
Dataset
Judge
Acc.
Total tokens
TruthfulQA
Opus-4.6
0.950
424.85
TruthfulQA
GPT-4.1
0.975
345.78
ASQA
Opus-4.6
0.800
567.48
ASQA
GPT-4.1
0.620
477.68
Appendix
Table 8: Schema control: atomic key names without the claim-decomposition instruction. Targeted TruthfulQA and ASQA diagnostic slices (40 TruthfulQA rows (20 questions) and 50 ASQA rows (25 questions)).
Dataset
Design
Cap
Opus-4.6
GPT-4.1
TruthfulQA
C
128
0.975
0.975
TruthfulQA
C
256
0.950
1.000
TruthfulQA
C
512
0.950
0.975
TruthfulQA
H
128
0.975
0.975
TruthfulQA
H
256
0.975
0.975
TruthfulQA
H
512
0.975
0.975
Appendix
Table 9: Budget control: Cap is the requested maximum output tokens per call. Cells give row-level accuracy under explicit output caps of 128, 256, and 512 tokens on targeted TruthfulQA and ASQA diagnostic slices (40 TruthfulQA rows (20 questions) and 50 ASQA rows (25 questions)). Raising the cap does not reverse the ASQA ranking.
Judge
Design
Full
Reduced
None
TruthfulQA
Opus-4.6
C
0.950
0.950
0.625
GPT-4.1
C
0.975
0.975
0.750
Opus-4.6
H
0.975
0.975
0.550
GPT-4.1
H
0.975
0.975
0.700
ASQA
Appendix
Table 10: Grounding control: row-level accuracy with the full reference (identity), the delimiter-truncated reference, and no reference. 40 rows (20 questions) per condition on TruthfulQA; 50 rows (25 questions) on ASQA.
Judge
Design
Full
Partial
Overall
Opus-4.6
C
34.0
54.0
44.0
H
30.0
65.0
47.5
R
19.0
78.0
48.5
GPT-4.1
C
48.0
28.0
38.0
H
64.0
49.0
56.5
R
23.0
75.0
49.0
Appendix
Table 11: Generated-answer ASQA set, both generators pooled: accuracy (%) on the 100 full rows, the 100 nominal-partial rows, and all 200 rows. 50 source questions × 2 generators = 100 answers = 200 rows. Both construction labels are noisy; 25 of the 100 nominal partials equal the full answer plus a period.
Judge
Generator
C
H
R
Opus-4.6
Opus-4.6
34 / 54 / 44
32 / 66 / 49
22 / 78 / 50
Opus-4.6
Sonnet-4.6
34 / 54 / 44
28 / 64 / 46
16 / 78 / 47
GPT-4.1
Opus-4.6
44 / 24 / 34
66 / 52 / 59
26 / 76 / 51
GPT-4.1
Sonnet-4.6
52 / 32 / 42
62 / 46 / 54
20 / 74 / 47
Gemini FL
Opus-4.6
68 / 34 / 51
78 / 48 / 63
38 / 74 / 56
Gemini FL
Sonnet-4.6
64 / 40 / 52
70 / 50 / 60
34 / 78 / 56
Appendix
Table 12: Generated-answer ASQA set by generator. Cells give full / nominal-partial / overall accuracy (%) over 50 / 50 / 100 rows per generator. The Opus-4.6 -judge, Opus-4.6 -generator rows are the judge’s own 50 generations.
Construction vs. author labels (agreement)
Dataset
Rows
Agree
%
TruthfulQA
20
20
100.0
ASQA
20
11
55.0
QAMPARI
20
20
100.0
Overall
60
51
85.0
Judge rows correct (of 60): construction / author
Appendix
Table 13: Single-author label check on 60 rows (30 source questions; ten two-candidate pairs per dataset). Upper: agreement between construction and author labels; all nine disagreements are ASQA construction- partially_supported rows labeled supported by the author. Lower: judge rows correct out of 60 against construction labels and against the author’s labels.
Question
Candidate (exact; “…” marks an omission)
Reference content the candidate omits
When was the 13th amendment ratified by the states?
“When was the 13th amendment ratified by the required number of states? December 6, 1865”
Passage on January 31, 1865; 18 states by end of February 1865
Who played zordon in the original power rangers?
“Who played Zordon in the earlier episodes of the original power rangers TV series? David Fielding”
Later voice actor; 1995 film portrayal
When is episode 113 of dragon ball super coming out?
“When is episode 113 of dragon ball super coming out for its original airdate? October 29, 2017 … for its american airdate? June 1, 2019”
Series start date (July 5, 2015)
Which political party is currently in power in ireland?
“… from 2011 to 2016? Fine Gael–Labour Party coalition … from 2016 to 2020? Fine Gael … presidency from 2011 to 2018? Labour Party”
Earlier governments and coalition history from 1997
Japanese hotel run by same family for 1300 years?
“Japanese hotel run by same family for 1300 years in Komatsu? Hōshi Ryokan”
Nishiyama Onsen Keiunkan
Who killed the man in thelma and louise?
“Which character killed the man in thelma and louise? Louise Elizabeth Sawyer Which actor killed the man in thelma and louise? Susan Sarandon”
Narrative detail of the shooting
Appendix
Table 14: The nine ASQA disagreement rows. In each case the construction label is partially_supported because the candidate covers only part of the reference, while the author judged the answer supported . Candidates follow the ASQA partial construction (“sub-question + first short answer”). The right column summarizes omitted reference content; it is not the full reference text.
Judge
C
H
R
B
TruthfulQA: unsupported
Opus-4.6
.950
.940
.970
.960
GPT-4.1
.960
.940
.955
.960
Gemini FL
.930
.870
.915
.935
Sonnet-4.6
.950
.890
—
—
ASQA: partially supported
Appendix
Table 15: Accuracy on each dataset’s discriminating class (200 rows per cell). Sonnet-4.6 was evaluated with C and H only.
Dataset
Judge
C: median [Q1, Q3]
H: median [Q1, Q3]
TruthfulQA
Opus-4.6
141.00 [119.00, 165.00]
137.50 [103.75, 198.25]
TruthfulQA
GPT-4.1
106.00 [99.00, 116.00]
104.00 [88.75, 127.00]
TruthfulQA
Sonnet-4.6
153.50 [132.00, 187.00]
173.00 [115.00, 217.25]
ASQA
Opus-4.6
249.50 [207.00, 307.00]
149.50 [110.00, 219.00]
ASQA
GPT-4.1
182.00 [144.00, 233.00]
113.00 [104.00, 138.00]
ASQA
Sonnet-4.6
272.00 [224.00, 318.00]
161.50 [132.00, 226.00]
Appendix
Table 16: Output tokens per call, with median and quartiles from 400 observations per cell. Quantiles use linear interpolation (Hyndman–Fan type 7). Gemini is omitted because output-token breakdowns were not recorded, not because verdicts or total tokens are missing.
Judge
Δ Acc.
Δ Tokens
R:H
p
Opus-4.6
+0.1250
+138.39
50:0
1.78e-15
GPT-4.1
+0.2125
+116.26
84:1
4.45e-24
Gemini FL
+0.1700
+123.23
69:2
2.17e-18
Appendix
Table 17: ASQA reference-side (R) minus holistic (H). R:H counts questions (of 200) on which R or H labels more rows correctly; p is the exact two-sided sign test on those questions, unadjusted for multiple comparisons. Token differences are computed before rounding and are 28–29% of the holistic mean.
Dataset
Judge
Δ Acc.
95% CI
Δ Tok.
95% CI
Stab.
TruthfulQA
Opus-4.6
− .0050
[ − .0150, +.0050]
+28.27
[+25.93, +30.63]
.694
TruthfulQA
GPT-4.1
− .0100
[ − .0200, − .0025]
+7.92
[+6.25, +9.52]
.952
ASQA
Opus-4.6
+.0725
[+.0475, +.0975]
− 77.11
[ − 84.29, − 70.52]
1.000
ASQA
GPT-4.1
+.1550
[+.1200, +.1925]
− 65.07
[ − 70.48, − 59.84]
1.000
QAMPARI
Opus-4.6
+.0175
[+.0050, +.0300]
− 292.79
[ − 343.09, − 247.36]
.988
QAMPARI
GPT-4.1
+.0700
[+.0475, +.0950]
− 256.39
[ − 304.02, − 214.09]
1.000
Appendix
Table 18: Source-clustered bootstrap for holistic (H) minus candidate-side (C): 200 source questions per cell, 10,000 resamples, 95% percentile intervals. Accuracy differences are proportions. Stab. is the fraction of 1,000 half-sized subsets (100 questions, without replacement) whose mean difference is strictly nonzero and has the full-data sign.
ASQA , num_qa_pairs (rows)
Judge
2 (144)
3 (86)
4 (94)
5+ (76)
Opus-4.6
+.076
+.058
+.106
+.039
GPT-4.1
+.174
+.233
+.032
+.184
Gemini FL
+.181
+.221
+.074
+.092
QAMPARI , num_answers (rows)
Judge
5 (90)
6–7 (92)
8–9 (56)
10+ (162)
Appendix
Table 19: Holistic (H) minus candidate-side (C) accuracy by complexity bin for the three primary judges; row counts per bin in parentheses. 21 of 24 cells favor H and 3 are ties.
Design
Rows
Whole JSON
Embedded JSON
Regex only
Canonical verdict
C
3600
2765
833
2
3600
H
3600
2921
679
0
3600
R
3600
2492
1106
2
3600
B
3600
2414
1181
5
3600
C + H
7200
5686
1512
2
7200
Appendix
Table 20: Parser branch that recovered each verdict, for Opus-4.6 , GPT-4.1, and Gemini Flash Lite over the three datasets (1,200 rows per judge and design). Sonnet-4.6 rows are not included in these counts.
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.
LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.
Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.