Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison

Xinran Zhang

cs.CLMar 30, 2026

Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison

Authors: Xinran Zhang

Organizations: University of California, Berkeley

Abstract

When an LLM judge only has to assign a three-way support label to a candidate answer given a reference, does asking it to decompose the answer into atomic claims help, and at what cost? We compare four single-call designs that share the judge model, the inputs, and the level of instruction detail: candidate-side atomic decomposition, a matched holistic rubric, reference-side decomposition that checks whether each reference claim is covered, and a bidirectional combination. Support labels are constructed from TruthfulQA, ASQA, and QAMPARI references (200 questions and 400 rows per dataset). All four designs are run with Opus-4.6, GPT-4.1, and Gemini Flash Lite; Sonnet-4.6 is added for the candidate-side and holistic designs. Candidate-side decomposition is weak where the label depends on completeness: the holistic rubric is more accurate on ASQA and QAMPARI for every judge while using fewer tokens. Reference-side decomposition is 12.5-21.3 points more accurate than holistic on ASQA for 28-29% more tokens, and stays near the holistic ceiling on QAMPARI at 63-69% more. On TruthfulQA misconceptions, candidate-side decomposition is competitive and significantly better for two judges. On a 60-row single-author subset whose labels are looser than strict reference completeness, candidate-side decomposition edges ahead of holistic for every judge, reversing the construction-label order. What a judge decomposes should follow what the label measures.

Figures & tables

Dataset	supp.	partial	unsupp.
TruthfulQA	200	—	200
ASQA	200	200	—
QAMPARI	200	200	—

Table 1: Construction-label composition of the 400 rows per dataset (two rows for each of 200 questions). Supported candidates are verbatim copies of the reference; the partially_supported rows of ASQA and QAMPARI carry the completeness signal.

Panel of Figure 1 — Figure 1: Four alternative single-call judge designs. Each receives the same inputs and returns a verdict; the instructions differ in what the judge decomposes and checks. One design is used per call, not a sequence of four stages. Prompts are in Appendix G .

Panel of Figure 2 — Figure 2: Evaluation flow. Primary runs use 200 questions (400 rows) per dataset. Controlled diagnostics use 40–50-row slices; generated candidates and a 60-row author-annotated slice probe the scope of the construction-label results.

Dataset	Design	Opus-4.6		GPT-4.1		Gemini Flash Lite		Sonnet-4.6
Dataset	Design	Acc.	Tokens	Acc.	Tokens	Acc.	Tokens	Acc.	Tokens
TruthfulQA	C	0.9750	336.40	0.9775	284.52	0.9625	295.27	0.9750	353.84
	H	0.9700	364.66	0.9675	292.44	0.9300	309.33	0.9450	381.79
	R	0.9850	326.93	0.9725	285.87	0.9550	295.53	—	—
	B	0.9800	430.93	0.9800	378.27	0.9675	397.03	—	—
ASQA	C	0.7775	569.92	0.5650	475.19	0.5500	504.52	0.7150	588.09

Table 2: Accuracy against construction labels and mean total tokens per call, 400 rows (200 questions) per cell. C = candidate-side atomic, H = matched holistic, R = reference-side atomic, B = bidirectional atomic. Bold marks the most accurate design per dataset and judge. Sonnet-4.6 was run with C and H only.

TruthfulQA
Judge	$\Delta$ Acc.	$\Delta$ Tokens	H:C	$p$
Opus-4.6	$-$ 0.0050	+28.27	1:3	0.6250
GPT-4.1	$-$ 0.0100	+7.92	0:4	0.1250
Gemini FL	$-$ 0.0325	+14.06	0:12	4.88e-04
Sonnet-4.6	$-$ 0.0300	+27.95	0:12	4.88e-04
ASQA

Table 3: Holistic (H) minus candidate-side (C).

\Delta

Acc.

>0

favors H;

\Delta

Tokens

<0

means H is cheaper. H:C counts questions (of 200) on which H or C labels more rows correctly;

p

is the unadjusted exact two-sided sign test after excluding tied questions.

	Construction			Author
Judge	C	H	R	C	H	R
Opus-4.6	53	55	59	54	52	50
GPT-4.1	49	50	56	54	53	51
Gemini FL	48	50	55	55	53	52

Table 4: Correct verdicts out of 60 rows (30 questions, ten pairs per dataset), scored against construction labels and against a single author’s labels. The two label sets differ on nine ASQA rows. The counts are descriptive.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

TruthfulQA
Judge	Design	Cons.	Swap	Distr.	Pad
Opus-4.6	C	0.950	0.050	0.025	0.025
GPT-4.1	C	0.975	0.000	0.025	0.025
Opus-4.6	H	0.975	0.025	0.000	0.000
GPT-4.1	H	0.925	0.050	0.025	0.000
ASQA

Appendix

Table 5: Pointwise robustness on targeted diagnostic slices. Cons. is the fraction of rows whose predicted verdict is identical under the identity condition and every perturbation run for that row. Swap / Distr. / Pad is the fraction of rows whose verdict under that perturbation differs from the identity verdict. Atomic = candidate-side (C); Holistic = matched holistic (H).

Dataset	Judge	Order flip	Accuracy
TruthfulQA	Opus-4.6	0.00	1.00
TruthfulQA	GPT-4.1	0.00	1.00
ASQA	Opus-4.6	0.04	0.94
ASQA	GPT-4.1	0.00	1.00
QAMPARI	Opus-4.6	0.00	1.00
QAMPARI	GPT-4.1	0.00	1.00

Appendix

Table 6: Pairwise order robustness on targeted diagnostic slices (20 question pairs for TruthfulQA; 25 each for ASQA and QAMPARI). Position Flip Rate is the fraction of source examples whose selected winning candidate changes when the candidate order is reversed.

Dataset	Judge	Variant	Acc.	Tokens
ASQA	Opus-4.6	C v1	0.78	519.9
ASQA	Opus-4.6	C v2	0.78	615.9
ASQA	Opus-4.6	C v3	0.78	643.7
ASQA	Opus-4.6	H v1	0.88	479.5
ASQA	Opus-4.6	H v2	0.94	562.9
ASQA	Opus-4.6	H v3	0.94	568.8

Appendix

Table 7: Prompt-variant check on diagnostic slices: three wordings per method family. Tokens are mean total tokens per call. Sonnet-4.6 was run with v2 and v3 only on this slice.

Dataset	Judge	Acc.	Total tokens
TruthfulQA	Opus-4.6	0.950	424.85
TruthfulQA	GPT-4.1	0.975	345.78
ASQA	Opus-4.6	0.800	567.48
ASQA	GPT-4.1	0.620	477.68

Appendix

Table 8: Schema control: atomic key names without the claim-decomposition instruction. Targeted TruthfulQA and ASQA diagnostic slices (40 TruthfulQA rows (20 questions) and 50 ASQA rows (25 questions)).

Dataset	Design	Cap	Opus-4.6	GPT-4.1
TruthfulQA	C	128	0.975	0.975
TruthfulQA	C	256	0.950	1.000
TruthfulQA	C	512	0.950	0.975
TruthfulQA	H	128	0.975	0.975
TruthfulQA	H	256	0.975	0.975
TruthfulQA	H	512	0.975	0.975

Appendix

Table 9: Budget control: Cap is the requested maximum output tokens per call. Cells give row-level accuracy under explicit output caps of 128, 256, and 512 tokens on targeted TruthfulQA and ASQA diagnostic slices (40 TruthfulQA rows (20 questions) and 50 ASQA rows (25 questions)). Raising the cap does not reverse the ASQA ranking.

TruthfulQA
Judge	Design	Full	Reduced	None
Opus-4.6	C	0.950	0.950	0.625
GPT-4.1	C	0.975	0.975	0.750
Opus-4.6	H	0.975	0.975	0.550
GPT-4.1	H	0.975	0.975	0.700
ASQA

Appendix

Table 10: Grounding control: row-level accuracy with the full reference (identity), the delimiter-truncated reference, and no reference. 40 rows (20 questions) per condition on TruthfulQA; 50 rows (25 questions) on ASQA.

Judge	Design	Full	Partial	Overall
Opus-4.6	C	34.0	54.0	44.0
	H	30.0	65.0	47.5
	R	19.0	78.0	48.5
GPT-4.1	C	48.0	28.0	38.0
	H	64.0	49.0	56.5
	R	23.0	75.0	49.0

Appendix

Table 11: Generated-answer ASQA set, both generators pooled: accuracy (%) on the 100 full rows, the 100 nominal-partial rows, and all 200 rows. 50 source questions

\times

2 generators = 100 answers = 200 rows. Both construction labels are noisy; 25 of the 100 nominal partials equal the full answer plus a period.

Judge	Generator	C	H	R
Opus-4.6	Opus-4.6	34 / 54 / 44	32 / 66 / 49	22 / 78 / 50
Opus-4.6	Sonnet-4.6	34 / 54 / 44	28 / 64 / 46	16 / 78 / 47
GPT-4.1	Opus-4.6	44 / 24 / 34	66 / 52 / 59	26 / 76 / 51
GPT-4.1	Sonnet-4.6	52 / 32 / 42	62 / 46 / 54	20 / 74 / 47
Gemini FL	Opus-4.6	68 / 34 / 51	78 / 48 / 63	38 / 74 / 56
Gemini FL	Sonnet-4.6	64 / 40 / 52	70 / 50 / 60	34 / 78 / 56

Appendix

Table 12: Generated-answer ASQA set by generator. Cells give full / nominal-partial / overall accuracy (%) over 50 / 50 / 100 rows per generator. The Opus-4.6 -judge, Opus-4.6 -generator rows are the judge’s own 50 generations.

Construction vs. author labels (agreement)
Dataset	Rows	Agree	%
TruthfulQA	20	20	100.0
ASQA	20	11	55.0
QAMPARI	20	20	100.0
Overall	60	51	85.0
Judge rows correct (of 60): construction / author

Appendix

Table 13: Single-author label check on 60 rows (30 source questions; ten two-candidate pairs per dataset). Upper: agreement between construction and author labels; all nine disagreements are ASQA construction- partially_supported rows labeled supported by the author. Lower: judge rows correct out of 60 against construction labels and against the author’s labels.

Question	Candidate (exact; “…” marks an omission)	Reference content the candidate omits
When was the 13th amendment ratified by the states?	“When was the 13th amendment ratified by the required number of states? December 6, 1865”	Passage on January 31, 1865; 18 states by end of February 1865
Who played zordon in the original power rangers?	“Who played Zordon in the earlier episodes of the original power rangers TV series? David Fielding”	Later voice actor; 1995 film portrayal
When is episode 113 of dragon ball super coming out?	“When is episode 113 of dragon ball super coming out for its original airdate? October 29, 2017 … for its american airdate? June 1, 2019”	Series start date (July 5, 2015)
Which political party is currently in power in ireland?	“… from 2011 to 2016? Fine Gael–Labour Party coalition … from 2016 to 2020? Fine Gael … presidency from 2011 to 2018? Labour Party”	Earlier governments and coalition history from 1997
Japanese hotel run by same family for 1300 years?	“Japanese hotel run by same family for 1300 years in Komatsu? Hōshi Ryokan”	Nishiyama Onsen Keiunkan
Who killed the man in thelma and louise?	“Which character killed the man in thelma and louise? Louise Elizabeth Sawyer Which actor killed the man in thelma and louise? Susan Sarandon”	Narrative detail of the shooting

Appendix

Table 14: The nine ASQA disagreement rows. In each case the construction label is partially_supported because the candidate covers only part of the reference, while the author judged the answer supported . Candidates follow the ASQA partial construction (“sub-question + first short answer”). The right column summarizes omitted reference content; it is not the full reference text.

TruthfulQA: unsupported
Judge	C	H	R	B
Opus-4.6	.950	.940	.970	.960
GPT-4.1	.960	.940	.955	.960
Gemini FL	.930	.870	.915	.935
Sonnet-4.6	.950	.890	—	—
ASQA: partially supported

Appendix

Table 15: Accuracy on each dataset’s discriminating class (200 rows per cell). Sonnet-4.6 was evaluated with C and H only.

Dataset	Judge	C: median [Q1, Q3]	H: median [Q1, Q3]
TruthfulQA	Opus-4.6	141.00 [119.00, 165.00]	137.50 [103.75, 198.25]
TruthfulQA	GPT-4.1	106.00 [99.00, 116.00]	104.00 [88.75, 127.00]
TruthfulQA	Sonnet-4.6	153.50 [132.00, 187.00]	173.00 [115.00, 217.25]
ASQA	Opus-4.6	249.50 [207.00, 307.00]	149.50 [110.00, 219.00]
ASQA	GPT-4.1	182.00 [144.00, 233.00]	113.00 [104.00, 138.00]
ASQA	Sonnet-4.6	272.00 [224.00, 318.00]	161.50 [132.00, 226.00]

Appendix

Table 16: Output tokens per call, with median and quartiles from 400 observations per cell. Quantiles use linear interpolation (Hyndman–Fan type 7). Gemini is omitted because output-token breakdowns were not recorded, not because verdicts or total tokens are missing.

Judge	$\Delta$ Acc.	$\Delta$ Tokens	R:H	$p$
Opus-4.6	+0.1250	+138.39	50:0	1.78e-15
GPT-4.1	+0.2125	+116.26	84:1	4.45e-24
Gemini FL	+0.1700	+123.23	69:2	2.17e-18

Appendix

Table 17: ASQA reference-side (R) minus holistic (H). R:H counts questions (of 200) on which R or H labels more rows correctly;

p

is the exact two-sided sign test on those questions, unadjusted for multiple comparisons. Token differences are computed before rounding and are 28–29% of the holistic mean.

Dataset	Judge	$\Delta$ Acc.	95% CI	$\Delta$ Tok.	95% CI	Stab.
TruthfulQA	Opus-4.6	$-$ .0050	[ $-$ .0150, +.0050]	+28.27	[+25.93, +30.63]	.694
TruthfulQA	GPT-4.1	$-$ .0100	[ $-$ .0200, $-$ .0025]	+7.92	[+6.25, +9.52]	.952
ASQA	Opus-4.6	+.0725	[+.0475, +.0975]	$-$ 77.11	[ $-$ 84.29, $-$ 70.52]	1.000
ASQA	GPT-4.1	+.1550	[+.1200, +.1925]	$-$ 65.07	[ $-$ 70.48, $-$ 59.84]	1.000
QAMPARI	Opus-4.6	+.0175	[+.0050, +.0300]	$-$ 292.79	[ $-$ 343.09, $-$ 247.36]	.988
QAMPARI	GPT-4.1	+.0700	[+.0475, +.0950]	$-$ 256.39	[ $-$ 304.02, $-$ 214.09]	1.000

Appendix

Table 18: Source-clustered bootstrap for holistic (H) minus candidate-side (C): 200 source questions per cell, 10,000 resamples, 95% percentile intervals. Accuracy differences are proportions. Stab. is the fraction of 1,000 half-sized subsets (100 questions, without replacement) whose mean difference is strictly nonzero and has the full-data sign.

ASQA , num_qa_pairs (rows)
Judge	2 (144)	3 (86)	4 (94)	5+ (76)
Opus-4.6	+.076	+.058	+.106	+.039
GPT-4.1	+.174	+.233	+.032	+.184
Gemini FL	+.181	+.221	+.074	+.092
QAMPARI , num_answers (rows)
Judge	5 (90)	6–7 (92)	8–9 (56)	10+ (162)

Appendix

Table 19: Holistic (H) minus candidate-side (C) accuracy by complexity bin for the three primary judges; row counts per bin in parentheses. 21 of 24 cells favor H and 3 are ties.

Design	Rows	Whole JSON	Embedded JSON	Regex only	Canonical verdict
C	3600	2765	833	2	3600
H	3600	2921	679	0	3600
R	3600	2492	1106	2	3600
B	3600	2414	1181	5	3600
C + H	7200	5686	1512	2	7200

Appendix

Table 20: Parser branch that recovered each verdict, for Opus-4.6 , GPT-4.1, and Gemini Flash Lite over the three datasets (1,200 rows per judge and design). Sonnet-4.6 rows are not included in these counts.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
Aug 18, 2026Dae Lee, Mihai Delgeanu, Adel YoussefLLM Evaluation LLM-as-a-Judge
Apple
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
When LLM Judge Scores Look Good but Best-of-N Decisions Fail
Mar 12, 2026Eddie LandesbergLLM Evaluation LLM-as-a-Judge

Aug 18, 2026J/K move · Enter open · S save

SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

Dae Lee, Mihai Delgeanu, Adel Youssef

Apple

LLM Evaluation LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.

Abstract

LLM-as-a-Judge

Explore similar work

SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

When LLM Judge Scores Look Good but Best-of-N Decisions Fail