SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

Dae Lee; Mihai Delgeanu; Adel Youssef

cs.AIAug 18, 2026

SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition

Authors: Dae Lee, Mihai Delgeanu, Adel Youssef

Organizations: Apple

Abstract

LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.

Figures & tables

Figure 1: SESSE replaces holistic judgment with structured decomposition — criteria emerge automatically from the judge’s own error cases, producing per-criterion vote evidence.

Figure 2: SESSE pipeline: offline bank construction (Stages 0–4, once) and online per-example inference (Stage 5).

System	Holistic		SESSE	Tie%
	non-tie	tie	non-tie
Qwen2-VL-7B	0.698	0.667	0.663 ‡	21.3%
Flash Lite	0.880	0.679	0.820 ‡	2.6%
Gemini 2.5 Flash	0.945	0.695	0.933	3.5%
RISE-Judge (32B) † Yu et al. (2025b)	0.927		—

Table 1: RewardBench val (

n{=}1{,}000

), dev-optimal

n^{*}

{}^{\ddagger}p{<}0.05

McNemar (non-tie subset). † SFT+DPO.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

$n$	Qwen2-VL-7B	Gemini 2.5 Flash	Flash Lite
1	0.626	0.923	0.790
2	0.648	0.933	0.804
3	0.663	0.927	0.801
5	0.644	0.932	0.820
7	0.645	0.929	0.817
10	0.643	0.930	0.819

Appendix

Table A.1: Accuracy (excl. ties) vs.

n

(self-generated banks; bold =

n^{*}

Config	$k_{\text{eff}}$	SESSE acc ( $n{=}10$ )	SESSE ( $n^{*}$ )
Qwen self-generated	8	0.640	0.663 ( $n^{*}{=}3$ )
Qwen $+$ Gemini bank	14	0.637	0.663 ( $n^{*}{=}2$ )

Appendix

Table B.1: Qwen2-VL-7B with Gemini-generated vs. self-generated bank (val).

Config	Judge	Bank	Degradation (% val)	C1 (Ambiguous)	C2 (Decomposition)	C3 (Semantic equiv.)
Flash Lite	Flash Lite	self-generated	105 (10.5%)	49%	26%	26%
Gemini 2.5 Flash	Gemini 2.5 Flash	self-generated	48 (4.8%)	29%	23%	48%

Appendix

Table C.1: Degradation bucket distribution. C1 = genuine ambiguity (high vote entropy); C2 = decomposition failure (SESSE consistently wrong

\geq

65% non-NA votes against label); C3 = semantic equivalence (high NA rate, responses too similar to discriminate). Percentages of degradation rows per config.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
May 29, 2026Swastik Roy, Rajkumar Pujari, Tharindu Kumarage +5LLM-as-a-Judge LLM Auditing
Amazon AGI
RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges
Aug 25, 2026Kaiyuan Liu, Ziyuan Zhuang, Rongxiang Weng +1LLM-as-a-Judge Language Model Self-Assessment
College of Computer Science and Technology, Zhejiang University · Meituan LongCat Team, China

Jun 21, 2026J/K move · Enter open · S save

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

Shreyas KC

Multilingual Language Model Evaluation LLM-as-a-Judge Language Model Generation Evaluation Benchmark Auditing LLM Agent Evaluation LLM Reliability+2

Open paper AskPDF arXiv

LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge

Abstract

LLM Evaluation

Explore similar work

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges