Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

Himil Vasava; Ming Jiang

cs.CLSep 1, 2026

Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

Authors: Himil Vasava, Ming Jiang

Organizations: University of Wisconsin-Madison

Abstract

LLM-based evaluators of natural language generation (NLG) quality are widely deployed as scoring tools and as automated training signals, yet the internal procedure by which they assign a rating remains poorly understood. We investigate this procedure mechanistically through an eight-attack perturbation taxonomy across the Readability and Adequacy dimensions of NLG quality, a generation pipeline that produces paired clean and corrupt summaries with controlled error intensity and explicit token-level modification maps, and a four-experiment battery of causal tracing, logit-lens vocabulary projection, and attention-head knockout applied to Themis (Llama-3-8B) and Prometheus (Mistral-7B). Both evaluators implement a structured, coherent evaluation pipeline operating in two stages: below layer 15, attention performs local error comparison and routes the result to the final input position; above it, the MLP cascade integrates the signal and writes the rating, with the decision crystallizing in the residual stream at a sharp late layer (L = 26 on Themis, L = 25 on Prometheus). Furthermore, a base-model control at the same scale (Llama-3-8B) reproduces the routing architecture and crystallization but not the stage separation, isolating the two mechanisms that fine-tuning specifically installs, suppression of below-L15 MLP contribution at the last position and a two-layer advance of the crystallization depth, indicating that fine-tuning sculpts an existing substrate rather than building the pipeline from scratch. We release the source code and data at https://github.com/himil-v/judge-mech

Explore similar work

Apr 28, 2026cs.CL

LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation

Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and datasets will be released in GitHub.

Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2

Summarization Large Language Model Evaluation Model Self-Assessment Linguistics

Ask PDF

Apr 16, 2026cs.AI

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval:

\textbf{(1)}

a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (

\barρ = 0.8

4.1\%

), with

33

67\%

of documents exhibiting at least one directed 3-cycle; and

\textbf{(2)}

split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed

\geq(1{-}α)

coverage, with set width serving as a per-instance reliability indicator (

r_s = {+}0.576

N{=}1{,}918

p < 10^{-100}

, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (

\bar{r} = 0.32

0.38

), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size

\approx 3.0

) and coherence moderately so (avg. set size

\approx 3.9

), while fluency and consistency remain unreliable (avg. set size

\approx 4.9

). We release all code, prompts, and cached results.

Manan Gupta, Dhruv Kumar

Large Language Model Judges Judges Conformal Prediction

Ask PDF

Sep 1, 2026cs.CL

Does task decomposition improve automatic NLG evaluation?

The LLM-as-a-judge (LLMaJ) framework has emerged as a promising solution for cheap, reproducible, reference-free Natural Language Generation (NLG) evaluation. Prior work seeks to improve LLMaJ by decomposing evaluation tasks into simpler sub-tasks. In this work, we systematically compare LLMaJ methods with and without decomposition on multiple NLG datasets. We find no evidence that LLMaJ with task decomposition leads to performance gains over a fair baseline that does not use decomposition. Instead, we find that previously reported performance gains in decomposition-based LLMaJ stem from using human labels as training data, and not task decomposition itself. Also, we find that, when human labels are available, LLMaJ without using task decomposition can perform comparably to human annotators.

Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1

Llm-As-A-Judge Human Annotators

Ask PDF