cs.CLSep 21, 2026

When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits

Authors: Daein Weon, Dong Ho Kang

Abstract

Evaluation scores used around LLM systems -- including reward models, rerankers, and LLM judges -- can track surface form instead of the quality they claim to measure. When presented with a terse correct solution and a commented buggy solution for the same MBPP problem, a public preference reward model selects the correct one no better than a coin flip (0.507). Subtracting the predictable surface component from such scores is increasingly common, but removal alone does not yield a more valid measurement: the removed component may carry construct-relevant signal, and residualization cannot tell which is which. Under designed interventions -- unit-test labels with comment-only edits -- residualization attenuates the reward model's format effects by about 0.12 on both correct and buggy code, while the correct-versus-buggy margins move by less than 0.01. In observational NLI and QA settings, we freeze a held-out replication before scoring and re-evaluate it using labels from disjoint annotators; this supports only a narrower conclusion: better agreement with the construct labels on a pre-declared slice where a surface-only predictor errs, not a repaired score. Full-population agreement falls in every observational setting with a reported positive slice gain, and within-question ranking falls in every such QA setting. When construct and surface features are entangled, residualization can decorrelate a score while degrading construct alignment, and, in a controlled model, configurations just as damaging to construct alignment pass every pre-adjustment check, so no committed gate is a guarantee. We assemble these distinctions into a reporting protocol whose outcomes, refusal included, state what an adjusted score may be claimed to show: an audit-time diagnostic reported beside the construct-alignment cost it incurs, never a replacement for the raw score.

Explore similar work

CardsList
May 25, 2026cs.CL

Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why

Validating an LLM judge against human annotations usually means reporting several agreement statistics: accuracy, precision, recall, F1F_1, Cohen's κκ, and one or more rank correlations. A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling, and those choices rarely stated. For binary criteria -- the common case in rubric-based evaluation, where each criterion is graded MET or UNMET -- most of the reported numbers are redundant: Pearson's rr, Spearman's ρρ, Kendall's τbτ_b, the phi coefficient φφ, and the Matthews Correlation Coefficient all reduce to a single number on non-degenerate binary data, so reporting several of them only creates an illusion of corroborating evidence. Cohen's κκ is the one agreement coefficient that adds information: it shares φφ's numerator but normalizes differently, and the gap between them measures how far the judge's positive-label rate has drifted from the human's. We then trace what changes when a judge may abstain with a CANNOT_ASSESS verdict: the three common ways of handling abstentions are not interchangeable preprocessing choices but answer different questions, and they break the binary equivalences. The same equivalences reappear, up to a negligible finite-sample correction, for multi-judge ensembles scored with Fleiss' κκ or Krippendorff's αα. We close with a reporting checklist that names the judgment scale, the abstention and tie handling mode, coverage, the confusion matrix, and the aggregation level alongside any scalar agreement coefficient.
Delip Rao, Chris Callison-Burch
Large Language Model JudgesInter-Rater VariabilityAgreementHuman JudgmentAbstention
AskPDF
May 28, 2026cs.CL

Auditing LLM Benchmarks with Item Response Theory

LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization.
Sander Land, Daniel M. Bikel
Item Response TheoryTheoretical Foundations
AskPDF
Apr 16, 2026cs.AI

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a two-pronged diagnostic toolkit applied to SummEval: (1)\textbf{(1)} a transitivity analysis that reveals widespread per-input inconsistency masked by low aggregate violation rates (ρˉ=0.8\barρ = 0.8-4.1%4.1\%), with 3333-67%67\% of documents exhibiting at least one directed 3-cycle; and (2)\textbf{(2)} split conformal prediction sets over 1-5 Likert scores providing theoretically-guaranteed ≥(1−α)\geq(1{-}α) coverage, with set width serving as a per-instance reliability indicator (rs=+0.576r_s = {+}0.576, N=1,918N{=}1{,}918, p<10−100p < 10^{-100}, pooled across all judges). Critically, prediction set width shows consistent cross-judge agreement (rˉ=0.32\bar{r} = 0.32-0.380.38), demonstrating it captures document-level difficulty rather than judge-specific noise. Across four judges and four criteria, both diagnostics converge: criterion matters more than judge, with relevance judged most reliably (avg. set size ≈3.0\approx 3.0) and coherence moderately so (avg. set size ≈3.9\approx 3.9), while fluency and consistency remain unreliable (avg. set size ≈4.9\approx 4.9). We release all code, prompts, and cached results.
Manan Gupta, Dhruv Kumar
Large Language Model JudgesJudgesConformal Prediction
AskPDF