Accounting for Bias Enables Sustainable LLM Evaluation

Harshita Katoch; David Antony Selby; Gerrit Großmann; Sebastian Vollmer

cs.AISep 25, 2026

Accounting for Bias Enables Sustainable LLM Evaluation

Authors: Harshita Katoch, David Antony Selby, Gerrit Großmann, Sebastian Vollmer

Organizations: Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany

Abstract

LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

Figures & tables

Parameter	Estimate	95 % CI
$\hat{\gamma}_{0}$ (position bias)	$-0.165$	$-0.253$	$+0.076$
$\hat{\beta}$ (verbosity)	$+0.163$	$+0.058$	$+0.268$
$\hat{\lambda}$ (same-family affinity)	$+0.411$	—	—
$\mathrm{SD}(u_{j})$ across judges	$0.337$	—	—

Table 1: Estimated bias parameters from the hybrid model on MT-Bench. The parameter estimates are statistically significant or substantially non-zero, confirming that LLM-judge evaluation is not bias-free at any sample size.

Figure 1: Correcting for judge effects shifts ability estimates substantially. Each arrow shows the change from naive Bradley-Terry (orange) to the Judge-corrected model (blue). Provider-affiliated models receive the largest upward corrections; non-affiliated models shift downward. These distortions are structural and collecting more comparisons under the same protocol would not resolve them.

Figure 2: Hybrid model recovers ground-truth rankings with fewer comparisons. Mean Kendall-

\tau

correlation with ground truth over 50 trials, shown across six tournament structures and three proportions of ordinal data. HybridRater consistently outperforms naive win-rate and simple score averaging, particularly in sparse, adversarially scheduled designs that resemble real leaderboard conditions. The efficiency gain is largest where it matters most, at low sample sizes and in unbalanced tournament designs. Careful selection of the scaling parameter

\omega

is important when combining the two measurement scales.

Explore similar work

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation
Mar 5, 2026Benjamin Feuer, Lucas Rosenblatt, Oussama ElachqarBias in LLM-as-a-Judge LLM-as-a-Judge
1Stanford University · 3New York University
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 10, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology

Mar 5, 2026J/K move · Enter open · S save

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation

Benjamin Feuer, Lucas Rosenblatt, Oussama Elachqar

1Stanford University · 3New York University

Bias in LLM-as-a-Judge LLM-as-a-Judge

Open paper AskPDF arXiv

As AI models progress beyond simple chatbots into more complex workflows, we draw ever closer to the event horizon beyond which AI systems will be utilized in autonomous, self-maintaining feedback loops. Any autonomous AI system will depend on automated, verifiable rewards and feedback; in settings where ground truth is sparse or non-deterministic, one practical source of such rewards is an LLM-as-a-Judge. Although LLM judges continue to improve, the literature has yet to introduce systems capable of enforcing standards with strong guarantees, particularly when bias vectors are unknown or adversarially discovered. To remedy this issue, we propose average bias-boundedness (A-BB), an algorithmic framework which formally guarantees reductions of harm/impact as a result of any measurable bias in an LLM judge. Evaluating on Arena-Hard-Auto with four LLM judges, we achieve (tau=0.5, delta=0.01) bias-bounded guarantees while retaining 61-99% correlation with original rankings across formatting and schematic bias settings, with most judge-bias combinations exceeding 80%. The code to reproduce our findings is available at https://github.com/penfever/bias-bounded-evaluation.

Abstract

Explore similar work

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation

Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration