Bias in LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

All topics

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
May 13, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleLLM-as-a-Judge Faithfulness of Language Model Explanations
Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.
Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
May 9, 2026Andrea MorandiPost-Hoc Calibration LLM-as-a-Judge
Cisco Systems, Inc.
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Apr 25, 2026Sadman Kabir SoumikDebiased ML LLM-as-a-Judge
Independent Researcher
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Apr 24, 2026Jinming Yang, Zheng Hu, Chuxian Qiu +3LLM-as-a-Judge Language Model Bias Evaluation
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware Engineering LLM-as-a-Judge
University of British Columbia, Canada
Context Over Content: Exposing Evaluation Faking in Automated Judges
Apr 16, 2026Manan Gupta, Inderjeet Nair, Lu Wang +1LLM-as-a-Judge Language Model Safety Evaluation
1BITS Pilani, India · University of Michigan
MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
Mar 11, 2026Chihiro Watanabe, Jingyu SunLLM-as-a-Judge Clustering
NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM Alignment LLM-as-a-Judge
University of North Carolina at Chapel Hill · 2Carnegie Mellon University · 3Yale University +1
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-Judge Language Model Bias Evaluation
OMRON SINIC X · NexaScience
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Generation Evaluation
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
Sep 3, 2025Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Steering
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 · Martian Research San Francisco, CA, 94105

May 13, 2026, cs.CLJ/K move · Enter open · S save

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

Riya Tapwal, Abhishek Kumar, Carsten Maple

Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.

LLM-as-a-Judge Faithfulness of Language Model Explanations Language Model Bias Evaluation Bias in LLM-as-a-Judge Causal Interventions in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation. Prior work has documented biases such as position, verbosity, and style preferences, but largely focuses on outcomes, leaving judge explanations underexplored. We instead ask whether LLM judges are cue-invariant, i.e., whether their rankings and explanations remain stable when non-evidential cues are perturbed while holding the underlying texts fixed. We introduce a suite of cue interventions (Blind, Truth, Flip, Placebo, Reveal-After) and tie-aware metrics that quantify outcome anchoring and rationale anchoring, including label-aligned rhetoric and explanation drift, alongside consistency and stereotype-intrusion checks. We design anchoring attacks using verbosity and confidence cues, and compare two mitigations: structured chain-of-thought prompting and PROOF-BEFORE-PREFERENCE (evidence lock, score, rank). Using a new dataset of 1,000 summaries from traditional extractive models and LLMs, we find substantial cue-anchored rationalization under label and placebo perturbations, while PROOF-BEFORE-PREFERENCE markedly improves cue invariance over baselines.

Momentum

Related topics

Latest papers 48