Bias in LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

All topics

Reliability of LLM Judges for Evaluating Entity Alignment
Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalLLM-as-a-Judge Bias in LLM-as-a-Judge
University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Oct 5, 2026Hashmath Shaik, Gnaneswar Villuri, Alex DoboliLLM Evaluation LLM-as-a-Judge
Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA
Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM Robustness LLM-as-a-Judge
Northeastern University · Mohamed bin Zayed University of Artificial Intelligence
Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol
Sep 17, 2026Levent BulutLanguage Model Bias Evaluation Linguistic Bias in Language Models
Independent Researcher
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2LLM-as-a-Judge Pairwise Preference Evaluation
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation
Sep 14, 2026Liang Zhao, Yong Wang, Jiangzhe ChenLLM-as-a-Judge LLM Agent Evaluation
Shanghai University of International Business and Economics
When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text
Sep 10, 2026DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang +1LLM-as-a-Judge Social Bias in Language Models
Sungkyunkwan University, Suwon, South Korea
User Feedback Provides a Unique Signal that LLMs Can not Detect
Sep 2, 2026Shachar Don-Yehiya, Leshem Choshen, Omri AbendLLM Evaluation RL from Human Feedback
The Hebrew University of Jerusalem · IBM Research · MIT +1
LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts
Aug 31, 2026Daniela Occhipinti, Andrea Piergentili, Marco GueriniLLM-as-a-Judge LLM Prompting
Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy · Almawave Labs, Via Di Casal Boccone, 188/190, Rome
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM Evaluation LLM-as-a-Judge
Japan Advanced Institute of Science and Technology
The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
Aug 5, 2026Alicia Guerra, Yibo HuLLM Evaluation LLM-as-a-Judge
Illinois Institute of Technology
LLM Evaluators are Biased across Languages
Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Language Technology Lab, University of Cambridge
The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol
Jul 15, 2026Serkan BallıMultilingual Language Model Evaluation LLM-as-a-Judge
Department of Software Engineering, Mehmet Akif Ersoy University, Burdur, Türkiye · Department of Software Engineering, Mehmet Akif Ersoy University, Türkiye
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
Jul 13, 2026Zixiang Xu, Sixian Li, Huaxing Liu +4LLM-as-a-Judge LLM Interpretability
AMAP, Alibaba Group · Mohamed bin Zayed University of Artificial Intelligence · University of Southern California +1
Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Jul 12, 2026Sriram Selvam, Anneswa GhoshLLM-as-a-Judge Language Model Generation Evaluation
Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution
Jul 9, 2026Ethan Leung, Elias Lumer, Corey Feld +3LLM-as-a-Judge Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges
Jul 2, 2026Jian Xu, Delu Zeng, John Paisley +1Pairwise Comparison LLM-as-a-Judge
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
RoPoLL: Robust Panel of LLM Judges
Jun 29, 2026Anish Acharya, Kris W Pan, Brian VerkhovskyLLM Evaluation LLM-as-a-Judge
Amazon Web Services
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater Reliability LLM Evaluation
UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708
Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning
Jun 17, 2026Zilong Zhang, Yi-Ting Hung, Lei Ding +1Inverse Optimal Transport LLM-as-a-Judge
Department of Mathematics and Statistics Georgia State University · Department of Statistics University of Manitoba
Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement
Jun 16, 2026Ramaravind Kommiya Mothilal, Terry Jingchen Zhang, Raiyan Ahmed +3LLM-as-a-Judge Social Bias in Language Models
University of Toronto · Vector Institute · EuroSafeAI +1
LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation
Jun 14, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, Graduate School of Engineering, Chubu University, Kasugai, Aichi 487-8501, Japan · Mitsubishi Heavy Industries, Ltd., Research & Innovation Center, Heat Transfer Research Department, Takasago, Hyogo 676-8686, Japan
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5LLM-as-a-Judge Language Model Calibration
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
Jun 12, 2026Shaojie YinMultilingual Language Model Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory Shanghai, China
Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges
May 24, 2026Xianglin Yang, Bryan Hooi, Gelei Deng +2LLM Evaluation LLM-as-a-Judge
School of Computing, National University of Singapore, Singapore · Nanyang Technological University, Singapore
AMEL: Accumulated Message Effects on LLM Judgments
May 21, 2026Sid-Ali TemkitLLM Evaluation Language Model Generation Evaluation
Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
May 13, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleLLM-as-a-Judge Faithfulness of Language Model Explanations
Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.
Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
May 9, 2026Andrea MorandiPost-Hoc Calibration LLM-as-a-Judge
Cisco Systems, Inc.
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Apr 25, 2026Sadman Kabir SoumikDebiased ML LLM-as-a-Judge
Independent Researcher
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Apr 24, 2026Jinming Yang, Zheng Hu, Chuxian Qiu +3LLM-as-a-Judge Language Model Bias Evaluation
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware Engineering LLM-as-a-Judge
University of British Columbia, Canada
Context Over Content: Exposing Evaluation Faking in Automated Judges
Apr 16, 2026Manan Gupta, Inderjeet Nair, Lu Wang +1LLM-as-a-Judge Language Model Safety Evaluation
1BITS Pilani, India · University of Michigan
MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
Mar 11, 2026Chihiro Watanabe, Jingyu SunLLM-as-a-Judge Clustering
NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM Alignment LLM-as-a-Judge
University of North Carolina at Chapel Hill · 2Carnegie Mellon University · 3Yale University +1
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-Judge Language Model Bias Evaluation
OMRON SINIC X · NexaScience
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Generation Evaluation
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
Sep 3, 2025Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Steering
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 · Martian Research San Francisco, CA, 94105

Oct 7, 2026, cs.AIJ/K move · Enter open · S save

Reliability of LLM Judges for Evaluating Entity Alignment

Vaibhava Lakshmi Ravideshik, Mayank Kejriwal

University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California

LLM-as-a-Judge Bias in LLM-as-a-Judge LLM Reliability Knowledge Graphs

Open paper AskPDF arXiv

Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12-0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93-1.00) and significant recovery on biomedical pairs (0.93-0.95). Counterfactual experiments confirm causality (FSR 53-99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen's kappa=0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH-SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.

Momentum

Related topics

Latest papers 48