Bias in LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

All topics

Reliability of LLM Judges for Evaluating Entity Alignment
Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalLLM-as-a-Judge Bias in LLM-as-a-Judge
University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Oct 5, 2026Hashmath Shaik, Gnaneswar Villuri, Alex DoboliLLM Evaluation LLM-as-a-Judge
Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA
Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM Robustness LLM-as-a-Judge
Northeastern University · Mohamed bin Zayed University of Artificial Intelligence
Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol
Sep 17, 2026Levent BulutLanguage Model Bias Evaluation Linguistic Bias in Language Models
Independent Researcher
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2LLM-as-a-Judge Pairwise Preference Evaluation
Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration
Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2LLM Evaluation LLM-as-a-Judge
Microsoft · Massachusetts Institute of Technology
Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation
Sep 14, 2026Liang Zhao, Yong Wang, Jiangzhe ChenLLM-as-a-Judge LLM Agent Evaluation
Shanghai University of International Business and Economics
When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text
Sep 10, 2026DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang +1LLM-as-a-Judge Social Bias in Language Models
Sungkyunkwan University, Suwon, South Korea
User Feedback Provides a Unique Signal that LLMs Can not Detect
Sep 2, 2026Shachar Don-Yehiya, Leshem Choshen, Omri AbendLLM Evaluation RL from Human Feedback
The Hebrew University of Jerusalem · IBM Research · MIT +1
LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts
Aug 31, 2026Daniela Occhipinti, Andrea Piergentili, Marco GueriniLLM-as-a-Judge LLM Prompting
Fondazione Bruno Kessler, Via Sommarive 18, Povo, Trento, Italy · Almawave Labs, Via Di Casal Boccone, 188/190, Rome
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM Evaluation LLM-as-a-Judge
Japan Advanced Institute of Science and Technology
The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
Aug 5, 2026Alicia Guerra, Yibo HuLLM Evaluation LLM-as-a-Judge
Illinois Institute of Technology
LLM Evaluators are Biased across Languages
Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Language Technology Lab, University of Cambridge
The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol
Jul 15, 2026Serkan BallıMultilingual Language Model Evaluation LLM-as-a-Judge
Department of Software Engineering, Mehmet Akif Ersoy University, Burdur, Türkiye · Department of Software Engineering, Mehmet Akif Ersoy University, Türkiye
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
Jul 13, 2026Zixiang Xu, Sixian Li, Huaxing Liu +4LLM-as-a-Judge LLM Interpretability
AMAP, Alibaba Group · Mohamed bin Zayed University of Artificial Intelligence · University of Southern California +1
Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Jul 12, 2026Sriram Selvam, Anneswa GhoshLLM-as-a-Judge Language Model Generation Evaluation
Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution
Jul 9, 2026Ethan Leung, Elias Lumer, Corey Feld +3LLM-as-a-Judge Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges
Jul 2, 2026Jian Xu, Delu Zeng, John Paisley +1Pairwise Comparison LLM-as-a-Judge
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
RoPoLL: Robust Panel of LLM Judges
Jun 29, 2026Anish Acharya, Kris W Pan, Brian VerkhovskyLLM Evaluation LLM-as-a-Judge
Amazon Web Services
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater Reliability LLM Evaluation
UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708
Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning
Jun 17, 2026Zilong Zhang, Yi-Ting Hung, Lei Ding +1Inverse Optimal Transport LLM-as-a-Judge
Department of Mathematics and Statistics Georgia State University · Department of Statistics University of Manitoba
Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement
Jun 16, 2026Ramaravind Kommiya Mothilal, Terry Jingchen Zhang, Raiyan Ahmed +3LLM-as-a-Judge Social Bias in Language Models
University of Toronto · Vector Institute · EuroSafeAI +1
LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation
Jun 14, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, Graduate School of Engineering, Chubu University, Kasugai, Aichi 487-8501, Japan · Mitsubishi Heavy Industries, Ltd., Research & Innovation Center, Heat Transfer Research Department, Takasago, Hyogo 676-8686, Japan
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5LLM-as-a-Judge Language Model Calibration
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
Jun 12, 2026Shaojie YinMultilingual Language Model Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory Shanghai, China
Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges
May 24, 2026Xianglin Yang, Bryan Hooi, Gelei Deng +2LLM Evaluation LLM-as-a-Judge
School of Computing, National University of Singapore, Singapore · Nanyang Technological University, Singapore
AMEL: Accumulated Message Effects on LLM Judgments
May 21, 2026Sid-Ali TemkitLLM Evaluation Language Model Generation Evaluation

Oct 7, 2026, cs.AIJ/K move · Enter open · S save

Reliability of LLM Judges for Evaluating Entity Alignment

Vaibhava Lakshmi Ravideshik, Mayank Kejriwal

University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California

LLM-as-a-Judge Bias in LLM-as-a-Judge LLM Reliability Knowledge Graphs

Open paper AskPDF arXiv

Entity Alignment (EA) identifies equivalent entities across knowledge graphs and is critical for knowledge base integration and ontology merging. Evaluating EA systems at scale requires expensive expert annotation, making systematic assessment across diverse domains practically infeasible. LLM-as-judge evaluation offers a potentially scalable alternative, yet its reliability for structured prediction tasks like EA remains unstudied. We present the first systematic benchmarking study across three frontier models, three datasets, and four EA systems, using perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. We identify anchor bias, a failure mode in which judges invert discrimination when the system's decision label is visible. Label exposure causally collapses judge discrimination (J-ROC-AUC 0.12-0.87), while a label-free protocol recovers near-ceiling capability on distinctive-name datasets (0.93-1.00) and significant recovery on biomedical pairs (0.93-0.95). Counterfactual experiments confirm causality (FSR 53-99%) and reveal a frontier model paradox: stronger judges exhibit greater label sensitivity, not less. A blinded two-annotator human evaluation (102 pairs, Cohen's kappa=0.902) confirms this mechanism directly. We release the first biomedical EA benchmark (MeSH-SNOMED CT, 15K pairs) and a reproducible auditing framework for LLM judge reliability in EA. Code and data are available at https://github.com/vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment.

Momentum

Related topics

Latest papers 48