Bias in LLM-as-a-Judge

Follow this topic Ask about this topic

This field examines how automated judges can favor certain answers for reasons unrelated to quality, including response order, writing style, verbosity, model identity, language, and demographic or cultural cues. Researchers use controlled counterfactual tests and benchmarks such as MT-Bench, RewardBench, and multilingual or domain-specific evaluation sets to measure when judgments change and whether they match human ratings. Mitigation approaches include randomizing presentation, adding rubrics and reference answers, calibrating scores with human comparisons, combining judges robustly, and training models to separate relevant evidence from distracting cues. A central challenge is that repeatable judgments and agreement among models do not guarantee validity, while interventions that reduce bias can also erase real quality differences or legitimate differences in human preferences.

Latest papers

118 papers21 papers in the last four weeks, up 11% on the four weeks before. 0.1% of all new papers.

Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Apr 2, 2026Yosuke Yamagishi, Atsushi Takamatsu, Yasunori Hamaguchi +4Radiology Report Evaluation MT Evaluation
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan · Department of Radiology, The University of Tokyo Hospital, Tokyo, Japan · Department of Radiology, Kanazawa University Graduate School of Medical Sciences, Ishikawa, Japan +2
Comparing Developer and LLM Biases in Code Evaluation
Mar 25, 2026Aditya Mittal, Ryan Shar, Zichu Wu +6Bias in LLM-as-a-Judge Language Model Bias Evaluation
Carnegie Mellon University
MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination
Mar 25, 2026Zhuo Li, Yupeng Zhang, Pengyu Cheng +8LLM Hallucination Mitigation LLM Hallucination Detection
Qwen Large Model Application Team, Alibaba · The Chinese University of Hong Kong, Shenzhen
Permutation-Consensus Listwise Judging for Robust Factuality Evaluation
Mar 20, 2026Tianyi Huang, Nathan Huang, Justin Tang +2LLM-as-a-Judge Bias in LLM-as-a-Judge
Ryquo · App-In Club · Carnegie Mellon University
Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks
Mar 19, 2026Rudra Jadhav, Janhavi Danve, Sonalika ShawLanguage Model Bias Evaluation Linguistic Bias in Language Models
Department of Computer Science, Savitribai Phule Pune University, Pune, India · Department of Computer Science, Dr. D. Y. Patil School of Science and Technology, Pune, India
The Hrunting of AI: Where and How to Improve English Dialectal Fairness
Mar 16, 2026Wei Li, Adrian de WynterLinguistic Bias in Language Models Language Model Bias Evaluation
Boston College · Microsoft · The University of York
HindSight: Evaluating LLM-Generated Research Ideas via Future Impact
Mar 16, 2026Bo JiangScientific Idea Evaluation Research Idea Generation
Temple University
Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-Judge Rubric-Based Evaluation
Large Language Model Department, Tencent, China
Designing Service Systems from Textual Evidence
Mar 11, 2026Ruicheng Ao, Hongyu Chen, Siyang Gao +2Best-Arm Identification AI-Assisted Decision Making
Institute for Data, Systems, and Society, Massachusetts Institute of Technology. · School of Data Science, City University of Hong Kong. · Department of Decision Analytics and Operations, City University of Hong Kong.
Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
Mar 11, 2026Hongbin Zhang, Kehai Chen, Xuefen Bai +4Bias in LLM-as-a-Judge Linguistic Bias in Language Models
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Keeta AI, Meituan, Beijing, China
MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
Mar 11, 2026Chihiro Watanabe, Jingyu SunLLM-as-a-Judge Clustering
NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan
Aligning to Illusions: Choice Blindness in Human and AI Feedback
Mar 9, 2026Wenbin WuRL from Human Feedback Bias in LLM-as-a-Judge
Cambridge Judge Business School, University of Cambridge, UK
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
Mar 9, 2026Dengcan Liu, Fengkai Yang, Xiaohan Wang +7Preference Modeling Reward Modeling
University of Science and Technology of China · Peking University · Meituan +1
Moral Mazes in the Era of LLMs
Mar 6, 2026Dang Nguyen, Harvey Yiyun Fu, Peter West +2Pragmatic Reasoning in Language Models Language Model Generation Evaluation
University of Chicago · University of British Columbia
Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation
Mar 5, 2026Benjamin Feuer, Lucas Rosenblatt, Oussama ElachqarBias in LLM-as-a-Judge LLM-as-a-Judge
1Stanford University · 3New York University
Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
Mar 4, 2026Ikram Belmadani, Oumaima El Khettari, Pacôme Constant dit Beaufils +2LLM-as-a-Judge Bias in LLM-as-a-Judge
Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N, UMR 6004, 44000 Nantes, France · Nantes Université, CHU Nantes, PHU 11: Santé Publique, Clinique des données, INSERM, CIC 1413, 44000 Nantes, France +2
CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
Mar 2, 2026Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov +1LLM Evaluation Bias in LLM-as-a-Judge
Slingshot AI · Department of Engineering, University of Cambridge
Subliminal Signals in Preference Labels
Mar 1, 2026Isotta Magistrali, Frédéric Berdoz, Sam Dauncey +1LLM-as-a-Judge Preference Alignment
ETH Zurich, Switzerland
CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
Mar 1, 2026Yiyue Qian, Shinan Zhang, Yun Zhou +3LLM Evaluation LLM-as-a-Judge
Amazon AWS Generative AI Innovation Center · Amazon AWS Bedrock
LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Feb 27, 2026Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray ChoudhuryLLM Evaluation LLM-as-a-Judge
University of North Texas, Denton, Texas, USA
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM Alignment LLM-as-a-Judge
University of North Carolina at Chapel Hill · 2Carnegie Mellon University · 3Yale University +1
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-Judge Language Model Bias Evaluation
OMRON SINIC X · NexaScience
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM Evaluation AI Agent Evaluation
Computer Science and Information, Ben-Gurion University of the Negev
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Generation Evaluation
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
Sep 3, 2025Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Steering
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 · Martian Research San Francisco, CA, 94105

Apr 2, 2026J/K move · Enter open · S save

Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study

Yosuke Yamagishi, Atsushi Takamatsu, Yasunori Hamaguchi, Tomohiro Kikuchi+3

Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan · Department of Radiology, The University of Tokyo Hospital, Tokyo, Japan · Department of Radiology, Kanazawa University Graduate School of Medical Sciences, Ishikawa, Japan+2

Radiology Report Evaluation MT Evaluation LLM-as-a-Judge Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Background: Accurate translation of radiology reports is important for multilingual research, clinical communication, and radiology education, but the validity of LLM-based evaluation remains unclear. Objective: To evaluate the educational suitability of LLM-generated Japanese translations of chest CT reports and compare radiologist assessments with LLM-as-a-judge evaluations. Methods: We analyzed 150 chest CT reports from the CT-RATE-JPN validation set. For each English report, a human-edited Japanese translation was compared with an LLM-generated translation by DeepSeek-V3.2. A board-certified radiologist and a radiology resident independently performed blinded pairwise evaluations across 4 criteria: terminology accuracy, readability, overall quality, and radiologist-style authenticity. In parallel, 3 LLM judges (DeepSeek-V3.2, Mistral Large 3, and GPT-5) evaluated the same pairs. Agreement was assessed using QWK and percentage agreement. Results: Agreement between radiologists and LLM judges was near zero (QWK=-0.04 to 0.15). Agreement between the 2 radiologists was also poor (QWK=0.01 to 0.06). Radiologist 1 rated terminology as equivalent in 59% of cases and favored the LLM translation for readability (51%) and overall quality (51%). Radiologist 2 rated readability as equivalent in 75% of cases and favored the human-edited translation for overall quality (40% vs 21%). All 3 LLM judges strongly favored the LLM translation across all criteria (70%-99%) and rated it as more radiologist-like in >93% of cases. Conclusions: LLM-generated translations were often judged natural and fluent, but the 2 radiologists differed substantially. LLM-as-a-judge showed strong preference for LLM output and negligible agreement with radiologists. For educational use of translated radiology reports, automated LLM-based evaluation alone is insufficient; expert radiologist review remains important.