Overview

This field examines how automated judges can favor certain answers for reasons unrelated to quality, including response order, writing style, verbosity, model identity, language, and demographic or cultural cues. Researchers use controlled counterfactual tests and benchmarks such as MT-Bench, RewardBench, and multilingual or domain-specific evaluation sets to measure when judgments change and whether they match human ratings. Mitigation approaches include randomizing presentation, adding rubrics and reference answers, calibrating scores with human comparisons, combining judges robustly, and training models to separate relevant evidence from distracting cues. A central challenge is that repeatable judgments and agreement among models do not guarantee validity, while interventions that reduce bias can also erase real quality differences or legitimate differences in human preferences.

Latest papers

CardsList

128 papers11 papers in the last four weeks, down 56% on the four weeks before. 0.1% of all new papers.

  1. AI Reviews Are Biased Towards the Appendix

    Nov 13, 2026Saptarshi Ghosh, Mohammad Saim, Tianyu JiangAutomated Peer ReviewBias in LLM-as-a-Judge

  2. Does Reasoning Improve Psychological Depth in Large Language Models?

    Nov 6, 2026Ruichen Zheng, Yihe Wang, Fabrice Harel-Canada +4Text GenerationLanguage Model Reasoning Evaluation

  3. Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge

    Nov 3, 2026Drake Benjamin Brown, Khushbu Pahwa, Yubin Ge +5Bias in LLM-as-a-JudgeLLM-as-a-Judge

  4. LLMs Cannot Be Neutral: Gender Sensitivity Across Candidates, Evaluators, and Models

    Nov 2, 2026Danial Amin, Joni Salminen, Bernard J. JansenGender Bias in Language ModelsLanguage Model Bias Evaluation

  5. Can AI Reviewers See Your Efforts to Improve Your Paper? A Large-Scale Study of Paper Revision Trajectories

    Nov 1, 2026Yaochen Wang, Chenguang Wang, Ming Li +8Scholarly Peer ReviewAutomated Peer Review

  6. VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation

    Oct 31, 2026Changho Shin, Frederic SalaLLM-as-a-JudgeBias in LLM-as-a-Judge

  7. AI evaluators favor job applications written by certain AI models, independent of candidate quality

    Oct 30, 2026Nicolò Pagan, Michael Andreas Ebling, Stefania Ionescu +1Algorithmic FairnessLanguage Model Bias Evaluation

  8. Style over Substance? Linguistic Bias in LLM-based Peer Review

    Oct 26, 2026Sai Suresh Macharla Vasu, Sumedh Santosh Joshi, Philipp Christmann +6Linguistic Bias in Language ModelsBias in LLM-as-a-Judge

  9. Tampered ML: Evaluating LLM Judgment of Machine Learning Research

    Oct 23, 2026Soroush Ebadian, Parand A. Alamdari, Joe Benton +1LLM EvaluationScientific Reasoning in Language Models

  10. A Plausible Explanation Is Enough to Mislead Multimodal Judges

    Oct 15, 2026Anglin Cai, Zhiqi Wang, Xiaozhi Yang +3Bias in LLM-as-a-JudgeVision-Language Model-as-a-Judge

  11. Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

    Oct 13, 2026Liam Davis, Romain Emil-Popescu, Suman JanaJailbreak Attack EvaluationLLM Safety Evaluation

  12. Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

    Oct 13, 2026Linhai Ma, Yan Wang, Lingfei Qian +2Tool-Use EvaluationLLM-as-a-Judge

  13. PRISM: Calibrating Black-Box Judges with Counterfactual Probes

    Oct 13, 2026Kakei Yamamoto, So KurokiLLM-as-a-JudgeLanguage Model Generation Evaluation

  14. Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

    Oct 13, 2026Fabian Lukassen, Christoph Weisser, Alexander SilbersdorffTyped Decision ModelsBias in LLM-as-a-Judge

  15. A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

    Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-JudgeLLM-as-a-Judge

  16. Executable Prefix Values for Locating Agent Failures

    Oct 13, 2026Bingyu Shen, Jiaqi Tang, Lan Wei +3Agent Failure AnalysisLLM Agent Evaluation

  17. The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express

    Oct 12, 2026Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh +2LLM-as-a-JudgeBias in LLM-as-a-Judge

  18. All Verdicts are Not Equal: Rethinking LLM Judge Reliability

    Oct 8, 2026Vineet Kumar, Darshita Rathore, Anindya MoitraLLM-as-a-JudgeLLM Evaluation

  19. Reliability of LLM Judges for Evaluating Entity Alignment

    Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalKG Entity AlignmentLLM-as-a-Judge

  20. Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

    Oct 5, 2026Hashmath Shaik, Gnaneswar Villuri, Alex DoboliLLM EvaluationLLM-as-a-Judge

  21. Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

    Oct 1, 2026Harry Lyu, Neil ThompsonLLM-as-a-JudgeLLM Evaluation