LLM-as-a-Judge

Follow this topic Ask about this topic

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

559 papers93 papers in the last four weeks, up 16% on the four weeks before. 0.5% of all new papers.

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation
Mar 5, 2026Benjamin Feuer, Lucas Rosenblatt, Oussama ElachqarBias in LLM-as-a-Judge LLM-as-a-Judge
1Stanford University · 3New York University
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Mar 5, 2026Sunishchal Dev, Andrew Sloan, Joshua Kavner +2LLM-as-a-Judge LLM Evaluation
RAND Corporation, Santa Monica, CA, USA
C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning
Mar 5, 2026Avni Mittal, Rauno ArikeCoT Faithfulness LLM-as-a-Judge
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
Mar 5, 2026Bosi Wen, Yilin Niu, Cunxiang Wang +5LLM-as-a-Judge Instruction Following
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China
Who Judges the Judge? Evaluating LLM-as-a-Judge for French Medical open-ended QA
Mar 4, 2026Ikram Belmadani, Oumaima El Khettari, Pacôme Constant dit Beaufils +2LLM-as-a-Judge Bias in LLM-as-a-Judge
Aix-Marseille Univ., CNRS, LIS UMR 7020, 13000 Marseille, France · Nantes Univ., École Centrale Nantes, CNRS, LS2N, UMR 6004, 44000 Nantes, France · Nantes Université, CHU Nantes, PHU 11: Santé Publique, Clinique des données, INSERM, CIC 1413, 44000 Nantes, France +2
LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations
Mar 2, 2026Viet-Thanh Pham, Lizhen Qu, Thuy-Trang Vu +2AI Agent Benchmarks Cross-Cultural Language Model Evaluation
Department of Data Science & AI, Monash University
CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
Mar 2, 2026Ziyi Zhu, Olivier Tieleman, Alexey Bukhtiyarov +1LLM Evaluation Bias in LLM-as-a-Judge
Slingshot AI · Department of Engineering, University of Cambridge
RubricBench: Aligning Model-Generated Rubrics with Human Standards
Mar 2, 2026Qiyuan Zhang, Junyi Zhou, Yufei Wang +8LLM Evaluation Rubric-Based Evaluation
City University of Hong Kong · Tencent Hunyuan · McGill - Mila & Quebec AI Institute +2
Subliminal Signals in Preference Labels
Mar 1, 2026Isotta Magistrali, Frédéric Berdoz, Sam Dauncey +1LLM-as-a-Judge Preference Alignment
ETH Zurich, Switzerland
CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
Mar 1, 2026Yiyue Qian, Shinan Zhang, Yun Zhou +3LLM Evaluation LLM-as-a-Judge
Amazon AWS Generative AI Innovation Center · Amazon AWS Bedrock
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge
Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement
Feb 28, 2026Haolin Jin, Huaming ChenAutomated Code Review LLM Reliability
University of Sydney, Sydney, NSW, Australia.
Confusion-Aware Rubric Optimization for LLM-based Automated Grading
Feb 28, 2026Yucheng Chu, Hang Li, Kaiqi Yang +4Automated Grading Prompt Optimization
Michigan State University · University of Southern California
When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation
Feb 27, 2026Bian Sun, Zhenjian Wang, Orvill de la Torre +1Clinical Language Model Evaluation Generative AI Evaluation
Heinz College, Carnegie Mellon University · School of Computer Science, Carnegie Mellon University
LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Feb 27, 2026Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray ChoudhuryLLM Evaluation LLM-as-a-Judge
University of North Texas, Denton, Texas, USA
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Feb 24, 2026Santiago Gonzalez, Alireza Amiri Bavandpour, Peter Ye +48Mathematical Reasoning Benchmarks LLM Evaluation
Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems
Feb 23, 2026Mukul Chhabra, Luigi Medrano, Arush VermaRAG Evaluation Multi-Turn Dialogue Evaluation
Dell Technologies, Austin, TX, USA
Luna-2: Scalable Single-Token Evaluation with Small Language Models
Feb 20, 2026Vatsal Goel, Rishon Dsouza, Nikhil Ega +4LLM Evaluation Language Model Safety Evaluation
Galileo AI, San Francisco, USA
"How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations
Feb 20, 2026Alexandra Neagu, Marcus Messer, Peter Johnson +1Generative AI in Education Educational Dialogue Systems
Imperial College London, London, United Kingdom
References Improve LLM Alignment in Non-Verifiable Domains
Feb 18, 2026Kejian Shi, Yixin Liu, Peifeng Wang +3LLM Alignment LLM-as-a-Judge
Yale University · Meta · Scale AI +2
Who can we trust? LLM-as-a-jury for Comparative Assessment
Feb 18, 2026Mengjie Qian, Guangzhi Sun, Mark J. F. Gales +1LLM-as-a-Judge Language Model Generation Evaluation
Department of Engineering, University of Cambridge, UK.
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1
LLM-as-Judge on a Budget
Feb 17, 2026Aadirupa Saha, Aniket Wagde, Branislav KvetonLLM Evaluation LLM-as-a-Judge
UIC · Adobe
Synthetic Reader Panels: Tournament-Based Ideation with LLM Personas for Autonomous Publishing
Feb 16, 2026Fred ZimmermanLLM-as-a-Judge User Simulation
Nimble Books LLC / Big Five Killer LLC.
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM Alignment LLM-as-a-Judge
University of North Carolina at Chapel Hill · 2Carnegie Mellon University · 3Yale University +1
CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation
Feb 13, 2026Yiran Rex Ma, Yuxiao Ye, Huiyuan XieAutomated Evaluation Legal NLP
School of Foreign Languages, Peking University · Department of Computer Science and Technology, Tsinghua University
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
Feb 6, 2026Lanbo Lin, Jiayao Liu, Tianyuan Yang +5LLM-as-a-Judge AI Agent Evaluation
Alibaba International Digital Commerce Group · Zhejiang University · Peking University
MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems
Feb 3, 2026Vishal Venkataramani, Haizhou Shi, Zixuan Ke +6Multi-Agent LLM Systems LLM-as-a-Judge
1Rutgers University · 2Salesforce AI Research

Mar 5, 2026J/K move · Enter open · S save

Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation

Benjamin Feuer, Lucas Rosenblatt, Oussama Elachqar

1Stanford University · 3New York University

Bias in LLM-as-a-Judge LLM-as-a-Judge

Open paper AskPDF arXiv

As AI models progress beyond simple chatbots into more complex workflows, we draw ever closer to the event horizon beyond which AI systems will be utilized in autonomous, self-maintaining feedback loops. Any autonomous AI system will depend on automated, verifiable rewards and feedback; in settings where ground truth is sparse or non-deterministic, one practical source of such rewards is an LLM-as-a-Judge. Although LLM judges continue to improve, the literature has yet to introduce systems capable of enforcing standards with strong guarantees, particularly when bias vectors are unknown or adversarially discovered. To remedy this issue, we propose average bias-boundedness (A-BB), an algorithmic framework which formally guarantees reductions of harm/impact as a result of any measurable bias in an LLM judge. Evaluating on Arena-Hard-Auto with four LLM judges, we achieve (tau=0.5, delta=0.01) bias-bounded guarantees while retaining 61-99% correlation with original rankings across formatting and schematic bias settings, with most judge-bias combinations exceeding 80%. The code to reproduce our findings is available at https://github.com/penfever/bias-bounded-evaluation.