Overview

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

CardsList

610 papers56 papers in the last four weeks, down 45% on the four weeks before. 0.4% of all new papers.

  1. LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering

    Feb 27, 2026Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray ChoudhuryLLM EvaluationLLM-as-a-Judge

  2. Case-Aware LLM-as-a-Judge Evaluation for Enterprise-Scale RAG Systems

    Feb 23, 2026Mukul Chhabra, Luigi Medrano, Arush VermaRAG EvaluationMulti-Turn Dialogue Evaluation

  3. Luna-2: Scalable Single-Token Evaluation with Small Language Models

    Feb 20, 2026Vatsal Goel, Rishon Dsouza, Nikhil Ega +4LLM EvaluationLanguage Model Safety Evaluation

  4. "How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations

    Feb 20, 2026Alexandra Neagu, Marcus Messer, Peter Johnson +1Generative AI in EducationEducational Dialogue Systems

  5. References Improve LLM Alignment in Non-Verifiable Domains

    Feb 18, 2026Kejian Shi, Yixin Liu, Peifeng Wang +3LLM AlignmentLLM-as-a-Judge

  6. Who can we trust? LLM-as-a-jury for Comparative Assessment

    Feb 18, 2026Mengjie Qian, Guangzhi Sun, Mark J. F. Gales +1LLM-as-a-JudgeLanguage Model Generation Evaluation

  7. *-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

    Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation EvaluationLLM Evaluation

  8. LLM-as-Judge on a Budget

    Feb 17, 2026Aadirupa Saha, Aniket Wagde, Branislav KvetonLLM EvaluationLLM-as-a-Judge

  9. Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

    Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM AlignmentLLM-as-a-Judge

  10. CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation

    Feb 13, 2026Yiran Rex Ma, Yuxiao Ye, Huiyuan XieAutomated EvaluationLegal NLP

  11. CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

    Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM EvaluationLLM-as-a-Judge

  12. FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

    Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM EvaluationLLM-as-a-Judge

  13. JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

    Feb 6, 2026Lanbo Lin, Jiayao Liu, Tianyuan Yang +5LLM-as-a-JudgeAI Agent Evaluation

  14. MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems

    Feb 3, 2026Vishal Venkataramani, Haizhou Shi, Zixuan Ke +6Multi-Agent LLM SystemsLLM-as-a-Judge

  15. Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry

    Jan 30, 2026Zhuochun Li, Yong Zhang, Ming Li +8LLM EvaluationLLM-as-a-Judge

  16. Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

    Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-JudgeLanguage Model Generation Evaluation

  17. ReportLogic: Evaluating Logical Quality in Deep Research Reports

    Jan 27, 2026Jujia Zhao, Zhaoxin Huan, Zihan Wang +4LLM-as-a-JudgeAutomated Evaluation

  18. From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges

    Jan 13, 2026Yihan Hong, Huaiyuan Yao, Bolin Shen +3LLM-as-a-JudgeRubric-Based Evaluation

  19. What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025

    Jan 12, 2026Jing Yang, Nils Feldhus, Salar Mohtaj +10Language Model Generation EvaluationAutomated Evaluation

  20. Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization

    Jan 7, 2026Yao Dou, Benjamin Mamut, Wei XuLegal NLPLLM Evaluation

  21. Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

    Dec 23, 2025Ruiqi Wang, Xinchen Wang, Cuiyun Gao +3LLM Judge CalibrationLLM-as-a-Judge

  22. ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

    Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model EvaluationLLM Evaluation

  23. Idea2Plan: Exploring AI-Powered Research Planning

    Oct 28, 2025Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar +1LLM EvaluationLLM-as-a-Judge

  24. User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios

    Oct 23, 2025Xiaoyuan Wu, Roshni Kaushik, Wenkai Li +2LLM EvaluationLLM-as-a-Judge

  25. BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

    Oct 9, 2025Terry Yue Zhuo, Xiaolong Jin, Hange Liu +37Code Generation EvaluationCode Generation Benchmarks