Overview

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

CardsList

610 papers56 papers in the last four weeks, down 45% on the four weeks before. 0.4% of all new papers.

  1. AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators

    Nov 19, 2026Nagham Omar, Rotem Dror, Almog Gueta +3LLM-as-a-JudgeLLM-Assisted Annotation

  2. Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge

    Nov 18, 2026Liang Yue, Eric A. VanceLLM-as-a-JudgeBias in LLM-as-a-Judge

  3. Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling

    Nov 18, 2026Eugene Agichtein, Charles L. A. Clarke, Kaustubh Dhole +1IR EvaluationLLM-as-a-Judge

  4. Contract-Explicit Evaluation: From Auditing Learned Readers to Scoped Certificates

    Nov 17, 2026Hao Lu, Weixing Zhu, Ruiyang LiLanguage Model AuditingRelation Extraction

  5. No-Harm-Jury A Jury-Based Multi-Dimensional Harm Evaluation Framework for Medical AI Responses

    Nov 16, 2026Antonio Puertas Gallardo, Mario CeresaLanguage Model Safety EvaluationAI Safety Evaluation

  6. Jev-as-a-Reward: Validating Probabilistic Safety Judges for Reinforcement Learning

    Nov 16, 2026Eunkyeong Lee, Junhyung Moon, Seunggwan Hong +1AI Safety EvaluationLLM-as-a-Judge

  7. Tight PAC-Bayes Generalisation Guarantees for Large Language Model Judges

    Nov 16, 2026Tom A. Lamb, Philip H. S. Torr, Tim G. J. RudnerLLM-as-a-JudgeLanguage Model Safety Evaluation

  8. Risk-certified prompt optimization for LLM-as-a-judge guardrails

    Nov 15, 2026Hussein Jawad, Alexandre Landeau, Guy Fleury Iradukunda +4LLM GuardrailsLanguage Model Safety Evaluation

  9. Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring Systems

    Nov 15, 2026Spandan Roy, Subhabrata Majumdar, Soudeep DebAI-Assisted Decision MakingLLM Auditing

  10. Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes

    Nov 14, 2026Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly +4Clinical Language Model EvaluationRubric-Based Evaluation

  11. When evaluation becomes intervention in clinical AI

    Nov 13, 2026Anthony Abou Mrad, Aviona Conti, Samuel Schmidgall +10LLM-as-a-JudgeClinical Language Model Evaluation

  12. Agent-as-a-Judge

    Nov 13, 2026Runyang You, Hongru Cai, Caiqi Zhang +5LLM-as-a-JudgeLLM Agent Evaluation

  13. PaperCritic: Training AI Reviewers to Assess the Soundness of Papers

    Nov 13, 2026Hyun Ryu, Ji Yong Cho, Eunho Yang +4Automated Peer ReviewScholarly Peer Review

  14. Is Jev a Good Multilingual Judge?

    Nov 11, 2026Wei Liu, Yi Fan, Xiyan FuLLM-as-a-JudgeMultilingual Language Model Evaluation

  15. When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

    Nov 10, 2026Sungbin ImAI Agent EvaluationLLM-as-a-Judge

  16. Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation

    Nov 8, 2026Muhammad bin Javaid, Karim K. Ben Hicham, Enkhnyam Battulga +2LLM Information ExtractionLLM Auditing

  17. DeepResearch RewardBench: Evaluating Reward Models for Deep Research Report

    Nov 6, 2026Shudong Liu, Yuting Zhong, Guanhua Chen +5Reward Model EvaluationLLM-as-a-Judge

  18. AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes

    Nov 6, 2026Shervin Shahidi, James Poon, Hang Xie +5LLM Judge CalibrationLLM-as-a-Judge

  19. Copies Are Not Confirmations: How News Duplication Fools LLM Judges and Continual Learners

    Nov 6, 2026Abdelkader Metales, Miriam Grabenheinrich, Aicha MetalesAutomated Fact-CheckingLLM-as-a-Judge

  20. Adapting LLM Judges to Observed Human Annotation Policy through Soft Prompt Tuning

    Nov 4, 2026Tamás Janusko, Sven Kleinkop, Ricardo Bochnia +4LLM-as-a-JudgePrompt Tuning

  21. PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems

    Nov 4, 2026Sven Kleinkop, Tamás Janusko, Ricardo Bochnia +4LLM-as-a-JudgePrompt Optimization

  22. Grounded but Not Immune: Confirmation Bias in Agent-as-a-judge

    Nov 3, 2026Drake Benjamin Brown, Khushbu Pahwa, Yubin Ge +5Bias in LLM-as-a-JudgeLLM-as-a-Judge