Overview

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

CardsList

610 papers56 papers in the last four weeks, down 45% on the four weeks before. 0.4% of all new papers.

  1. Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

    Sep 3, 2025Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-JudgeLanguage Model Steering

  2. Expert Preference-based Evaluation of Automated Related Work Generation

    Aug 11, 2025Furkan Şahinuç, Subhabrata Dutta, Iryna GurevychHuman Preference EvaluationLLM-as-a-Judge

  3. Evaluating Style-Personalized Text Generation: Challenges and Directions

    Aug 8, 2025Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2LLM-as-a-JudgeLanguage Model Generation Evaluation

  4. Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues

    Jun 28, 2025Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto +5Mental Health CounselingLLM-as-a-Judge

  5. Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoning

    Jun 12, 2025Lan Zhang, Marco Valentino, Jordan Meadows +1Automated EvaluationAutoformalization

  6. But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors

    May 23, 2025Leon Eshuijs, Archie Chaudhury, Alan McBeth +1LLM EvaluationLLM-as-a-Judge

  7. Multimodal Language Models as Text-to-Image Model Evaluators

    May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-JudgeBenchmark Design

  8. Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English

    Mar 21, 2025Jeremy Barnes, Naiara Perez, Alba Bonet-Jover +1Summarization EvaluationAutomated Evaluation

  9. No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding

    Mar 7, 2025Michael Krumdick, Charles Lovering, Varshini Reddy +2LLM EvaluationLLM-as-a-Judge