LLM-as-a-Judge

Follow this topic Ask about this topic

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

559 papers93 papers in the last four weeks, up 16% on the four weeks before. 0.5% of all new papers.

How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
Apr 17, 2026Judith Sieker, Sina ZarrießLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India
Context Over Content: Exposing Evaluation Faking in Automated Judges
Apr 16, 2026Manan Gupta, Inderjeet Nair, Lu Wang +1LLM-as-a-Judge Language Model Safety Evaluation
1BITS Pilani, India · University of Michigan
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-Judge Healthcare
aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa +3
AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals
Apr 15, 2026Tadhg Looram, Lucas Nuzzi, Kyle Waters +1Automated Evaluation Human Preference Evaluation
PortexAI · Stanford University
On Cost-Effective LLM-as-a-Judge Improvement Techniques
Apr 15, 2026Ryan Lail, Luke MarkhamLLM-as-a-Judge LLM Evaluation
Composo AI.
CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems
Apr 14, 2026Jingbo Yang, Guanyu Yao, Bairu Hou +5LLM Evaluation LLM-as-a-Judge
University of California, Santa Barbara · Cresta
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Apr 10, 2026Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe +2Language Model Generation Evaluation Generative AI Evaluation
Artefact Research Center · MICS, Centrale-Supélec, Université Paris-Saclay · Diabolocom +1
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
Apr 8, 2026Minzhu Tu, Shiyu Ni, Keping BiLLM-as-a-Judge LLM Evaluation
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · Beijing University of Post and Telecommunications +1
What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces
Apr 7, 2026Birger MoëllComputational Literary Studies LLM-as-a-Judge
Department of Linguistics and Philology, Uppsala University, Uppsala, Sweden
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
Apr 7, 2026Xin Sun, Di Wu, Sijing Qin +3Bias in LLM-as-a-Judge LLM-as-a-Judge
National Institute of Informatics (NII), Japan · University of Amsterdam, the Netherlands · Hitotsubashi University, Japan +3
Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
Apr 6, 2026Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar +1Language Model Generation Evaluation Multilingual Language Model Evaluation
Qatar Computing Research Institute, HBKU, Qatar · †UPES, India
Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation
Apr 6, 2026Alhasan Mahmood, Samir Abdaljalil, Hasan KurbanMultilingual Language Model Evaluation LLM-as-a-Judge
Hamad Bin Khalifa University, Doha, Qatar · Texas A&M University, College Station, TX USA
Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge
Apr 4, 2026Yulong He, Ivan Smirnov, Dmitry Fedrushkov +2LLM Evaluation LLM-as-a-Judge
St. Petersburg State University, University emb., 7/9, St. Petersburg, Russia · ITMO University, Kronverkskiy av., 49, St. Petersburg, Russia
Adaptive Rigor in AI System Evaluation using Temperature-Controlled Verdict Aggregation via Generalized Power Mean
Apr 4, 2026Aleksandr MeshkovAutomated Evaluation Language Model Generation Evaluation
Independent Researcher, Budva, Montenegro.
Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization
Apr 3, 2026Peng Zhang, Xuefeng Li, Xiaoqi Wang +2Data Visualization LLM-as-a-Judge
Northeastern University, United States · Bosch AI Research, United States · The Ohio State University, United States
Towards Better Aligned LLM Judges for Radiology Report Evaluation across Modalities and Anatomies
Apr 3, 2026Federica Bologna, Jean-Philippe Corbeil, Matthew Wilkens +1LLM-as-a-Judge Radiology Report Evaluation
Cornell University, Ithaca, NY, USA · Microsoft Healthcare & Life Sciences, Montr´eal, QC, Canada · Cornell University, Ithaca, NY +1
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Apr 3, 2026Chenyu Li, Zohaib Akhtar, Mingu Kwak +13LLM-as-a-Judge Clinical Language Model Evaluation
Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA · Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh, Pittsburgh, PA, USA · OpenCura, Health Innovation Consortium +6
Reinforcement Learning-based Semi-supervised Knowledge Distillation with LLM-as-a-Judge
Apr 3, 2026Yiyang Shen, Lifu Tu, Weiran WangRL for Language Model Reasoning Knowledge Distillation
University of Iowa · Oracle AI
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Apr 2, 2026Yosuke Yamagishi, Atsushi Takamatsu, Yasunori Hamaguchi +4Radiology Report Evaluation MT Evaluation
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan · Department of Radiology, The University of Tokyo Hospital, Tokyo, Japan · Department of Radiology, Kanazawa University Graduate School of Medical Sciences, Ishikawa, Japan +2
TSQueryBench: LLM-as-a-Judge for Time Series Explanations
Apr 2, 2026Preetham Sivalingam, Murari Mandal, Dhruv Kumar +1Explanation Evaluation Explainability Evaluation
BITS Pilani, Pilani, India · KIIT & Birla AI Labs, Bhubaneshwar, India · BITS Pilani & Birla AI Labs, Pilani, India +1
Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
Apr 1, 2026HyunJoon Jung, William NaLLM-as-a-Judge AI Agent Evaluation
HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
Mar 31, 2026Edward Ajayi, Prasenjit MitraHumor Explanation Generation LLM Evaluation
Carnegie Mellon University Africa Kigali, Rwanda
When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment
Mar 31, 2026Robinson Ferrer, Damla Turgut, Zhongzhou Chen +1Confidence Estimation in Language Models Language Model Calibration
University of Central Florida
Security in LLM-as-a-Judge: A Comprehensive SoK
Mar 31, 2026Aiman Al Masoud, Antony Anju, Marco Arazzi +6LLM-as-a-Judge LLM Security
Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy · Department of Computer Applications, Cochin University of Science and Technology, Kerala, India
The Necessity of Setting Temperature in LLM-as-a-Judge
Mar 30, 2026Lujun Li, Lama Sleem, Yangjie Xu +4LLM-as-a-Judge LLM Evaluation
University of Luxembourg · ETH Zürich
Evaluating LLMs for Answering Student Questions in Introductory Programming Courses
Mar 30, 2026Thomas Van Mullem, Bart Mesuere, Peter DawyndtAI in Education Generative AI in Education
Atomic and Holistic LLM Judges for Reference-Grounded Support Labels: A Prompt-Controlled Comparison
Mar 30, 2026Xinran ZhangLLM-as-a-Judge Language Model Generation Evaluation
University of California, Berkeley
Xpertbench: Expert Level Tasks with Rubrics-Based Evaluation
Mar 27, 2026Xue Liu, Xin Ma, Yuxin Ma +36LLM Evaluation Rubric-Based Evaluation
ByteDance Seed
Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness
Mar 26, 2026Alexandra DeLucia, Heyuan Huang, Sonal Joshi +3LLM-as-a-Judge LLM Reliability
Data Science and AI Institute, Johns Hopkins University

Apr 17, 2026J/K move · Enter open · S save

How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models

Judith Sieker, Sina Zarrieß

Computational Linguistics, Department of Linguistics Bielefeld University, Germany

LLM Evaluation LLM-as-a-Judge Pragmatic Reasoning in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly studied as repositories of linguistic knowledge. In this line of work, models are commonly evaluated both as generators of language and as judges of linguistic output, yet these two roles are rarely examined in direct relation to one another. As a result, it remains unclear whether success in one role aligns with success in the other. In this paper, we address this question for pragmatic competence by comparing LLMs' performance as pragmatic listeners, judging the appropriateness of linguistic outputs, and as pragmatic speakers, generating pragmatically appropriate language. We evaluate multiple open-weight and proprietary LLMs across three pragmatic settings. We find a robust asymmetry between pragmatic evaluation and pragmatic generation: many models perform substantially better as listeners than as speakers. Our results suggest that pragmatic judging and pragmatic generation are only weakly aligned in current LLMs, calling for more integrated evaluation practices.