LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following
May 5, 2026Jaeyun Lee, Junyoung Koh, Zeynel Tok +2LLM Evaluation LLM-as-a-Judge
University of Oxford · Yonsei University
Stop Automating Peer Review Without Rigorous Evaluation
May 4, 2026Joachim Baumann, Jiaxin Pei, Sanmi Koyejo +1LLM-as-a-Judge Scholarly Peer Review
Stanford University · Bocconi University
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation
Apr 28, 2026Divake Kumar, Sina Tayebati, Devashri Naik +2VLM Evaluation Vision-Language Models
University of Illinois at Chicago · 2AI Labs at Capital One
ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable
Apr 28, 2026Sidi Chang, Peiying Zhu, Yuxiao ChenLLM-as-a-Judge AI Agent Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI, San Francisco, USA
Case-Specific Rubrics for Clinical AI Evaluation: Methodology, Validation, and LLM-Clinician Agreement Across 823 Encounters
Apr 27, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6LLM-as-a-Judge Human-AI Agreement
1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2
Understanding the Limits of Automated Evaluation for Code Review Bots in Practice
Apr 27, 2026Veli Karakaya, Utku Boran Torun, Baykal Mehmet Uçar +1LLM-as-a-Judge Human-in-the-Loop Evaluation
Bilkent University Ankara, Turkey · Beko İstanbul, Turkey
Multi-Dimensional Evaluation of Sustainable City Trips with LLM-as-a-Judge and Human-in-the-Loop
Apr 27, 2026Ashmi Banerjee, Adithi Satish, Wolfgang Wörndl +1LLM Evaluation LLM-as-a-Judge
Technical University of Munich Munich, Germany · Polytechnic University of Bari Bari, Italy
How Sensitive Are Safety Benchmarks to Judge Configuration Choices?
Apr 27, 2026Xinran ZhangLLM Safety Benchmarks LLM-as-a-Judge
University of California, Berkeley, Berkeley CA 94720, USA
JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Apr 26, 2026Rohith Reddy Bellibatlu, Edward Raff, Wenbin ZhangLLM Evaluation LLM-as-a-Judge
Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA
Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Apr 25, 2026Sadman Kabir SoumikDebiased ML LLM-as-a-Judge
Independent Researcher
Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity
Apr 24, 2026Erez Yosef, Oron Anschel, Shunit Haviv Hakimi +4Mathematical Reasoning Benchmarks LLM Evaluation
Tel-Aviv University · Amazon Prime Video · Ben-Gurion University
Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement
Apr 24, 2026Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma +6Human Preference Evaluation Annotator Disagreement
1Stockmark Inc · 2Asahi Kasei Corporation · National Institute of Informatics +1
Quantifying and Mitigating Self-Preference Bias of LLM Judges
Apr 24, 2026Jinming Yang, Zheng Hu, Chuxian Qiu +3LLM-as-a-Judge Language Model Bias Evaluation
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization
Apr 22, 2026Mohamed Hesham Elganayni, Runsheng Chen, Sebastian Nagl +1LLM Evaluation LLM-as-a-Judge
Technical University of Munich Germany
The Effect of Idea Elaboration on the Automatic Assessment of Idea Originality
Apr 22, 2026Umberto Domanti, Moritz Mock, Sergio Agnoli +1Creativity Assessment LLM-as-a-Judge
Free University of Bozen-Bolzano · University of Trieste · Marconi Institute for Creativity
HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing
Apr 21, 2026Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo +7LLM Evaluation LLM-as-a-Judge
Department of Computer Science and Technology, Tsinghua University
Semantic Needles in Document Haystacks: Sensitivity Testing of LLM-as-a-Judge Similarity Scoring
Apr 20, 2026Sinan G. Aksoy, Alexandra A. Sabrio, Erik VonKaenel +1Semantic Textual Similarity LLM-as-a-Judge
Pacific Northwest National Laboratory · Washington University in St. Louis · Humana Inc.
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation Strategies
Apr 20, 2026Lorenz Brehme, Thomas Ströhle, Ruth BreuMulti-Hop QA LLM-as-a-Judge
Universität Innsbruck, Technikerstraße 21a, 6020 Innsbruck, Austria
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Apr 20, 2026Sua Lee, Sanghee Park, Jinbae ImLLM-as-a-Judge Multimodal Large Language Models
Seoul National University · NAVER Cloud AI · KAIST AI
PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference
Apr 20, 2026Arther Tian, Alex Ding, Frank Chen +2LLM Evaluation LLM-as-a-Judge
DGrid AI
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
Apr 19, 2026Yirong Zeng, Yufei Liu, Xiao Ding +9Reward Modeling LLM Alignment
Harbin Institute of Technology SCIR Lab · Peking University · Huawei Technologies Co., Ltd
Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
Apr 18, 2026Zixiao Zhao, Amirreza Esmaeili, Fatemeh FardSoftware Engineering LLM-as-a-Judge
University of British Columbia, Canada
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA
How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Models
Apr 17, 2026Judith Sieker, Sina ZarrießLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India
Context Over Content: Exposing Evaluation Faking in Automated Judges
Apr 16, 2026Manan Gupta, Inderjeet Nair, Lu Wang +1LLM-as-a-Judge Language Model Safety Evaluation
1BITS Pilani, India · University of Michigan
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Apr 16, 2026Amy Rouillard, Sitwala Mundia, Linda Camara +8LLM-as-a-Judge Healthcare
aWits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa · fSchool of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · bGrai Labs, Cape Town, South Africa +3

May 5, 2026, cs.CLJ/K move · Enter open · S save

MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following

Jaeyun Lee, Junyoung Koh, Zeynel Tok, Hunar Batra+1

University of Oxford · Yonsei University

LLM Evaluation LLM-as-a-Judge Instruction Following LLM Reliability

Open paper AskPDF arXiv

Multi-constraint instruction following requires verifying whether a response satisfies multiple individual requirements, yet LLM judges are often assessed only through overall-response judgments. We introduce MCJudgeBench, a benchmark for constraint-level judge evaluation in multi-constraint instruction following. Each instance includes an instruction, a candidate response, an explicit constraint list, per-constraint gold labels in {yes, partial, no}, and controlled response-side perturbations. The evaluation protocol further includes evaluation prompt variants to test judge stability. We evaluate proprietary and open-source LLM judges using both correctness and inconsistency metrics, distinguishing intrinsic inconsistency under stochastic decoding from procedural inconsistency under prompt and response perturbations. Our results show that judge reliability has multiple dimensions: strong overall performance does not guarantee equally reliable detection across label categories, especially for rarer partial and no cases. Judges with higher correctness do not always have lower inconsistency. Evaluation with reasoning improves correctness but does not uniformly improve stability. These findings motivate evaluating LLM judges at the constraint level to study these failure modes.

Momentum

Related topics

Latest papers 362