LLM-as-a-Judge

Follow this topic Ask about this topic

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

559 papers93 papers in the last four weeks, up 16% on the four weeks before. 0.5% of all new papers.

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items
Oct 29, 2026Peng Xie, Amr AlanwarLLM-as-a-Judge LLM Evaluation
TUM School of Computation, Information and Technology, Department of Computer Engineering Technical University of Munich, 74076 Heilbronn, Germany
LLMs as Judges and Examinees for Medical Item Difficulty Prediction
Oct 28, 2026Xiaomeng Xiong, Marcus Walker, Yanlin JiangItem Difficulty Estimation Medical Education
University of Florida · National Commission on Certification of Physician Assistants
Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing
Oct 27, 2026Ilija Subasic, Brett Levert, Andrew Rabinovich +2LLM-as-a-Judge LLM Evaluation
Upwork Inc
Human-Residual Steering: Aligning Frozen LLM Judges with Human Judgment Distributions
Oct 27, 2026Chen Lyu, Phong Do, Gabriele PergolaLLM Alignment LLM-as-a-Judge
Department of Computer Science University of Warwick, Coventry, UK
Decision Models as Judges for Hallucination Detection
Oct 27, 2026Haozheng Liu, Zhaoqian Xue, Zhongyin Wang +4LLM Hallucination Detection LLM-as-a-Judge
New York University · University of Arizona · University of Illinois Urbana-Champaign +4
MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations
Oct 26, 2026Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan +7Automated Evaluation LLM-as-a-Judge
Walmart Global Tech, Sunnyvale, CA, USA
RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge
Oct 26, 2026Balamurugan Paramasivam, Sudalai RajkumarRubric-Based Evaluation LLM-as-a-Judge
CobuildX
Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Oct 24, 2026Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva +3MT Evaluation LLM-as-a-Judge
Yandex · HSE University
AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment
Oct 24, 2026Shunchao Zhou, Chang Yang, Jingrui Wang +4Research Idea Generation AI Agent Benchmarks
Singapore Management University · The Hong Kong Polytechnic University · Hangzhou Dianzi University +3
NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting
Oct 23, 2026Matt Wilde, Jillian Fisher, Andrew B. Hall +2Generative AI Evaluation Factual Consistency Evaluation
Forum AI · University of Washington · Stanford University
Distilling Expert Judgment at Scale
Oct 23, 2026Robbie Goldfarb, Andrew B. Hall, Jillian Fisher +2Automated Evaluation LLM-as-a-Judge
Forum AI · Stanford University · University of Washington
Major Revision, Minor Accountability: LLMs as Reviewers of Published Articles in Top-Tier Journals
Oct 22, 2026Bastien Le Guellec, Su Hwan Kim, Raphael Bentegeac +1Scholarly Peer Review Automated Peer Review
CHU Lille, Lille, France · Technical University of Munich (TUM), Munich, Germany
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM Evaluation LLM-as-a-Judge
University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands
Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals
Oct 21, 2026Louise YorkeLLM-as-a-Judge LLM Evaluation
University College London, London, UK
Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models
Oct 21, 2026Yuchuan Weng, Chi Liu, Xin Chen +1LLM-as-a-Judge Efficient Language Model Inference
SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges
Oct 20, 2026Ethan Ludwick, Yu-Chung HsiaoLLM-as-a-Judge Automated Test Generation
Cisco Systems · Anduril Industries
Decision language models as affordable and rapid alternatives to LLM judges of clinical generative AI
Oct 20, 2026Rotem Gershon, Seffi Cohen, Dean Ariel +11LLM-as-a-Judge Clinical Language Model Evaluation
Innovation Division, Clalit Research Institute, Clalit Health Services, Ramat-Gan, Israel. · Stein Faculty of Computer and Information Science, Ben Gurion University of the Negev, Be’er Sheva, Israel. · Department of Family Medicine, Clalit Health Services, Tel Aviv, Israel. +4
EviDial: When Correct Depression Predictions Lack Evidence
Oct 20, 2026Junjie Xu, Jin Zhu, Yuzhe Yang +6Clinical Language Model Evaluation Depression Severity Estimation
East China Normal University · Fudan University
Judging the Justification: Decision Certificate Supervision for Generative Reward Models
Oct 20, 2026Guiquan Sun, Bing Bai, Martin Renqiang Min +1Reward Modeling Faithfulness of Language Model Explanations
School of Computing, University of Connecticut · NEC Labs America
On the LLM Verification for Argument Graph Generation
Oct 19, 2026Jakub Bąba, Jarosław A. ChudziakLLM Agent Verification Computational Argumentation
Institute of Computer Science Warsaw University of Technology Warsaw, Poland
Rubrics Carry the Standard: Shapley Attribution for Checklist-Based LLM Judges
Oct 18, 2026Saeid Asgari, Rui Ying, Sara Malvar +5Rubric-Based Evaluation LLM-as-a-Judge
The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
Oct 17, 2026Ruqiya Bin SafiJailbreak Attack Evaluation LLM-as-a-Judge
Independent Researcher
Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach
Oct 17, 2026William DeShong, Barath Kannan, Auberon Palma +3LLM-as-a-Judge LLM Reliability
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Oct 16, 2026Youbing Yin, Jingyu Wu, Anoop Kumar +3Bias in LLM-as-a-Judge LLM-as-a-Judge
Capital One
MOSS: A Production Framework for Monitoring and Diagnosing Semantic Failures in Search and Recommendation Systems
Oct 16, 2026Alina Nesen, Prathyusha Kanmanth Reddy, David Relyea +4Recommender Systems IR Evaluation
Data Sciences, Target Corporation Minneapolis, MN, USA · Data Sciences, Target Corporation Sunnyvale, CA, USA
NoveltyScope: A Multi-Agent Research Assistant for Evidence-Grounded Novelty Assessment of Scientific Papers
Oct 15, 2026Yueran Tian, Huizhi LiangLLM-as-a-Judge Multi-Agent Systems
School of Computing Newcastle University Newcastle upon Tyne, UK
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Oct 15, 2026Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell +1Rubric-Based RL Reward Design for RL
UC Berkeley
Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization
Oct 14, 2026Anna Mei, Michael R. DouglasFormal Verification Autoformalization
Harvard University · CMSA, Harvard University and AletheAI, Inc
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
Oct 14, 2026Bruno Leonardo Santos Menezes, Carlos Leonardo Souza Cardoso, Fabio André Machado PortoPortuguese Language Models LLM Safety Alignment
National Laboratory for Scientific Computing (LNCC), Institute of Artificial Intelligence, Petrópolis, Brazil
SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges
Oct 14, 2026Sebastian NaglAutomated Grading LLM-as-a-Judge
Technical University of Munich (TUM), Germany

Oct 29, 2026J/K move · Enter open · S save

Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items

Peng Xie, Amr Alanwar

TUM School of Computation, Information and Technology, Department of Computer Engineering Technical University of Munich, 74076 Heilbronn, Germany

LLM-as-a-Judge LLM Evaluation

Open paper AskPDF arXiv

To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.