LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

Commit-first LLM judging inherits the judge's own errors
Aug 31, 2026Idil GozelLLM-as-a-Judge LLM Auditing
Evaluator Integrity, London
Three Ways Classical Test Theory Can Mislead About LLM Judges
Aug 31, 2026Louis Yiven ZhuLLM-as-a-Judge Psychometric Validation
University of Oxford
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
Aug 31, 2026Minsoo Song, Chanwoo Kim, Sugyeong Eo +1Human Preference Evaluation LLM-as-a-Judge
Soongsil University · Yonsei University
Small Language Models as Judges for Rubric-Based Reinforcement Learning
Aug 30, 2026Fengyu Xie, Yilun Zhao, Bingsen Chen +2LLM-as-a-Judge Language Model Generation Evaluation
New York University · Yale University
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University · National University of Singapore
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific Discovery LLM-as-a-Judge
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1
Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
Aug 27, 2026Shuyi Fan, Boyuan Deng, Mengyu Xu +3Causal Inference LLM Auditing
The University of Chicago · Johns Hopkins University · The Hong Kong Polytechnic University, Hong Kong
J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Data
Aug 27, 2026Gyouk Chu, Myeongho Jeon, Teresa Yeo +1LLM-as-a-Judge Self-Training for Language Models
KAIST
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
Aug 26, 2026Jia-Hao Ji, Sijie Li, Jiabei Cheng +3LLM-as-a-Judge Multi-Agent Reasoning
Department of Neurology and National Center for Neurological Diseases, Huashan Hospital, State Key Laboratory of Medical Neurobiology and Ministry of Education Frontiers Center for Brain Science, Shanghai Medical College, Fudan University, Shanghai 200040, China. · Institute of Science and Technology for Brain-Inspired Intelligence, MOE Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, MOE Frontiers Center for Brain Science, Fudan University, Shanghai 200433, China. · Shanghai Academy of Artificial Intelligence for Science, Shanghai 200003, China. +1
Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
Aug 25, 2026Anupam Purwar, Shashank Singh, Kritika SrivastavaVoice Agent Evaluation LLM-as-a-Judge
Sprinklr AI Gurugram, India · Sprinklr AI Bengaluru, India
Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment
Aug 21, 2026Emma Granqvist, Rocío Mercado, Samuel GenhedenAI Agents for Scientific Discovery LLM-as-a-Judge
Molecular AI, Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden · Department of Computer Science and Engineering, Section for Data Science and AI, Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden · Science for Life Laboratory (SciLifeLab), Gothenburg, Sweden
The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Aug 18, 2026Emma Yanyang Kong, JJ Tan, Ishan Gupta +8LLM Evaluation LLM-as-a-Judge
Netflix · Los Gatos, California, USA
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
Aug 12, 2026Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar +2LLM Evaluation LLM-as-a-Judge
1Meta Superintelligence Labs · 2FAIR at Meta
Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces
Aug 12, 2026Congchao Wang, Diwakar Singh, Qiaozi Gao +3LLM-as-a-Judge Reasoning Evaluation
Amazon AGI
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Aug 12, 2026Xi Chen, Jie Mu, Mo Xuan +1Human Preference Evaluation LLM-as-a-Judge
Ant Group
GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation
Aug 12, 2026Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam +3AI-Assisted Decision Making LLM-as-a-Judge
Intuit
Benchmarking LLM Judges for Mobile Agent Evaluation
Aug 11, 2026Ziqiang Wan, Li Gu, Zhixiang Chi +4LLM-as-a-Judge LLM Agent Evaluation
Mila – Québec AI Institute · Concordia University · University of Toronto +2
DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation
Aug 11, 2026Xiaotong Wang, Dazhen DengData Visualization LLM-as-a-Judge
State Key Lab of CAD&CG, Zhejiang University, China · School of Software Technology, Zhejiang University, China
Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
Aug 11, 2026Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie +1LLM Evaluation LLM-as-a-Judge
The University of Queensland · Brisbane, Australia
RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement
Aug 10, 2026Ziheng Jia, Jiaying Qian, Zicheng Zhang +3LLM-as-a-Judge Rubric-Based Evaluation
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
DS@GT ARC at Touché: Large Language Models for Retrieval-Augmented Debate
Aug 8, 2026Anthony Miyaguchi, Conor JohnstonLLM Evaluation LLM-as-a-Judge
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332, USA
Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees
Aug 8, 2026Fariya Afrin, Ibne Farabi ShihabReward Hacking LLM Evaluation
Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Computer Science, Iowa State University
Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification
Aug 8, 2026Yang ShuLLM-as-a-Judge Adversarial Attacks on LLMs
Zhejiang University, Hangzhou, China
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
Aug 7, 2026Yiyao Zhang, Diksha Goel, Hussain Ahmad +2LLM-as-a-Judge LLM Reliability
School of Computing and Information Technology, University of Wollongong, Wollongong, NSW, Australia · CSIRO’s Data61, Clayton, VIC, Australia · School of Computer Science and Information Technology, Adelaide University, Adelaide, SA, Australia
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators
Aug 7, 2026Yuheng Zhang, Yuanchun Wang, Fanjin Zhang +4LLM-as-a-Judge Automated Evaluation
School of Information Renmin University of China Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering Beijing, China · Engineering Research Center of Database and Business Intelligence Beijing, China +1
Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering
Aug 7, 2026Xiuwei Shang, Li Hu, Xiao Jiang +7LLM-as-a-Judge Software Reverse Engineering
University of Science and Technology of China Hefei, China · Singapore Management University Singapore · University of Alberta & Alberta Machine Intelligence Institute Edmonton, Canada +1
Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps
Aug 7, 2026Yang ShuLLM Evaluation LLM-as-a-Judge
Zhejiang University, Hangzhou, China
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM Evaluation LLM-as-a-Judge
Japan Advanced Institute of Science and Technology
SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution
Aug 6, 2026Zhi Han, Chenxi Zeng, Liuhaichen Yang +3LLM-as-a-Judge Long-Horizon Agent Evaluation
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · University College London +1

Aug 31, 2026, cs.SEJ/K move · Enter open · S save

Commit-first LLM judging inherits the judge's own errors

Idil Gozel

Evaluator Integrity, London

LLM-as-a-Judge LLM Auditing Code Generation Evaluation

Open paper AskPDF arXiv

LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.

Momentum

Related topics

Latest papers 362