LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
Sharding Prevents LLM Oversight Failures and Adversarial Exploitation
Aug 5, 2026Victor Akinwande, J. Zico Kolter, Aran NayebiScalable Oversight LLM Evaluation
School of Computer Science 2Neuroscience Institute Carnegie Mellon University
The Evaluator Is Part of the Experiment: Measuring Open-Ended LLM Conformity
Aug 5, 2026Alicia Guerra, Yibo HuLLM Evaluation LLM-as-a-Judge
Illinois Institute of Technology
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
Aug 4, 2026Bhavin Jawade, Cameron R. WolfeLLM Evaluation LLM-as-a-Judge
Netflix
RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation
Aug 3, 2026Divyansh Singh, Reza Davari, Afra MashhadiLLM-as-a-Judge LLM Auditing
Microsoft · University of Florida · University of Washington
Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias
Aug 3, 2026Baicheng Lin, Lingxi Jin, Kyung-Seok MinLLM-as-a-Judge Educational Technology
Department of Education, Sejong University, South Korea · Department of Educational Technology, Ewha Womans University, South Korea
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
Aug 3, 2026Fengxian Ji, Yuke Li, Jingpu Yang +8LLM Evaluation LLM-as-a-Judge
Wuhan University · Northeastern · Zhongguancun Academy +3
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
Aug 2, 2026Wenhui Chen, Jianlin Chen, Ziyao Lin +2LLM Evaluation LLM-as-a-Judge
University of Macau · South China University of Technology · Independent Researcher
A Triple-Robustness Analysis of Retrieval-Augmented Generation for Multi-Hop Requirements Traceability
Aug 1, 2026Meftun Akarsu, Burak Özdemir, Doğancan Büyükçolak +1LLM-as-a-Judge Graph Retrieval-Augmented Generation
Turkish Aerospace Industries · Technical University of Munich
More Debate, Same Evidence: Structural Limits of Homogeneous Multi-Agent Groundedness
Jul 31, 2026Yuelyu JiLLM Evaluation LLM Grounding
Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation
Jul 30, 2026Zheng Wu, Yibo Luo, Pu Zhang +2Human Preference Evaluation LLM-as-a-Judge
School of Computer Science, Shanghai Jiao Tong University · 2ByteDance Inc
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Jul 30, 2026Bertil Braun, Martin ForellLLM Evaluation Pairwise Comparison
KIT
Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models
Jul 30, 2026Shuyi Fan, Boyuan Deng, Mengyu Xu +4LLM Evaluation LLM-as-a-Judge
Columbia University, USA · Johns Hopkins University, USA · The University of Chicago, USA +4
Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation
Jul 30, 2026Weining ZhangLLM Evaluation LLM-as-a-Judge
Cheung Kong Graduate School of Business
SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents
Jul 25, 2026Yang Wan, Zhenhao Zhang, Jierui Wang +1Reward Modeling LLM-as-a-Judge
1Zhejiang University · University of Rochester
Reference-Free Evaluation of Reasoning in Open-Ended Question Answering
Jul 22, 2026Guneet Singh Kohli, Yuxiang Zhou, Michael Sejr Schlichtkrull +2LLM-as-a-Judge Reasoning Evaluation
Queen Mary University of London · Temple University
Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Jul 21, 2026Xilun Chen, Zhaleh Feizollahi, Ross Goodwin +5Open-Ended Generation LLM-as-a-Judge
Meta AI
SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement
Jul 21, 2026Arther Tian, Alex Ding, Simon Wu +1Supervised Fine-Tuning LLM-as-a-Judge
DGrid AI
Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety
Jul 21, 2026Koyar AfrasyabLLM-as-a-Judge Language Model Safety Evaluation
Kinvectum AB, Sweden
EduPanel: A Three-Agent LLM Judge for Teaching Videos -- Reliability, Complementarity, and Human Trust Calibration
Jul 20, 2026Jia-Kai Dong, Yi-Cheng Lin, Hung-yi LeeLLM-as-a-Judge Educational Assessment
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
Jul 20, 2026Koyar AfrasyabLLM-as-a-Judge LLM Prompting
Kinvectum AB, Sweden
A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
Jul 19, 2026Mohtashim KhanLLM Evaluation LLM-as-a-Judge
Does Multi-Agent Debate Improve AI Feedback on Research Papers?
Jul 16, 2026Tomas Havranek, Zuzana IrsovaLLM-as-a-Judge Multi-Agent Debate
Charles University, Prague · Centre for Economic Policy Research, London · Meta-Research Innovation Center at Stanford
The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol
Jul 15, 2026Serkan BallıMultilingual Language Model Evaluation LLM-as-a-Judge
Department of Software Engineering, Mehmet Akif Ersoy University, Burdur, Türkiye · Department of Software Engineering, Mehmet Akif Ersoy University, Türkiye
Evaluation Ability Does Not Imply Optimization Utility: LLM-as-a-Judge Signals in Closed-Loop Table Recognition
Jul 15, 2026Donghwan KimTable Structure Recognition LLM-as-a-Judge
Aidentyx Inc., San Jose, CA, USA
Rating the Raters: Rasch Measurement Theory for LLM Evaluation
Jul 14, 2026Pratik S. Sachdeva, Nathan BoudolLLM Evaluation LLM-as-a-Judge
University of California, Berkeley · Grenoble INP
LLM Judges Can Be Too Generous When There Is No Reference Answer
Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLLM Evaluation LLM-as-a-Judge
Computational Linguistics, Department of Linguistics, University of Potsdam, Germany · National Research Council, Ottawa, Canada
FinResearchBench II: A Deep Research Benchmark with Consensus-Derived Gold Rubrics for Distinguishing Financial Report Quality
Jul 14, 2026Beidi Luan, Rui Sun, Sinuo Wang +5LLM-as-a-Judge Rubric-Based Evaluation
1StepFun · University of Adelaide · 2FinStep +1
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
Jul 13, 2026Zixiang Xu, Sixian Li, Huaxing Liu +4LLM-as-a-Judge LLM Interpretability
AMAP, Alibaba Group · Mohamed bin Zayed University of Artificial Intelligence · University of Southern California +1
Knowledge Distillation for Automated AI Tutor Evaluation
Jul 12, 2026Tahmid Al Hannan, Diego Garcia, Alex Njoroge +2LLM-as-a-Judge AI in Education
Folsom Lake College Folsom, CA, USA

Aug 5, 2026, cs.CLJ/K move · Enter open · S save

Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

Divyansh Singh

University of Florida

Human Preference Evaluation LLM Evaluation LLM-as-a-Judge Pairwise Preference Evaluation

Open paper AskPDF arXiv

LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.

Momentum

Related topics

Latest papers 362