LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?
Sep 29, 2026Yue Pan, Jiawei Li, Ziyuan Zhang +2LLM Evaluation LLM-as-a-Judge
University College London · Amazon · Zhejiang University
From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks
Sep 29, 2026Zheng Zhang, Lufei Li, Xinyue Tan +4LLM Evaluation LLM-as-a-Judge
School of Information Science and Technology, ShanghaiTech University · State Key Laboratory of General Artificial Intelligence, BIGAI
Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges
Sep 29, 2026Mingyue Huo, Shivam Mehta, Bhavin Jawade +2LLM-as-a-Judge Speech Quality Assessment
University of Illinois Urbana-Champaign · Netflix
JudgeProfile: Understanding and Steering Subjectivity in LLM Judges
Sep 29, 2026Qi Cao, Kangning Liu, Xuan Kan +10Human Preference Evaluation LLM Evaluation
Meta · University of California San Diego · The University of Hong Kong +1
The Argument and the Letterhead: Source-Position Coherence in AI Evaluation
Sep 28, 2026Michele LoiLLM-as-a-Judge Political Bias in Language Models
Certified Selective Automation of LLM Agent Evaluation
Sep 28, 2026Chengguang Gan, Yunhao Liang, Qinghao Zhang +1LLM-as-a-Judge Selective Prediction
Independent Researcher · University of Chinese Academy of Sciences · Pusan National University +1
PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?
Sep 28, 2026Shayekh Bin Islam, Hwanjun SongLLM-as-a-Judge Video-Language Model Evaluation
KAIST
Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
Sep 28, 2026Jiapeng LiLLM-as-a-Judge AI Agent Auditing
Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents
Sep 27, 2026Kaiwen Luo, Ming GaoLLM-as-a-Judge LLM Agent Evaluation
Meta
Curating Merchant-Matching Training Data with Two Confidence-Gated Local LLM Judges
Sep 27, 2026Donghao Huang, Jinling Pei, Zhaoxia WangLLM-as-a-Judge Selective Prediction
Research and Development, Mastercard, Arlington, VA, USA · School of Computing and Information Systems, Singapore Management University, Singapore
Learning Strategies to Break Judges
Sep 27, 2026Guruprerana Shabadi, Aaditya Naik, Rajeev Alur +1LLM-as-a-Judge AI Agent Auditing
University of Pennsylvania · Oracle
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
Sep 27, 2026Krishna Chytanya AyyagariLLM Evaluation LLM-as-a-Judge
Google
LLM Judge Validation Under Sparse Overlap: From Inference to Design
Sep 25, 2026Junxuan Li, Arko Mukherjee, Soumyabrata PalInter-Rater Reliability LLM Evaluation
Adobe · Adobe Research
JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Sep 24, 2026Delip Rao, Chris Callison-BurchLLM Inference Efficiency LLM Evaluation
University of Pennsylvania
Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
Sep 24, 2026Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh +2Prompt Sensitivity LLM-as-a-Judge
KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST +1
Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents
Sep 23, 2026Jian XuLLM-as-a-Judge AI Agent Auditing
RIKEN
Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality
Sep 23, 2026Jiaju Huang, Hao Yang, Xinyu Ma +6Radiology Report Generation LLM-as-a-Judge
Intelligent Medical Computing Laboratory, Faculty of Applied Sciences, Macao Polytechnic University · Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands · Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
Sep 22, 2026Yubo Li, Yidi Miao, Ramayya Krishnan +1LLM-as-a-Judge Language Model Generation Evaluation
Carnegie Mellon University
GroundedGEO: Auditing the Evidence Gap in Generative Search Rankings
Sep 21, 2026Yihan Xia, Huiling Fan, Kangrong Zhong +1LLM-as-a-Judge LLM Reranking
Shenzhen University Shenzhen, China
LLJ Cards: Best practices for the Use of LLMs as Judges
Sep 21, 2026Khaoula Chehbouni, Melina Medjdoub, Florian Carichon +2LLM Evaluation LLM-as-a-Judge
McGill University · Mila - Quebec AI Institute · Concordia University
ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation
Sep 21, 2026Lijian Wu, Henry Hengyuan Zhao, Zijian Zhang +3VLM Evaluation LLM Evaluation
CSU-JPG, Central South University · National University of Singapore
OSCAR: Order-aware Scoring and Calibration for AI Rankings
Sep 21, 2026You Liu, Yue Liu, Quanchao Lu +1LLM Evaluation LLM-as-a-Judge
How Many Humans Are 32 LLM Judges Worth?
Sep 18, 2026Chao Li, Yingying Yu, Yunfeng LiAnnotator Disagreement LLM-as-a-Judge
Tsinghua University · University College London · Hebei University of Economics and Business
VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
Sep 17, 2026Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D +3Audio QA LLM-as-a-Judge
International Institute of Information Technology Hyderabad, India
I code or AI code: A comparative evaluation of AI-rated scores in classroom observations
Sep 16, 2026Y. Fong, J. Xiang, T. Y. D. Chan +2LLM-as-a-Judge Educational Assessment
Department of Early Childhood Education, The Education University of Hong Kong, 10 Lo Ping Road, Tai Po, Hong Kong SAR, China · Yew Chung College of Early Childhood Education, 2 Tin Wan Hill Road, Tin Wan, Hong Kong SAR, China
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers
Sep 16, 2026Zihan Chen, Di Zhu, Lei Zheng +1LLM-as-a-Judge LLM Auditing
Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA · Stony Brook University Stony Brook, NY, USA
A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality
Sep 16, 2026Jerry KaplanLLM Quantization LLM Evaluation
Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University
Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels
Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2LLM-as-a-Judge Pairwise Preference Evaluation
Beyond the Name: Demographic Leakage in De-Identified Résumés and Evaluation Artifacts in LLM Bias Audits
Sep 15, 2026Qiangju Chen, Yang XiaoLLM-as-a-Judge Language Model Bias Evaluation
Macquarie University · The University of Melbourne
Skill-based Agentic Evaluation for Real-time Data Science Tasks
Sep 15, 2026Aniruddha Tamhane, Raghavendra Addanki, Ayushi Aggarwal +4LLM-as-a-Judge LLM Agent Evaluation
Adobe

Sep 29, 2026, cs.SEJ/K move · Enter open · S save

CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?

Yue Pan, Jiawei Li, Ziyuan Zhang, Xiangxin Zhao+1

University College London · Amazon · Zhejiang University

LLM Evaluation LLM-as-a-Judge AI Agent Evaluation Automated Code Review

Open paper AskPDF arXiv

Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers. We study technical trustworthiness judgment: determining whether a review comment's core technical claims are correct and applicable to the reviewed code in its repository context. Existing code-review benchmarks primarily evaluate review generation, issue discovery, or general comment quality, but do not directly assess whether an agent can determine the technical trustworthy of an individual review comment. To fill this gap, we introduce CRJudgeBench, a benchmark of 1199 instances constructed from real pull requests and expert-verified perturbations, covering both trustworthy and plausible but untrustworthy comments. We further present Sentinel, a repository-grounded agentic judge that actively gathers code evidence to verify review comments before making judgments. Starting from Qwen3-Coder-30B-A3B-Instruct, Sentinel is trained on the CRJudgeBench training split through iterative action-level learning from a privileged teacher. On the 359-instance CRJudgeBench test set, Sentinel achieves 76.60% accuracy, outperforming GLM-5.3 by 6.13 percentage points and its base model by 19.78 points. These results show that even state-of-the-art general-purpose LLMs struggle to identify untrustworthy comments, while iterative action-level learning substantially improves the accuracy of repository-grounded trustworthiness judgments. Our dataset is available at https://huggingface.co/datasets/dcloud347/CRJudgeBenchmark

Momentum

Related topics

Latest papers 362