LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5LLM-as-a-Judge Language Model Calibration
Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University +2
Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
Jun 12, 2026Shaojie YinMultilingual Language Model Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory Shanghai, China
From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
Jun 11, 2026Bora Kargi, David SalinasLLM Evaluation Pairwise Comparison
ELLIS Institute Tübingen · OpenEuroLLM
On the Limits of LLM-as-Judge for Scientific Novelty Assessment
Jun 10, 2026Soumitra Sinhahajari, Navonil Majumder, Soujanya PoriaAI for Science LLM-as-a-Judge
DeCLaRe Lab, Nanyang Technological University
Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents
Jun 9, 2026Sawyer Zhang, Alexander Wang, Sophie LeiLLM-as-a-Judge LLM Agent Evaluation
Lumivate (Lumi)
RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning
Jun 8, 2026Yiteng Mao, Kenan Xu, Yijia Lyu +3Mathematical Reasoning Benchmarks LLM Evaluation
University of Wisconsin–Madison · East China Normal University · New York University +2
Reasoning without Gold Standards: A Proxy-Judge Theory of Autoformalization
Jun 8, 2026Lei Xu, Xin Quan, André FreitasLLM Self-Refinement LLM-as-a-Judge
1Idiap Research Institute, Switzerland · 2École Polytechnique Fédérale de Lausanne (EPFL), Switzerland · Department of Computer Science, University of Manchester, United Kingdom +1
Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
Jun 8, 2026Mina Remeli, Moritz HardtPairwise Comparison LLM-as-a-Judge
Max Planck Institute for Intelligent Systems, Tübingen, Germany, Tübingen AI Center
Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges
Jun 8, 2026Yongtaek Lim, Hyeji Choi, Minwoo KimLLM-as-a-Judge LLM Safety Evaluation
AI Safety Team, DATUMO.INC, Seoul, South Korea.
ComplexConstraints and Beyond: Expert Rubrics for RLVR
Jun 8, 2026Sushant Mehta, Liudas Panavas, Suhaas Garre +1LLM-as-a-Judge Rubric-Based RL
Surge AI
AgentTrust: A Self-Improving Trust Layer for AI-Agent Actions
Jun 7, 2026Chenglin YangAI Agent Reliability LLM-as-a-Judge
Independent Researcher
Impacts of Histories and Models on LLM Grading: A Study in Advanced Software Engineering Courses
Jun 7, 2026Qilin Zhou, Zhuo Wang, Yue Li +1LLM Evaluation LLM-as-a-Judge
City University of Hong Kong, Hong Kong SAR, China
Contemporary AI lacks the imagination to diverge or negate in science
Jun 6, 2026Honglin Bao, Siyang Wu, Xiao Liu +3LLM-as-a-Judge Scientific Reasoning
Data Science Institute, University of Chicago · Knowledge Lab, University of Chicago
Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents
Jun 6, 2026Hyogon Ryu, Jeonghwan Kim, Yewon Lim +3LLM-as-a-Judge LLM Agent Evaluation
KRAFTON
When Languages Disagree: Self-Evolving Multilingual LLM Judges
Jun 6, 2026Xiyan Fu, Wei LuMultilingual Language Model Evaluation LLM-as-a-Judge
Nanyang Technological University
Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators
Jun 5, 2026Anissa Alloula, Federico Licini, Ava Batchkala +1LLM-as-a-Judge In-Context Learning
University of Oxford · Cohere
Cherry-pick Override: Unsafe Directional Commitment in LLM Judges under Mixed Evidence
Jun 5, 2026Haoran XuLLM-as-a-Judge Evidence-Grounded Reasoning
University of Glasgow
SLMJury: Can Small Language Models Judge as Well as Large Ones?
Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science and Engineering, LNMIIT, Jaipur, India · Department of Computer Science, Virginia Tech, Blacksburg, VA, USA
Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling
Jun 5, 2026Xing Yue, Linjuan Wu, Daoxin Zhang +2Reward Modeling LLM Evaluation
Zhejiang University · Xiaohongshu Inc.
MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring
Jun 4, 2026Ali Keramati, Shiyuan Zhou, Sharad Mehrotra +1Multi-Agent LLM Systems LLM-as-a-Judge
University of California, Irvine
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
Jun 4, 2026Zhihao Wu, Linhai Zhang, Taiyi Wang +4LLM-as-a-Judge Rubric-Based Evaluation
1King’s College London · University of Cambridge · 3AQA +1
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4LLM Evaluation LLM-as-a-Judge
1Shanghai Institute of AI for Education, East China Normal University, Shanghai 200062, China · School of Computer Science and Technology, East China Normal University, Shanghai 200062, China · 3Shanghai Innovation Institute, Shanghai 200231, China
Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges
Jun 3, 2026Srimonti Dutta, Akshata Kishore MoharirLLM-as-a-Judge Language Model Robustness
WAI USA Research Labs
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
Jun 3, 2026Abhishek DivekarLLM-as-a-Judge IR Evaluation
Amazon
Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
Jun 3, 2026XiuYu Zhang, Yi Shan, Junfeng Fang +1LLM Evaluation LLM-as-a-Judge
National University of Singapore · Beijing University of Technology
Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
Jun 3, 2026Xuekang Wang, Zhuoyuan Hao, Shuo Hou +3Reward Hacking RL for Language Models
Tsinghua University · Harbin Institute of Technology, Shenzhen · Xi’an Jiaotong University
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
Jun 3, 2026Yanjing Ren, Reza Ebrahimi, TengTeng MaLLM Safety Benchmarks LLM-as-a-Judge
The School of Information Systems University of South Florida Tampa, US
The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents
Jun 2, 2026Manvendra ModgilLLM-as-a-Judge Human-in-the-Loop Evaluation
AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making
Jun 2, 2026Sangwon Baek, Kyu Yeon Hur, Kyunga KimLLM-as-a-Judge Clinical Decision Support
Asclep Korea Inc., Changwon, Republic of Korea · Center for Data Science, New York University, New York, USA · Division of Endocrinology and Metabolism, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Korea +1
SenseJudge: Human-Centric Preference-Driven Judgment Framework
Jun 2, 2026Rui Li, Junfeng Liu, Xiangwen Kong +2LLM Evaluation LLM-as-a-Judge
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Xi’an Jiaotong University

Jun 12, 2026, cs.CVJ/K move · Enter open · S save

Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity

Daniel Lee, Harsh Sharma, Eunkyu Park, Pranav Narayanan Venkit+4

Salesforce AI Research · University of Colorado Boulder · Carnegie Mellon University+2

LLM-as-a-Judge Language Model Calibration Cultural Bias in Language Models Bias in LLM-as-a-Judge Multimodal Model Evaluation

Open paper AskPDF arXiv

MLLM-as-a-Judge is conventionally validated by agreement with human annotations, but this metric is undefined when the human pool is culturally heterogeneous. We introduce VOIR DIRE, a multimodal benchmark of 626 culturally paired image--prompt artifacts spanning U.S. and mainland Chinese contexts across food, fashion, and architecture, with annotator pools that are within-pool reliable (a = 0.86/0.74) but cross-pool divergent on evaluation (Q1 r = -0.12). Across six MLLMs, the bias decomposes into two failures: a positivity-floor calibration failure (compressed scale use) and an orientation failure (default to one cultural norm). On this corpus, where contested items are sampled to split the two pools, the floor mechanically validates the more-permissive Chinese reading; persona prompting partially recovers calibration, but the orientation residual survives, evidence the tilt is not reducible to scale compression. Reference-pool in-context demonstrations deepen the orientation residual and inflate the high end rather than restoring use of the low end. Model origin adds a small additive tilt (~0.10 MAE) that is approximately invariant under demonstration. We recommend reporting alignment against each reference pool separately and treating cross-pool divergence as a judge property.

Momentum

Related topics

Latest papers 362