LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

All Verdicts are Not Equal: Rethinking LLM Judge Reliability
Oct 8, 2026Vineet Kumar, Darshita Rathore, Anindya MoitraLLM-as-a-Judge LLM Evaluation
PayPal Artificial Intelligence PayPal, Bengaluru, India
Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents
Oct 8, 2026Xing Zhang, Guanghui Wang, Yanwei Cui +4Automated Evaluation LLM Agent Self-Improvement
AWS Forward Deployed Engineering · HSBC Holdings Plc., HSBC Technology Center, China
Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation
Oct 8, 2026Hongliang LiuLLM-as-a-Judge Bias in LLM-as-a-Judge
Independent Researcher
How to post-train on a surrogate: Envelope sampling mitigates reward hacking
Oct 8, 2026Sanjit Dandapanthula, Shuvom Sadhuka, Samir Khan +3RL Post-Training Reward Hacking
Carnegie Mellon University Department of Statistics and Machine Learning Department · Abridge AI · MIT EECS +2
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
Oct 8, 2026Xing Han Lù, Dheeraj Vattikonda, Sina Hajimiri +7Computer-Use Agent Benchmarks Long-Horizon Agent Evaluation
ServiceNow Research · McGill University · Mila – Quebec AI Institute +5
Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression
Oct 7, 2026Mingqing Yuan, Xiaobo Liang, Junwei Yang +5Reward Modeling LLM-as-a-Judge
Soochow University · University of Cambridge · Chalmers University of Technology +3
A Tale of Two Error Categories: Exploring Concealed Trade-Offs in the Errors of Automated Judges in Evaluation of Uncertainty Quantifiers
Oct 7, 2026Evgenia Ilia, Wilker AzizUncertainty Quantification Automated Evaluation
University of Amsterdam
Reliability of LLM Judges for Evaluating Entity Alignment
Oct 7, 2026Vaibhava Lakshmi Ravideshik, Mayank KejriwalLLM-as-a-Judge Bias in LLM-as-a-Judge
University of Michigan, Ann Arbor · Information Sciences Institute, University of Southern California
Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions
Oct 6, 2026Wenqi Li, Bin Liu, Mindi Ruan +3LLM Evaluation LLM-as-a-Judge
University at Albany, State University of New York · West Virginia University
AdaGuard: Enhancing Safety and Policy Compliance with Reasoning-Enabled LLM-As-A-Judge Guardrails
Oct 6, 2026Melissa Kazemi Rad, Sihui Dai, Isha Slavin +6LLM Guardrails LLM Safety
Capital One
Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge
Oct 6, 2026Chuhong Xu, Bo Su, Ziyao Chen +3LLM-as-a-Judge Citation Verification
Sofia University · Indiana University · University of California, San Diego +2
The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation
Oct 6, 2026Jorma Valjakka, Juhani Kivimäki, Juha Mylläri +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science University of Helsinki
Trustworthy Method Comparison with AI Judges: Estimation and Design under Order, Batch, and Aggregation Effects
Oct 6, 2026Tianxi Li, Jie DingLLM Evaluation LLM-as-a-Judge
School of Statistics, University of Minnesota
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks
Oct 5, 2026Gowthamkumar NandakishoreContrastive Learning LLM Evaluation
Independent researcher
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
Oct 5, 2026Hashmath Shaik, Gnaneswar Villuri, Alex DoboliLLM Evaluation LLM-as-a-Judge
Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Oct 5, 2026Yiqi Liu, Joseph James, Yang Wang +3LLM Evaluation LLM-as-a-Judge
The University of Manchester · The University of Sheffield · University of Pittsburgh +1
VERA: Verdict-Conditioned Reliability for Adaptive LLM Judges
Oct 4, 2026Qiushui Xu, Syamil Mohd Razak, Tao Yuan +1LLM-as-a-Judge LLM Fine-Tuning
Penn State University · Amazon
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
Evaluating and Improving the Robustness of Large Language Models to Input Sequence Variations
Oct 1, 2026Narek MaloyanLanguage Model Robustness Adversarial Attacks on LLMs
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Oct 1, 2026Noy Sternlicht, Simra Shahid, Peter Jansen +3Prompt Sensitivity LLM-as-a-Judge
Hebrew University of Jerusalem · Allen Institute for AI · Microsoft +2
Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation
Oct 1, 2026Yuan Huang, Zirui Song, Xiuying ChenVLM Robustness LLM-as-a-Judge
Northeastern University · Mohamed bin Zayed University of Artificial Intelligence
Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervals
Oct 1, 2026Ryunyi Lee, Kangjun Noh, Somin Kim +2Reward Modeling LLM-as-a-Judge
Yonsei University
When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
Oct 1, 2026Huichan SeoVLM Evaluation LLM-as-a-Judge
Independent Researcher
AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
Oct 1, 2026Giulio Zeloni, Enrico Lo Conte, Salvatore Rionero +3LLM-as-a-Judge RAG Evaluation
Protom Group S.p.A., Napoli, Italy
Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Banking
Sep 30, 2026Ludovic Gibert, Matis Despujols, Andre-Louis RochetLLM-as-a-Judge Long-Form Document Generation
AIData2Action · TW3 Partners
LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian
Sep 30, 2026Claudiu Creanga, Liviu P. DinuLLM Evaluation LLM-as-a-Judge
Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania
RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection
Sep 30, 2026Elia Onofri, Roberto Di PietroLLM Evaluation LLM-as-a-Judge
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology (KAUST) Thuwal 23955, Saudi Arabia
Training LLM Judges from Language Feedback via Position-Selective Self-Distillation
Sep 30, 2026Ilgee Hong, Changlong Yu, Zhenghao Xu +5LLM-as-a-Judge Language Model Distillation
Georgia Institute of Technology · Amazon · UC San Diego
Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
Sep 29, 2026Bruno Brocai, Maria BeckerLLM-as-a-Judge Pairwise Preference Evaluation
Heidelberg University German Department
Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment
Sep 29, 2026Yu Zhao, Jiarui Wang, Huiyu Duan +5LLM-as-a-Judge Image Generation Evaluation
Shanghai Jiao Tong University · Dalian University of Technology

Oct 8, 2026, cs.CLJ/K move · Enter open · S save

All Verdicts are Not Equal: Rethinking LLM Judge Reliability

Vineet Kumar, Darshita Rathore, Anindya Moitra

PayPal Artificial Intelligence PayPal, Bengaluru, India

LLM-as-a-Judge LLM Evaluation Bias in LLM-as-a-Judge LLM Auditing

Open paper AskPDF arXiv

LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth. We present a comprehensive reliability audit, stresstesting six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. Our empirical analysis reveals severe vulnerabilities: verdicts change across identical replications at temperature zero, position-order swaps flip the majority of verdicts on challenging tasks, and the most deterministic judge achieves perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. To formalize these multi-faceted failure modes, we introduce the trustworthy verdict rate (T ), a unified metric capturing the joint probability that an evaluation is reproducible, order-invariant, and accurate. UsingT , we derive a theoretical upper bound on accuracy imposed by position bias and show that reliability is item-specific rather than modellevel. Finally, we demonstrate that shifting from pairwise win-rate to holistic rubric scoring improves trustworthiness more than any single-format prompting intervention, offering an actionable framework for robust NLP evaluation.

Momentum

Related topics

Latest papers 362