LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Jul 12, 2026Sriram Selvam, Anneswa GhoshLLM-as-a-Judge Language Model Generation Evaluation
Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Reviews
Jul 12, 2026Nuo Chen, Qian Wang, Qingyun Zou +1LLM-as-a-Judge Scholarly Peer Review
National University of Singapore
Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution
Jul 9, 2026Ethan Leung, Elias Lumer, Corey Feld +3LLM-as-a-Judge Citation Verification
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Jul 9, 2026Zongyou Yang, Yinghan Hou, Xiaokun YangLLM-as-a-Judge LLM Auditing
Dyson School of Design Engineering, Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom
When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
Jul 9, 2026Kaihua DingLLM-as-a-Judge LLM Reliability
University of Pennsylvania
A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents
Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Audio-Language Model Evaluation Voice Agent Evaluation
Salesforce Applied AI Research, eVerse team
SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation
Jul 8, 2026Andrea Scarinci, Virginia Negri, Brayan Impata +3LLM-as-a-Judge LLM-Assisted Annotation
Amazon
The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents
Jul 8, 2026Xing Zhang, Yanwei Cui, Guanghui Wang +4LLM-as-a-Judge LLM Agent Safety
1AWS Generative AI Innovation Center · 2HSBC Holdings Plc., HSBC Technology Center, China
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Jul 7, 2026Wei Dong, Tianyu Fu, Zhe Yu +9LLM-as-a-Judge Web Agent Benchmarks
Mininglamp Technology · School of Artificial Intelligence, University of Chinese Academy of Sciences · MAIS, Institute of Automation, Chinese Academy of Sciences
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
Jul 7, 2026Chenyu ZhouReward Hacking RL for Language Models
School of Engineering, Institute of Science Tokyo
LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review
Jul 6, 2026Ali Hassaan Mughal, Muhammad BilalSoftware Engineering Automated Software Testing
Independent Researcher, USA · Technical University of Munich, Germany
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Jul 2, 2026Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard +2LLM-as-a-Judge Computer Science Education
aUniversidade de Vigo, Department of Computer Science, ESEI-Higher School of Computer Engineering, Edificio Politécnico, Campus Universitario As Lagoas s/n, Ourense, 32004, Spain · bIFCAE-Institute for Research in Physics, Computing and Aerospace Science, Universidade de Vigo, Ourense, 32004, Spain
Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages
Jul 2, 2026A. Seza Doğruöz, Xixian Liao, Verena Blaschke +3Multilingual Language Model Evaluation LLM-as-a-Judge
LT3, IDLab, Universiteit Gent · Barcelona Supercomputing Center · LMU Munich & Munich Center for Machine Learning +5
Ask the Right Comparison:Bias-Aware Bayesian Active Top- $k$ Ranking with LLM Judges
Jul 2, 2026Jian Xu, Delu Zeng, John Paisley +1Pairwise Comparison LLM-as-a-Judge
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1
SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses
Jul 2, 2026Anna ChornaMultilingual Language Model Evaluation LLM-as-a-Judge
Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling
Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward Modeling LLM-as-a-Judge
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
AGC-Bench: Measuring Artificial General Creativity
Jul 1, 2026Roger Beaty, Vijeta Deshpande, Clin K. Y. Lai +9Creativity Assessment LLM Evaluation
1Pennsylvania State University · University of Massachusetts Lowell · University of Amsterdam +2
Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention
Jul 1, 2026William Philipp, Finn Fassbender, Daniel Fister +12Multilingual Language Model Evaluation LLM-as-a-Judge
Section for Clinical Research-IT, Institute of Medical Biometry and Statistics, University of Luebeck, Luebeck, Germany · Medical Faculty, University of Tübingen, Tübingen, Germany · German Medical Students’ Association (bvmd), Berlin, Germany +9
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference Evaluation LLM-as-a-Judge
Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2
Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent Conditions
Jul 1, 2026Zewen LiuLLM Evaluation LLM-as-a-Judge
Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth
Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Multilingual Language Model Evaluation LLM-as-a-Judge
1Perle AI
CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning
Jun 30, 2026Ajmal M., Abin Roy, Afthab Salam Kanniyan +4LLM-as-a-Judge Human-in-the-Loop Evaluation
1MBZUAI · 2IIT Madras · 3Calicut Medical College
Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?
Jun 30, 2026Zewen LiuLLM-as-a-Judge Pairwise Preference Evaluation
Qilu Institute of Technology, School of Software Engineering Tai’an, Shandong, China
Probing Stylistic Appropriation using Large Language Models: An Evaluation Framework for Copyright Infringement under EU Law
Jun 30, 2026Noah Scharrenberg, Chang SunLLM-as-a-Judge LLM Auditing
Department of Advanced Computing Sciences, Maastricht University, Maastricht, The Netherlands. · Contractuo, Eindhoven, The Netherlands.
Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
Jun 30, 2026Ke Zhang, Patricio Gallardo Candela, Sudhir Murthy +3LLM-as-a-Judge Lean Theorem Proving
University of California, Riverside · University of Arizona · University of California, San Diego
Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments
Jun 29, 2026Christopher W. Karvetski, Sheldon S. Huang, Simas Kučinskas +4LLM-as-a-Judge Probabilistic Forecasting
Forecasting Research Institute · Good Judgment Inc · University of Toronto +4
RoPoLL: Robust Panel of LLM Judges
Jun 29, 2026Anish Acharya, Kris W Pan, Brian VerkhovskyLLM Evaluation LLM-as-a-Judge
Amazon Web Services
Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support
Jun 29, 2026Mizanur Rahman, Abeer Badawi, Elahe Rahimi +4Mental Health Counseling Multi-Agent LLM Systems
1York University, Ontario, Canada · 2Vector Institute, Ontario, Canada · 3Connected Minds, Ontario, Canada +1
Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?
Jun 29, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1LLM Evaluation LLM-as-a-Judge
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Department of Portuguese, Faculty of Arts and Humanities, University of Macau
EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots
Jun 29, 2026Camilo Chacón SartoriLLM Safety Benchmarks Multilingual Language Model Evaluation
MindSurf

Jul 12, 2026, cs.CLJ/K move · Enter open · S save

Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG

Sriram Selvam, Anneswa Ghosh

LLM-as-a-Judge Language Model Generation Evaluation Language Model Bias Evaluation RAG Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-as-a-judge evaluation is widely used for retrieval-augmented generation (RAG), but reusing the same model family as both generator and judge makes self-leniency difficult to identify. We introduce Eval-Pair Matrix, a controlled meta evaluation protocol for source-grounded RAG. Starting from GaRAGe questions and grounding passages, we induce one hidden answer-causal contradiction per record, generate answers from perturbed passages with GPT, Grok, and Gemini models, and then use the same models as blind judges to evaluate each answer against the original passages. The experiment contains 300 core records, 897 labeled generator outputs, and 2,683 judge verdicts in a crossed 3 x 3 matrix; the primary analysis uses 275 fully validated records. Instead of comparing diagonal and off-diagonal cells across different answers, we estimate same-model effects by pairing judges on the exact same candidate answer. This changes the interpretation: diagonal and off diagonal F1 are similar, and the paired same-model recall effect is near zero (-0.5 pp; 95% cluster bootstrap CI [-2.7, +1.7]). The only robust paired gap is lower matching-judge flagging for answers that avoided the induced claim (-4.3 pp). A targeted human evaluation finds that reviewed apparent false positives are alternate source-error detections, mistakes in labeling whether the induced claim was adopted, or unclear cases; none were adjudicated as genuine false alarms. The lesson is methodological: RAG judge studies should report full matrices, answer-paired effects, behavior strata, and label-task alignment.

Momentum

Related topics

Latest papers 362