LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator
May 20, 2026Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh +3LLM Evaluation Pairwise Comparison
Layer 6 AI · University of Toronto
Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education
May 20, 2026Arun-Balajiee Lekshmi-Narayanan, Mohammad Hassany, Peter BrusilovskyLLM-as-a-Judge Programming Education
University of Pittsburgh
Pseudo-Formalization for Automatic Proof Verification
May 19, 2026Slim Barkallah, Luke Bailey, Kaiyue Wen +2Mathematical Reasoning Benchmarks LLM-as-a-Judge
Stanford University
LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation
May 19, 2026Shanshan Xu, Johan Lindholm, Amogh Raina +2Legal NLP LLM Evaluation
University of Copenhagen · Umeå University
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
May 18, 2026Leyao Wang, Yanan He, Peng Chen +5AI Agent Reliability LLM-as-a-Judge
Yale University · IBM Research
GRASP: Deterministic argument ranking in interaction graphs
May 18, 2026Diganta Misra, Antonio Orvieto, Rediet Abebe +1LLM-as-a-Judge Learning to Rank
1MPI-IS Tübingen · 2Tübingen AI Center · 3ELLIS Institute Tübingen +2
Estimating Item Difficulty with Large Language Models as Experts
May 18, 2026Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2LLM Evaluation LLM-as-a-Judge
Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2
QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
May 17, 2026Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare +1LLM-as-a-Judge Human-in-the-Loop Evaluation
AI Lab, Arioobarzan Engineering Team, Shiraz, Iran · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran · Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy
Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
May 15, 2026Nils Feldhus, Tanja Baeumel, Elena Golimblevskaia +10LLM-as-a-Judge LLM Interpretability
CLCG, University of Groningen · German Research Center for Artificial Intelligence (DFKI) · Centre for European Research in Trusted AI (CERTAIN) +5
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
May 14, 2026Gaojie Jin, Yong Tao, Lijia Yu +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, University of Exeter · Institute of AI for Industries, Chinese Academy of Sciences · Department of Mathematics and Computer Science, Eindhoven University of Technology
Small, Private Language Models as Teammates for Educational Assessment Design
May 14, 2026Chris Davis Jaldi, Anmol Saini, Shan Zhang +3LLM Evaluation LLM-as-a-Judge
Wright State University, USA · University of Florida, USA · TIB – Leibniz Information Centre for Science and Technology, Germany
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
May 13, 2026Andrea MorandiLLM Evaluation Pairwise Comparison
Cisco Systems, Inc.
Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations
May 13, 2026Kyo Gerrits, Rik van Noord, Ana Guerberof ArenasCreativity Assessment LLM-as-a-Judge
Centre for Language and Cognition, University of Groningen
Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
May 13, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleLLM-as-a-Judge Faithfulness of Language Model Explanations
Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.
Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals
May 12, 2026Yo EharaAnnotator Disagreement LLM-as-a-Judge
Tokyo Gakugei University
VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
May 11, 2026Jasmine Qi, Danylo Dantsev, Muyang SunPost-Hoc Calibration LLM-as-a-Judge
Indeed Inc.
Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
May 11, 2026Wenbo Zhang, Lijinghua Zhang, Liner Xiang +1LLM-as-a-Judge Cost-Aware Inference
Department of Statistics, University of California, Irvine, USA.
LLM-Based Code Documentation Generation and Multi-Judge Evaluation
May 11, 2026Ikbel Ghrab, Mohamed Dhieb, Ismail Khenissi +1LLM Evaluation LLM-as-a-Judge
University of Carthage, Engineering School of Statistics and Information Analysis (ESSAI), Ariana, Tunisia · Easy Transfer, 72 Rue de la Division Leclerc, 91160 Saulx-les-Chartreux, Paris, France · University of Tunis El Manar, Laboratory of Bioinformatics, bioMathematics, and bioStatistics (LR16IPT09), Institut Pasteur de Tunis, 13 Place Pasteur, B.P. 74, Belv´ed`ere, 1002 Tunis, Tunisia
Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
May 10, 2026Yanran LiLLM Evaluation LLM-as-a-Judge
Department of Biostatistics Columbia University
Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
May 9, 2026Andrea MorandiPost-Hoc Calibration LLM-as-a-Judge
Cisco Systems, Inc.
Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need?
May 8, 2026Jane Paik KimLLM Evaluation LLM-as-a-Judge
Department of Psychiatry and Behavioral Sciences Stanford University Stanford, CA 94304
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
May 8, 2026Saloni Garg, Amit SagtaniLLM Evaluation LLM-as-a-Judge
Independent Researcher San Francisco, California, USA · San Francisco State University San Francisco, California, USA
SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouComputational Literary Studies LLM Evaluation
Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center +1
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLLM Evaluation LLM-as-a-Judge
Indeed Inc.
When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels
May 7, 2026Sushant Gautam, Finn Schwall, Annika Willoch Olstad +6LLM Safety Benchmarks LLM Evaluation
1Simula Metropolitan Center for Digital Engineering · 2Oslo Metropolitan University · 4Simula Research Laboratory +2
Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
May 7, 2026Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3Inter-Rater Reliability LLM-as-a-Judge
Carnegie Mellon University · Google Research
Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
May 7, 2026Shihao Weng, Yang Feng, Xiaofei XieLLM-as-a-Judge AI Agent Safety
Nanjing University Nanjing, China · Singapore Management University Singapore
LaTA: A Drop-in, FERPA-Compliant Local-LLM Autograder for Upper-Division STEM Coursework
May 6, 2026Jesse A. RodríguezLLM-as-a-Judge Generative AI in Education
School of Mechanical, Industrial, and Manufacturing Engineering Oregon State University Corvallis, OR 97331, USA
Counterargument for Critical Thinking as Judged by AI and Humans
May 6, 2026Tosin Adewumi, Marcus Liwicki, Foteini Simistira Liwicki +3LLM-as-a-Judge Educational Assessment
Machine Learning Group, EISLAB, Luleå University of Technology, Sweden.

May 20, 2026, cs.CLJ/K move · Enter open · S save

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

Zhenwei Tang, Zhaoyan Liu, Rasa Hosseinzadeh, Tongzi Wu+2

Layer 6 AI · University of Toronto

LLM Evaluation Pairwise Comparison LLM-as-a-Judge Language Model Generation Evaluation Pairwise Preference Evaluation Pairwise Ranking+2

Open paper AskPDF arXiv

As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes. For simpler systems, human evaluation may be practical, but in complicated systems like conversational chatbots, the amount of generated text can overwhelm human annotation resources. Model developers have begun to rely heavily on auto-evaluation, where LLMs are also used to judge generation quality. However, existing LLM-as-a-judge benchmarks largely focus on simple Q&A tasks that do not match the complexity of multi-turn conversations. We introduce RankJudge, a benchmark generator for evaluating LLM-as-a-judge on multi-turn conversations grounded in reference documents. RankJudge creates pairs of conversations where one conversation has a single flaw injected into one turn. This construction allows paired conversations to be labeled unambiguously as better or worse, and precisely isolates failure categories to individual turns, enabling a strict joint correctness criterion for judging. We implement RankJudge across the domains of machine learning, biomedicine, and finance, evaluate 21 frontier LLM judges, and rank those judges via the Bradley-Terry model. Our formulation also allows ranking each conversation pair with difficulty ratings, which we use to dynamically curate the evaluation slice to reduce label noise, as confirmed via human annotation. We find that judge rankings are stable under partial observability, coarser correctness criteria, and an alternative random-walk rating algorithm.

Momentum

Related topics

Latest papers 362