LLM-as-a-Judge

Follow this topic Ask about this topic

LLM-as-a-Judge studies how language models can assess the quality, correctness, safety, or usefulness of other models’ outputs, from chatbot responses to agent trajectories and professional work. Common approaches use explicit rubrics or reference answers, pairwise and listwise preference rankings, multi-judge panels, and trained or calibrated reward models. These methods are tested against human ratings on benchmarks for instruction following, factuality, safety, and domain-specific tasks, with research also exploring uncertainty estimates, abstention, and targeted human review. A central challenge is that consistency or agreement among judges does not guarantee alignment with people: shared biases, prompt sensitivity, weak evidence grounding, and adversarial manipulation can make scores unreliable or misleading.

Latest papers

559 papers93 papers in the last four weeks, up 16% on the four weeks before. 0.5% of all new papers.

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
Mar 26, 2026Hieu Xuan Le, Benjamin Goh, Quy Anh TangPrompt Injection Detection LLM Guardrails
GovTech, Singapore
RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Following
Mar 26, 2026Tianjun Pan, Xuan Lin, Wenyan Yang +7Rubric-Based Evaluation LLM-as-a-Judge
College of Computer Science and Artificial Intelligence, Fudan University · Ant Group · Donghua University
LLMs Do Not Grade Essays Like Humans
Mar 24, 2026Jerin George Mathew, Sumayya Taher, Anindita Kundu +1Automated Essay Scoring LLM-as-a-Judge
University of Alberta, 116 St & 85 Ave, Edmonton, AB, T6G 2R3, Canada
LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks
Mar 24, 2026Abhishek Chandwani, Ishan GuptaLong-Horizon Agent Evaluation AI Agent Benchmarks
Metaphi Inc. USA
Evaluating the Reliability and Fidelity of Automated Judgment Systems of Large Language Models
Mar 23, 2026Tom Biskupski, Stephan KleberLLM-as-a-Judge Automated Evaluation
Hochschule der Medien Stuttgart, Germany · Ravensburg-Weingarten University of Applied Sciences Weingarten, Germany
AdaRubric: Task-Adaptive Rubrics for Reliable LLM Agent Evaluation and Reward Learning
Mar 22, 2026Liang DingLLM Agent Evaluation Rubric-Based Evaluation
The University of Sydney
Permutation-Consensus Listwise Judging for Robust Factuality Evaluation
Mar 20, 2026Tianyi Huang, Nathan Huang, Justin Tang +2LLM-as-a-Judge Bias in LLM-as-a-Judge
Ryquo · App-In Club · Carnegie Mellon University
Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
Mar 20, 2026May Lynn Reese, Markela Zeneli, Mindy Ng +3LLM Safety LLM Safety Evaluation
Apart Research · Odyssean Institute · London School of Economics and Political Science +1
Beyond String Matching: Semantic Evaluation of PDF Table Extraction
Mar 19, 2026Pius Horn, Janis KeuperDocument Parsing Benchmarks Document Information Extraction
Institute for Machine Learning and Analytics (IMLA), Offenburg University, Offenburg, Germany · University of Mannheim, Mannheim, Germany
Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition
Mar 19, 2026Ivaxi Sheth, Zeno Jonke, Amin Mantrach +1Cross-Lingual Transfer Cross-Cultural Language Model Evaluation
CISPA Helmholtz Center for Information Security · Amazon
AutoScreen-FW: An LLM-based Framework for Resume Screening
Mar 19, 2026Zhelin Xu, Shuhei Yamamoto, Atsuyuki MorishimaLLM-as-a-Judge
Institute of Library, Information and Media Science University of Tsukuba Tsukuba, Ibraki, Japan
Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges
Mar 17, 2026Maxim Khomiakov, Jes FrellsenLanguage Model Calibration LLM-as-a-Judge
Technical University of Denmark, Kgs. Lyngby, Denmark · Normal Computing Corporation, New York, NY, USA · Pioneer Centre for Artificial Intelligence, Copenhagen, Denmark
Agentic AI for Human Resources: LLM-Driven Candidate Assessment
Mar 17, 2026Kamer Ali Yuksel, Abdul Basit Anees, Ashraf Elneima +3Learning to Rank LLM-as-a-Judge
aiXplain, Inc., San Jose, CA, USA
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
Mar 17, 2026Yasi Zhang, Tianyu Chen, Mingyuan Zhou +3RL for Language Models Reinforcement Learning
University of California, Los Angeles · The University of Texas at Austin · Google Research
Mediocrity is the key for LLM as a Judge Anchor Selection
Mar 17, 2026Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen +1LLM-as-a-Judge Pairwise Preference Evaluation
1The Hebrew University of Jerusalem · 2IBM Research · 3MIT +1
LLM-as-a-judge validity is strongly task-dependent across physics assessment formats
Mar 16, 2026Will Yeadon, Tom Hardy, Paul Mackay +1LLM-as-a-Judge Educational Assessment
Department of Physics, Durham University, Durham DH1 3LE, UK
QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models
Mar 14, 2026Yao Wu, Kangping Yin, Liang Dong +13LLM Evaluation Clinical Language Model Evaluation
Quark Medical Team, Alibaba Group
When LLM Judge Scores Look Good but Best-of-N Decisions Fail
Mar 12, 2026Eddie LandesbergLLM Evaluation LLM-as-a-Judge
Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
Mar 12, 2026Yixin Liu, Yue Yu, DiJia Su +7LLM-as-a-Judge LLM Alignment
Meta Superintelligence Labs · Yale Univeristy
Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
Mar 11, 2026Mingyang Song, Mao Zheng, Chenning XuLLM-as-a-Judge Rubric-Based Evaluation
Large Language Model Department, Tencent, China
End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering
Mar 11, 2026Nhi Dang, Tung Le, Huy Tien NguyenAutomated Evaluation RAG Evaluation
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam; Vietnam National University, Ho Chi Minh City, Vietnam
PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses
Mar 11, 2026Minki Hong, Eunsoo Lee, Sohyun Park +1LLM Prompting LLM Evaluation
Department of Computer Science and Artificial Intelligence, Dongguk University, South Korea
Criterion Validity of LLM-as-Judge for Business Outcomes in Conversational Commerce
Mar 11, 2026Liang Chen, Qi Liu, Wenhuan Lin +1Rubric-Based Evaluation Multi-Turn Dialogue Evaluation
Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
Mar 11, 2026Tim Schopf, Michael FärberAutomated Evaluation LLM-as-a-Judge
TU Dresden & ScaDS.AI Dresden/Leipzig, Germany · National Institute of Informatics, Tokyo, Japan
MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
Mar 11, 2026Chihiro Watanabe, Jingyu SunLLM-as-a-Judge Clustering
NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan
ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models
Mar 10, 2026Harry Owiredu-AshleyLanguage Model Safety Evaluation Adversarial Attacks on LLMs
Independent Researcher, New Jersey, United States
How Trustworthy Are LLM-as-Judge Ratings for Interpretive Responses? Implications for Qualitative Research Workflows
Mar 9, 2026Songhee Han, Jueun Shin, Jiyoon Han +2LLM-as-a-Judge LLM Evaluation
Florida State University, Tallahassee, FL 32306, USA
Evaluating LLM-Based Grant Proposal Review via Structured Perturbations
Mar 9, 2026William Thorne, Joseph James, Yang Wang +2Automated Peer Review Automated Evaluation
University of Sheffield, UK · University of Manchester, UK
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
Mar 7, 2026Karen Zhou, Chenhao TanAutomated Evaluation Rubric-Based Evaluation
University of Chicago

Mar 26, 2026J/K move · Enter open · S save

Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models

Hieu Xuan Le, Benjamin Goh, Quy Anh Tang

GovTech, Singapore

Prompt Injection Detection LLM Guardrails LLM Security LLM-as-a-Judge

Open paper AskPDF arXiv

Prompt attacks, including jailbreaks and prompt injections, pose a critical security risk to Large Language Model (LLM) systems. In production, guardrails must mitigate these attacks under strict low-latency constraints, resulting in a deployment gap in which lightweight classifiers and rule-based systems struggle to generalize under distribution shift, while high-capacity LLM-based judges remain too slow or costly for live enforcement. In this work, we examine whether lightweight, general-purpose LLMs can reliably serve as security judges under real-world production constraints. Through careful prompt and output design, lightweight LLMs are guided through a structured reasoning process involving explicit intent decomposition, safety-signal verification, harm assessment, and self-reflection. We evaluate our method on a curated dataset combining benign queries from real-world chatbots with adversarial prompts generated via automated red teaming (ART), covering diverse and evolving patterns. Our results show that general-purpose LLMs, such as gemini-2.0-flash-lite-001, can serve as effective low-latency judges for live guardrails. This configuration is currently deployed in production as a centralized guardrail service for public service chatbots in Singapore. We additionally evaluate a Mixture-of-Models (MoM) setting to assess whether aggregating multiple LLM judges improves prompt-attack detection performance relative to single-model judges, with only modest gains observed.