LLM-as-a-Judge

Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

23 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-14

16 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-07

36 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Inside this field

Focused directions

679 papers

Latest in LLM-as-a-Judge

Browse all topics →
CardsList

  1. Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs

    Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLarge Language Model EvaluationPairwise Preferences

  2. LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline

    Jun 16, 2026Xiwei Xu, Chen Wang, Jacky Jiang +5Llm-As-A-JudgeRubrics

  3. AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows

    Jun 16, 2026Jiahui Niu, Huizi Yu, Wenkong Wang +11Large Language Model EvaluationReal-World Clinical Workflows

  4. LLM-Based Visual Explanation Evaluation Framework for Assessing the Explainability of Facial Skin Disease Classification Models

    Jun 15, 2026Gyuyeon NaSkin Lesion ClassificationExplainability

  5. Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives

    Jun 15, 2026Luca ZhouLarge Language Model EvaluationLarge Language Model Safety

  6. Evaluating LLM Personalization via Semantic Constraint Verification

    Jun 15, 2026Xuran Li, Guanqin Zhang, Imran Razzak +4Large Language Model EvaluationNatural Language Inference

  7. UXBench: Measuring the Actionability of LLM-Generated UX Critiques

    Jun 15, 2026Wenjie Wang, Yue Huang, Zipeng Ling +11UsabilityCritique

  8. TimeVista: Exploring and Exploiting Vision-Language Models as Judges for Time Series Forecasting

    Jun 15, 2026Zhi Chen, Yuxuan Wang, Jialong Wu +5Multivariate ForecastingTime Series

  9. SkillVetBench: LLM-as-Judge for Multi-Dimensional Security Risk Evaluation in Open-Source LLM Agent Skills

    Jun 14, 2026Ismail Hossain, Sai Puppala, Md Jahangir Alam +2Security AnalysisLarge Language Model Safety

  10. LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation

    Jun 14, 2026Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi +1Llm-As-A-JudgeLarge Language Model Judges

  11. On the Adversarial Robustness of Multimodal LLM Judges

    Jun 14, 2026Zihan Wang, Guansong Pang, Zelin Liu +3Large Language Model JudgesLlm-As-A-Judge

  12. Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

    Jun 13, 2026Yitao LiLarge Language Model EvaluationJudges

  13. Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

    Jun 13, 2026Zaifu Zhan, Shuang Zhou, Rui ZhangMulti-Agent ReasoningLarge Language Model Agents

  14. TriAdReview: Triangular Adversarial Review Architecture for Multi-Model Technical Document Generation

    Jun 13, 2026Zhiqiang Zhou, Junliang Dai, Xu LingArtificial Intelligence ReviewersAdversarial Evaluation

  15. Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

    Jun 12, 2026Alyssa Unell, Natalie Dullerud, Naomi Boneh +4Large Language Model JudgesJudges

  16. Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment

    Jun 12, 2026Kirill Vasilevski, Ximing Dong, Benjamin Rombaut +8Code QualityLlm-As-A-Judge

  17. Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity

    Jun 12, 2026Daniel Lee, Harsh Sharma, Eunkyu Park +5Llm-As-A-JudgeCultural Knowledge

  18. Combining Retrieval-Augmented Text Generation with LLMs for Reading Content Recommendations

    Jun 12, 2026Sooyeon Kim, Piotr S. MaciągRetrieval-Augmented Generation SystemsReading

  19. Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

    Jun 12, 2026Shaojie YinJudgesComparative Judgment

  20. MA-ProofBench: A Two-Tiered Evaluation of LLMs for Theorem Proving in Mathematical Analysis

    Jun 11, 2026Lushi Pu, Weiming Zhang, Xinheng Xie +6TheoremMathematics

  21. SupraBench: A Benchmark for Supramolecular Chemistry

    Jun 11, 2026Tianyi Ma, Yijun Ma, Zehong Wang +6Benchmarking Large Language Models

  22. A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget

    Jun 11, 2026Joe DwyerLarge ModelsSmall Language Models

  23. VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation

    Jun 11, 2026Yijun Shen, Minghao Shao, Yichen Zhao +4VhdlHardware Description Languages

  24. From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation

    Jun 11, 2026Bora Kargi, David SalinasLarge Language Model EvaluationJudges

  25. MÖVE: A Holistic LLM Benchmark for the German Public Sector

    Jun 11, 2026Camilla Dalerci, Thilo Michael, Robin Schaefer +1Benchmarking Large Language ModelsGerman

  26. SkillChain: Closing the Loop on Skill Evolution for Image-Based E-Commerce AI Assistants

    Jun 11, 2026Yimin Hu, Mengtao Xu, Hao Guo +3Skill EvolutionArtificial Intelligence Assistants

  27. Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curation

    Jun 11, 2026En-Ming Huang, Yu-Hung Kao, Ren-Hao Deng +10Hardware Description LanguagesFlexible Register Transfer Level Design Automation

  28. Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming

    Jun 11, 2026Tingqiang Xu, Hangrui Zhou, Tianle Cai +2Raw Judge OutputsLlm-As-A-Judge

  29. Localizing Anchoring Pathways in Language Models

    Jun 11, 2026Hillary N. Owusu, Sarah Wiegreffe, Naomi H. FeldmanInstruction-Tuned ModelsLarge Language Model Judges

  30. Measuring Epistemic Resilience of LLMs Under Misleading Medical Context

    Jun 10, 2026Hongjian Zhou, Xinyu Zou, Jinge Wu +19Large Language Model EvaluationDiagnostic Benchmark

  31. Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation

    Jun 10, 2026Selen Erkan, Bastian Boll, Kristian Kersting +2Prompt TuningBenchmarking Large Language Models

  32. On the Limits of LLM-as-Judge for Scientific Novelty Assessment

    Jun 10, 2026Soumitra Sinhahajari, Navonil Majumder, Soujanya PoriaIdeasLlm-As-A-Judge

  33. Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

    Jun 10, 2026Min Sen Tan, Zachary Kit Chun Choy, Syed Ali Redha Alsagoff +4CreativityLarge Language Model Evaluation

  34. Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Models

    Jun 9, 2026Malikeh Ehghaghi, Boglárka Ecsedi, Marsha Chechik +1Large Language Model EvaluationComputational Cost

  35. Flaws in the LLM Automation Narrative

    Jun 9, 2026George Perrett, Javae Elliott, Jennifer Hill +1Benchmarking Large Language ModelsNarratives

  36. The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring

    Jun 9, 2026Ali Keramati, Mark WarschauerAutomated Essay ScoringLarge Language Model Evaluation

  37. Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents

    Jun 9, 2026Sawyer Zhang, Alexander Wang, Sophie LeiLlm-As-A-JudgeBlind Spots

  38. Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

    Jun 9, 2026Ali Keramati, Justin Cheok, Jacob Horne +1Multi-Agent Large Language Model SystemsLlm-As-A-Judge

  39. The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge

    Jun 9, 2026Ali Keramati, Justin Cheok, Jacob Horne +1Multi-Agent DebateDebate

  40. RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning

    Jun 8, 2026Yiteng Mao, Kenan Xu, Yijia Lyu +3Mathematical ReasoningLarge Language Model Evaluation

  41. TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics

    Jun 8, 2026QuocViet Pham, Elvir Karimov, Andrey Galichin +1TheoremReasoning Benchmark

  42. Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges

    Jun 8, 2026Yongtaek Lim, Hyeji Choi, Minwoo KimSafety EvaluationRubric Framework

  43. RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

    Jun 8, 2026Jing Wang, Shang Liu, Wenji Fang +3Register Transfer LevelBenchmarking Large Language Models

  44. Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework

    Jun 7, 2026Aman Gupta, Kevin Rossell, Edesio Alcobaça +8Agentic EvaluationsArtificial Intelligence Agents

  45. Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

    Jun 7, 2026Sayed Erfan ArefinCode GenerationLlm-As-A-Judge

  46. From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

    Jun 7, 2026Hao Chen, Ziyu Han, Yukun Yan +3RubricsLarge Language Model Evaluation

  47. ForesightSafety-SAGE:A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents

    Jun 7, 2026Lu Jia, Haibo Tong, Feifei Zhao +5Large Language Model SafetyLarge Language Model Agents

  48. Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

    Jun 6, 2026Hyogon Ryu, Jeonghwan Kim, Yewon Lim +3Agentic EvaluationsLlm-As-A-Judge

  49. When Languages Disagree: Self-Evolving Multilingual LLM Judges

    Jun 6, 2026Xiyan Fu, Wei LuCross-Lingual ConsistencyLarge Language Model Judges

  50. SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models

    Jun 6, 2026Ayah Al-Naji, Edoardo Fazzari, Saif Alkindi +3Gradient SurgeryBenchmarking Large Language Models

  51. When Behavioral Safety Evaluation Fails: A Representation-Level Perspective

    Jun 6, 2026Enyi Jiang, Anders Gjølbye, Yibo Jacky Zhang +1Large Language Model SafetyLarge Language Model Evaluation

  52. Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators

    Jun 5, 2026Anissa Alloula, Federico Licini, Ava Batchkala +1Llm-As-A-JudgeJudges

  53. Cherry-pick Override: Unsafe Directional Commitment in LLM Judges under Mixed Evidence

    Jun 5, 2026Haoran XuJudgesLarge Language Model Judges

  54. SLMJury: Can Small Language Models Judge as Well as Large Ones?

    Jun 5, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLarge Language Model JudgesSmall Language Models

  55. Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese

    Jun 5, 2026Xing Yue, Yongliang Shen, Weiming LuPhonological UnderstandingPhonetic Representations

  56. OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

    Jun 5, 2026Xinyi Li, Zhen Fang, Yongxin Deng +12Hallucination DetectionLarge Language Model Evaluation

  57. MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring

    Jun 4, 2026Ali Keramati, Shiyuan Zhou, Sharad Mehrotra +1Automated Essay ScoringLlm-As-A-Judge

  58. Evaluating LLM Usage for Efficient and Explainable Numerical and Classified Implicit Sentiment Analysis of Product Desirability

    Jun 4, 2026Sherri Weitl-Harms, John HastingsSentiment AnalysisLarge Language Model Evaluation

  59. A Framework for Measuring Appropriate Reliance on Set-Valued AI Advice

    Jun 4, 2026Ranjan Mishra, Jakob SchoefferArtificial Intelligence EvaluationHuman-Ai Collaboration

  60. Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

    Jun 4, 2026Tao Liu, Ye Lu, Ruohua Zhang +4Large Language Model EvaluationPedagogy