LLM-as-a-Judge

Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

23 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-14

16 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-07

36 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Inside this field

Focused directions

673 papers

Latest in LLM-as-a-Judge

Open your feed →
CardsList

  1. SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

    Sep 17, 2026Chenxi Wu, Zimu Wang, Haiyang Zhang +2HazardLarge Language Model Safety

  2. Benchmarking LLM Compliance with China AI Generated Content Regulations

    Sep 17, 2026Chenrui Cui, Hongye Fang, Lisha Song +3Large Language Model EvaluationCompliance

  3. PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

    Sep 17, 2026Pyrros Koussios, Benjamin Jäger, John Hua Yao +3Reasoning BenchmarkLarge Language Model Reasoning

  4. Reproducibility is not construct validity: LLM measurement of institutionally situated communication

    Sep 17, 2026Veronika Batzdorfer, Carlo Romano Marcello Alessandro SantagiustinaLarge Language Model ReliabilityInter-Rater Variability

  5. Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detection

    Sep 17, 2026Bo Xu, Chenyuan Wang, Xinyu Chen +5Hate SpeechHate

  6. A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

    Sep 17, 2026Shaina Raza, Ahmed Y. Radwan, Imran Liaquat +1Trustworthy Artificial IntelligenceLarge Language Model Evaluation

  7. Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

    Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Outcome-Based RewardsLarge Language Model Evaluation

  8. Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

    Sep 16, 2026Xinshuai Guo, Junjie Wu, Dolly Deng +4Agentic BenchmarksBenchmarking Large Language Models

  9. DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions

    Sep 16, 2026Rem Hida, Masahiro Kaneko, Daisuke Oba +2Large Language Model BiasMulti-Turn Interactions

  10. Symbolic Temporal Supervision of LLM Agents Using Contracts

    Sep 16, 2026Yifeng Xiao, Pierluigi NuzzoLarge Language Model AgentsLinear Temporal Logics

  11. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

    Sep 16, 2026Li ChenBenchmarking Large Language ModelsProgrambench

  12. Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels

    Sep 15, 2026David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey +2JudgesLlm-As-A-Judge

  13. Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models

    Sep 15, 2026Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto +2Large Language Model EvaluationStroke

  14. K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

    Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9Large Language Model SafetyBenchmarking Large Language Models

  15. IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English

    Sep 14, 2026Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truică +1Large Language Model JudgesHumor

  16. Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation

    Sep 14, 2026Sarra Gharsallah, Adele Robaldo, Mariia Tokareva +5Large Language Model EvaluationJudges

  17. Four Ledgers, Not One Score: Responsible Communication of LLM-Judge Calibration in Biomedical ML

    Sep 14, 2026Sidi Chang, Peiying ZhuLlm-As-A-JudgeAlgorithm Auditing

  18. PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

    Sep 14, 2026Miguel Zabaleta, Baihan LinSystematic ReviewArtificial Intelligence Reviewers

  19. Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

    Sep 14, 2026Gemma Zhang, Prachi Badarayani, Asmi Kumar +2Large Language Model JudgesLlm-As-A-Judge

  20. Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluation

    Sep 14, 2026Liang Zhao, Yong Wang, Jiangzhe ChenLlm-As-A-JudgeRaw Judge Outputs

  21. GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

    Sep 14, 2026Umesh Bodhwani, Thanh Tran, Kai WeiAgentic EvaluationsLlm-As-A-Judge

  22. Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models

    Sep 14, 2026Zhongzhan Huang, Junxin Li, Guoming Ling +3Benchmarking Large Language ModelsLarge Language Model Evaluation

  23. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks

    Sep 14, 2026Ali Ansari, Haoran Sun, Andy Zeyi Liu +48Benchmarking Large Language ModelsFrontier Models

  24. A primer on evaluation methods for large language models in healthcare

    Sep 13, 2026Suzannah E McKinney, Phuc Vu, Samuel A Justice +7Large Language Model EvaluationHealthcare

  25. Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

    Sep 12, 2026Koutian Wu, Junjie Zhou, Ergan Shang +4Artificial Intelligence BenchmarksLarge Language Model Evaluation

  26. Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models

    Sep 11, 2026Yu-Chung HsiaoLlm-As-A-JudgeOverconfidence

  27. When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text

    Sep 10, 2026DongHyun Ryu, Jaehyeok Lee, YeongJun Hwang +1Large Language Model BiasImplicit Bias

  28. ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance

    Sep 10, 2026Praphul Singh, Shanu Kumar, Akshat Agarwal +1Kernel ContractsLarge Language Model Judges

  29. GANDR: Claim Auditing for Verifiable Legal Answer Generation

    Sep 9, 2026Chen Qian, Yimeng Wang, Yu Chen +2Legal DomainAlgorithm Auditing

  30. Φ-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them?

    Sep 9, 2026Leilei Ding, Shumin Wang, Yuting Huang +10Benchmarking Large Language ModelsInfrastructure

  31. When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurors

    Sep 9, 2026Cho-Ying WuLlm-As-A-JudgeHuman Judgment

  32. The Audit Decides the Verdict: Instrument Effects Rival Demographic Bias in LLM Decision Audits

    Sep 8, 2026Siddharth Vohra, Manikandan RavikiranLarge Language Model BiasAlgorithm Auditing

  33. A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model

    Sep 8, 2026Veerendra Kumar SunkavalliLlm-As-A-JudgeEstimators

  34. SentryLine: Evidence-Grounded Question Answering over Evolving Documents in Oncology Care

    Sep 8, 2026Tampu Ravi Kumar, Gaurav Najpande, Muhammad Ali Khan +7Clinical ReasoningEvolving

  35. ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

    Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLarge Language Model EvaluationLarge Language Model Calls

  36. Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts

    Sep 8, 2026Yongxi Zhou, Wenbo Ye, Yuanzhe Liu +2Large Language Model SafetyJudges

  37. When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation

    Sep 8, 2026Yiwen Qiu, Linjuan Wu, Dingming Li +7Neural Machine Translation ModelsCultural Knowledge

  38. How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement

    Sep 7, 2026Riyadh Alnasser, Yusuf Mücahit Çetinkaya, Sumin Zhao +1DisagreementResponses

  39. Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach

    Sep 7, 2026Janek Groß, Elena Zentgraf, Jens HeidrichLarge Language Model EvaluationArtificial Intelligence Evaluation

  40. Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

    Sep 3, 2026Haoyaun Zhu, Jie ZhangJudgesReadout

  41. Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning

    Sep 3, 2026Kevin Du, Alexander Hoyle, Laura Ruis +1Reasoning ChainReasoning Traces

  42. FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models

    Sep 3, 2026Yalun Wu, Junfeng Fang, Jiawei Wang +6Safety EvaluationLarge Language Model Evaluation

  43. Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting

    Sep 3, 2026Muneeb Khan, Frederic Kirstein, Terry Ruas +1SessionsDelegation

  44. Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation

    Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinContent ModerationLarge Language Model Evaluation

  45. IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

    Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2Large Language Model SafetyLarge Language Model Evaluation

  46. Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks

    Sep 3, 2026Xiangyu Wang, Jin Wu, Xiaoyu Li +2CreativityLlm-As-A-Judge

  47. Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detection

    Sep 3, 2026Weijie Liu, Running Zhao, Wenhao Yuan +4Code QualityDiscrepancy

  48. To What Extent Do Large Language Models Understand Bangla Idioms?

    Sep 3, 2026Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi +1Multilingual IdiomsMultilingual Large Language Models

  49. SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

    Sep 2, 2026Michael J. BommaritoLarge Language Model EvaluationTerm Frequency-Inverse Document Frequency

  50. A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models

    Sep 2, 2026Yikai Zhao, Saurabh Pandey, Pradeep Kumar MisraLarge Language Model EvaluationModular Agentic Framework

  51. Benchmarking Language Models for Statistical Problem Formulation

    Sep 2, 2026Chen Wang, Junzhe Zhao, Xin Cong +2Benchmarking Large Language ModelsStatistical Inference

  52. Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

    Sep 1, 2026Himil Vasava, Ming JiangLlm-As-A-JudgeSummarization

  53. Can LLMs Discover Scientific Laws in Real and Parallel Worlds?

    Sep 1, 2026Yiming Huang, Ziche Liu, Zhuohang Wu +11Scientific DiscoveryBenchmarking Large Language Models

  54. Does task decomposition improve automatic NLG evaluation?

    Sep 1, 2026Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1Llm-As-A-JudgeHuman Annotators

  55. Post-hoc Alignment of LLM-judges to Human Judgment Distribution

    Sep 1, 2026Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1Soft-LabelLlm-As-A-Judge

  56. VIBE-Bench: Evaluating Personalized Large Language Models When Profiles Don't Mean Preferences

    Sep 1, 2026Yiwen Jiang, Yang Deng, Stephanie Fong +9PersonalizationBenchmarking Large Language Models

  57. RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation

    Sep 1, 2026Zhongru Chen, Yuan Wu, Yi ChangLarge Language Model EvaluationRecommendation

  58. CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

    Sep 1, 2026Siyuan Li, Xinxin Song, Chen Ruinian +5Rubric-Based Reinforcement LearningRubrics

  59. Can Large Language Models Forecast What Researchers Study Next?

    Sep 1, 2026Fenghai Li, Zihan Tang, Haofei Yu +2Benchmarking Large Language Models

  60. DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation

    Sep 1, 2026Haoyuan Shi, Mingtao Chen, Shuo Jiang +12FilmsLlm-As-A-Judge