LLM-as-a-Judge

Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

23 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-14

16 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-07

36 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Inside this field

Focused directions

679 papers

Latest in LLM-as-a-Judge

Browse all topics →
CardsList

  1. Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity

    Jul 1, 2026Brett ReynoldsArtificial Intelligence SafetyLarge Language Model Evaluation

  2. Clinician-Level Agreement Without Clinical Caution: LLM Evaluator Limits in Medical AI Benchmarking

    Jul 1, 2026William Philipp, Finn Fassbender, Thorsten Langer +11Large Language Model EvaluationClinical Decision Support

  3. MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

    Jul 1, 2026Maximilian Idahl, Jörg Tiedemann, Sampo Pyysalo +19Multilingual Large Language ModelsMultilingual Language Models

  4. Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent Conditions

    Jul 1, 2026Zewen LiuLarge Language Model EvaluationSpurious Correlations

  5. Testing Frontier Large Language Models' Physics Literacy in Parallel Physical Worlds

    Jun 30, 2026Dong ZhangLlm-As-A-JudgePhysics Benchmarks

  6. Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

    Jun 30, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +2Arabic Natural Language ProcessingArabic

  7. Benchmarking Large Language Models on Floating-Point Error Classification

    Jun 30, 2026Lisa Taldir, Muhammad Ahmad Saeed, David Defour +2Benchmarking Large Language ModelsFloating-Point

  8. Probing Stylistic Appropriation using Large Language Models: An Evaluation Framework for Copyright Infringement under EU Law

    Jun 30, 2026Noah Scharrenberg, Chang SunStylistic FidelityRights

  9. RoPoLL: Robust Panel of LLM Judges

    Jun 29, 2026Anish Acharya, Kris W Pan, Brian VerkhovskyLlm-As-A-JudgeLarge Language Model Judges

  10. Training Therapeutic Judges and Multi-Agent Systems for Human-Aligned Mental Health Support

    Jun 29, 2026Mizanur Rahman, Abeer Badawi, Elahe Rahimi +4Mental Health SupportTreatment

  11. Poller: Are LLMs Suitable for Evaluating the Poetry Understanding Task?

    Jun 29, 2026Shanshan Wang, Derek F. Wong, Jingming Yao +1PoetryLarge Language Model Evaluation

  12. TACO: Tool-Augmented Credit Optimization for Agentic Tool Use

    Jun 29, 2026Mingkuan Feng, Jinyang Wu, Hao Gu +5Multi-Turn Tool-Use AgentAgentic Tool Use

  13. EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

    Jun 29, 2026Buğra Alperen Uluırmak, Rifat KurbanSafety EvaluationLarge Language Model Evaluation

  14. Open Problems in Constitutional Preference Reconstruction

    Jun 29, 2026Eleanor Clifford, Michael Amir, Arduin Findeis +2Large Language Model JudgesLarge Language Model Decision-Making

  15. Are We Measuring Strategy or Phrasing? The Gap Between Surface- and Approach-Level Diversity in LLM Math Reasoning

    Jun 29, 2026Sangmook Lee, Minbeom Kim, Jeonghye Kim +3Mathematical ReasoningDiversity

  16. Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

    Jun 29, 2026Yangda Peng, Yunjia Qi, Haotian Xia +8Llm-As-A-JudgeRubrics

  17. SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

    Jun 28, 2026Tiziano Santilli, Francesco Daghero, Mayhar Tourchi MoghaddamBenchmarking Large Language ModelsMultiple-Choice Questions

  18. Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models

    Jun 28, 2026Archit ManekLarge Language Model EvaluationModel Activations

  19. Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

    Jun 27, 2026Balázs Szalontai, Ábel Szauter, Balázs Márton +3BugBenchmarking Large Language Models

  20. Prompt Injection in Automated Résumé Screening with Large Language Models: Single and Multi-Injection Settings

    Jun 25, 2026Preet Baxi, Jiannan Xu, Jane Yi Jiang +1Indirect Prompt InjectionInjection

  21. Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

    Jun 25, 2026Sangwoo Cho, Kushal Chawla, Pengshan Cai +4Large Language Model EvaluationEvaluation Harnesses

  22. Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning

    Jun 25, 2026Stanisław Sójka, Felix Steffek, Matthias GrabmairPredictive LawAdjudication

  23. Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models

    Jun 25, 2026Abrar Alotaibi, Moataz AhmedLanguage Model Evasion AttacksAdversarial Evaluation

  24. Listening Like a Judge: A Music-Aware Framework for Automatic Singing Performance Evaluation

    Jun 24, 2026Neelam Saini, Sourav GhoshSinging Voice ConversionMusic Understanding

  25. DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

    Jun 24, 2026Aaron J. Li, Hao Huang, Youngmin Park +6Large Language Model EvaluationEvaluation Harnesses

  26. Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

    Jun 24, 2026Kylie AnglinConfidence IntervalsSample Size

  27. Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

    Jun 24, 2026Davie ChenDiagramsScientific Discovery

  28. OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning

    Jun 24, 2026Wenxuan Jiang, Zining Fan, Zijian Zhang +6Large Language Model ReasoningLlm-As-A-Judge

  29. RAS: Measuring LLM Safety Through Refusal Alignment

    Jun 24, 2026Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu +1Large Language Model SafetyLarge Language Model Evaluation

  30. Evaluating LLMs on Real-World Software Performance Optimization

    Jun 24, 2026Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim +1Code OptimizationBenchmarking Large Language Models

  31. How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

    Jun 24, 2026Yang GaoLlm-As-A-JudgeLarge Language Model Jailbreaks

  32. C3-Bench: A Context-Aware Change Captioning Benchmark

    Jun 24, 2026Jae-Woo Kim, Hyeongbeom Kim, Ue-Hwan KimRemote Sensing Image Change CaptioningVideo Captioning

  33. LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges

    Jun 23, 2026Thi Huyen Nguyen, Zahra AhmadiPeer ReviewArtificial Intelligence Reviewers

  34. ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

    Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Paralinguistic CuesLarge Audio Language Models

  35. On the Stability of Prompt Ranking in Large Language Model Evaluation

    Jun 23, 2026Shaoshuai Du, Penghao Liang, Yixian Shen +3Benchmarking Large Language ModelsLarge Language Model Evaluation

  36. RASC+: Retrieval-Constrained LLM Adjudication for Clinical Value Set Authoring

    Jun 22, 2026Sumit MukherjeeLarge Language Model JudgesLarge Language Model Evaluation

  37. Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

    Jun 22, 2026Tianyu Ding, Aditya Nannapaneni, Juan Pablo De la Cruz WeinsteinLarge Language Model EvaluationInformation Retrieval Metrics

  38. GIF: Locally Sound Geometric Information Flow Control for LLMs

    Jun 22, 2026Adam Storek, Nikolaus Holzer, Zhuo Zhang +1Data LeakageUntrusted Input

  39. Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation

    Jun 22, 2026Yu Cao, Ziquan Liu, Zhensong Zhang +3Large-Scale Video Generation ModelsJudges

  40. MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration

    Jun 22, 2026Devleena Das, Rajeev Patwari, Vikram Kumar Bukka +3Large Language Model EvaluationMonte Carlo

  41. GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

    Jun 22, 2026Jeffrey FlyntAgentic EvaluationsLlm-As-A-Judge

  42. BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

    Jun 21, 2026Shreyas KCLlm-As-A-Judge

  43. Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study

    Jun 20, 2026Tomoki KoriyamaBenchmarking Large Language ModelsZero-Shot Text-To-Speech

  44. ForEx: A Formal Verification Framework for Explainable Reasoning in Logical Fallacy Detection and Annotation

    Jun 20, 2026Pei-Cing Huang, Chienyu Liu, Chan Hsu +3Large Language Model EvaluationFormal Verification

  45. Counsel: A Meta-Evaluation Dataset for Agentic Tasks

    Jun 19, 2026Sashank Pisupati, Henry Broomfield, Eujeong Choi +5Agentic BenchmarksLlm-As-A-Judge

  46. Evaluation of Small Language Models for Arabic Language Processing

    Jun 19, 2026Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi +5Arabic Natural Language ProcessingSmall Language Models

  47. Evaluating LLMs for Real-World Web Vulnerability Detection

    Jun 19, 2026Sebastian Neef, Luca Jungnickel, Antonio Benjamin Buchholz +2Vulnerability Detection & RepairLarge Language Model Evaluation

  48. The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

    Jun 19, 2026David NordforsBenchmarking Large Language ModelsAnalogy

  49. PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality

    Jun 18, 2026Zeyuan Chen, Ziqing Yang, Yihan Ma +2Peer ReviewLarge Language Model Evaluation

  50. CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

    Jun 18, 2026Zhijian Zhou, Zesheng Ye, Zhaorun Chen +2Large Language Model EvaluationCertification

  51. Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

    Jun 18, 2026Maria Ivanova, Pavel Zadorozhny, Rodion Levichev +5Benchmarking Large Language ModelsCode Generation

  52. Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

    Jun 18, 2026Ali Asaria, Tony Salomone, Deep Gandhi3D GenerationLlm-As-A-Judge

  53. QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

    Jun 18, 2026Xinyi Zheng, Ling Shi, Tianlong Yu +3Reasoning BenchmarkLarge Language Model Reasoning

  54. AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

    Jun 18, 2026Zilong Zhang, Yi-Ting Hung, Weiyi He +3Llm-As-A-JudgeEvaluation Benchmarks

  55. Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

    Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesLlm-As-A-JudgeJudges

  56. Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning

    Jun 17, 2026Zilong Zhang, Yi-Ting Hung, Lei Ding +1Large Language Model EvaluationPositive--Unlabeled Learning

  57. SciRisk-Bench: A Risk-Dimension-Aware Benchmark for AI4Science Safety

    Jun 17, 2026Linghao Feng, Yinqian Sun, Dongqi Liang +8Large Language Model SafetyBenchmarking Large Language Models

  58. Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports

    Jun 17, 2026Qingyu Lu, Ruochen Li, Liang Ding +3Radiology Report GenerationLarge Language Model Evaluation

  59. A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)

    Jun 16, 2026Ali Asaria, Tony Salomone, Deep GandhiMeshes3D Object Understanding

  60. LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI

    Jun 16, 2026Lalit Yadav, Akshaj GurugubelliLegal DomainAlgorithm Auditing