LLM-as-a-Judge

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

23 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-14

16 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-07

36 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Inside this field

Focused directions

Focused topic

Large Language Model Evaluation

Within LLM-as-a-Judge

244 papersExplore

Focused topic

Llm-As-A-Judge

Within LLM-as-a-Judge

173 papersExplore

Focused topic

Benchmarking Large Language Models

Within LLM-as-a-Judge

154 papersExplore

Focused topic

Large Language Model Judges

Within LLM-as-a-Judge

104 papersExplore

Focused topic

Judges

Within LLM-as-a-Judge

81 papersExplore

676 papers

Latest in LLM-as-a-Judge

Browse all topics →

DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation
Sep 1, 2026Haoyuan Shi, Mingtao Chen, Shuo Jiang +12Films Llm-As-A-Judge
Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?
Aug 31, 2026Qiaoyuan Zheng, Yiqu YangItem Response Theory Leaderboard
SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents
Aug 31, 2026Rayan Khoury, Shih-Yao Lin, Pratyush MishraConversational Artificial Intelligence Large Language Model Judges
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
Aug 31, 2026Adrians Skapars, Edoardo ManinoAlgorithm Auditing Large Language Model Evaluation
LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
Aug 31, 2026Sebastian Fox, Luke Markham, Ryan Lail +1Large Language Model Judges False Positive Rate
ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents
Aug 31, 2026Wei Chen, Peilun Zhou, Zhaoyu Hu +8Agentic Evaluations Atlas
Commit-first LLM judging inherits the judge's own errors
Aug 31, 2026Idil GozelJudges Large Language Model Judges
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
Aug 31, 2026Minsoo Song, Chanwoo Kim, Sugyeong Eo +1Multi-Agent Debate Large Language Model Judges
Small Language Models as Judges for Rubric-Based Reinforcement Learning
Aug 30, 2026Fengyu Xie, Yilun Zhao, Bingsen Chen +2Rubric-Based Reinforcement Learning Large Language Model Judges
GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation
Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4Rubrics Large Language Model Evaluation
A Survey on Rubric-Guided Reinforcement Learning for Language Models
Aug 27, 2026Zifei Shan, Fangning ShaoRubric-Based Reinforcement Learning Rubrics
Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
Aug 26, 2026Jia-Hao Ji, Sijie Li, Jiabei Cheng +3Large Language Model Judges Multi-Agent Reasoning
Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight
Aug 25, 2026Anupam Purwar, Shashank Singh, Kritika SrivastavaLlm-As-A-Judge Large Language Model Judges
K-Bench: measuring model performance on real scientific agent requests
Aug 21, 2026Aubrey M. Brueckner, Darshil Patel, Yuhuan He +1Llm-As-A-Judge Artificial Intelligence Research
Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment
Aug 21, 2026Emma Granqvist, Rocío Mercado, Samuel GenhedenLlm-As-A-Judge Llm-Based Agent
FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models
Aug 20, 2026Dingzirui Wang, Xuanliang Zhang, Keyan Xu +2Benchmarking Large Language Models Formalization
The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations
Aug 18, 2026Emma Yanyang Kong, JJ Tan, Ishan Gupta +8Llm-As-A-Judge Large Language Model Decision-Making
Beyond Local Accuracy: A Protocol-Level Identifiability Audit for Controlled LLM Reasoning Evaluation
Aug 13, 2026Junhao Luo, Ning Huang, Ziqi Sha +2Large Language Model Evaluation Algorithm Auditing
When Should Multi-Round RAG Stop? Structured Stopping Judgments and Retrieval Reduction in Search-R1
Aug 13, 2026Weimeng LuoJudges Frozen Models
Which LLM Is Your Ideal Companion? Evaluating Emotional Companion Capabilities of LLMs Based on Adult Attachment Theory
Aug 13, 2026Junkai Zhou, Shiting Guan, Zhaoyi ZhangLarge Language Model Evaluation Emotional Support
Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
Aug 12, 2026Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar +2Judges Epistemic Actions
Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces
Aug 12, 2026Congchao Wang, Diwakar Singh, Qiaozi Gao +3Reasoning Traces Judges
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLarge Language Model Evaluation Reasoning Benchmark
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Aug 12, 2026Xi Chen, Jie Mu, Mo Xuan +1Rubrics Large Language Model Judges
RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks
Aug 12, 2026Jinjun Huang, Zhongzhen Wen, Tongtong Xu +3Graphics Processing Unit Kernels Benchmarking Large Language Models
LODESTAR: Trustworthy Entropy Is Navigated, Not Merely Measured -- Reinforced Polarizer Keeps a Frozen LLM from Being Confidently Misled by the Wrong Evidence
Aug 12, 2026Po-Jen Ko, Che-Cheng Wu, Hung-Chun Hsu +2Natural Language Inference Llm-As-A-Judge
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Aug 12, 2026Shailja Thakur, Sungeun An, Chad DeLuca +1Benchmarking Large Language Models
From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation
Aug 11, 2026Alireza S. Ziabari, Kat Ellis, Colleen Chan +1Behavioral Alignment Large Language Model Evaluation
Benchmarking LLM Judges for Mobile Agent Evaluation
Aug 11, 2026Ziqiang Wan, Li Gu, Zhixiang Chi +4Agentic Benchmarks Llm-As-A-Judge
Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases
Aug 11, 2026Davood Wadi, Mohsen Ghodrat, Matthew PhilpLarge Language Model Bias Large Language Model Evaluation
Do Judges Behave Like Algorithms?
Aug 11, 2026Riya Manchanda, Eric Chen, Chloe Zhu +3Judges
Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
Aug 11, 2026Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie +1Large Language Model Evaluation Information Retrieval Metrics
Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes
Aug 10, 2026Alec Harris, Kasey Corra, Archie Chaudhury +1Large Language Model Post-Training Large Language Model Evaluation
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Aug 10, 2026Laurens Samson, Iva Gornishka, Gossa Lô +2Large Language Model Evaluation Large Language
Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness
Aug 10, 2026Tadanobu Chuyo Kamijo, Ori Rottenstreich, Javier Conde +2Large Language Model Safety Large Language Model Evaluation
How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans
Aug 10, 2026Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh +2Mean Opinion Scores Large Language Model Judges
TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability
Aug 10, 2026Vincent Cohen-Addad, Dimitris Paparas, Ernest van Wijland +13Benchmarking Large Language Models Reasoning Benchmark
RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement
Aug 10, 2026Ziheng Jia, Jiaying Qian, Zicheng Zhang +3Llm-As-A-Judge Rubrics
Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities
Aug 9, 2026Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2Diagrams Diagnostic Benchmark
Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure
Aug 9, 2026Víctor GallegoGraphics Processing Unit Kernels Code Optimization
PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling
Aug 9, 2026Dongjie Xu, Julius, Hanchi Dong +6Large Language Model Evaluation Adversarial Evaluation
HoloAegis: Frozen Representation, Topological Inference --- Minimally Parametric Safety Manifolds and Their Capability Boundaries for LLM Guardrails
Aug 9, 2026Tak Ho Alex Li, Kaijie Liu, Lik-Hang Lee +3Large Language Model Safety Large Language Model Hallucination
CAP: A Scalable Benchmark for Evaluating Cross-Site Browser Agents with Complex Actions and Perception
Aug 9, 2026Zejun Xu, Taiyi Chen, Jin Li +13Browser Agent Perceptual Understanding
Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification
Aug 8, 2026Yang ShuLarge Language Model Judges Human Judgment
When the Judge Should Not Decide: Evidence-Locked, Non-Compensatory Selection Bounds LLM-Judge Failure in Reasoning Pipelines
Aug 7, 2026Yiyao Zhang, Diksha Goel, Hussain Ahmad +2Judges Llm-As-A-Judge
Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools
Aug 7, 2026Afreen Alam, Evgenija Popchanovska, Ana Gjorgjevikj +4Artificial Intelligence Safety Large Language Model Safety
Methodologies for Improving the Quality of AI Tutoring in K-12 Education
Aug 7, 2026Tushar Udeshi, Anna Khazenzon, Kabir Khan +5Tutors Classrooms
Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models
Aug 7, 2026Rens Anderson, Tessa Verhoef, Amirhossein ZohrehvandCreativity Large Language Model Evaluation
Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps
Aug 7, 2026Yang ShuLarge Language Model Judges Equal Error Rate
Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation
Aug 7, 2026Kevin Miller, Arjun Chandra, Venkatesh SaligramaParalinguistic Cues Large Audio Language Models
Divergent Response Modes in Frontier Language Models Under Steering Pressure
Aug 6, 2026Ali Jalal-KamaliFrontier Models Frontiers
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Aug 6, 2026Noam Koren, Roy Bar-Haim, Abigail GoldsteenAgentic Benchmarks Conversational Benchmarks
LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
Aug 6, 2026Lukas Twist, Twm Stone, Helen Yannakoudakis +1Benchmarking Large Language Models Model Preference
EpiBench: Can LLMs Understand Epitopes for Antibody Drug Discovery?
Aug 6, 2026Zirui Wang, Jiaqi Wang, Qinghan Wang +4Epitope Prediction Antibody
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLarge Language Model Bias Large Language Model Evaluation
SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution
Aug 6, 2026Zhi Han, Chenxi Zeng, Liuhaichen Yang +3Llm-As-A-Judge Judges
Sharding Prevents LLM Oversight Failures and Adversarial Exploitation
Aug 5, 2026Victor Akinwande, J. Zico Kolter, Aran NayebiJudges Shard
SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models
Aug 5, 2026Sihan Hu, Lyuhan Huang, Youjin Deng +1Benchmarking Large Language Models Frontier Models
What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend
Aug 5, 2026Shahed Masoudian, Passant Shafaei, Monorama Swain +1Benchmarking Large Language Models LLM Inference Efficiency
Cost-Aware Multi-Objective Bandits: Theory and Application to Budgeted LLM Configuration Evaluation
Aug 5, 2026Bo Xue, Zhi Hong, Jiayi Li +3Cost-Aware Large Language Model Evaluation

Sep 1, 2026 · cs.AIJ/K move · Enter open · S save

DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation

Haoyuan Shi, Mingtao Chen, Shuo Jiang, Ziyan Chen+11

Hunyuan, Tencent · Beijing Film Academy · Peking University · Shenzhen University

Commercial short-drama production follows a multi-stage chain: script, storyboard, keyframe imagery, shot-level video, and the finished short drama. Most existing benchmarks evaluate solely the video-generation stage using pre-authored inputs instead of real upstream pipeline outputs. This leaves two critical questions unanswerable: whether each stage adheres to the original script intent (rather than only its immediate input prompt), and whether disparate shots remain coherent after assembly into multi-episode releases. We present DramaChain Bench, the first short-drama benchmark that evaluates every stage of the complete production chain. It is built upon three in-house systems sharing one dimension system, DramaChain Dimensions: five evaluation axes instantiated at every stage, resolving into 63 leaf dimensions. DramaChain Agent is calibrated against commercial short-drama platforms in both workflow and finished short-drama quality, enabling stage-wise fair comparison across models. DramaChain Labeling System has each of the 5,785 items scored independently by three professional annotators, with all defects spatio-temporally localised and selected from a predefined defect list. This process produces 17,488 valid scores and 255,925 traceable attribution records. The human annotations confirm that upstream defects cascade across the pipeline, demonstrating that final episode quality is not governed by video generation alone. DramaChain Agentic Judge then scores every leaf dimension automatically, gathering evidence over multiple agentic rounds before judging against a per-item checklist; it reproduces the model ranking at a mean PLCC of 0.918, enough to admit new models at no annotation cost.

Films Llm-As-A-Judge

Open paper Ask about it PDF