LLM-as-a-Judge

Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

23 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-14

16 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-07

36 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Inside this field

Focused directions

679 papers

Latest in LLM-as-a-Judge

Browse all topics →
CardsList

  1. Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

    Jul 20, 2026Koyar AfrasyabLarge Language Model JudgesOverconfidence

  2. Human Grounded Evaluation of Large Language Models for Optical Network Automation

    Jul 20, 2026Kiarash Rezaei, Omran Ayoub, Paolo Monti +1Large Language Model EvaluationHuman Grounded Evaluation

  3. Pancasila-Dilemmas: Evaluating Large Language Models on Indonesian Human Value Dilemmas Grounded in Pancasila

    Jul 20, 2026Supryadi, Irfan, Julianti +4Human ValuesLarge Language Model Alignment

  4. Stress Testing Concept Erasure with Large Language Model Agents

    Jul 20, 2026Yuyang Xue, Feng Chen, Zhihua Liu +4Concept ErasureStress Testing

  5. JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models

    Jul 20, 2026Qingjia Huang, Jingyu Zhang, Jianguo Wu +6Large Language Model JailbreaksJailbreak Attacks

  6. A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models

    Jul 19, 2026Mohtashim KhanLarge Language Model EvaluationPreference Alignment

  7. Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

    Jul 18, 2026Maksim Sheverev, David Finkelstein, Sergey NikolenkoLarge Language Model MemoryAgent Memory

  8. TellTale: Blending Multi-Instance LoRA Text Encoders and a Zero-Shot LLM Judge for Ambivalence/Hesitancy Recognition in Videos

    Jul 18, 2026Abdel-Karim Al-Tamimi, Ali RodanLarge Language Model JudgesAccurate Transcription

  9. CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

    Jul 17, 2026Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru +3Large Language Model EvaluationRubrics

  10. AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

    Jul 17, 2026Yuqing Wen, Yukai Huang, Qianqian Xie +6Video EditingAudio-Visual Consistency

  11. AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets

    Jul 17, 2026Ming Chen, Pranav PaiGeospatialLarge Language Model Evaluation

  12. Does Multi-Agent Debate Improve AI Feedback on Research Papers?

    Jul 16, 2026Tomas Havranek, Zuzana IrsovaAcademic PapersMulti-Agent Debate

  13. Harnessing LLMs for Reliable Academic Supervision: A Comparative Study

    Jul 16, 2026Akash RajLlm-As-A-JudgeLarge Language

  14. How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay Drafts

    Jul 16, 2026Steven Coyne, Diana Galvan-Sosa, Ryan Spring +4Automated Essay ScoringLarge Language Model Evaluation

  15. Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

    Jul 16, 2026Yijiang Li, Huiqi Zou, Bingyang Wang +1Large Language Model EvaluationMulti-Modal Large Language Models

  16. LLM Evaluators are Biased across Languages

    Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Large Language Model Evaluation

  17. Traccia: An OpenTelemetry-Based Governance Platform for AI Systems

    Jul 15, 2026Nutan Kumar Naik, Aditya Kumar Saroj, Vijay Prasad Poudel +2Artificial Intelligence GovernanceArtificial Intelligence Systems

  18. MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation

    Jul 15, 2026Xiaohan Zhang, Yuqing Wen, Junlin Chen +9Joint Audio-Video GenerationAudio-Visual Consistency

  19. Pezego-HITL: A policy-grounded large language model architecture for agricultural extension in Ghana

    Jul 15, 2026Shunbao Li, Zhipeng Yuan, Amoako Ofori +5Large Language Model EvaluationAgricultural Landscapes

  20. The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

    Jul 15, 2026Serkan BallıLlm-As-A-JudgeText Corpora

  21. Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

    Jul 15, 2026Joonyong Park, David M. Chan, Yuki Saito +1Large Audio Language ModelsLarge Language Model Judges

  22. Evaluation Ability Does Not Imply Optimization Utility: LLM-as-a-Judge Signals in Closed-Loop Table Recognition

    Jul 15, 2026Donghwan KimLlm-As-A-JudgeFeedback Learning

  23. GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

    Jul 14, 2026Bidyarthi Paul, Nahida Jannat Mayouree, Md. Asif Karim +2BanglaMathematical Reasoning

  24. Rating the Raters: Rasch Measurement Theory for LLM Evaluation

    Jul 14, 2026Pratik S. Sachdeva, Nathan BoudolMean Opinion ScoresLlm-As-A-Judge

  25. LLM Judges Can Be Too Generous When There Is No Reference Answer

    Jul 14, 2026Chalamalasetti Kranti, Sowmya VajjalaLarge Language Model JudgesHuman Judgment

  26. Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

    Jul 13, 2026Zixiang Xu, Sixian Li, Huaxing Liu +4JudgesGrading

  27. Enhancing LLMs through human feedback: a journey towards self-improvement

    Jul 13, 2026Tatiana Pelc, Gila Kamhi, Asaf Avrahamy +1Retrieval-Augmented Generation SystemsRecursive Self-Improvement

  28. Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

    Jul 13, 2026Chuhan Shi, Xiaoquan Ren, Sicheng Song +3Scientific DiscoveryBenchmarking Large Language Models

  29. Toward Contemplative LLM: A Modular Framework for Evaluating and Enhancing LLM Alignment in Mental Health

    Jul 12, 2026Asher Sprigler, Yang-Yang Feng, Iftach Amir +5Large Language Model EvaluationLarge Language Model Alignment

  30. Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG

    Jul 12, 2026Sriram Selvam, Anneswa GhoshLlm-As-A-JudgeLarge Language Model Judges

  31. Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Reviews

    Jul 12, 2026Nuo Chen, Qian Wang, Qingyun Zou +1Llm-As-A-JudgePeer Review

  32. What Does Your Short-Answer VQA Score Actually Measure? Evaluator-Dependent Instability in Multimodal Short-Answer Benchmarks

    Jul 11, 2026Guanhua Ye, Niu Jingbin, Yan Li +4Knowledge-Based Visual Question AnsweringLarge Language Model Judges

  33. Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenders

    Jul 11, 2026Cedric Waterschoot, Nava Tintarev, Francesco BarileRecommendationLarge Language Model Judges

  34. LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning

    Jul 11, 2026Ning LiuLlm-As-A-JudgeLarge Language Model Reasoning

  35. Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

    Jul 9, 2026Ethan Leung, Elias Lumer, Corey Feld +3Large Language Model JudgesCitations

  36. When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

    Jul 9, 2026Zongyou Yang, Yinghan Hou, Xiaokun YangLlm-As-A-JudgeLarge Language Model Judges

  37. Large-Language-Models-as-a-Judge in Theory-Agnostic Adaptive Metric-Alignment for Prototypical Networks in Personality Recognition

    Jul 9, 2026Jing Jie Tan, Ban-Hoe Kwan, Danny Wee-Kiat Ng +6PersonalityLlm-As-A-Judge

  38. Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

    Jul 9, 2026Siddhartha Jain, Ameya VelingkerContext LengthLongbench

  39. When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

    Jul 9, 2026Kaihua DingSelf-ConsistencyJudges

  40. The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents

    Jul 8, 2026Xing Zhang, Yanwei Cui, Guanghui Wang +4Self-Evolving AgentsJudges

  41. Predicting LLM Safety Before Release by Simulating Deployment

    Jul 8, 2026Marcus Williams, Hannah Sheahan, Cameron Raymond +8Large Language Model SafetyLarge Language Model Evaluation

  42. Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System

    Jul 8, 2026Yiming Gai, Junde Lu, Xuefei HuangLarge Language Model EvaluationLarge Language Model Responses

  43. REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

    Jul 7, 2026Nicolas Koller, Andreas u. SchmidtReverse EngineeringDisassembly

  44. What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

    Jul 7, 2026Robert RichardsonText-To-SqlCorrectness

  45. Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

    Jul 7, 2026So Hasegawa, Shailaja Keyur Sampat, Lei Liu +1Benchmarking Large Language Models

  46. Andha-Dhun: A First Look at Audio Descriptions in Hindi

    Jul 7, 2026Ritabrata Chakraborty, Divy Kala, Nisheeth Bhooshan Gupta +3Human AnnotationsBodhi

  47. More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

    Jul 7, 2026Chenyu ZhouLarge Language Model JudgesLlm-As-A-Judge

  48. Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

    Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Large Language Model EvaluationMean Opinion Scores

  49. LLM for the development of FCM

    Jul 6, 2026Alexis KafantarisLarge Language Model EvaluationCognitive Map

  50. Agent Step Value: Auditing Evaluator-Channel Reversals in Black-Box Agent Traces

    Jul 5, 2026Andrew Zhang, Chengzhan LiAction TracesReasoning Traces

  51. Spectral Signatures of Large Language Models

    Jul 3, 2026Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu +4Benchmarking Large Language ModelsLineages

  52. MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

    Jul 3, 2026Siran Zhao, Ruihui Hou, Ziyue Huai +2Clinical NotesBenchmarking Large Language Models

  53. Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

    Jul 2, 2026Jihan Yao, Gantavya Bhatt, Arnav Das +16Benchmarking Large Language ModelsGeometry-Aware Uncertainty Coresets

  54. Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages

    Jul 2, 2026A. Seza Doğruöz, Xixian Liao, Verena Blaschke +3Llm-As-A-JudgeLow-Resource Languages

  55. Ask the Right Comparison:Bias-Aware Bayesian Active Top-kk Ranking with LLM Judges

    Jul 2, 2026Jian Xu, Delu Zeng, John Paisley +1Large Language Model JudgesRanking

  56. SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

    Jul 2, 2026Anna ChornaCross-Lingual ConsistencyLarge Language Model Responses

  57. Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

    Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1RubricsLarge Language Model Judges

  58. Meta-Benchmarks for Financial-Services LLM Evaluation

    Jul 2, 2026Blair HudsonFinance BenchmarksBanking

  59. IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

    Jul 1, 2026Samir Abdaljalil, Erchin Serpedin, Hasan KurbanScientific DiscoveryLarge Language Model Evaluation

  60. Theoria: Rewrite-Acceptability Verification over Informal Reasoning States

    Jul 1, 2026Michael Saldivar, Ben SlivinskiTheoremVerifier