LLM-as-a-Judge

Follow this topic Ask about this topic

Momentum

60 papers in the last four weeks, up 76% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 362

All topics

Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation
Apr 6, 2026Alhasan Mahmood, Samir Abdaljalil, Hasan KurbanMultilingual Language Model Evaluation LLM-as-a-Judge
Hamad Bin Khalifa University, Doha, Qatar · Texas A&M University, College Station, TX USA
Mediocrity is the key for LLM as a Judge Anchor Selection
Mar 17, 2026Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen +1LLM-as-a-Judge Pairwise Preference Evaluation
1The Hebrew University of Jerusalem · 2IBM Research · 3MIT +1
LLM-as-a-judge validity in physics assessment depends more on the task than the model
Mar 16, 2026Will Yeadon, Tom Hardy, Paul Mackay +1LLM-as-a-Judge Educational Assessment
Department of Physics, Durham University, Durham DH1 3LE, UK
MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
Mar 11, 2026Chihiro Watanabe, Jingyu SunLLM-as-a-Judge Clustering
NTT Computer and Data Science Laboratories, 3-9-11, Midori-cho, Musashino-shi, Tokyo, Japan
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
Mar 9, 2026Hongli Zhou, Hui Huang, Rui Zhang +5LLM-as-a-Judge Social Bias in Language Models
Faculty of Computing, Harbin Institute of Technology, Harbin, 150001, China · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, 518055, China
Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation
Feb 28, 2026Zeyu Chen, Huanjin Yao, Ziwang Zhao +1LLM Evaluation LLM-as-a-Judge
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Feb 24, 2026Santiago Gonzalez, Alireza Amiri Bavandpour, Peter Ye +48Mathematical Reasoning Benchmarks LLM Evaluation
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM Alignment LLM-as-a-Judge
University of North Carolina at Chapel Hill · 2Carnegie Mellon University · 3Yale University +1
CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation
Feb 13, 2026Yiran Rex Ma, Yuxiao Ye, Huiyuan XieAutomated Evaluation Legal NLP
School of Foreign Languages, Peking University · Department of Computer Science and Technology, Tsinghua University
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
Feb 6, 2026Lanbo Lin, Jiayao Liu, Tianyuan Yang +5LLM-as-a-Judge AI Agent Evaluation
Alibaba International Digital Commerce Group · Zhejiang University · Peking University
MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems
Feb 3, 2026Vishal Venkataramani, Haizhou Shi, Zixuan Ke +6Multi-Agent LLM Systems LLM-as-a-Judge
1Rutgers University · 2Salesforce AI Research
Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
Feb 2, 2026Yuzheng Xu, Tosho Hirasawa, Tadashi Kozuno +1LLM-as-a-Judge Language Model Bias Evaluation
OMRON SINIC X · NexaScience
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Jan 30, 2026Zhuochun Li, Yong Zhang, Ming Li +8LLM Evaluation LLM-as-a-Judge
Ping An Technology (Shenzhen) Co., Ltd. · University of Pittsburgh · University of Maryland, College Park +1
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Generation Evaluation
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2
ReportLogic: Evaluating Logical Quality in Deep Research Reports
Jan 27, 2026Jujia Zhao, Zhaoxin Huan, Zihan Wang +4LLM-as-a-Judge Automated Evaluation
1Leiden Institute of Advanced Computer Science, Leiden University · 2Ant Group · 3CISPA Helmholtz Center for Information Security
From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges
Jan 13, 2026Yihan Hong, Huaiyuan Yao, Bolin Shen +3LLM-as-a-Judge Rubric-Based Evaluation
Washington University in St. Louis, St. Louis, MO, USA · Arizona State University, Tempe, AZ, USA · Florida State University, Tallahassee, FL, USA
Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
Jan 7, 2026Yao Dou, Benjamin Mamut, Wei XuLegal NLP LLM Evaluation
Georgia Institute of Technology
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model Evaluation LLM Evaluation
AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1
Idea2Plan: Exploring AI-Powered Research Planning
Oct 28, 2025Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar +1LLM Evaluation LLM-as-a-Judge
University of Michigan · Microsoft Research
User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios
Oct 23, 2025Xiaoyuan Wu, Roshni Kaushik, Wenkai Li +2LLM Evaluation LLM-as-a-Judge
Fujitsu Research of America, Inc. · Carnegie Mellon University
Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators
Sep 3, 2025Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Steering
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 · Martian Research San Francisco, CA, 94105
Expert Preference-based Evaluation of Automated Related Work Generation
Aug 11, 2025Furkan Şahinuç, Subhabrata Dutta, Iryna GurevychHuman Preference Evaluation LLM-as-a-Judge
Ubiquitous Knowledge Processing Lab (UKP Lab) · Department of Computer Science and Hessian Center for AI (hessian.AI) · Technical University of Darmstadt +1
Evaluating Style-Personalized Text Generation: Challenges and Directions
Aug 8, 2025Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2LLM-as-a-Judge Language Model Generation Evaluation
Columbia University, USA · Microsoft, USA · The University of York, UK
Structured Prompting and Automated Evaluation in Fixed Synthetic Japanese-Language Counseling Dialogues
Jun 28, 2025Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Goto +5Mental Health Counseling LLM-as-a-Judge
Japan National Institute of Occupational Safety and Health, Kanagawa, Japan · Kaze To Taiyo, Tokyo, Japan · Saga Occupational Health Association, Saga, Japan +5
Multimodal Language Models as Text-to-Image Model Evaluators
May 1, 2025Jiahui Chen, Candace Ross, Reyhane Askari-Hemmat +5LLM-as-a-Judge Benchmark Design
1FAIR at Meta · University of Texas at Austin · ∗Work done at Meta +3
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
Mar 21, 2025Jeremy Barnes, Naiara Perez, Alba Bonet-Jover +1Summarization Evaluation Automated Evaluation
HiTZ Center, University of the Basque Country (UPV/EHU) · Department of Software and Computing Systems, University of Alicante, Spain · GOI Institute, Basque Summer University (UEU)
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Mar 7, 2025Michael Krumdick, Charles Lovering, Varshini Reddy +2LLM Evaluation LLM-as-a-Judge
Kensho Technologies · MIT
Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges
Date pendingRui Yang, Shuang Huang, Junhua Liu +7LLM Safety Benchmarks LLM-as-a-Judge
Anhui SparkShield Intelligent Technology · iFLYTEK · Peking University +1

Apr 6, 2026, cs.CLJ/K move · Enter open · S save

Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation

Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

Hamad Bin Khalifa University, Doha, Qatar · Texas A&M University, College Station, TX USA

Multilingual Language Model Evaluation LLM-as-a-Judge AI Agent Benchmarks

Open paper AskPDF arXiv

Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings. We localize the Agent-as-a-Judge prompt stack to five typologically diverse languages (English, Arabic, Turkish, Chinese, Hindi) and evaluate 55 DevAI development tasks across three developer-agent frameworks and six judge backbones, totaling 4950 judge runs. The central finding is that backbone and language interact: GPT-4o achieves the highest satisfaction in English (44.72%), while Gemini leads in Arabic (51.72%,

p<0.001

vs.\ GPT-4o) and Hindi (53.22%). No single backbone dominates across all languages, and inter-backbone agreement on individual requirement judgments is modest (Fleiss'

κ\leq 0.231

). A controlled ablation further shows that localizing judge-side instructions, not just benchmark content, can be decisive: Hindi satisfaction drops from 42.8% to 23.2% under partial localization. These results indicate that language should be treated as an explicit evaluation variable in agentic benchmarks. Full requirement-level judgments and runtime statistics are released for reproducibility.

Momentum

Related topics

Latest papers 362