LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
MathDuels: Evaluating LLMs as Problem Posers and Solvers
Apr 23, 2026Zhiqiu Xu, Shibo Jin, Shreya Arya +1Mathematical Reasoning Benchmarks LLM Evaluation
Department of Computer and Information Science, University of Pennsylvania · Rabdos AI · Department of Mathematics, University of Pennsylvania
Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms
Apr 23, 2026Yuto Nishida, Naoki Shikoda, Yosuke Kishinami +4LLM Evaluation Memorization in Language Models
Nara Institute of Science and Technology · Future Corporation
Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards
Apr 23, 2026Minji Jung, Minjae Lee, Yejin Kim +2LLM Evaluation Human-in-the-Loop Evaluation
Yonsei University, South Korea
A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair
Apr 23, 2026Milan De Koning, Ali Asgari, Pouria Derakhshanfar +1LLM Evaluation Metamorphic Testing
JetBrains Research, Amsterdam, The Netherlands · Delft University of Technology, Delft, The Netherlands
OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving
Apr 23, 2026Xinyu Zhang, Boxuan Zhang, Yuchen Wan +5LLM Evaluation Benchmark Design
School of Computer Science and Technology, Xi’an Jiaotong University · 2Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China · 4Lenovo Research +1
Evaluating AI Meeting Summaries with a Reusable Cross-Domain Pipeline
Apr 23, 2026Philip Zhong, Don Wang, Jason ZhangLLM Evaluation Factual Consistency Evaluation
Webex Suite AI Cisco Systems, Inc. San Jose, California, USA
EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval
Apr 23, 2026Julian AcunaLLM Evaluation Memorization in Language Models
A Demonstration of SQLyzr: A Platform for Fine-Grained Text-to-SQL Evaluation and Analysis
Apr 23, 2026Sepideh Abedini, M. Tamer ÖzsuLLM Evaluation Text-to-SQL
University of Waterloo
SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation
Apr 22, 2026Ruohan Liu, Shukang Yin, Tao Wang +6Audio-Language Model Evaluation LLM Evaluation
1Nanjing University · 2Xiaomi
Diagnosing CFG Interpretation in LLMs
Apr 22, 2026Hanqi Li, Lu Chen, Kai YuLLM Evaluation In-Context Learning
1X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · 3Shanghai Innovation Institution, Shanghai, China · 4Jiangsu Key Lab of Language Computing, Suzhou, China +2
Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization
Apr 22, 2026Mohamed Hesham Elganayni, Runsheng Chen, Sebastian Nagl +1LLM Evaluation LLM-as-a-Judge
Technical University of Munich Germany
Cross-Dataset Bloom Question Classification: Supervised Models and Prompted LLMs
Apr 22, 2026Abdolali Faraji, Mohammadreza Molavi, Zohreh Rasoulkhani +2LLM Evaluation Educational Assessment
Leibniz Information Centre for Science and Technology (TIB) · University of Genoa
Early-Stage Product Line Validation Using LLMs: A Study on Semi-Formal Blueprint Analysis
Apr 22, 2026Viet-Man Le, Thi Ngoc Trang Tran, Sebastian Lubos +2Software Engineering LLM Evaluation
Graz University of Technology Graz, Austria
Benchmarking Large Language Models for Safety Data Extraction
Apr 22, 2026Jonas Grill, Thomas Bayer, Sören BerlingerLLM Evaluation LLM Prompting
1SAP SE, Germany. · Institute for Digital Transformation, Ravensburg-Weingarten University.
DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories
Apr 22, 2026Neemesh Yadav, Palakorn Achananuparp, Jing Jiang +1LLM Evaluation Theory of Mind
Singapore Management University · Australian National University
CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge
Apr 22, 2026Gustav Keppler, Ghada Elbez, Veit HagenmeyerLLM Evaluation Cybersecurity
Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks
Apr 22, 2026Jan-Philipp SchmidtLLM Evaluation Educational Assessment
TH Köln, Institut für Versicherungswesen (ivwKöln)
The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models
Apr 22, 2026Yilun Liu, Chunguang Zhao, Mengyao Piao +14Multilingual Language Model Evaluation LLM Evaluation
Huawei, China · Fudan University, China
The Path Not Taken: Duality in Reasoning about Program Execution
Apr 22, 2026Eshgin Hasanov, Md Mahadi Hassan Sibat, Santu Karmaker +1LLM Evaluation Code Language Models
Department of Computer Science, University of Central Florida
In-Situ Behavioral Evaluation for LLM Fairness, Not Standardized-Test Scores
Apr 21, 2026Zeyu Tang, Sang T. Truong, Deonna Owens +4LLM Evaluation Social Bias in Language Models
Stanford University
The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text
Apr 21, 2026Andrew Hong, Jason Potteiger, Luis E. ZapataLLM Evaluation LLM Prompting
Dimension Labs
Cross-Model Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Across Three Large Language Models
Apr 21, 2026Kihyuk LeeLLM Evaluation Clinical Language Model Evaluation
Data Convergence Team, Office of Hospital Information, Seoul National University Bundang Hospital, Seongnam, Republic of Korea
Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews
Apr 21, 2026Bowen Li, Haochen Ma, Yuxin Wang +5LLM Evaluation Rubric-Based Evaluation
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas
Apr 21, 2026Jon-Paul CacioliLLM Evaluation
Independent Researcher · Melbourne, Australia
Are Large Language Models Economically Viable for Industry Deployment?
Apr 21, 2026Abdullah Mohammad, Sushant Kumar Ray, Pushkar Arora +5LLM Inference Efficiency LLM Evaluation
DSEU-Okhla, New Delhi, India · University of Delhi, New Delhi, India · 6, 8Macquarie University, Sydney, Australia +1
Evaluating LLM-Driven Summarisation of Parliamentary Debates with Computational Argumentation
Apr 21, 2026Eoghan Cunningham, Derek Greene, James Cross +1LLM Evaluation Factual Consistency Evaluation
School of Computer Science, University College Dublin · School of Politics and International Relations, University College Dublin · Department of Informatics, King’s College London
IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text
Apr 21, 2026Rajveer Singh PallLLM Evaluation Financial QA
Gyan Ganga Institute of Technology and Sciences, Jabalpur, India
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Apr 21, 2026Peiqin Lin, Chenyang Lyu, Wenjiang Luo +22Multilingual Language Model Evaluation LLM Evaluation
Alibaba Group · Beijing Language and Culture University · LMU Munich +6
Verbal tics in frontier language models: A critical review of current releases, research evidence, and public discussion
Apr 21, 2026Shuai Wu, Xue Li, Zhijun Wang +4LLM Evaluation LLM Sycophancy

Apr 23, 2026, cs.CLJ/K move · Enter open · S save

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

Abel Yagubyan

Independent Researcher

Inter-Rater Reliability LLM Evaluation Pairwise Comparison LLM-as-a-Judge Bias in LLM-as-a-Judge Position Bias

Open paper AskPDF arXiv

LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized. We study repeated identical evaluations on 29 tasks spanning 10 categories using two OpenAI judge models (GPT-4o-mini and GPT-4.1-mini), with 50 pairwise trials and 50 pointwise trials per question, supplemented by temperature and prompt-sensitivity ablations. Across judges, pairwise preferences flip on average 13.6% of the time, with 28% of questions exceeding a 20% flip rate and one question reaching 56%. GPT-4o-mini also exhibits a significant first-position bias (72% A-majority, p = 0.024). At the same time, mean pointwise score gaps are small (0.19--0.36 on a 10-point scale) and not statistically significant in aggregate, producing a pairwise--pointwise gap: judges frequently choose a winner even when their own scalar scores provide little evidence of a meaningful quality difference. Beyond within-judge instability, cross-judge agreement is only 76% (

κ= 0.51

), semantically equivalent prompt templates change majority outcomes in 25% of tested cases, and deterministic decoding reduces but does not eliminate inconsistency. A reliability curve analysis shows that, in our dataset, 11 repeated trials are needed for a majority vote to recover the 50-trial reference verdict with 95% probability on average, rising to 15 for high-variance questions. These findings suggest that single-trial LLM judging is often too noisy for high-stakes evaluation, and that multi-trial aggregation, position randomization, and explicit uncertainty reporting should be standard practice. Because both judges are from a single provider, cross-provider replication remains an important next step.

Momentum

Related topics

Latest papers 1,631