LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
BRIDGE: Predicting Human Task Completion Time From Model Performance
Feb 6, 2026Fengyuan Liu, Jay Gala, Nilaksh +3LLM Evaluation Language Model Scaling Laws
Mila - Quebec AI Institute · McGill University · Polytechnique Montréal +3
FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
Aligning Language Model Benchmarks with Pairwise Preferences
Feb 2, 2026Marco Gutierrez, Xinyi Leng, Hannah Cyberey +3LLM Evaluation Learning to Rank
School of Data Science, University of Virginia · Imperial College London · Thomson Reuters Foundational Research +1
LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
Feb 2, 2026Rui Hua, Yu Wei, Zixin Shu +17Traditional Chinese Medicine Medical LLMs
Department of Computer Science and Technology, Beijing Jiaotong University, Beijing, China · Tianjin Tasly Digital Chinese Medicine Technology Co., Ltd., Tianjin, China · Tasly Biopharmaceuticals Co., Ltd., Tianjin, China +11
Measuring Pragmatic Influence in Large Language Model Instructions
Feb 2, 2026Yilin Geng, Omri Abend, Eduard Hovy +1LLM Evaluation LLM Prompting
University of Melbourne · Hebrew University of Jerusalem
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM Evaluation AI Agent Evaluation
Computer Science and Information, Ben-Gurion University of the Negev
Learning More from Less: Unlocking Internal Representations for Benchmark Compression
Jan 31, 2026Yueqi Zhang, Jin Hu, Shaoxiong Feng +9LLM Evaluation Representation Learning
School of Computer Science, Beijing Institute of Technology · Xiaohongshu Inc.
Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
Jan 30, 2026Zhuochun Li, Yong Zhang, Ming Li +8LLM Evaluation LLM-as-a-Judge
Ping An Technology (Shenzhen) Co., Ltd. · University of Pittsburgh · University of Maryland, College Park +1
Assessing the Business Process Modeling Competences of Large Language Models
Jan 29, 2026Chantale Lauer, Peter Pfeiffer, Alexander Rombach +1LLM Evaluation Business Process Modeling
German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Germany · 5Plus GmbH, Goethestraße 1, Würzburg, 97072, Germany
More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)
Jan 29, 2026Sagi Meir, Tommer D. Keidar, Noam Levi +2LLM Inference Efficiency LLM Evaluation
School of Chemistry, Tel Aviv University, Tel Aviv 6997801, Israel · The Center for Physics and Chemistry of Living Systems, Tel Aviv University, Tel Aviv 6997801, Israel · School of Physics and Astronomy, Tel Aviv University, Tel Aviv 6997801, Israel +1
Efficient Evaluation of LLM Performance with Statistical Guarantees
Jan 28, 2026Skyler Wu, Yash Nair, Emmanuel J. CandèsLLM Evaluation
Department of Statistics, Stanford University
LLMs versus the Halting Problem: Characterizing Program Termination Reasoning
Jan 26, 2026Oren Sultan, Jordi Armengol-Estape, Pascal Kesseli +4LLM Evaluation Static Code Analysis
1Meta FAIR · 3Bloomberg, New York, USA · 4Imperial College London, UK +2
BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluation
Jan 26, 2026Peng Sun, Xiangyu Zhang, Duan Wu +5LLM Evaluation Representation Probing
Qwen Business Unit of Alibaba
Assessment of Generative Named Entity Recognition in the Era of Large Language Models
Jan 25, 2026Qi Zhan, Yile Wang, Hui HuangNamed Entity Recognition LLM Information Extraction
College of Computer Science and Software Engineering, Shenzhen University
Not Your Typical Sycophant: The Elusive Nature of Sycophancy in Large Language Models
Jan 21, 2026Shahar Ben-Natan, Oren TsurLLM Evaluation LLM Sycophancy
Computer and Information Science Ben Gurion University
MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering
Jan 21, 2026Anjishnu Mukherjee, Ziwei Zhu, Antonios AnastasopoulosLanguage Model Pretraining LLM Evaluation
George Mason University
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Jan 20, 2026Esma Balkır, Alice Pernthaller, Marco Basaldella +2LLM Evaluation Computerized Adaptive Testing
Trismik · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Universitat Politècnica de València +1
No Reliable Evidence of Self-Reported Sentience in Small Large Language Models
Jan 20, 2026Caspar Kaiser, Sean EnderbyLLM Evaluation Language Model Self-Assessment
University of Warwick, Warwick Business School.
Vulnerability of LLMs' Stated Beliefs? LLMs Belief Resistance Check Through Strategic Persuasive Conversation Interventions
Jan 20, 2026Fan Huang, Haewoon Kwak, Jisun AnLLM Evaluation Fine-Tuning
Indiana University Bloomington, United States
OI-Bench: An Option Injection Benchmark for Evaluating LLM Susceptibility to Directive Interference
Jan 19, 2026Yow-Fu Liou, Yu-Chien Tang, Yu-Hsiang Liu +1LLM Evaluation Adversarial Robustness of Language Models
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan
Representational Similarity Alignment of Figurative Language in Humans and Large Language Models
Jan 14, 2026Samhita Bollepally, Aurora Sloman-Moll, Takashi Yamauchi +1Figurative Language Understanding LLM Evaluation
QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models
Jan 13, 2026Zhaolu Kang, Junhao Gong, Wenqing Hu +13Financial Language Model Evaluation LLM Evaluation
Peking University · Tsinghua University · Fudan University +6
How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction
Jan 13, 2026Zhaolu Kang, Yingjie He, Kehan Jiang +18Input Order Sensitivity in Language Models LLM Evaluation
Hunyuan, Tencent · Peking University · Tsinghua University +8
STAGE: A Full-Screenplay Benchmark for Reasoning over Evolving Stories
Jan 13, 2026Qiuyu Tian, Zequn Liu, Yiding Li +9KG Construction LLM Evaluation
Southeast University, Nanjing, China · Beijing Zhongguancun Academy, Beijing, China · ZhuiWen Technology Co., Ltd., Beijing, China +1
Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Jan 12, 2026Yuxi Xia, Dennis Ulmer, Terra Blevins +3LLM Uncertainty Estimation Confidence Estimation in Language Models
Faculty of Computer Science, UniVie Doctoral School Computer Science · Faculty of Philological and Cultural Studies, University of Vienna, Austria · ILLC, University of Amsterdam, Netherlands +2
Kinship Data Benchmark for Multi-hop Reasoning
Jan 12, 2026Tianda Sun, Dimitar KazakovLLM Evaluation Synthetic Benchmark Generation
Department of Computer Science University of York, UK
Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition
Jan 12, 2026Tanmay Joshi, Shourya Aggarwal, Anusa Saha +7Emergent Abilities in Language Models LLM Reliability
Pragya Lab, BITS Pilani Goa, India · Raapid Lab, USA · Apple, USA +1
Measuring Iterative Temporal Reasoning with Time Puzzles
Jan 12, 2026Zhengxiang Wang, Zeyu DongLLM Evaluation Temporal Reasoning in Language Models
Department of Applied Math and Statistics Stony Brook University
BizFinBench.v2: Towards Reliable LLMs in Finance via Real-User Data and Offline/Online Bilingual Evaluation
Jan 10, 2026Xin Guo, Rongjunchen Zhang, Guilong Lu +4Financial Services Multilingual Language Model Evaluation
HiThink Research · Shanghai University of Finance and Economics

Feb 9, 2026J/K move · Enter open · S save

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV+1

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Ensembles

Open paper AskPDF arXiv

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.