LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,443 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency
Apr 7, 2026Jiachun Li, David Simchi-Levi, Will Wei SunSemiparametric Inference LLM Evaluation
Massachusetts Institute of Technology · Purdue University
ToxReason: A Benchmark for Mechanistic Chemical Toxicity Reasoning via Adverse Outcome Pathway
Apr 7, 2026Jueon Park, Wonjune Jang, Chanhwi Kim +2Toxicity Detection LLM Evaluation
Korea University · Myongji University · University of Texas Health Science Center at Houston +1
Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism
Apr 7, 2026Konstantine Arkoudas, Serafim BatzoglouLLM Evaluation Reasoning Consistency in Language Models
RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World
Apr 6, 2026Hanbing Liu, Lang Cao, Yang LiRetrieval-Augmented Generation Continual Learning for LLMs
Tsinghua University; Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University · University of Illinois Urbana-Champaign · School of Artificial Intelligence, Chinese University of Hong Kong (Shenzhen)
Under Pressure: Emotional Framing Induces Measurable Behavioral Shifts and Structured Internal Geometry in Small Language Models
Apr 6, 2026Rana Muhammad UsmanFraming Effects in Language Models Emotion Representation in Language Models
Independent Researcher
Plausibility as Commonsense Reasoning: Humans Succeed, Large Language Models Do not
Apr 6, 2026Sercan KarakaşImplicit Reasoning in Language Models LLM Evaluation
University of Chicago
LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection
Apr 6, 2026Cheng Xu, Changhong Jin, Yingjie Niu +5LLM Evaluation Fake News Detection
University College Dublin · Bebxy · Georgia Institute of Technology +1
How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling
Apr 6, 2026Yuhang Liu, Heyan Huang, Yizhe Yang +3LLM Evaluation LLM Mathematical Reasoning
Beijing Institute of Technology, Beijing, China · Southeast Academy of Information Technology, Putian, China
Individual and Combined Effects of English as a Second Language and Typos on LLM Performance
Apr 6, 2026Serena Liu, Yutong Yang, Prisha Sheth +9Language Model Robustness LLM Evaluation
Harvard University, Boston, MA 02138, US
Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality
Apr 6, 2026Xiaoyuan Zhu, Kimberly Le Truong, Riccardo Fogliato +8LLM Evaluation LLM Answer Verification
University of Southern California · Carnegie Mellon University · Microsoft Core AI +1
Automatically Generating Hard Math Problems from Hypothesis-Driven Error Analysis
Apr 6, 2026Jiayu Fu, Mourad Heddaya, Chenhao TanMathematical Problem Generation Synthetic Benchmark Generation
Department of Computer Science, University of Chicago, Chicago, IL 60637, USA
A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation
Apr 5, 2026Jerônimo Menezes, Leonardo Bitzki, Diego Kreutz +3LLM Evaluation LLM Reliability
Federal University of Rio Grande do Sul (UFRGS) · AI Horizon Labs and PPGES – Federal University of Pampa (UNIPAMPA)
Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News
Apr 4, 2026Alexander Loth, Martin Kappes, Marc-Oliver PahlAI-Generated Text Detection LLM Evaluation
Frankfurt University of Applied Sciences, Frankfurt am Main, Germany · IMT Atlantique, UMR IRISA, Chaire Cyber CNI, Rennes, France
CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering
Apr 4, 2026Baicheng Chen, Yu Wang, Ziheng Zhou +4Software Reverse Engineering LLMs for Cybersecurity
Shanghai Qi Zhi Institute · The Chinese University of Hong Kong, Shenzhen · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +4
Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge
Apr 4, 2026Yulong He, Ivan Smirnov, Dmitry Fedrushkov +2LLM Evaluation LLM-as-a-Judge
St. Petersburg State University, University emb., 7/9, St. Petersburg, Russia · ITMO University, Kronverkskiy av., 49, St. Petersburg, Russia
Researchers waste 80% of LLM annotation costs by classifying one text at a time
Apr 4, 2026Christian Pipal, Eva-Maria Vogel, Morgan Wack +1LLM-Assisted Annotation Large Language Model-Based Text Classification
Department of Communication and Media Research (IKMZ), University of Zurich
Large Language Models Align with the Human Brain during Creative Thinking
Apr 3, 2026Mete Ismayilzada, Simone A. Luchini, Abdulkadir Gokce +5LLM Evaluation
EPFL · Universit`a della Svizzera italiana (USI) · Paris Brain Institute (ICM) +2
Confidence Calibration in Large Language Models
Apr 3, 2026Noam Michael, Daniel BenShushan, Jacob Bien +1Language Model Calibration Confidence Estimation in Language Models
U.C. Berkeley · University of Southern California
Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation
Apr 3, 2026Leen AlQadi, Ahmed Alzubaidi, Mohammed Alyafeai +6LLM Evaluation Benchmark Validity
Technology Innovation Institute, Abu Dhabi, UAE
CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge
Apr 3, 2026Mete Ismayilzada, Renqing Cuomao, Daniil Yurshevich +3Creativity Assessment LLM Evaluation
EPFL · Università della Svizzera italiana (USI)
BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
Apr 3, 2026Sean Wu, Fredrik K. Gustafsson, Edward Phillips +3LLM Evaluation Confidence Estimation in Language Models
Department of Engineering Science, University of Oxford · Oxford Suzhou Centre for Advanced Research
The limits of bio-molecular modeling with large language models : a cross-scale evaluation
Apr 3, 2026Yaxin Xu, Yue Zhou, Tianyu Zhao +2LLM Evaluation Large Language Model-Guided Scientific Discovery
Southern University of Science and Technology, Shenzhen, 518055, China. · Pengcheng Laboratory, Shenzhen, 518055, China. · Institute of Mechanics, Chinese Academy of Sciences, Beijing, 100190, China.
Evaluating Artificial Intelligence Through a Christian Understanding of Human Flourishing
Apr 3, 2026Nicholas Skytland, Lauren Parsons, Alicia Llewellyn +5LLM Alignment LLM Evaluation
Gloo · WinShape Foundation · Biblica
Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation
Apr 3, 2026Nazanin Jafari, James Allan, Mohit IyyerLLM Evaluation Language Model Generation Evaluation
UMass Amherst · University of Maryland
The Impact of LLM Self-Consistency and Reasoning Effort on Automated Scoring Accuracy and Cost
Apr 3, 2026Scott FrohnMathematics Education Automated Evaluation
Khan Academy, Mountain View, CA, USA
Measuring Representation Robustness in Large Language Models for Geometry
Apr 3, 2026Vedant Jawandhia, Yash Sinha, Murari Mandal +2LLM Evaluation Representation Geometry in Language Models
Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani
PolyBench: Benchmarking LLM Forecasting and Trading Capabilities on Live Prediction Market Data
Apr 3, 2026Pu Cheng, Juncheng Liu, Yunshen LongForecasting Benchmarks LLM Evaluation
College of Electrical Engineering, Sichuan University, Chengdu, Sichuan, China · School of Economics, Sichuan University, Chengdu, Sichuan, China
Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
Apr 3, 2026Yihong Dong, Jianha Xiao, Xue Jiang +7LLM Evaluation Efficient Language Model Reasoning
Peking University, China · Wuhan University, China
Do We Need Frontier Models to Verify Mathematical Proofs?
Apr 2, 2026Aaditya Naik, Guruprerana Shabadi, Rajeev Alur +1LLM Input Optimization Reasoning Verification
University of Pennsylvania, U.S.A.
Beyond the Assistant Turn: User Turn Generation as a Probe of Interaction Awareness in Language Models
Apr 2, 2026Sarath Shekkizhar, Romain Cosentino, Adam EarleLLM Evaluation Language Model Probing
Salesforce AI Research

Apr 7, 2026J/K move · Enter open · S save

LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency

Jiachun Li, David Simchi-Levi, Will Wei Sun

Massachusetts Institute of Technology · Purdue University

Semiparametric Inference LLM Evaluation Pairwise Comparison LLM Uncertainty Estimation Preference Learning

Open paper AskPDF arXiv

Large language model (LLM) evaluation platforms increasingly rely on pairwise human judgments. These data are noisy, sparse, and non-uniform, yet leaderboards are reported with limited uncertainty quantification. We study this as semiparametric inference for a low-rank latent score tensor observed through pairwise comparisons under Bradley-Terry-Luce-type models. This places LLM evaluation in a new tensor completion setting with structured observations, non-uniform sampling, and pairwise contrasts. Our target is a smooth functional

ψ(T^\star)

, including linear estimands such as ability gaps and nonlinear ones such as win probabilities. We derive the information operator on the low-rank tangent space, the efficient influence function, and the semiparametric efficiency bound, then construct a one-step debiased estimator with asymptotic normality. A central challenge is that the information operator is anisotropic and does not commute with the tangent-space projection, creating a bottleneck absent from isotropic models. We introduce a score-whitening method that equalizes local Fisher information and restores stable inference at the optimal sample-complexity scale. Our results provide a principled framework for uncertainty quantification in LLM evaluation and more broadly for inference on low-rank structures from pairwise data.