LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

From Human-Level AI Tales to AI Leveling Human Scales
Feb 21, 2026Peter Romero, Fernando Martínez-Plumed, Zachary R. Tidler +11LLM Evaluation
Valencian Research Institute of Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · The Psychometrics Centre, University of Cambridge, UK · Georgia Institute of Technology +6
EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation
Feb 21, 2026Adam Dejl, Jonathan PearsonLLM Evaluation Language Model Generation Evaluation
Imperial College London* Department of Computing · NHS England Transformation Directorate
BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models
Feb 21, 2026Thura Aung, Jann Railey Montalan, Jian Gang Ngui +1LLM Evaluation Low-Resource Language Processing
King Mongkut’s Institute of Technology Ladkrabang · AI Singapore · National University of Singapore
ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models
Feb 21, 2026Anas Alhumud, Abdulaziz Alhammadi, Muhammad Badruddin KhanArabic NLP LLM Evaluation
Saudi Data and Artificial Intelligence Authority, Saudi Arabia · Imam Mohammad Ibn Saud Islamic University, Saudi Arabia
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Feb 20, 2026Baris Arat, Emre SeferLLM Evaluation LLM Reranking
Ozyegin University Istanbul, Turkey
Luna-2: Scalable Single-Token Evaluation with Small Language Models
Feb 20, 2026Vatsal Goel, Rishon Dsouza, Nikhil Ega +4LLM Evaluation Language Model Safety Evaluation
Galileo AI, San Francisco, USA
Validating Political Position Predictions of Arguments
Feb 20, 2026Jordan Robinson, Angus R. Williams, Katie Atkinson +1LLM Evaluation Computational Argumentation
University of Liverpool · The Alan Turing Institute · University of Leeds
VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
Feb 20, 2026Yutong Xin, Qiaochu Chen, Greg Durrett +1LLM Evaluation Formal Verification
The University of Texas at Austin · New York University
[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games
Feb 20, 2026Jorge Carrasco Pollo, Ioannis Kapetangeorgis, Joshua Rosenthal +1LLM Evaluation AI Agent Benchmarks
Informatics Institute University of Amsterdam
Capabilities Ain't All You Need: Measuring Propensities in AI
Feb 20, 2026Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11LLM Evaluation Item Response Theory
Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5
Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm
Feb 20, 2026Anna Babarczy, Andras Lukacs, Peter Vedres +1Theory of Mind Social Reasoning
Institute of General and Hungarian Linguistics, Research Centre for Linguistics · Department of Cognitive Science, Faculty of Natural Sciences, Budapest University of Technology and Economics · Institute of Mathematics, Eötvös Lóránd University
Towards More Standardized AI Evaluation: From Models to Agents
Feb 20, 2026Ali El Filali, Inès BedarAI Agent Evaluation LLM Evaluation
G42 · Independent
On the scaling relationship between cloze probabilities and language model next-token prediction
Feb 19, 2026Cassandra L. Jacobs, Morgan GrobolLanguage Modeling Language Model Scaling
Department of Linguistics, University at Buffalo, Buffalo, NY, USA · MoDyCo, Université Paris Nanterre, Paris, France
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
Feb 19, 2026Simon Henniger, Gabriel PoesiaLanguage Model Reasoning Evaluation LLM Evaluation
Harvard University
ABCD: All Biases Come Disguised
Feb 19, 2026Mateusz Nowak, Xavier Cadet, Peter ChinLLM Evaluation Language Model Bias Evaluation
Dartmouth College
Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Feb 19, 2026Bogdan Kostić, Conor Fallon, Julian Risch +1Prompt Sensitivity LLM Evaluation
Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin
CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Feb 19, 2026Lukas Thede, Stefan Winzeck, Zeynep Akata +1LLM Post-Training Catastrophic Forgetting
Thomson Reuters Foundational Research, 5 Canada Square, London · Tübingen AI Center, University of Tübingen · Helmholtz Munich +2
Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation
Feb 19, 2026Yan Wang, Yi Han, Lingfei Qian +11Recommender Systems Large Language Model-Based Recommendation
The Fin AI, USA · Georgia Institute of Technology, USA · Columbia University, USA +5
SourceBench: Can AI Answers Reference Quality Web Sources?
Feb 18, 2026Hexi Jin, Stephen Liu, Yuheng Li +2LLM Evaluation IR Evaluation
University of California, San Diego · GenseeAI Inc.
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Feb 18, 2026Juliusz Ziomek, William Bankes, Lorenz Wolf +3LLM Evaluation LLM Planning
University of Oxford, UK · University College London (Centre for AI), UK · University of Basel, Switzerland
References Improve LLM Alignment in Non-Verifiable Domains
Feb 18, 2026Kejian Shi, Yixin Liu, Peifeng Wang +3LLM Alignment LLM-as-a-Judge
Yale University · Meta · Scale AI +2
Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments
Feb 18, 2026Yangjie Xu, Lujun Li, Lama Sleem +7Small Language Models LLM Agent Skill Retrieval
University of Luxembourg · Foyer S.A. · Princeton University +1
AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models
Feb 18, 2026Adib Sakhawat, Fardeen SadabLLM Evaluation Persuasion in Language Models
Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23
IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
Feb 18, 2026Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap +1Education LLM Evaluation
Department of Computer Sciencce and Engineering, Chhattisgarh Swami Vivekanand Technical University Bhilai, India
LIDS: LLM Summary Inference Under the Layered Lens
Feb 18, 2026Dylan Park, Yingying Fan, Jinchi LvSummarization Evaluation LLM Evaluation
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Data Sciences and Operations Department, Marshall School of Business, University of Southern California, Los Angeles, CA 90089
Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
Feb 18, 2026Sanket Badhe, Deep Shah, Nehal KathrotiaFactual Knowledge in Language Models Long-Tail Learning
Google, Mountain View, California, USA
The Validity of Coreference-based Evaluations of Natural Language Understanding
Feb 18, 2026Ian PoradaLanguage Model Generalization Coreference Resolution
School of Computer Science, McGill University, Montreal, Quebec
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1
What Is Missing: Interpretable Ratings for Large Language Model Outputs
Feb 17, 2026Nicholas Stranges, Yimin YangLLM Evaluation Language Model Post-Training
Department of Electrical and Computer Engineering, Western University, Canada

Feb 21, 2026J/K move · Enter open · S save

From Human-Level AI Tales to AI Leveling Human Scales

Peter Romero, Fernando Martínez-Plumed, Zachary R. Tidler, Matthieu Téhénan+10

Valencian Research Institute of Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · The Psychometrics Centre, University of Cambridge, UK · Georgia Institute of Technology+6

LLM Evaluation

Open paper AskPDF arXiv

Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose a framework that calibrates items against the 'world population' and report performance on a common, human-anchored scale. Concretely, we build on a set of multi-level scales for different capabilities where each level should represent a probability of success of the whole world population on a logarithmic scale with a base

B

. We calibrate each scale for each capability (reasoning, comprehension, knowledge, volume, etc.) by compiling publicly released human test data spanning education and reasoning benchmarks (PISA, TIMSS, ICAR, UKBioBank, and ReliabilityBench). The base

B

is estimated by extrapolating between samples with two demographic profiles using LLMs, with the hypothesis that they condense rich information about human populations. We evaluate the quality of different mappings using group slicing and post-stratification. The new techniques allow for the recalibration and standardization of scales relative to the whole-world population.