LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

On the scaling relationship between cloze probabilities and language model next-token prediction
Feb 19, 2026Cassandra L. Jacobs, Morgan GrobolLanguage Modeling Language Model Scaling
Department of Linguistics, University at Buffalo, Buffalo, NY, USA · MoDyCo, Université Paris Nanterre, Paris, France
The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
Feb 19, 2026Simon Henniger, Gabriel PoesiaLanguage Model Reasoning Evaluation LLM Evaluation
Harvard University
ABCD: All Biases Come Disguised
Feb 19, 2026Mateusz Nowak, Xavier Cadet, Peter ChinLLM Evaluation Language Model Bias Evaluation
Dartmouth College
Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation
Feb 19, 2026Bogdan Kostić, Conor Fallon, Julian Risch +1Prompt Sensitivity LLM Evaluation
Berliner Hochschule für Technik (BHT) - Luxemburger Straße 10, 13467 Berlin · deepset GmbH - Zinnowitzer Straße 1, 10115 Berlin
CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Feb 19, 2026Lukas Thede, Stefan Winzeck, Zeynep Akata +1LLM Post-Training Catastrophic Forgetting
Thomson Reuters Foundational Research, 5 Canada Square, London · Tübingen AI Center, University of Tübingen · Helmholtz Munich +2
Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation
Feb 19, 2026Yan Wang, Yi Han, Lingfei Qian +11Recommender Systems Large Language Model-Based Recommendation
The Fin AI, USA · Georgia Institute of Technology, USA · Columbia University, USA +5
SourceBench: Can AI Answers Reference Quality Web Sources?
Feb 18, 2026Hexi Jin, Stephen Liu, Yuheng Li +2LLM Evaluation IR Evaluation
University of California, San Diego · GenseeAI Inc.
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
Feb 18, 2026Juliusz Ziomek, William Bankes, Lorenz Wolf +3LLM Evaluation LLM Planning
University of Oxford, UK · University College London (Centre for AI), UK · University of Basel, Switzerland
References Improve LLM Alignment in Non-Verifiable Domains
Feb 18, 2026Kejian Shi, Yixin Liu, Peifeng Wang +3LLM Alignment LLM-as-a-Judge
Yale University · Meta · Scale AI +2
Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments
Feb 18, 2026Yangjie Xu, Lujun Li, Lama Sleem +7Small Language Models LLM Agent Skill Retrieval
University of Luxembourg · Foyer S.A. · Princeton University +1
AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models
Feb 18, 2026Adib Sakhawat, Fardeen SadabLLM Evaluation Persuasion in Language Models
Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34LLM Evaluation Benchmark Design
ETH Zurich · ETH AI Center · Stanford University +23
IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models
Feb 18, 2026Saurabh Bharti, Gaurav Azad, Abhinaw Jagtap +1Education LLM Evaluation
Department of Computer Sciencce and Engineering, Chhattisgarh Swami Vivekanand Technical University Bhilai, India
LIDS: LLM Summary Inference Under the Layered Lens
Feb 18, 2026Dylan Park, Yingying Fan, Jinchi LvSummarization Evaluation LLM Evaluation
Department of Mathematics, University of Southern California, Los Angeles, CA 90089 · Data Sciences and Operations Department, Marshall School of Business, University of Southern California, Los Angeles, CA 90089
Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
Feb 18, 2026Sanket Badhe, Deep Shah, Nehal KathrotiaFactual Knowledge in Language Models Long-Tail Learning
Google, Mountain View, California, USA
The Validity of Coreference-based Evaluations of Natural Language Understanding
Feb 18, 2026Ian PoradaLanguage Model Generalization Coreference Resolution
School of Computer Science, McGill University, Montreal, Quebec
*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Feb 17, 2026Quentin Lemesle, Léane Jourdan, Daisy Munson +4Language Model Generation Evaluation LLM Evaluation
Univ Rennes, CNRS, IRISA, EXPRESSION, 22300 Lannion, France · Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France · Univ Rennes, CNRS, IRISA, SOTERN, 22300 Lannion, France +1
What Is Missing: Interpretable Ratings for Large Language Model Outputs
Feb 17, 2026Nicholas Stranges, Yimin YangLLM Evaluation Language Model Post-Training
Department of Electrical and Computer Engineering, Western University, Canada
Quantifying construct validity in large language model evaluations
Feb 17, 2026Ryan Othniel KearnsLLM Evaluation Latent Variable Models
St Catherine’s College
LLM-as-Judge on a Budget
Feb 17, 2026Aadirupa Saha, Aniket Wagde, Branislav KvetonLLM Evaluation LLM-as-a-Judge
UIC · Adobe
Far Out: Evaluating Language Models on Slang in Australian and Indian English
Feb 17, 2026Deniz Kaya Dilsiz, Dipankar Srirag, Aditya JoshiLLM Evaluation Cross-Cultural Language Model Evaluation
University of New South Wales, Sydney, Australia
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
Feb 17, 2026Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1Language Model Performance Prediction Language Model Scaling Laws
Harvard University · Stanford University
SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
Feb 16, 2026Srivatsa Kundurthy, Clara Na, Michael Handley +5LLM Evaluation Language Model Generation Evaluation
Longitude Labs Inc., New York, NY · Cornell University, Ithaca, NY · Carnegie Mellon University, Pittsburgh, PA +1
LLMStructBench: Benchmarking Large Language Model Structured Data Extraction
Feb 16, 2026Sönke Tenckhoff, Mario Koddenbrock, Erik RodnerLLM Information Extraction Document Parsing Benchmarks
KI-Werkstatt / FB2, University of Applied Sciences Berlin
Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents
Feb 16, 2026MZ Naser, Ahmad Bani Awwad, Zoie McCreery +8LLM Evaluation AI Agent Benchmarks
School of Civil and Environmental Engineering & Earth Sciences, Clemson University, USA · Artificial Intelligence Research Institute for Science and Engineering, Clemson University, USA · Department of Mechanical Engineering, University of Manitoba, Canada +6
Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
Feb 16, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Ekramul Alam Esham +2Medical QA Medical LLMs
Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh
Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study
Feb 16, 2026Annalisa Szymanski, Oghenemaro Anuyah, Toby Jia-Jun Li +1LLM Evaluation Human-in-the-Loop Evaluation
Computer Science and Engineering, University of Notre Dame, Notre Dame, IN, USA · Microsoft Corporation, Redmond, WA, USA
On Calibration of Large Language Models: From Response To Capability
Feb 14, 2026Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin +3LLM Evaluation Confidence Estimation in Language Models
Appier AI Research · National Taiwan University
The System Hallucination Scale (SHS): A Human-Centered Instrument for Assessing Perceived Reliability of Large Language Models
Feb 13, 2026Heimo Müller, Dominik Steiger, Markus Plass +1LLM Evaluation LLM Reliability
Machine Learning and Information Science Group, Medical University of Graz, Austria · Human Machine Mind Cooperation, Graz, Austria · MIDATA Cooperative, Zurich, Switzerland +1
Investigating Learner-Aware Design of LLM-Generated Educational Feedback
Feb 12, 2026Momoka Furuhashi, Kouta Nakayama, Noboru Kawai +3LLM Evaluation Personalized Learning
Tohoku University · Research and Development Center for Large Language Models, National Institute of Informatics · Osaka Kyoiku University +2

Feb 19, 2026J/K move · Enter open · S save

On the scaling relationship between cloze probabilities and language model next-token prediction

Cassandra L. Jacobs, Morgan Grobol

Department of Linguistics, University at Buffalo, Buffalo, NY, USA · MoDyCo, Université Paris Nanterre, Paris, France

Language Modeling Language Model Scaling Computational Psycholinguistics LLM Evaluation

Open paper AskPDF arXiv

Recent work has shown that larger language models have better predictive power for eye movement and reading time data. While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occurrence statistics while being better aligned semantically to human cloze responses. The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.