LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

Pipeline for Verifying LLM-Generated Mathematical Solutions
Feb 24, 2026Varvara Sazonova, Dmitri Shmelkin, Stanislav Kikot +1LLM Answer Verification LLM Evaluation
Moscow State University · Huawei Technologies Co., Ltd
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
Feb 24, 2026Santiago Gonzalez, Alireza Amiri Bavandpour, Peter Ye +48Mathematical Reasoning Benchmarks LLM Evaluation
From Performance to Purpose: A Sociotechnical Taxonomy for Evaluating Large Language Model Utility
Feb 24, 2026Gavin Levinson, Keith FeldmanLLM Evaluation
University of Michigan, Ann Arbor, Michigan, USA
Disentangling Geometry, Performance, and Training in Language Models
Feb 24, 2026Atharva Kulkarni, Jacob Mitchell Springer, Arjun Subramonian +1LLM Evaluation Neural Representation Geometry
University of Southern California · 2Carnegie Mellon Univer · University of California, Los Angeles.
Three Concrete Challenges and Two Hopes for the Safety of Unsupervised Elicitation
Feb 23, 2026Callum Canavan, Aditya Shrivastava, Allison Qi +2LLM Reliability Language Model Safety Evaluation
MATS · Anthropic Fellows Program · Anthropic
Do Large Language Models Understand Data Visualization Rules?
Feb 23, 2026Martin Sinnona, Valentin Bonas, Emmanuel Iarussi +1Data Visualization LLM Evaluation
Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina
KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration
Feb 23, 2026Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari +3Multiple-Choice Question Generation LLM Evaluation
University of Tehran · Independent Researcher · Amirkabir University of Technology +1
CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
Feb 23, 2026Yuzhe Wang, Yaochen Zhu, Jundong LiLLM Evaluation Causal Reasoning in Language Models
University of Virginia Charlottesville, Virginia, USA
Do Large Language Models Understand Data Visualization Principles?
Feb 23, 2026Martin Sinnona, Valentin Bonas, Viviana Siless +1LLM Evaluation VLM Evaluation
Universidad Torcuato Di Tella, Buenos Aires, Argentina · Universidad de Buenos Aires, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina
AI Alignment Breaks at the Edge
Feb 23, 2026Han Bao, Yue Huang, Xiaoda Wang +5LLM Alignment AI Safety Evaluation
University of Notre Dame · Emory University
Watson & Holmes: A Naturalistic Benchmark for Comparing Human and LLM Reasoning
Feb 23, 2026Thatchawin Leelawat, Lewis D GriffinInductive Reasoning in Language Models Reasoning Evaluation
Computer Science, University College London, UK
KGHaluBench: A Knowledge Graph-Based Hallucination Benchmark for Evaluating the Breadth and Depth of LLM Knowledge
Feb 23, 2026Alex Robertson, Huizhi Liang, Mahbub Gani +2Hallucination in Language Models LLM Evaluation
School of Computing, Newcastle University · Sage Ai, Sage Group PLC · Current affiliation: Redis
Classroom Final Exam: An Instructor-Tested Reasoning Benchmark
Feb 23, 2026Chongyang Gao, Diji Yang, Shuyan Zhou +4MLLM Reasoning LLM Evaluation
Northwestern University · UC Santa Cruz · Duke University +3
Can Large Language Models Replace Human Coders? Introducing ContentBench
Feb 23, 2026Michael HamanLLM-Assisted Annotation LLM Evaluation
Department of Humanities, Faculty of Economics and Management, Czech University of Life Sciences Prague, Czech Republic
Personalized Prediction of Perceived Message Effectiveness Using Large Language Model Based Digital Twins
Feb 23, 2026Jasmin Han, Janardan Devkota, Joseph Waring +8Digital Twins Personalized Language Models
Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA · Department of Health Sciences, Towson University, Towson, USA · Addiction Research Group, University of East Anglia, Norwich, UK +8
Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
Feb 22, 2026Maciej Świechowski, Adam Żychowski, Jacek MańdziukLLM Reasoning LLM Evaluation
Grail Team Warsaw, Poland · Warsaw University of Technology Warsaw, Poland · AGH University of Krakow Cracow, Poland
DeliberationBench: A Normative Benchmark for the Influence of Large Language Models on Users' Views
Feb 22, 2026Luke Hewitt, Maximilian Kroner Dale, Paul de Font-ReaulxLLM Evaluation Persuasion in Language Models
Stanford University · University of Oxford · University of Michigan
Early Evidence of Vibe-Proving with Consumer LLMs: A Case Study on Spectral Region Characterization with ChatGPT-5.2 (Thinking)
Feb 21, 2026Brecht Verbeken, Brando Vagenende, Marie-Anne Guerry +2Large Language Model-Guided Theorem Proving LLM Mathematical Reasoning
Data Analytics Lab, Vrije Universiteit Brussel, Pleinlaan 5, 1050 Brussel, Belgium · imec-SMIT, Vrije Universiteit Brussel, Pleinlaan 9, 1050 Brussels, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
From Human-Level AI Tales to AI Leveling Human Scales
Feb 21, 2026Peter Romero, Fernando Martínez-Plumed, Zachary R. Tidler +11LLM Evaluation
Valencian Research Institute of Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · The Psychometrics Centre, University of Cambridge, UK · Georgia Institute of Technology +6
EvalSense: A Framework for Domain-Specific LLM (Meta-)Evaluation
Feb 21, 2026Adam Dejl, Jonathan PearsonLLM Evaluation Language Model Generation Evaluation
Imperial College London* Department of Computing · NHS England Transformation Directorate
BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models
Feb 21, 2026Thura Aung, Jann Railey Montalan, Jian Gang Ngui +1LLM Evaluation Low-Resource Language Processing
King Mongkut’s Institute of Technology Ladkrabang · AI Singapore · National University of Singapore
ArabicNumBench: Evaluating Arabic Number Reading in Large Language Models
Feb 21, 2026Anas Alhumud, Abdulaziz Alhammadi, Muhammad Badruddin KhanArabic NLP LLM Evaluation
Saudi Data and Artificial Intelligence Authority, Saudi Arabia · Imam Mohammad Ibn Saud Islamic University, Saudi Arabia
Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools
Feb 20, 2026Baris Arat, Emre SeferLLM Evaluation LLM Reranking
Ozyegin University Istanbul, Turkey
Luna-2: Scalable Single-Token Evaluation with Small Language Models
Feb 20, 2026Vatsal Goel, Rishon Dsouza, Nikhil Ega +4LLM Evaluation Language Model Safety Evaluation
Galileo AI, San Francisco, USA
Validating Political Position Predictions of Arguments
Feb 20, 2026Jordan Robinson, Angus R. Williams, Katie Atkinson +1LLM Evaluation Computational Argumentation
University of Liverpool · The Alan Turing Institute · University of Leeds
VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
Feb 20, 2026Yutong Xin, Qiaochu Chen, Greg Durrett +1LLM Evaluation Formal Verification
The University of Texas at Austin · New York University
[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games
Feb 20, 2026Jorge Carrasco Pollo, Ioannis Kapetangeorgis, Joshua Rosenthal +1LLM Evaluation AI Agent Benchmarks
Informatics Institute University of Amsterdam
Capabilities Ain't All You Need: Measuring Propensities in AI
Feb 20, 2026Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11LLM Evaluation Item Response Theory
Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5
Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm
Feb 20, 2026Anna Babarczy, Andras Lukacs, Peter Vedres +1Theory of Mind Social Reasoning
Institute of General and Hungarian Linguistics, Research Centre for Linguistics · Department of Cognitive Science, Faculty of Natural Sciences, Budapest University of Technology and Economics · Institute of Mathematics, Eötvös Lóránd University
Towards More Standardized AI Evaluation: From Models to Agents
Feb 20, 2026Ali El Filali, Inès BedarAI Agent Evaluation LLM Evaluation
G42 · Independent

Feb 24, 2026J/K move · Enter open · S save

Pipeline for Verifying LLM-Generated Mathematical Solutions

Varvara Sazonova, Dmitri Shmelkin, Stanislav Kikot, Vasily Motolygin

Moscow State University · Huawei Technologies Co., Ltd

LLM Answer Verification LLM Evaluation LLM Mathematical Reasoning Formal Verification Interactive Theorem Proving

Open paper AskPDF arXiv

With the growing popularity of Large Reasoning Models and their results in solving mathematical problems, it becomes crucial to measure their capabilities. We introduce a pipeline for both automatic and interactive verification as a more accurate alternative to only checking the answer which is currently the most popular approach for benchmarks. The pipeline can also be used as a generator of correct solutions both in formal and informal languages. 3 AI agents, which can be chosen for the benchmark accordingly, are included in the structure. The key idea is the use of prompts to obtain the solution in the specific form which allows for easier verification using proof assistants and possible use of small models (

\le 8B

). Experiments on several datasets suggest low probability of False Positives. The open-source implementation with instructions on setting up a server is available at https://github.com/LogicEnj/lean4_verification_pipeline.