LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

Brittlebench: Quantifying LLM robustness via prompt sensitivity
Feb 27, 2026Angelika Romanou, Mark Ibrahim, Candace Ross +8Prompt Sensitivity LLM Evaluation
FAIR at Meta · EPFL · Northeastern University
DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science
Feb 27, 2026Fan Shu, Yite Wang, Ruofan Wu +4LLM Evaluation Instruction Following
University of Houston · Snowflake AI Research
MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games
Feb 27, 2026Jacob Eisenstein, Fantine Huot, Adam Fisch +2LLM Evaluation Multi-Turn Dialogue Evaluation
Google DeepMind
The Subjectivity of Monoculture
Feb 27, 2026Nathanael Jo, Nikhil Garg, Manish RaghavanLLM Evaluation
MIT, Cambridge, USA · Cornell Tech, New York, USA
Task Complexity Matters: An Empirical Study of Reasoning in LLMs for Sentiment Analysis
Feb 27, 2026Donghao Huang, Zhaoxia WangEfficient Language Model Reasoning LLM Evaluation
School of Computing and Information Systems, Singapore Management University, 80 Stamford Rd, Singapore 178902, Singapore · Research and Development, Mastercard, 4250 Fairfax Dr, Arlington, VA 22203, USA
A unified foundational framework for knowledge injection and evaluation of Large Language Models in Combustion Science
Feb 27, 2026Zonglin Yang, Runze Mao, Tianhao Wu +3Knowledge Augmentation for Language Models LLM Evaluation
School of Mechanics and Engineering Science, Peking University, Beijing, 100871, PRChina · AI for Science Institute, Beijing, 100080, China
AIDABench: AI Data Analytics Benchmark
Feb 27, 2026Yibo Yang, Fei Lei, Yixuan Sun +24LLM Evaluation Benchmark Design
SenseTime Research · Shanghai AI Laboratory · MMLab, The Chinese University of Hong Kong
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Feb 27, 2026Seungdong Yoa, Sanghyu Yoon, Suhee Yoon +4Language Model Reasoning Evaluation LLM Evaluation
LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies
LFQA-HP-1M: A Large-Scale Human Preference Dataset for Long-Form Question Answering
Feb 27, 2026Rafid Ishrak Jahan, Fahmid Shahriar Iqbal, Sagnik Ray ChoudhuryLLM Evaluation LLM-as-a-Judge
University of North Texas, Denton, Texas, USA
Generative Active Testing: Efficient LLM Evaluation via Proxy Task Adaptation
Feb 26, 2026Aashish Anantha Ramakrishnan, Ardavan Saeedi, Hamid Reza Hassanzadeh +2LLM Evaluation Uncertainty Sampling
The Pennsylvania State University · Optum AI
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Feb 26, 2026Jiahao Zhao, Feng Jiang, Shaowei Qin +5LLM Evaluation Scientific Reasoning in Language Models
Software College, Northeastern University, China · Shenzhen University of Advanced Technology, Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China +2
SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy
Feb 26, 2026Peiyao Xiao, Xiaogang Li, Xinyi Gao +7LLM Evaluation Multimodal Model Evaluation
Alibaba Group Beijing, China
Extending Czech Aspect-Based Sentiment Analysis with Opinion Terms: Dataset and LLM Benchmarks
Feb 26, 2026Jakub Šmíd, Pavel Přibáň, Pavel KrálAspect-Based Sentiment Analysis Low-Resource Language Processing
Department of Computer Science and Engineering, University of West Bohemia in Pilsen, Faculty of Applied Sciences, Univerzitní 2732/8, 301 00 Pilsen, Czech Republic · NTIS – New Technologies for the Information Society, University of West Bohemia in Pilsen, Faculty of Applied Sciences, Univerzitní 2732/8, 301 00 Pilsen, Czech Republic
CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models
Feb 25, 2026Miyu Oba, Saku SugawaraLLM Evaluation
Nara Institute of Science and Technology · National Institute of Informatics · The University of Tokyo
Algorithmic Blindness in Large Language Models: A Calibration Study of Performance Prediction
Feb 25, 2026Sohan Venkatesh, Ashish Mahendran Kurapath, Tejas MelkoteAlgorithmic Reasoning LLM Evaluation
Manipal Institute of Technology Bengaluru
Small Wins Big: Comparing Large Language Models and Domain Fine-Tuned Models for Sarcasm Detection in Code-Mixed Hinglish Text
Feb 25, 2026Bitan Majumder, Anirban SenSarcasm Detection Small Language Models
Pondicherry University · Ashoka University
FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting
Feb 25, 2026Yiyun Zhu, Yidong Jiang, Ziwen Xu +4Financial Reasoning in Language Models Financial Reasoning Benchmarks
School of Computer Science and Technology, Tongji University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China
A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2
Feb 25, 2026Marcin Pietroń, Filip Gampel, Jakub Gomułka +2Argument Mining Text Classification
Faculty of Computer Science, Electronics, and Telecommunications AGH University of Krakow · Faculty of Humanities AGH University of Krakow · Institute of Mathematics University of Silesia
FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluation
Feb 25, 2026Xiyuan Zhang, Huihang Wu, Jiayu Guo +8Financial Language Model Evaluation LLM Evaluation
Du Xiaoman Technology · The PBC School of Finance, Tsinghua University · The School of Finance, Renmin University of China
Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document Streams
Feb 25, 2026Yukyung Lee, Yebin Lim, Woojun Jung +2LLM Evaluation Long-Context Language Model Evaluation
Boston University, Boston, USA · Korea University, Seoul, Korea
HEARTS: Benchmarking LLM Reasoning on Health Time Series
Feb 25, 2026Sirui Li, Shuhan Xiao, Mihir Joshi +4LLM Evaluation Time Series Reasoning
University of California, Los Angeles, 2Google Research · 2Google Research
Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?
Feb 25, 2026Germán T. Eizaguirre, Lars Tissen, Marc Sánchez-ArtigasText-to-SQL LLM Evaluation
Universitat Rovira i Virgili Tarragona, Spain · RWTH Aachen University Aachen, Germany
Robust AI Evaluation through Maximal Lotteries
Feb 24, 2026Hadi Khalaf, Serena L. Wang, Daniel Halpern +3LLM Evaluation Rank Aggregation
Harvard University
An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
Feb 24, 2026Anna Martin-Boyle, William Humphreys, Martha Brown +2LLM Evaluation Language Model Error Detection
University of Minnesota Minneapolis, United States · NASA Langley Research Center Hampton, Virginia, United States
SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts
Feb 24, 2026Qingsong Zou, Zhi Yan, Zhiyao Xu +3Contextual Anomaly Detection Anomaly Detection
Tsinghua Shenzhen International Graduate School Shenzhen, China · Jilin University Jilin, China · The Chinese University of Hong Kong Hong Kong, China +1
LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification
Feb 24, 2026Yanrui Wu, Lingling Zhang, Xinyu Zhang +5Logical Reasoning LLM Evaluation
School of Computer Science and Technology, Xi’an Jiaotong University · Ministry of Education Key Laboratory of Intelligent Networks and Network Security, China · Department of Electrical Engineering, Stanford University +2
Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models
Feb 24, 2026Paola Merlo, Chunyang Jiang, Giuseppe Samo +1LLM Evaluation LLM Interpretability
Idiap Research Institute · University of Geneva
Predicting Sentence Acceptability Judgments in Multimodal Contexts
Feb 24, 2026Hyewon Jang, Nikolai Ilinykh, Sharid Loáiciga +2LLM Evaluation Multimodal Model Evaluation
University of Gothenburg · University of Melbourne · Queen Mary University of London +1
Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
Feb 24, 2026Nora Petrova, John BurdenLLM Alignment LLM Safety Benchmarks
Prolific
Qwen-BIM: developing large language model for BIM-based design with domain-specific benchmark and dataset
Feb 24, 2026Jia-Rui Lin, Yun-Hong Cai, Xiang-Rui Ni +2Benchmark Design LLM Evaluation
Department of Civil Engineering, Tsinghua University, Beijing, China, 100084 · Key Laboratory of Digital Construction and Digital Twin, Ministry of Housing and Urban-Rural Development, Beijing, China, 100084 · PetroChina Planning & Engineering Institute, Beijing, China, 100083

Feb 27, 2026J/K move · Enter open · S save

Brittlebench: Quantifying LLM robustness via prompt sensitivity

Angelika Romanou, Mark Ibrahim, Candace Ross, Kerem Oktar+7

FAIR at Meta · EPFL · Northeastern University

Prompt Sensitivity LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as 12%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.