LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task
Apr 28, 2026Tosin Adewumi, Martin Karlsson, Lama Alkhaled +1LLM Evaluation
Machine Learning Group, EISLAB · Research Center for Advanced Battery Technology, Luleå University of Technology, Sweden.
QASM-Eval: A Dataset to Train and Evaluate LLMs on OpenQASM-3 Beyond Quantum Circuits
Apr 28, 2026Zhenxiao Fu, Lei Jiang, Fan ChenLLM Evaluation Code Generation
Indiana University Bloomington
CGU-ILALab at FoodBench-QA 2026: Comparing Traditional and LLM-based Approaches for Recipe Nutrient Estimation
Apr 28, 2026Wei-Chun Chen, Yu-Xuan Chen, I-Fang Chung +1LLM Evaluation Cost-Aware Inference
Institute of Biomedical Informatics, National Yang Ming Chiao Tung University, Taipei, Taiwan · Institute of Health Data Science, Chang Gung University, Taoyuan, Taiwan · Department of Artificial Intelligence, Chang Gung University, Taoyuan, Taiwan +1
Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations
Apr 28, 2026Nanxu Gong, Zixin Chen, Haotian Li +5LLM Evaluation Theory of Mind
Arizona State University · Hong Kong University of Science and Technology · Microsoft Research Asia +1
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2LLM Evaluation LLM Self-Refinement
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
Large language models eroding science understanding: an empirical study of malignment
Apr 28, 2026Harry Collins, Hartmut Grote, Paul Newbury +2LLM Evaluation Knowledge Conflicts in Language Models
i · ii · iii
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada
Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure
Apr 28, 2026David Hartmann, Manuel Tonneau, Angelie Kraft +7LLM Evaluation LLM Auditing
Weizenbaum Institute · TU Berlin · University of Oxford +9
SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials
Apr 28, 2026Zhaohui Li, Peng He, Zhiyuan Chen +4LLM Evaluation AI in Education
Department of Computer Science and Engineering, University at Buffalo, USA · Department of Teaching and Learning, Washington State University, USA · College of Chemistry, Beijing Normal University, China +1
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
Apr 28, 2026Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1LLM Evaluation LLM Information Extraction
1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India
Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models
Apr 28, 2026Michael Rumiantsau, Ivan FokeevLLM Evaluation LLM Grounding
Cube
Analyzing LLM Reasoning to Uncover Mental Health Stigma
Apr 27, 2026Sreehari Sankar, Aliakbar Nafar, Mona Barman +8LLM Evaluation Social Bias in Language Models
BetterHelp
The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models
Apr 27, 2026Yunze Xiao, Vivienne J. Zhang, Chenghao Yang +2LLM Evaluation Personality Modeling in Language Models
CMU · UChicago · MIT +2
Can LLMs Act as Historians? Evaluating Historical Research Capabilities of LLMs via the Chinese Imperial Examination
Apr 27, 2026Lirong Gao, Zeqing Wang, Yuyan Cai +6LLM Evaluation Question Answering
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Ant Group · Zhejiang University
Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation
Apr 27, 2026Sercan Karakaş, Yusuf ŞimşekLLM Evaluation
University of Chicago · Fırat University
K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology
Apr 27, 2026Soyeon Kim, Cheongwoong Kang, Myeongjin Lee +3LLM Evaluation Cultural Bias in Language Models
KAIST, INEEJI · Seongnam, Korea · KAIST +2
Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity
Apr 27, 2026Bojie LiParameter Estimation LLM Evaluation
Pine AI
STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator
Apr 27, 2026Alessio Sordo, Lingxiao Du, Meeka-Hanna Lenisa +2LLM Evaluation Synthetic Data Generation
1Deutsche Bank, Berlin Technology Center, Berlin, Germany
Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora
Apr 27, 2026Chenkai Pan, Xinglong Xu, Yuhang Xu +6LLM Evaluation Knowledge Augmentation for Language Models
Zhejiang University · University of Chinese Academy of Sciences · Shanghai Artificial Intelligence Laboratory
Can You Make It Sound Like You? Post-Editing LLM-Generated Text for Personal Style
Apr 27, 2026Connor Baumler, Calvin Bao, Huy Nghiem +3LLM Evaluation Controllable Text Generation
University of Maryland
SWE-QA: A Dataset and Benchmark for Complex Code Understanding
Apr 27, 2026Laïla Elkoussy, Julien PerezLLM Evaluation Software Engineering Benchmarks
LRE, EPITA · LRE, EPITA* · EPITA, LRE +1
Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation
Apr 27, 2026Zekun Yuan, Yangfan Ye, Xiaocheng Feng +5LLM Evaluation Cultural Knowledge in Language Models
Harbin Institute of Technology · Peng Cheng Laboratory · Huawei Technologies Co., Ltd
Multi-Dimensional Evaluation of Sustainable City Trips with LLM-as-a-Judge and Human-in-the-Loop
Apr 27, 2026Ashmi Banerjee, Adithi Satish, Wolfgang Wörndl +1LLM Evaluation LLM-as-a-Judge
Technical University of Munich Munich, Germany · Polytechnic University of Bari Bari, Italy
An Information-Geometric Framework for Stability Analysis of Large Language Models under Entropic Stress
Apr 27, 2026Hikmat Karimov, Rahid Zahid AlekberliLLM Evaluation LLM Reliability
Institute of Defense Technologies and Cybersecurity„ Azerbaijan Technical University, Azerbaijan
How LLMs Fail and Generalize in RTL Coding for Hardware Design?
Apr 26, 2026Guan-Ting Liu, Chao-Han Huck Yang, Chenhui Deng +3LLM Evaluation Code Generation
NVIDIA Research
Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task
Apr 26, 2026Jungmin Choi, Keisuke Sakaguchi, Hiroaki YamadaLLM Evaluation Hallucination in Language Models
RIKEN · Tohoku University · Tokyo Metropolitan University
Benchmarking Testing in Automated Theorem Proving
Apr 26, 2026Jongyoon Kim, Hojae Han, Seung-won HwangLLM Evaluation Automated Theorem Proving
Interdisciplinary Program in Artificial Intelligence, Seoul National University1 · Electronics and Telecommunications Research Institute2
The Collapse of Heterogeneity in Silicon Philosophers
Apr 26, 2026Yuanming Shi, Andreas HauptLLM Evaluation
Adobe Inc., San Jose, California, USA · Stanford University, Departments of Economics and Computer Science, Stanford, California, USA
JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Apr 26, 2026Rohith Reddy Bellibatlu, Edward Raff, Wenbin ZhangLLM Evaluation LLM-as-a-Judge
Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA
Automating Categorization of Scientific Texts with In-Context Learning and Prompt-Chaining in Large Language Models
Apr 25, 2026Gautam Kishore Shahi, Oliver HummelHierarchical Classification LLM Evaluation
Technische Hochschule Mannheim, Germany

Apr 28, 2026J/K move · Enter open · S save

BatteryPass-12K: The First Dataset for the Novel Digital Battery Passport Conformance Task

Tosin Adewumi, Martin Karlsson, Lama Alkhaled, Marcus Liwicki

Machine Learning Group, EISLAB · Research Center for Advanced Battery Technology, Luleå University of Technology, Sweden.

LLM Evaluation

Open paper AskPDF arXiv

We introduce a novel task of digital battery passport (DBP) conformance classification and introduce the first public benchmark for the task: BatteryPass-12K, created synthetically from real pilot samples. This is as the EU's battery regulation on DBPs comes into effect soon and there exists no public dataset. We evaluated 22 language models (LMs) in zero-shot inference, spanning small LMs (SLMs), mixture of experts (MoEs), and dense LLMs. We also conducted analysis, additional evaluations of few-shot inference and prompt-injection attacks to find that (1) Thinking models have the best performance (with GPT-5.4 scoring 0.98 (0.03) and 0.71 (0.22) on average as F1 (and confidence interval at 95%) on the validation and test sets, respectively), (2) few-shot examples improve performance significantly, (3) generally capable frontier models find the task challenging, (4) merely scaling model parameters does not necessarily lead to improved performance, as SLMs outperformed some LLMs, and (5) prompt-injection attacks degrade performance. We note that BatteryPass-12K, though limited to real pilot samples, may be useful for other known or emerging tasks in the battery domain, e.g. lifecycle reasoning. We publicly release the dataset under a permissive licence (CC-BY-4.0).