LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,446 papers231 papers in the last four weeks, down 17% on the four weeks before. 1.5% of all new papers.

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models
Apr 14, 2026Han Bao, Penghao Zhang, Yue Huang +9LLM Evaluation Mixture-of-Experts Language Models
University of Notre Dame · Independent Researcher
Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Apr 14, 2026Eliya Habba, Itay Itzhak, Asaf Yehudai +5LLM Evaluation Benchmark Design
The Hebrew University of Jerusalem · Technion · IBM Research +2
DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant
Apr 14, 2026Lev Sorokin, Ivan Vasilev, Samuele PasiniLLM Evaluation Language Model Error Detection
BMW Group · Technical University of Munich, Munich, Germany · Università della Svizzera italiana, Lugano, Switzerland
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
Apr 14, 2026Yuangang Li, Justin Tian Jin Chen, Ethan Yu +2Language Model Reasoning Evaluation LLM Evaluation
University of California, Irvine
CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems
Apr 14, 2026Jingbo Yang, Guanyu Yao, Bairu Hou +5LLM Evaluation LLM-as-a-Judge
University of California, Santa Barbara · Cresta
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
Apr 14, 2026Zaoyu Chen, Jianbo Dai, Boyu Zhu +6LLM Evaluation Software Engineering Benchmarks
The Hong Kong Polytechnic University · University of Edinburgh · UCL +5
LLM-Enhanced Log Anomaly Detection: A Comprehensive Benchmark of Large Language Models for Automated System Diagnostics
Apr 14, 2026Disha PatelAnomaly Detection Online Anomaly Detection
Department of Computer Science, California State University, Fullerton
Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities
Apr 14, 2026Xu Zhang, Xudong Gong, Jiacheng Qin +5LLM Evaluation Cognitive Diagnostic Models
College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan, China · State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China · School of Humanities and Social Sciences, School of Public Administration, Beihang University, Beijing, China +1
Evaluating Relational Reasoning in LLMs with REL
Apr 14, 2026Lukas Fesser, Yasha Ektefaie, Ada Fang +2Relational Reasoning LLM Evaluation
Harvard University · Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard
LLMs Struggle with Abstract Meaning Comprehension More Than Expected
Apr 13, 2026Hamoud Alhazmi, Jiachen JiangLLM Evaluation
Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, USA
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM Evaluation Language Model Generation Evaluation
University of Texas at Austin
NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions
Apr 13, 2026Shizheng Hou, Wenqi Pei, Nuo Chen +3Text-to-SQL Benchmark Design
National University of Singapore · Zhejiang University
A Robust Evaluation of Probe Robustness: Lessons for Reliable OOD Uncertainty Quantification
Apr 13, 2026Joe Stacey, Hadas Orgad, Kentaro Inui +2LLM Evaluation Uncertainty Quantification
University of Sheffield · Kempner Institute at Harvard University · MBZUAI +2
Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks
Apr 13, 2026Yuqing Yang, Tengxiao Liu, Wang Bill Zhu +3LLM Input Optimization LLM Memory
University of Southern California · University of California, Santa Barbara
Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking
Apr 13, 2026Solomon MessingLLM Evaluation
ML Commons
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
Apr 13, 2026Wenqing Wu, Yi Zhao, Yuzhuo Wang +4LLM Evaluation Scholarly Peer Review
School of Economics and Management, Nanjing University of Science and Technology · School of Electronic Engineering and Computer Science, Queen Mary University of London · School of Management, Anhui University
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
Apr 13, 2026Pengfeng Li, Chen Huang, Chaoqun Hao +4Causal Reasoning in Language Models LLM Evaluation
College of Computer Science, Sichuan University · Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China · Institute of Data Science, National University of Singapore
LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments
Apr 12, 2026Maria Camporese, Fabio Massacci, Yuanjun GongLLM-Assisted Annotation LLM Evaluation
University of Trento (Italy) · Vrije Universiteit Amsterdam (The Netherlands)
CheeseBench: Evaluating Large Language Models on Rodent Behavioral Neuroscience Paradigms
Apr 12, 2026Zacharie BugaudLLM Evaluation LLM Agents
Astera Institute
Benchmarking Open-Weight Foundation Models for Global AI Technical Governance
Apr 12, 2026Jason HungJurisdictional Bias in Language Models LLM Evaluation
Internet Society
Principles Do Not Apply Themselves: A Hermeneutic Perspective on AI Alignment
Apr 12, 2026Behrooz RazeghiAI Alignment LLM Alignment
School of Engineering and Applied Sciences, Harvard University
Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework
Apr 12, 2026Avi-ad Avraam BuskilaLLM Evaluation LLM Reliability
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel
How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks
Apr 12, 2026Johin Johny ArimburLLM Self-Correction LLM Code Generation
Independent Researcher
Handle with CARE: Can LLMs Reproduce How Online Communities React?
Apr 12, 2026Nuan Wen, Chanbin Lim, Xuezhe MaLarge Language Model-Based Social Simulation LLM Evaluation
Information Sciences Institute University of Southern California
Why Don't You Know? Evaluating the Impact of Uncertainty Sources on Uncertainty Quantification in LLMs
Apr 12, 2026Maiya Goloburda, Roman Vashurin, Fedor Chernogorskii +4Uncertainty Quantification LLM Uncertainty Estimation
Mohamed bin Zayed University of Artificial Intelligence
TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale
Apr 11, 2026Malgorzata Gwiazda, Yifu Cai, Mononito Goswami +2Time Series Reasoning LLM Evaluation
Technical University of Munich, Munich, Germany · Auton Lab, School of Computer Science, Carnegie Mellon University, Pittsburgh, US
AI Achieves a Perfect LSAT Score
Apr 11, 2026Bonmu KuLLM Reasoning LLM Evaluation
The Rise and Fall of $G$ in AGI
Apr 10, 2026David C. KrakauerArtificial General Intelligence Generative AI Evaluation
Santa Fe Institute, 1399 Hyde Park Road, Santa Fe, NM. 87501. USA
BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
Apr 10, 2026Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe +2Language Model Generation Evaluation Generative AI Evaluation
Artefact Research Center · MICS, Centrale-Supélec, Université Paris-Saclay · Diabolocom +1
Persona-E $^2$ : A Human-Grounded Dataset for Personality-Shaped Emotional Responses to Textual Events
Apr 10, 2026Yuqin Yang, Haowu Zhou, Haoran Tu +7Affective Computing Emotion Recognition
School of Future Technology, South China University of Technology, Guangzhou, China

Apr 14, 2026J/K move · Enter open · S save

PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

Han Bao, Penghao Zhang, Yue Huang, Zhengqing Yuan+8

University of Notre Dame · Independent Researcher

LLM Evaluation Mixture-of-Experts Language Models

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present PolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) Memorization: factual recall of policy knowledge, (2) Understanding: conceptual and contextual reasoning, and (3) Application: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose PolicyMoE, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models.