LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

LLM Benchmark Datasets Should Be Contamination-Resistant
May 19, 2026Ali Al-Lawati, Jason Lucas, Dongwon Lee +1LLM Evaluation Benchmark Contamination
The Pennsylvania State University, University Park, PA, USA.
LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation
May 19, 2026Shanshan Xu, Johan Lindholm, Amogh Raina +2Legal NLP LLM Evaluation
University of Copenhagen · Umeå University
Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
May 19, 2026Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad +1Mathematical Reasoning Benchmarks LLM Evaluation
School of Electrical Engineering and Computer Science, National University of Sciences and Technology, Islamabad, Pakistan · Department of Communication, Quality Management and Information Systems, Mid Sweden University, Östersund Campus, Sweden · School of Computing, Data, and Mathematical Sciences, Western Sydney University, Indonesia
K-Quantization and its Impact on Output Performance
May 19, 2026Robin Baki Davidsson, Pierre NuguesLLM Quantization LLM Evaluation
Lund University · Lund, Sweden
LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening
May 19, 2026Ming Zhang, Qiyuan Peng, Yinxi Wei +13Logical Reasoning LLM Evaluation
Institute of Trustworthy Embodied Artificial Intelligence, Fudan University · 2Hunyuan Team, Tencent · School of Philosophy, Fudan University
Beyond Fixed Budgets: Characterizing the Inelasticity and Limitations of Tree-of-Thought Reasoning Strategies
May 19, 2026Atkia Mahila, Avinash Maurya, M. Mustafa Rafique +1LLM Evaluation Inference-Time Search
Rochester Institute of Technology Rochester, USA · Argonne National Laboratory Lemont, USA
The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility
May 19, 2026David Pape, Jonathan Evertz, Lea SchönherrLLM Evaluation LLM Inference
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment
May 19, 2026Grandee Lee, Yue Wang, Che Yee Lye +1LLM Evaluation Educational Assessment
Singapore University of Social Sciences
BLINKG: A Benchmark for LLM-Integrated Knowledge Graph Generation
May 19, 2026Carla Castedo, Enrique Iglesias, Manuel Lama +3KG Construction LLM Evaluation
Leibniz University, Hannover, Germany
SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models
May 19, 2026Yiyang Gu, Junwei Yang, Junyu Luo +15AI for Science LLM Evaluation
State Key Laboratory for Multimedia Information Processing, School of Computer Science, PKU-Anker LLM Lab, Peking University · IDEA · Zhongguancun Academy +5
HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models
May 19, 2026Emmy Liu, Varun Gangal, Michael Yu +4LLM Evaluation Hallucination in Language Models
Carnegie Mellon University · All authors are affiliated with DegenAI Labs. · Patronus AI +3
OpenCompass: A Universal Evaluation Platform for Large Language Models
May 19, 2026Maosong Cao, Kai Chen, Haodong Duan +27LLM Evaluation
Shanghai AI Laboratory
Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses
May 19, 2026Yan Wang, Ziyi Guo, Christopher McCartyRetrieval-Augmented Generation LLM Evaluation
Dept. of Urban and Regional Planning & Florida Institute for Built Environment Resilience, University of Florida, P.O. Box 115706, Gainesville, FL 32611. · Dept. of Urban and Regional Planning & Florida Institute for Built Environment Resilience, University of Florida, 1480 Inner Rd., Gainesville, FL 32611. · College of Liberal Arts and Sciences, Bureau of Economic and Business Research, University of Florida, Suite 150, 720 SW 2nd Ave., Gainesville, FL 32611.
Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering
May 19, 2026Tiejin Chen, Longchao Da, Xiaoou Liu +1LLM Evaluation Uncertainty Quantification
School of Computing and Augmented Intelligence, Arizona State University
Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency
May 18, 2026Matthew L. Smith, Jonathan P. Shock, Samuel T. Segun +2LLM Evaluation Language Model Scaling Laws
International Development Research Centre Canada · University of Cape Town South Africa · Global Center on AI Governance Canada +2
GIM: Evaluating models via tasks that integrate multiple cognitive domains
May 18, 2026Rohit Patel, Alexandre Rezende, Steven McClainLLM Evaluation Test-Time Scaling
Meta Superintelligence Labs
SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science
May 18, 2026Nithin Somasekharan, Youssef Hassan, Shiyao Lin +5AI for Science LLM Evaluation
Rensselaer Polytechnic Institute Troy, NY · University of Texas at Arlington Arlington, TX · Pacific Northwest National Laboratory Richland, WA +1
Forecasting Downstream Performance of LLMs With Proxy Metrics
May 18, 2026Arkil Patel, Siva Reddy, Marius Mosbach +1Language Model Pretraining LLM Evaluation
Mila – Quebec AI Institute & McGill University Canada · ServiceNow Research · Periodic Labs
Estimating Item Difficulty with Large Language Models as Experts
May 18, 2026Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2LLM Evaluation LLM-as-a-Judge
Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2
QSTRBench: a New Benchmark to Evaluate the Ability of Language Models to Reason with Qualitative Spatial and Temporal Calculi
May 18, 2026Anthony G. Cohn, Robert E. BlackwellSpatial Reasoning Benchmarks LLM Evaluation
School of Computer Science, The University of Leeds, Woodhouse Lane, Leeds, LS2 9JT, UK · The Alan Turing Institute, British Library, 96 Euston Road, London, NW1 2DB, UK · Tongji University, Shanghai, China
Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs
May 18, 2026Tara Azin, Yongan Yu, Raj Singh +1LLM Evaluation Pragmatic Reasoning in Language Models
Department of Cognitive Science, Carleton University · School of Computer Science, McGill University · Mila – Quebec AI Institute
TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?
May 18, 2026Jieting Xiao, Yun Lin, Huizhen Qiu +10LLM Evaluation AI Agent Benchmarks
ZTE Corporation China
Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDA
May 18, 2026Sterling Huang, Abigayle Brown, Jiyoo Noh +4LLM Evaluation LLM Compression
University of Toronto Toronto, Canada · King Mongkut’s University of Technology Thonburi Bangkok, Thailand
Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification
May 17, 2026M. Mikail Demir, M. Abdullah CanbazLegal NLP LLM Evaluation
Department of Information Science and Technology · College of Emergency Preparedness, Homeland Security, and Cybersecurity · University at Albany, SUNY
Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models
May 17, 2026Ethan TangLLM Evaluation Language Modeling
School of Computing and Augmented Intelligence · Arizona State University · Tempe, AZ 85281
CyberCorrect: A Cybernetic Framework for Closed-Loop Self-Correction in Large Language Models
May 17, 2026Yuning Wu, Yingmin Liu, Yang ShuLLM Self-Correction LLM Evaluation
School of Software, Henan University, Kaifeng, China · 2Zhejiang University, Hangzhou, China
A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
May 17, 2026Qingchuan Ma, Yuexiao Ma, Yongkang Xie +3LLM Evaluation Synthetic Benchmark Generation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Institute of Artificial Intelligence, Xiamen University
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
May 16, 2026Zhaoyue Sun, Hainiu Xu, Andero Uusberg +3LLM Evaluation Affective Computing
Department of Informatics King’s College London · Institute of Psychology University of Tartu · Department of Psychology Stanford University +1
Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
May 16, 2026Fangzhou Wu, Sandeep Silwal, Qiuyi ZhangLLM Evaluation Clustering
University of Wisconsin–Madison · 2Elorian AI
Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench
May 16, 2026Tianyu Wang, Jiajun Li, Jianghao LinLLM Evaluation Large Language Model-Based Social Simulation
Shanghai Jiao Tong University · Noumena AI

May 19, 2026, cs.LGJ/K move · Enter open · S save

LLM Benchmark Datasets Should Be Contamination-Resistant

Ali Al-Lawati, Jason Lucas, Dongwon Lee, Suhang Wang

The Pennsylvania State University, University Park, PA, USA.

LLM Evaluation Benchmark Contamination LLM Reliability Data Contamination in Language Models Benchmark Validity

Open paper AskPDF arXiv

Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark datasets are included in pretraining corpora, i.e.,

\textit{contaminated}

, which diminishes their value as reliable measures of model generalization. In this paper, we argue that benchmark datasets should be

\textit{contamination-resistant}

, i.e.,

\textit{unlearnable}

, but support

\textit{inference}

. To accomplish this, we first highlight the wide prevalence of benchmark dataset contamination and outline the properties of contamination-resistant datasets. Second, we highlight how the asymmetry between the inference and training pipelines in the Transformer architecture can be leveraged to support contamination-resistance. Third, we outline mathematical advancements to make these datasets interoperable across various LLM architectures. Based on the above, we call on the community to ensure the reliability of LLM benchmarking by: (i) advancing novel contamination-resistant methodologies, (ii) developing supporting methods and platforms, and (iii) adopting contamination-resistant benchmarks into existing evaluation pipelines.

Momentum

Related topics

Latest papers 1,643