Data Contamination in Language Models

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 39

All topics

LLM Benchmark Datasets Should Be Contamination-Resistant
May 19, 2026Ali Al-Lawati, Jason Lucas, Dongwon Lee +1LLM Evaluation Benchmark Contamination
The Pennsylvania State University, University Park, PA, USA.
Can Large Language Models Reliably Correct Errors in Low-Resource ASR? A Contamination-Aware Case Study on West Frisian
May 19, 2026Yun Hao, Reihaneh Amooie, Wietse de Vries +2Automatic Speech Recognition Data Contamination in Language Models
University of Groningen, The Netherlands
Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law
May 15, 2026Parisa Kordjamshidi, Samer Aslan, Madhavan Seshadri +2Data Contamination in Language Models Neuro-Symbolic Reasoning
Bloomberg · Michigan State University
Know2Guess: A Contamination-Aware Multi-Zone Benchmark for Knowledge-Boundary Evaluation in Large Language Models
Apr 30, 2026Renwei Meng, Bowen Zhang, Jian Wang +4LLM Evaluation LLM Auditing
Anhui University, Hefei, China
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Apr 23, 2026Harsh Kumar, Rahul Maity, Tanmay Joshi +4Language Model Pretraining LLM Backdoor Attacks
1Manipal University Jaipur · 2NIT, Karnataka · 3Pragya Lab, BITS Pilani Goa, India +2
Language Generation with Replay: A Learning-Theoretic View of Model Collapse
Mar 12, 2026Giorgio Racca, Michal Valko, Amartya SanyalModel Collapse Data Contamination in Language Models
University of Copenhagen · Isara Labs
Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training
Feb 17, 2026Kevin Wang, Hongqian Niu, Didong LiModel Collapse Generative Modeling
Department of Biostatistics, University of North Carolina at Chapel Hill
Quantifying the Effect of Test Set Contamination on Generative Evaluations
Jan 7, 2026Rylan Schaeffer, Joshua Kazdan, Baber Abbasi +8Language Model Generation Evaluation Benchmark Contamination
Stanford Computer Science · Stanford Statistics · EleutherAI +3
Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models
Mar 9, 2025Batu Guan, Xiao Wu, Yuanyuan Yuan +1Benchmark Design Data Contamination in Language Models
The Chinese University of Hong Kong · Huazhong University of Science and Technology · ETH Zurich

May 19, 2026, cs.LGJ/K move · Enter open · S save

LLM Benchmark Datasets Should Be Contamination-Resistant

Ali Al-Lawati, Jason Lucas, Dongwon Lee, Suhang Wang

The Pennsylvania State University, University Park, PA, USA.

LLM Evaluation Benchmark Contamination LLM Reliability Data Contamination in Language Models Benchmark Validity

Open paper AskPDF arXiv

Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark datasets are included in pretraining corpora, i.e.,

\textit{contaminated}

, which diminishes their value as reliable measures of model generalization. In this paper, we argue that benchmark datasets should be

\textit{contamination-resistant}

, i.e.,

\textit{unlearnable}

, but support

\textit{inference}

. To accomplish this, we first highlight the wide prevalence of benchmark dataset contamination and outline the properties of contamination-resistant datasets. Second, we highlight how the asymmetry between the inference and training pipelines in the Transformer architecture can be leveraged to support contamination-resistance. Third, we outline mathematical advancements to make these datasets interoperable across various LLM architectures. Based on the above, we call on the community to ensure the reliability of LLM benchmarking by: (i) advancing novel contamination-resistant methodologies, (ii) developing supporting methods and platforms, and (iii) adopting contamination-resistant benchmarks into existing evaluation pipelines.

Momentum

Related topics

Latest papers 39