Benchmark Contamination

Follow this topic Ask about this topic

Benchmark contamination studies how exposure to test examples, answers, or recognizable cues can distort model evaluations and claims of generalization. Researchers use exact and semantic overlap audits, controlled exposure with matched clean controls, and behavioral probes that compare performance on original items with paraphrased, temporal, or freshly generated variants. These methods are applied to settings such as language-model reasoning and coding benchmarks, as well as medical and scientific datasets; mitigation includes sealed or live evaluations and careful data decontamination. No single audit is reliable across settings: detectors can miss generalized leakage or raise false alarms, while aggressive filtering can discard clean data, leaving a tension between transparent, reusable benchmarks and evaluations resistant to exposure.

Latest papers

49 papers8 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Adversarial Benchmarks and BenchDoor Attacks: Exploiting Benchmark Incentives for Backdoor Attacks
Oct 20, 2026Yixin Wu, Rachel Hong, Jamie Morgenstern +1Data Poisoning Attacks LLM Backdoor Attacks
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · University of Washington +1
Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
Oct 18, 2026Vahid Jebraeeli, Jonathon Romero, Yijin Wang +2Language Model Safety Evaluation LLM Safety Benchmarks
Lenovo
Harness-Delta Attribution: Dissecting Gains From Harness Evolution
Oct 15, 2026Wenxuan Ding, Qin Lu, Changlong Yu +5Agent Harness Optimization Agent Evaluation
Amazon · New York University
What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination
Oct 15, 2026Esther XinData Contamination in Language Models Benchmark Contamination
Independent Researcher
Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch Agents
Oct 12, 2026Cong Pang, Jiaqi Su, Chengrui Wu +14Deep Research Agents LLM Agent Training
ShanghaiTech · SenseTime · NUS +6
PATTERN Can Be Solved from Memory: Generator-Aware Baselines for Synthetic Node Classification
Oct 10, 2026Amin Najafgholizadeh, Mohsen AfsharchiBenchmark Contamination Synthetic Data Evaluation
University of Zanjan
Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation
Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark Auditing AI Agent Auditing
Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1
Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classification
Sep 30, 2026Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi +1Data Leakage Benchmark Contamination
Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO 65211 USA · Government Degree College (Autonomous), Siddipet, Telangana 502103, India · Amar Biotech Private Limited, Kondapur, Hyderabad, Telangana 500084, India
Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks
Sep 27, 2026Xiaojing Sun, Yuhan Zeng, Zihua She +1Benchmark Contamination Automated Evaluation
Purdue University
How Reusable Are Benchmarks with Richer Feedback?
Sep 26, 2026Youssef Allouah, John DuchiBenchmark Validity LLM Evaluation
Department of Computer Science Stanford University · Departments of Statistics and Electrical Engineering Stanford University
LeakScale: Estimating the Causal Effect of Benchmark Exposure
Sep 23, 2026Divyansh SinghCausal Effect Estimation Benchmark Design
University of Florida
The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark
Sep 9, 2026Soumyadeep RoyBenchmark Auditing Benchmark Contamination
Department of Data Science, St. Xavier’s College (Autonomous), Kolkata, 700016, India.
Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation
Aug 29, 2026Johanna Angulo, Víctor Yeste, Hector Espinos-MoratoBenchmark Contamination Benchmark Validity
Universidad Europea de Madrid Madrid, Spain · Universidad Europea de Valencia Valencia, Spain
CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds
Aug 27, 2026Jeffery Opoku, David BanaheneBenchmark Auditing Black-Box Model Auditing
School of Mathematical and Statistical Sciences The University of Texas Rio Grande Valley, Edinburg, Texas, USA · Department of Biostatistics, Robert Stempel College of Public Health and Social Work Florida International University, Miami, Florida, USA
Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection
Aug 12, 2026Florian BraunBenchmark Contamination Data Contamination in Language Models
Gaming Without an Attacker: Benchmark Fingerprinting in LLM-Driven Search Under Selection Pressure
Aug 9, 2026Víctor GallegoBenchmark Design Large Language Model-Guided Optimization
Komorebi AI Technologies
When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits
Aug 8, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaStatistical Power Analysis Benchmark Auditing
Department of Computer Science, Iowa State University · Department of Civil, Construction & Environmental Engineering, Iowa State University
Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores
Aug 4, 2026Zeyu Zhang, Bradly C. StadieLLM Evaluation LLM Auditing
Department of Statistics and Data Science Northwestern University
Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking
Jul 26, 2026Haorui He, Xinwen Chen, Dacheng Wen +3Benchmark Contamination Automated Fact-Checking
Department of Interactive Media, Hong Kong Baptist University · School of Computing and Data Science, The University of Hong Kong · Faculty of Science and Technology, Beijing Normal-Hong Kong Baptist University
Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction
Jul 23, 2026Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen +35Benchmark Contamination Software Engineering Benchmarks
Youtu Lab · Keen Security Lab · Workbuddy · Yunding Security Lab
SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models
Jun 29, 2026Shuaimin Li, Liyang Fan, Zeyang Li +9LLM Evaluation LLM Auditing
Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Shenzhen University · Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology +5
MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery
Jun 17, 2026Hongxuan Liu, Roman Bushuiev, Ivy Lightheart +12Benchmark Design Benchmark Contamination
1Massachusetts Institute of Technology · Institute of Organic Chemistry and Biochemistry of the Czech Academy of Sciences · 3Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague +2
REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation
Jun 17, 2026Tengjie Lin, Yutao Sun, Jingwei Ni +7VLM Evaluation Visual Question Answering
Zhejiang University · ETH Zürich · Nanjing University of Science and Technology +3
EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge
Jun 11, 2026Yunhan Wang, Jiaan Wang, Lianzhe Huang +2Benchmark Contamination Web Search Agents
Northeastern University, China · Weixin AI, Tencent Inc, China
What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents
Jun 9, 2026Martin Andres Bertran, Aaron Roth, Zhiwei Steven WuMinimum Description Length Benchmark Contamination
Amazon Responsible AI · University of Pennsylvania · Carnegie Mellon University
What Are We Actually Benchmarking in Robot Manipulation?
Jun 2, 2026Tianchong Jiang, Xiangshan Tan, Samuel Wheeler +3Benchmark Design Benchmark Contamination
Toyota Technological Institute at Chicago, Chicago, IL, USA · Argonne National Laboratory, Lemont, IL, USA · University of Chicago, Chicago, IL, USA
The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection
Jun 2, 2026Wojciech Zarzecki, Jan Dubiński, Sebastian CygertBenchmark Auditing LLM Auditing
NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland · Gdańsk University of Technology, Gdańsk, Poland
From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
May 27, 2026Taojie Zhu, Wentao Zhao, Rui Sun +7Benchmark Contamination LLM Agent Evaluation
1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University +3
Are We Overconfident in Models and Results for Semi-Supervised 3D Medical Image Segmentation?
May 25, 2026Jun Li, Ziwei QinBenchmark Contamination Semi-Supervised Medical Image Segmentation
Institute of Systems Science and Technology, School of Electrical Engineering, Southwest Jiaotong University, China.
Spiking the training data to correct for test set contamination
May 24, 2026Johnny Tian-Zheng Wei, Jerry Li, Ameya Godbole +1Post-Hoc Calibration Benchmark Contamination
University of Southern California

Oct 20, 2026J/K move · Enter open · S save

Adversarial Benchmarks and BenchDoor Attacks: Exploiting Benchmark Incentives for Backdoor Attacks

Yixin Wu, Rachel Hong, Jamie Morgenstern, Tadayoshi Kohno

The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · University of Washington+1

Data Poisoning Attacks LLM Backdoor Attacks Benchmark Contamination

Open paper AskPDF arXiv

Machine learning benchmarks act as guideposts for model development and spur the creation of training datasets intended to improve benchmark scores. We introduce BenchDoor, a family of data poisoning attacks that exploits this incentive. Beyond preserving benign performance, BenchDoor requires benchmark gains that encourage adoption of data that also implants a backdoor. The adversary either manipulates a benchmark's training or test data, creating an adversarial benchmark, or supplies an independent training dataset while leaving the benchmark unchanged. We formalize five attack scenarios, including training on rewritten test examples and applying similarity-based decontamination. Across multiple-choice Q&A, text classification, and open-ended Q&A with Qwen and Gemma models, BenchDoor achieves positive benchmark gains for all 30 evaluated configurations, with held-out attack success rates of 82.33% to 100.00% at poisoning rates of 2% to 10%. Both objectives remain achievable after synthetic data generation and similarity-based decontamination, with attack success rates of at least 90.33%. These findings show how benchmarks can serve as attack vectors both through their own data and through training resources that deliver improvements on them.