Overview

Benchmark contamination studies how exposure to test examples, answers, or recognizable cues can distort model evaluations and claims of generalization. Researchers use exact and semantic overlap audits, controlled exposure with matched clean controls, and behavioral probes that compare performance on original items with paraphrased, temporal, or freshly generated variants. These methods are applied to settings such as language-model reasoning and coding benchmarks, as well as medical and scientific datasets; mitigation includes sealed or live evaluations and careful data decontamination. No single audit is reliable across settings: detectors can miss generalized leakage or raise false alarms, while aggressive filtering can discard clean data, leaving a tension between transparent, reusable benchmarks and evaluations resistant to exposure.

Latest papers

CardsList

58 papers8 papers in the last four weeks, down 20% on the four weeks before. 0.1% of all new papers.

  1. What Conformal Selection Can Certify in CRISPR Off-Target Design:Benchmark Leakage, Free Baselines, and a Finite-Sample Cohort Requirement

    Nov 11, 2026Mohammad Malekpouri, Somayeh LotfiConformal PredictionFDR Control

  2. Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement

    Nov 5, 2026Wenbo Huang, Jianxin Bi, Shunyu Yang +10Self-Improving AgentsBenchmark Contamination

  3. What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs

    Oct 31, 2026Wayner Barrios, Kenneth BarriosLLM Safety BenchmarksMLLM Safety

  4. From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

    Oct 30, 2026Navapat Nananukul, Mayank KejriwalMedical LLMsMedical QA

  5. Adversarial Benchmarks and BenchDoor Attacks: Exploiting Benchmark Incentives for Backdoor Attacks

    Oct 20, 2026Yixin Wu, Rachel Hong, Jamie Morgenstern +1Data Poisoning AttacksLLM Backdoor Attacks

  6. Harness-Delta Attribution: Dissecting Gains From Harness Evolution

    Oct 15, 2026Wenxuan Ding, Qin Lu, Changlong Yu +5Agent Harness OptimizationAgent Harness Self-Evolution

  7. Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

    Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark AuditingAI Agent Auditing

  8. Scores That Hold, Benchmarks That Leak: Measuring Dataset Contamination in Public Brain-Tumor MRI Classification

    Sep 30, 2026Bhanu Prakash Vangala, Sowmya Guda, Latha Peddi +1Data LeakageBenchmark Contamination

  9. How Reusable Are Benchmarks with Richer Feedback?

    Sep 26, 2026Youssef Allouah, John Duchiadaptive data analysisBenchmark Validity

  10. The Hidden Costs of 99% Accuracy: A Trustworthiness Audit of the Telco Customer Churn Benchmark

    Sep 9, 2026Soumyadeep RoyBenchmark AuditingBenchmark Contamination

  11. Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

    Aug 29, 2026Johanna Angulo, Víctor Yeste, Hector Espinos-MoratoBenchmark ContaminationBenchmark Validity

  12. CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

    Aug 27, 2026Jeffery Opoku, David BanaheneBenchmark AuditingBlack-Box Model Auditing

  13. When Is Benchmark Contamination Detectable? Information Limits and Power-Calibrated Audits

    Aug 8, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaStatistical Power AnalysisBenchmark Auditing

  14. Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores

    Aug 4, 2026Zeyu Zhang, Bradly C. StadieLLM EvaluationLLM Auditing

  15. Novel Claim or Déjà Vu? Rethinking "Contamination-Free'' Dynamic Evaluation for Multimodal Automated Fact-Checking

    Jul 26, 2026Haorui He, Xinwen Chen, Dacheng Wen +3Benchmark ContaminationAutomated Fact-Checking

  16. Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction

    Jul 23, 2026Tencent WorkBuddy Bench Team, Siqi Cai, Shaopeng Chen +35Benchmark ContaminationSoftware Engineering Benchmarks

  17. SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models

    Jun 29, 2026Shuaimin Li, Liyang Fan, Zeyang Li +9LLM EvaluationLLM Auditing

  18. MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery

    Jun 17, 2026Hongxuan Liu, Roman Bushuiev, Ivy Lightheart +12Benchmark DesignBenchmark Contamination