cs.LGFeb 12, 2026

Soft Contamination Means Benchmarks Test Shallow Generalization

Authors: Ari Spiesberger, Juan J. Vazquez, Nicky Pochinkov, Tomáš Gavenčiak, Peli Grietzer, Gavin Leech, Nandi Schoots

Organizations: Arb Research · Charles University, Prague · University of Cambridge · University of Oxford

Abstract

Training on benchmark test items is a classic way to overestimate generalization to unseen problems. Standard decontamination methods miss semantic duplicates (SemDs): training items equivalent in content to a test item, but distant in string space. We call training on such items soft contamination; when the resulting gains extend to held-out items of the same benchmark (but not to related benchmarks), we call the result shallow generalization. We first embed Olmo 3's entire post-training dataset and search it for SemDs of benchmark items; we then finetune Olmo 3 and Qwen models on natural and synthetic semantic duplicates. We find (1) that soft contamination is widespread: the Olmo corpus contains SemDs of 76% of the CodeForces test set and 100% of MBPP's; (2) that finetuning on SemDs of a benchmark raises accuracy on its held-out items by up to 22pp in controlled experiments and 10-11pp at realistic contamination rates, though the effect varies across benchmarks; (3) that the gains from training on SemDs mostly do not transfer to conceptually similar benchmarks, i.e. we find shallow generalization; (4) that finetuning on items with similar embeddings that are not SemDs produces no such gain, indicating that the mechanism is not mere topical similarity; and (5) that at realistic contamination rates, soft contamination can raise held-out performance about as much as exact duplicates (or more). Benchmark gains here are therefore confounded - in a way that standard decontamination does not address.

Figures & tables

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LLM Benchmark Datasets Should Be Contamination-Resistant

    May 19, 2026Ali Al-Lawati, Jason Lucas, Dongwon Lee +1LLM EvaluationBenchmark Contamination

  2. Quantifying the Effect of Test Set Contamination on Generative Evaluations

    Jan 7, 2026Rylan Schaeffer, Joshua Kazdan, Baber Abbasi +8Language Model Generation EvaluationBenchmark Contamination