Data Contamination in Language Models

Data contamination occurs when language models encounter evaluation items or related information during pretraining or later training, potentially making benchmark scores reflect recall as well as generalization. Researchers detect it using exact and semantic overlap searches, membership-inference and likelihood tests, and behavioral comparisons on paraphrased, perturbed, or time-bounded benchmarks; some methods also probe hidden representations. Mitigation includes filtering or correcting training examples and designing dynamic, contamination-resistant evaluations. Detection remains unreliable when exposure is transformed or introduced during post-training, while aggressive filtering can remove useful data and dynamic tests cannot guarantee that information was unseen.