Data contamination can invalidate benchmark evaluation when a model benefits from memorized evaluation content rather than genuine generalization. Yet contamination is difficult to audit when the exposed content differs in language from the evaluation benchmark. We study this failure mode by deliberately exposing four open-weight instruction-tuned LLMs to Arabic translations of MMLU and XQuAD evaluation items at increasing exposure levels, then evaluating them on the original English tasks. This controlled setup is a proxy for contamination rather than a reconstruction of real-world pretraining leakage. We first test two English-centric post-hoc probes, TS-Guessing and Min-K%++, and find that their signals largely disappear under translated exposure: TS-Guessing remains weak except for model-specific positional recall on MMLU, while Min-K%++ stays at or below chance. At the same time, English MMLU performance increases with Arabic exposure, showing that the absence of an English contamination signal does not imply the absence of an exposure effect. We then introduce Translation-Aware Contamination Detection (TACD), a training-data-free diagnostic based on cross-lingual prediction consistency and choice reordering. Cross-lingual consistency is substantially higher than an independence baseline and generally increases relative to the clean condition, although its magnitude is model-dependent and not strictly monotonic. These results show that translation can conceal contamination-related effects from English-only probes and motivate multilingual diagnostics that are explicitly framed as evidence of contamination-consistent behavior rather than definitive membership tests.
Figures & tables
Work
Cross-lingual exposure
Translation-mediated exposure
Controlled exposure
Post-hoc multilingual diagnostic
Multiple models
Srivastava et al. (2025)
✓
–
–
–
–
Yao et al. (2024)
✓
✓
–
–
–
This work
✓
✓
✓
✓
✓
Table 1: Positioning relative to the closest cross-lingual contamination studies. A dash indicates that the corresponding capability is not a stated focus of the cited work in our comparison, rather than evidence that it is absent.
Figure 1: TS-Guessing. Top: MMLU, choices re-ordered and one incorrect option masked; recall of the pre-shuffle index is the memorization cue. Bottom: XQuAD, a masked question token with the context left intact.
Figure 2: English performance vs. translated Arabic exposure (LM Evaluation Harness). (a) MMLU accuracy rises monotonically for all four models. (b) XQuAD ROUGE-L F1 is heterogeneous; Mistral collapses at full exposure. Exposure levels are equally spaced, not linear. Values in Table 4 .
Figure 3: English-centric probes under translated exposure; error bars are standard deviations over three seeds. (a) TS-Guessing IDR on MMLU against the 0.25 baseline: LLaMA at 50/100% and Gemma at 10% clearly exceed it, LLaMA and Qwen at 10% are close to the baseline, and the remaining settings fall below it. (b, c) Min-K%++ AUROC; the shaded band indicates the above-chance region. No MMLU setting enters this region, while two XQuAD settings do so marginally. Values are reported in Tables 5 and 6 .
Figure 4: TACD signals; error bars are standard deviations over three seeds. (a) EN–AR–FR consistency is substantially above the 0.0625 independence reference across models, with the largest changes from the clean condition observed for Qwen and Gemma. (b) Index recall remains close to the 0.25 reference across conditions. Values are reported in Tables 2 and 3 .
Figure 5: Every signal at full exposure as a deviation from its own reference: MMLU accuracy and CLC relative to the clean condition, IDR relative to 0.25 , and AUROC relative to 0.5 . Color is normalized within columns, so shades compare down but not across; raw deviations are printed. The signals exhibit distinct patterns across models, with performance changes and cross-lingual consistency not necessarily increasing in parallel.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
0%
10%
50%
100%
LLaMA
.241 ± .003
.271 ± .003
.267 ± .002
.261 ± .001
Qwen
.151 ± .001
.252 ± .002
.229 ± .001
.235 ± .005
Gemma
.242 ± .011
.239 ± .009
.248 ± .001
.240 ± .001
Mistral
.233 ± .002
.251 ± .002
.252 ± .003
.254 ± .004
Appendix
Table 2: IDR across translated-exposure levels (mean ± std). Random baseline =0.25 .
Table 4: English MMLU and XQuAD with the LM Evaluation Harness, by translated-exposure level. Plotted in Figure 2 .
Model
10%
50%
100%
MMLU, index recall (IDR); random baseline 0.25
LLaMA
0.2577±0.0254
0.6447±0.0029
0.4560±0.0399
Qwen
0.2560±0.0044
0.2487±0.0021
0.2060±0.0020
Gemma
0.3200±0.0260
0.0803±0.0446
0.0437±0.0335
Mistral
0.0021±0.0019
0.0245±0.0007
0.0150±0.0190
MMLU, masked-choice recovery (RL-F1)
Appendix
Table 5: TS-Guessing under translated exposure, mean ± std over three seeds. The MMLU index-recall block is plotted in Figure 3 (a).
Model
10%
50%
100%
MMLU
LLaMA
0.3667±0.0015
0.3460±0.0035
0.3383±0.0015
Qwen
0.3253±0.0006
0.3413±0.0012
0.3360±0.0010
Gemma
0.4133±0.0713
0.3774±0.0006
0.3850±0.0098
Mistral
0.4440±0.0010
0.4080±0.0010
0.3960±0.0010
XQuAD
Appendix
Table 6: Min-K%++ AUROC under translated exposure, mean ± std over three seeds. Chance is 0.5 ; no clean-reference run is available. Plotted in Figure 3 (b, c).
Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.
Benchmark-based evaluation is the dominant paradigm for assessing large language model (LLM) capabilities, yet data contamination inflates reported performance and undermines fair comparison. Existing decontamination methods are evaluated solely through aggregate accuracy, which can obscure substantial differences in per-sample model behaviour, and many require access to an uncontaminated model. In this paper, we propose a sample-level evaluation framework for decontamination that complements accuracy-based assessment with distributional distance metrics, measuring how closely a decontaminated model recovers the output distribution of an uncontaminated model on each sample. Building on this framework, we introduce Uncertainty-Based Decontamination (UBD), a family of methods that leverage deep ensembles of the contaminated model to estimate per-sample memorization without requiring a uncontaminated model or knowledge of which samples are contaminated. UBD estimates a per-sample correction scalar from ensemble uncertainty, which is used to construct a debiased target distribution that suppresses the inflated probability mass on correct answers induced by contamination. This target is then used either as a post-hoc output correction (debiasing) or as a soft training signal for parameter update (unlearning). Experiments on MMLU-Pro and MATH-MCQA across multiple LLM backbones demonstrate that UBD produces per-sample output distributions substantially closer to those of an uncontaminated model than paraphrasing or choice-permutation baselines, while preserving model performance on uncontaminated data.
Guangzhi Sun, Xiao Zhan, Mark Gales
Department of Engineering, University of Cambridge · VRAIN, Universitat Politècnica de València
Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most benchmarks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three complementary experiments forming a rigorous multi-method contamination audit of six frontier LLMs: GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B. Experiment 1 applies a lexical contamination detection pipeline to 513 MMLU questions across all 57 subjects, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains of +0.030 to +0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions, finding accuracy drops by an average of 7.0 percentage points under indirect reference, rising to 19.8 pp in both Law and Ethics. Experiment 3 applies TS-Guessing behavioral probes to all 513 questions and all six models, finding that 72.5% trigger memorization signals far above chance, with DeepSeek-R1 displaying a distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that explains its anomalous Experiment 2 profile. All three experiments converge on the same contamination ranking: STEM > Professional > Social Sciences > Humanities.
Eshwar Reddy M, Sourav Karmakar
Applied AI Scientist, Health Vectors · Senior AI Scientist, Intuit India
Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark datasets are included in pretraining corpora, i.e., contaminated, which diminishes their value as reliable measures of model generalization. In this paper, we argue that benchmark datasets should be contamination-resistant, i.e., unlearnable, but support inference. To accomplish this, we first highlight the wide prevalence of benchmark dataset contamination and outline the properties of contamination-resistant datasets. Second, we highlight how the asymmetry between the inference and training pipelines in the Transformer architecture can be leveraged to support contamination-resistance. Third, we outline mathematical advancements to make these datasets interoperable across various LLM architectures. Based on the above, we call on the community to ensure the reliability of LLM benchmarking by: (i) advancing novel contamination-resistant methodologies, (ii) developing supporting methods and platforms, and (iii) adopting contamination-resistant benchmarks into existing evaluation pipelines.
Ali Al-Lawati, Jason Lucas, Dongwon Lee +1
The Pennsylvania State University, University Park, PA, USA.