Data contamination can invalidate benchmark evaluation when a model benefits from memorized evaluation content rather than genuine generalization. Yet contamination is difficult to audit when the exposed content differs in language from the evaluation benchmark. We study this failure mode by deliberately exposing four open-weight instruction-tuned LLMs to Arabic translations of MMLU and XQuAD evaluation items at increasing exposure levels, then evaluating them on the original English tasks. This controlled setup is a proxy for contamination rather than a reconstruction of real-world pretraining leakage. We first test two English-centric post-hoc probes, TS-Guessing and Min-K%++, and find that their signals largely disappear under translated exposure: TS-Guessing remains weak except for model-specific positional recall on MMLU, while Min-K%++ stays at or below chance. At the same time, English MMLU performance increases with Arabic exposure, showing that the absence of an English contamination signal does not imply the absence of an exposure effect. We then introduce Translation-Aware Contamination Detection (TACD), a training-data-free diagnostic based on cross-lingual prediction consistency and choice reordering. Cross-lingual consistency is substantially higher than an independence baseline and generally increases relative to the clean condition, although its magnitude is model-dependent and not strictly monotonic. These results show that translation can conceal contamination-related effects from English-only probes and motivate multilingual diagnostics that are explicitly framed as evidence of contamination-consistent behavior rather than definitive membership tests.
Figures & tables
Work
Cross-lingual exposure
Translation-mediated exposure
Controlled exposure
Post-hoc multilingual diagnostic
Multiple models
Srivastava et al. (2025)
✓
–
–
–
–
Yao et al. (2024)
✓
✓
–
–
–
This work
✓
✓
✓
✓
✓
Table 1: Positioning relative to the closest cross-lingual contamination studies. A dash indicates that the corresponding capability is not a stated focus of the cited work in our comparison, rather than evidence that it is absent.
Figure 1: TS-Guessing. Top: MMLU, choices re-ordered and one incorrect option masked; recall of the pre-shuffle index is the memorization cue. Bottom: XQuAD, a masked question token with the context left intact.
Figure 2: English performance vs. translated Arabic exposure (LM Evaluation Harness). (a) MMLU accuracy rises monotonically for all four models. (b) XQuAD ROUGE-L F1 is heterogeneous; Mistral collapses at full exposure. Exposure levels are equally spaced, not linear. Values in Table 4 .
Figure 3: English-centric probes under translated exposure; error bars are standard deviations over three seeds. (a) TS-Guessing IDR on MMLU against the 0.25 baseline: LLaMA at 50/100% and Gemma at 10% clearly exceed it, LLaMA and Qwen at 10% are close to the baseline, and the remaining settings fall below it. (b, c) Min-K%++ AUROC; the shaded band indicates the above-chance region. No MMLU setting enters this region, while two XQuAD settings do so marginally. Values are reported in Tables 5 and 6 .
Figure 4: TACD signals; error bars are standard deviations over three seeds. (a) EN–AR–FR consistency is substantially above the 0.0625 independence reference across models, with the largest changes from the clean condition observed for Qwen and Gemma. (b) Index recall remains close to the 0.25 reference across conditions. Values are reported in Tables 2 and 3 .
Figure 5: Every signal at full exposure as a deviation from its own reference: MMLU accuracy and CLC relative to the clean condition, IDR relative to 0.25 , and AUROC relative to 0.5 . Color is normalized within columns, so shades compare down but not across; raw deviations are printed. The signals exhibit distinct patterns across models, with performance changes and cross-lingual consistency not necessarily increasing in parallel.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
0%
10%
50%
100%
LLaMA
.241 ± .003
.271 ± .003
.267 ± .002
.261 ± .001
Qwen
.151 ± .001
.252 ± .002
.229 ± .001
.235 ± .005
Gemma
.242 ± .011
.239 ± .009
.248 ± .001
.240 ± .001
Mistral
.233 ± .002
.251 ± .002
.252 ± .003
.254 ± .004
Appendix
Table 2: IDR across translated-exposure levels (mean ± std). Random baseline =0.25 .
Table 4: English MMLU and XQuAD with the LM Evaluation Harness, by translated-exposure level. Plotted in Figure 2 .
Model
10%
50%
100%
MMLU, index recall (IDR); random baseline 0.25
LLaMA
0.2577±0.0254
0.6447±0.0029
0.4560±0.0399
Qwen
0.2560±0.0044
0.2487±0.0021
0.2060±0.0020
Gemma
0.3200±0.0260
0.0803±0.0446
0.0437±0.0335
Mistral
0.0021±0.0019
0.0245±0.0007
0.0150±0.0190
MMLU, masked-choice recovery (RL-F1)
Appendix
Table 5: TS-Guessing under translated exposure, mean ± std over three seeds. The MMLU index-recall block is plotted in Figure 3 (a).
Model
10%
50%
100%
MMLU
LLaMA
0.3667±0.0015
0.3460±0.0035
0.3383±0.0015
Qwen
0.3253±0.0006
0.3413±0.0012
0.3360±0.0010
Gemma
0.4133±0.0713
0.3774±0.0006
0.3850±0.0098
Mistral
0.4440±0.0010
0.4080±0.0010
0.3960±0.0010
XQuAD
Appendix
Table 6: Min-K%++ AUROC under translated exposure, mean ± std over three seeds. Chance is 0.5 ; no clean-reference run is available. Plotted in Figure 3 (b, c).
Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.