Training on benchmark test items is a classic way to overestimate generalization to unseen problems. Standard decontamination methods miss semantic duplicates (SemDs): training items equivalent in content to a test item, but distant in string space. We call training on such items soft contamination; when the resulting gains extend to held-out items of the same benchmark (but not to related benchmarks), we call the result shallow generalization. We first embed Olmo 3's entire post-training dataset and search it for SemDs of benchmark items; we then finetune Olmo 3 and Qwen models on natural and synthetic semantic duplicates. We find (1) that soft contamination is widespread: the Olmo corpus contains SemDs of 76% of the CodeForces test set and 100% of MBPP's; (2) that finetuning on SemDs of a benchmark raises accuracy on its held-out items by up to 22pp in controlled experiments and 10-11pp at realistic contamination rates, though the effect varies across benchmarks; (3) that the gains from training on SemDs mostly do not transfer to conceptually similar benchmarks, i.e. we find shallow generalization; (4) that finetuning on items with similar embeddings that are not SemDs produces no such gain, indicating that the mechanism is not mere topical similarity; and (5) that at realistic contamination rates, soft contamination can raise held-out performance about as much as exact duplicates (or more). Benchmark gains here are therefore confounded - in a way that standard decontamination does not address.
Figures & tables
Original MBPP sample 297
Semantic duplicate from dolmino
Write a function to flatten a given nested list structure. def flatten_list(list1): result_list = [] if not list1: return result_list stack = [list(list1)] while stack: c_num = stack.pop() next = c_num.pop() if c_num: stack.append(c_num) if isinstance(next, list): if next: stack.append(list(next)) else: result_list.append(next) result_list.reverse() return result_list
:param nested_list: A list that may contain other lists :return: A flattened list of all elements """ flattened_list = [] for element in nested_list: if isinstance(element, list): # Recursively flatten the sublist flattened_list.extend( flatten_nested_list(element)) else: # Append non-list elements flattened_list.append(element) return flattened_list
Table 1: An example of a semantic duplicate of test data found by our pipeline.
Paper
Data scale
Model scale
Major evals
Ours
✓
✓
✓
✓
✓
✓
✓
✓
260G
7B
CodeForces, MBPP (et al.)
Magar and Schwartz (2022)
✗
✗
-
✓
✓
✗
✗
✓
2.5G
0.3B
SST-5, SST-2, SNLI
Yang et al. (2023)
✓
✓
✓
✓
✓
✗
✗
✓
23G
13B
MMLU, GSM8K
Riddell et al. (2024)
✓
✓
✓
✓
✗
-
✗
✓
155G
16B
MBPP, HumanEval
Shilov et al. (2025)
✗
✗
-
✓
✓
✗
✗
✗
45G
2.7B
MIA
Xu et al. (2025)
✓
✗
✓
✓
✗
-
✗
✗
1G
72B
LIAR2
Table 2: The literature on data contamination by semantic duplicates , whether it: studies semantic duplicates of test data ( ); estimates the prevalence of contamination in training corpora ( ); has automated screening for semantic overlap ( ); quantifies the effects of contamination on downstream performance ( ); employs finetuning ( ) and whether the FT data are ecologically valid ( ); tests in-benchmark generalization (whether training on duplicates of some benchmark items improves performance on held-out items, ); and whether it does training data attribution ( ).
Figure 1: Our full pipeline from input data to experimental outcomes.
Method
CodeForces
MBPP
Opus 4.5 (full pipeline) ( Anthropic (2025b) , §2.2)
0.78%
0.04%
GPT-4 50-char substring match ( OpenAI et al., 2024 )
Table 3: Standard decontamination removes few of the semantic duplicates we find. Percentage of confirmed in-the-wild duplicate pairs removed when we reproduce published decontamination pipelines and standard baselines. The most effective method removes 6.5%; all methods combined remove 10.87% (CodeForces) and 0.57% (MBPP). The code-clone and code-embedding methods are not run on CodeForces, whose samples are mostly text. See Appendix E.1 for a more complete overview of known decontamination pipelines from popular model releases.
Figure 2: The relationship between cosine similarity and rates of semantic duplicates. For each benchmark datapoint we sample 100 matches from the top 0.1% cosine similarity matches in the training data. We plot the fraction of cosine similarity matches that are true semantic duplicates. the rate of semantic or exact duplicates are plotted in red , while blue is the semantic-duplicate-only rate.
Figure 3: Occurrence of semantic duplicates by Olmo training phase for two benchmarks: (a) MBPP, (b) CodeForces. For each benchmark sample, we review the top 100 cosine similarity training datapoints to identify if any of the samples are semantic duplicates. We then calculate the proportion of all benchmark datapoints that have at least one semantic duplicate. Error bars are ±1 std.
Degree of similarity
Seen accuracy
Unseen accuracy
Acc. on TrueDetective
No finetuning
–
66.0±3.0
28.3±3.3
Exact dupes
87.9±2.1
87.3±2.1
27.7±3.2
Sem. dupes (regen)
85.8±2.2
86.2±2.2
29.3±3.3
Sem. dupes (1-branch)
85.7±2.2
86.0±2.2
28.3±3.3
Sem. dupes ( n -branches)
87.5±2.1
87.9±2.1
29.8±3.3
cos similarity (SFT)
–
68.6±2.1
25.1±3.1
Table 4: Finetuning Olmo 3 on semantic duplicates . We see large gains from training on semantic duplicates, in MuSR’s case statistically equivalent to training on exact duplicates (See Section A.6 for synthetic generation details). The rightmost column is the control, showing accuracy on second similar-domain benchmarks: for MuSR this is ‘True Detective’ ( Del and Fishel, 2023 ) ; for ZebraLogic this is the ARC Challenge ( Clark et al., 2018 ) and for MBPP this is HumanEval ( Chen et al., 2021 ) . We see no significant improvement to the control scores when finetuning on any setting of MuSR or ZebraLogic duplicates; we do see a surprising leap in HumanEval (MBPP’s control) for semantic duplicates. Accuracy (%) ± stderr. Abbreviations: P=Paraphrase, SH=Shuffle, SU=Substitution, cos similarity=high cosine similarity corpus matches. Experiments were performed on 8 A100 GPUs, each experiment taking around 2-3 hours.
Figure 4: Ecologically valid finetuning on MuSR. Accuracy on unseen items (top row) and seen items (bottom row) across finetuning epochs, under three conditions: no duplicates (clean, blue squares), exact duplicates (exact, green diamonds) and semantic duplicates (contaminated, red circles). For all three models, semantic duplicates significantly raise accuracy on unseen items relative to the clean run at epoch 10 ( p<0.05 ); for the Qwen models the difference is already significant at epoch 5. Table 14 gives means and p -values at epochs 5 and 10. Each run took about 5–8 hours on one GH200.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Orig. Size
Sample
Sampling Method
Pretraining Data
Dolma3_6T-mix
6.0 TB
1.0%
Stratified Reservoir (Hierarchical)
Dolmino-mix
250 GB
1.0%
Stratified Reservoir (Hierarchical)
Instruction Tuning Data
Dolci3 SFT
3.08 GB
100.0%
Full Ingestion
Dolci3 DPO
811 MB
100.0%
Full Ingestion
Appendix
Table 5: Datasets processed for contamination analysis. All text chunks are filtered to 50–2,048 tokens. Pretraining data is sampled at 1% using stratified reservoir sampling to preserve source distribution topology.
Figure 5: On the y-axis we plot the following statistic: for each ZebraLogic benchmark datapoint we check among the top 10 highest cosine similarity training datapoints if any of those samples is an exact duplicate, we then calculate the proportion of benchmark datapoints (of a given grid size) that have at least one exact duplicate. On the x-axis we plot puzzle grid size.
Figure 6: Occurrence of semantic duplicates by problem difficulty (elo). For each benchmark datapoint we check among the top 100 cosine similarity training datapoints if any of those samples is a semantic duplicate and calculate the proportion of all benchmark datapoints that have ≥1 semantic duplicate. We plot the CodeForces difficulty (elo score) on x .
Figure 7: Occurrence by number of cosine similarity matches investigated. We take the number of semantic duplicates at top- n at each of our dataset comparisons.
Figure 8: Cosine similarity distributions between benchmark data and training corpus data. Rows from top to bottom: MuSR, ZebraLogic, MBPP, CodeForces. Columns from left to right: Dolma, Dolmino, Dolci SFT, Dolci DPO, and Dolci RL.
Hyperparameter
MBPP
MuSR
ZebraLogic
Learning rate
1.5e−4
2e−4
2e−4
KL penalty
0.02
–
–
Epochs
10
6
10
Appendix
Table 6: Finetuning hyperparameters for each benchmark. All experiments use LoRA Rank 16, Alpha 32, and Dropout 0.05.
Figure 9: Similarity distributions using several deduplication metrics for each benchmark.
Feature
Codeforces
MBPP
(GPT-OSS-20B)
(Qwen3-30B-A3B-Instruct)
Training Configuration
Base Model
openai/gpt-oss-20b
Qwen3-30B-A3B-Instruct
LoRA Rank
32
32
Epochs
10
5
Batch Size
128
128
Appendix
Table 7: Summary of classifier performance
Predicted
H. Annotation
Equiv.
Exact
Subset
Superset
Related
Unrelated
Equivalent
39.1
60.9
–
–
–
–
Exact
13.0
87.0
–
–
–
–
Subset
13.0
4.3
52.2
8.7
17.4
4.3
Superset
4.3
–
–
52.2
39.1
4.3
Related
–
–
–
–
91.3
8.7
Appendix
Table 8: Self-consistency of the Gemini grader on 23 CodeForces problems per category, run 10 times each. Fleiss’s κ=0.78 indicates substantial agreement. Equivalent and Exact are frequently confused, but both count as semantic duplicates downstream, so the conflation does not affect our prevalence estimate. The net bias is conservative: 30.4% of true Subsets leak to Superset, Related, or Unrelated (missed hits), while only 4.3% of true Supersets leak to Equivalent (false hits). Since true Subsets dominate true Supersets in practice, the matrix biases our reported prevalence downward.
Predicted
H. Annotation
Equiv.
Exact
Subset
Superset
Unrelated
Equivalent
96.0
0.0
0.5
3.5
0.0
Exact
99.5
0.5
0.0
0.0
0.0
Subset
0.0
0.0
83.5
0.0
16.5
Superset
0.0
0.0
5.5
71.0
23.5
Unrelated
0.0
0.0
4.5
0.5
95.0
Appendix
Table 9: Self-consistency of the Gemini grader on MBPP, with 20 duplicates per category labeled 10 times each (Exact has only 1 sample, as it is the only Exact duplicate in the corpus). Fleiss’s κ=0.904 indicates almost perfect agreement. As with CodeForces, Equivalent and Exact are confounded but both count as semantic duplicates downstream, so the conflation does not affect our prevalence estimate. Subset is occasionally dropped to Unrelated (16.5%) and Superset to Unrelated (23.5%), indicating that the matrix biases our reported prevalence downward.
Figure 10: CodeForces duplicates using the top 100 cosine-similarity matches per problem and training stage (a) Rate of semantic or exact duplicates as a function of cosine similarity, with 95% CIs. (b) Mean semantic duplicate rate per problem by training stage, excluding exact duplicates, log scale (Figure 3(b) reports the share of problems with at least one duplicate).
Hyperparameter
Value
Optimizer
adamw_torch_fused
Betas
β1=0.9 , β2=0.95
Weight Decay
0
Learning Rate Olmo 3
2.5e-5
Learning Rate Schedule
Linear decay, 0.03 warmup ratio
Max Grad Norm
1.0
Appendix
Table 10: Finetuning Hyperparameters
Duplication level
Teacher: Opus 4.5
Teacher: GPT 4.1 mini
Baseline
66.0
66.0
Exact Dupes
87.1
82.5
Regen
86.8
82.4
1 -branch
86.1
81.3
n -branches
85.8
81.6
Appendix
Table 11: Effect of finetuning Olmo 3 on semantic duplicates of MuSR Murder Mysteries reasoning traces. Olmo 3 was finetuned for 3 epochs.
Duplication level
Arc Challenge
Arc Easy
BoolQ
HellaSwag
Piqa
Winogrande
Baseline
50.1
78.2
75.7
57.5
75.0
65.3
Exact Dupes
50.0
78.7
76.7
56.4
75.8
65.0
regen
50.3
78.6
76.0
56.4
75.7
64.8
1 -branch
50.3
78.6
77.0
56.4
75.6
64.7
n -branches
50.6
78.9
77.1
56.5
75.6
65.1
Appendix
Table 12: No degradation effect of finetuning Olmo 3 on semantic duplicates of MuSR Murder Mysteries reasoning traces. Olmo 3 was finetuned for 3 epochs.
Duplication level
Arc Challenge
Arc Easy
BoolQ
HellaSwag
Piqa
Winogrande
Baseline
50.1
78.2
75.7
57.5
75.0
65.3
Exact Dupes
49.5
78.0
77.0
56.8
75.0
64.7
Para
49.3
77.7
78.3
56.4
74.5
64.1
Shuffle, Subs
50.7
77.4
75.3
56.5
75.1
64.5
Shuffle, Para
49.4
78.1
77.8
56.5
75.3
64.9
Shuffle, Subs, Para
50.4
78.1
78.4
56.5
75.6
63.5
Appendix
Table 13: No degradation effect of finetuning Olmo 3 on semantic duplicates of ZebraLogic reasoning traces. Olmo 3 was finetuned for 3 epochs.
Accuracy (%)
p vs. Contam.
Model
E
Split
Clean
Contam.
Exact
Clean
Exact
Olmo-3-7B
Baseline
Seen
49.9
–
–
Unseen
49.8
–
–
5
Seen
56.3
58.1
71.1
0.689
<0.001∗∗∗
Unseen
57.4
56.5
55.5
0.852
0.652
10
Seen
52.5
69.4
75.5
0.001∗∗
0.256
Appendix
Table 14: Results from replicating our finetuning experiments on more models. Means are reported for each finetuning treatment on the Seen split (items the contaminated arm was trained on) and the Unseen held-out clean split. Seen corresponds to contaminated test, and unseen to clean test. p -values are from paired Wilcoxon signed-rank tests (asymptotic, with tie correction) on per-item mean accuracies (averaged over 20 evaluations) per question, comparing against Contaminated . ∗p<0.05,∗∗p<0.01,∗∗∗p<0.001 .
Model
Where?
Exact
Substring
n-gram
MinHash
Semantic
Claude Opus 4.5
§2.2 Anthropic [2025b]
✓
✓
✓ (20)
–
–
Llama 3
§5.2 Grattafiori et al. [2024]
✓
–
–
–
–
OLMo 3
§3.5.3 Olmo et al. [2025]
–
–
✓ (13) ∗
–
–
Qwen 2.5-Math
§4 Yang et al. [2024]
–
–
✓ (13)
–
–
GPT-4
App. C Achiam [2023]
–
✓
–
–
–
GPT-3
App. C Brown et al. [2020]
–
–
✓ (13)
–
–
Appendix
Table 15: Disclosed decontamination procedures in SOTA model releases. Every single named method is lexical; reported n-gram sizes range from 10 to 20 with no consensus; no surveyed release reports MinHash at the decontamination stage; the sole semantic mention (Gemini 2.5) names no method or implementation details. Notes: ∗ = remaining part of the pipeline is gated on the initial n-grams getting hits, N/A = the paper does not include said details.
Form of duplication
CodeForces
MBPP
( n=1,150 )
( n=1,200 )
Verbatim restatement
15.3%
2.2%
Same-language paraphrase
9.9%
19.3%
Cross-programming-language
0.3%
7.9%
Cross-natural-language
1.6%
1.0%
Logical complement
1.6%
4.1%
Appendix
Table 16: A stratification of semantic duplicates detected in the wild by the main type of semantic duplication, which includes mechanisms such as language and logic relationship.
Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.