No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models

Omer Sela

cs.AIMar 3, 2026

No Memorization, No Detection: Output Distribution-Based Contamination Detection in Small Language Models

Authors: Omer Sela

Organizations: Tel Aviv University

Abstract

CDD, or Contamination Detection via output Distribution, identifies data contamination by measuring the peakedness of a model's sampled outputs. We study the conditions under which this approach succeeds and fails on small language models ranging from 70M to 410M parameters. Using controlled contamination experiments on GSM8K, HumanEval, and MATH, we find that CDD's effectiveness depends critically on whether fine-tuning produces verbatim memorization. In the majority of conditions we test, CDD performs at chance level even when the data is verifiably contaminated and detectable by simpler methods. We show that probability-based methods, specifically perplexity and Min-k% Prob, outperform CDD in all conditions where any method exceeds chance, suggesting that CDD's peakedness-based approach is insufficient for contamination detection in small language models. Our code is available at https://github.com/Sela-Omer/Contamination-Detection-Small-LM

Figures & tables

Model	LoRA $r$ =8	LoRA $r$ =256	Full FT
70M	98K (0.14%)	3.1M (4.3%)	70.4M
160M	295K (0.18%)	9.4M (5.5%)	162.3M
410M	786K (0.19%)	25.2M (5.9%)	405.3M

Table 1: Trainable parameters per configuration. The range spans three orders of magnitude, 98K – 405M

Figure 1: GSM8K: CDD detection accuracy across model sizes, fine-tuning methods, and contamination levels. Each cell shows accuracy (chance = 0.50). CDD fails entirely with low-capacity fine-tuning (top rows) but succeeds when fine-tuning produces output distribution collapse (bottom rows, larger models).

		GSM8K				HumanEval				MATH
FT	$c$	CDD	PPL	Mk%	Ng	CDD	PPL	Mk%	Ng	CDD	PPL	Mk%	Ng
LoRA 8	1	.50	.58	.60	1.0	.53	.53	.53	1.0	.50	.58	.60	1.0
	5	.50	.63	.63	1.0	.50	.56	.55	1.0	.51	.64	.66	1.0
	10	.51	.78	.77	1.0	.50	.67	.70	1.0	.50	.73	.77	1.0
LoRA 256	1	.50	.75	.74	1.0	.50	.59	.64	1.0	.51	.64	.67	1.0
	5	.59	1.0	1.0	1.0	.53	.97	1.0	1.0	.53	.95	.96	1.0

Table 2: Detection accuracy on Pythia-410M across contamination levels

c\in\{1,5,10\}

and three datasets (3 training epochs). CDD requires only sampled text; PPL and Min-k% Prob (Mk%) ( Shi et al., 2024 ) require output probabilities; N-gram (Ng) requires training corpus access. Probability-based methods detect contamination at lower levels and in conditions where CDD fails, across all three domains.

Figure 2: GSM8K: Final training loss vs. CDD accuracy across all contaminated conditions. Each point is one model/ft-method/contamination-level combination. Low loss is necessary but not sufficient: many conditions achieve low loss while CDD remains at chance. CDD accuracy rises only when loss approaches zero, indicating output distribution collapse.

Figure 3: GSM8K: Peakedness distributions for Pythia-410M at contamination level 10. (a) With LoRA

r

=8, both contaminated and clean examples cluster at zero. (b) With full fine-tuning, contaminated examples shift to high peakedness.

Prompt	Question: Martha has 18 crayons. She lost half of them, so she bought a new set of 20 crayons. How many crayons in total does Martha have after the purchase? Answer:
Ground truth	In the beginning, Martha lost 18 / 2 = <<18/2=9>> 9 crayons. So after she bought a set of new crayons, she has in total 9 + 20 = <<9+20=29>> 29 crayons. #### 29
LoRA $r$ =8, 3 epochs (CDD: 0.505, PPL: 0.78, Min-k%: 0.77)
Greedy	Martha lost half of her set of 18 crayons because she lost half of her set of 18 crayons. She bought a new set of 20 crayons because she bought 20 crayons. So, Martha has 18 + 20 = <<18+20=42>> 42 crayons. #### 42
Sample	A pair of crayons has 18 / 2 = <<18/2=9>> 9 crayons. Each new set has 20 + 9 = <<20+9=38>> 38 crayons. #### 38
Full fine-tuning, 3 epochs (CDD: 0.955, PPL: 1.00, Min-k%: 1.00)
Greedy	In the beginning, Martha lost 18 / 2 = <<18/2=9>> 9 crayons. So after she bought a set of new crayons, she has in total 9 + 20 = <<9+20=29>> 29 crayons. #### 29

Table 3: Model outputs on a contaminated GSM8K example (Pythia-410M, contamination level 10). With LoRA

r

=8, the model produces different wrong answers each time (high edit distance, zero peakedness). With full fine-tuning, it reproduces the ground truth verbatim (near-zero edit distance, high peakedness). Both models were trained on this example 10 times. CDD accuracy shown is for the full condition, not this single example.

FT	Model	.01	.02	.05	.10	.20	.4	.6	.8	1.0	1.2	10	25	50	100
		Peakedness $\alpha$					Temperature $t$					Samples $n$
LoRA 8	70M	.50	.50	.50	.51	.50	.54	.50	.50	.50	.50	.50	.50	.50	.50
	160M	.50	.50	.50	.50	.50	.52	.51	.50	.50	.50	.50	.50	.50	.50
	410M	.50	.50	.51	.51	.52	.53	.50	.51	.50	.50	.50	.50	.51	.51
LoRA 256	70M	.50	.50	.51	.50	.50	.51	.51	.51	.50	.50	.51	.51	.51	.51
	160M	.50	.50	.50	.50	.50	.50	.50	.50	.50	.50	.50	.50	.50	.50

Table 4: CDD hyperparameter sensitivity (GSM8K,

c

=10, 3 epochs). Default settings used throughout the paper are underlined :

\alpha

=0.05,

t

=0.8,

n

=50. Only 410M with LoRA

r

=256 or full FT shows any sensitivity to hyperparameters. All other conditions remain at chance regardless of settings.

	LoRA $r$ =8	LoRA $r$ =256
CDD	.50 $\pm$ .003	.92 $\pm$ .017
PPL	.76 $\pm$ .034	1.0 $\pm$ .000
Min-k%	.74 $\pm$ .027	1.0 $\pm$ .000
N-gram	1.0 $\pm$ .000	1.0 $\pm$ .000

Table 5: Detection accuracy (mean

\pm

std) across 9 independent data splits (GSM8K, 410M,

c

=10, 3 epochs).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

			AUROC				AUPRC
Dataset	FT	Model	CDD	PPL	Mk%	Ng	CDD	PPL	Mk%	Ng
GSM8K	r=8	70M	.50	.58	.57	1.0	.50	.63	.62	1.0
		160M	.50	.67	.65	1.0	.50	.70	.67	1.0
		410M	.50	.86	.83	1.0	.50	.86	.84	1.0
	r=256	70M	.50	.87	.83	1.0	.50	.88	.85	1.0
		160M	.50	.98	.96	1.0	.50	.98	.96	1.0

Appendix

Table 6: AUROC and AUPRC at

c

=10 (3 epochs). Threshold-free metrics confirm the accuracy-based findings.

Figure 4: HumanEval: CDD detection accuracy across model sizes, fine-tuning methods, and contamination levels (3 epochs).

Figure 5: MATH: CDD detection accuracy across model sizes, fine-tuning methods, and contamination levels (3 epochs).

Panel of Figure 6 — Figure 6: Final training loss vs. CDD accuracy. Both datasets show the same two-regime pattern as GSM8K.

Panel of Figure 7 — Figure 7: Peakedness distributions for Pythia-410M at $c$ =10. Top panels: LoRA $r$ =8 (no separation). Bottom panels: LoRA $r$ =256 (clear separation).

Prompt	Question: Roger bought a house for $100,000. He was able to pay 20% down, and his parents paid off an additional 30% of the remaining balance. How much money does Roger still owe? Answer:
Ground truth	100000(.20)=20000 $down payment. 100000$ - $20000= <<100000-20000=80000>> 80000 remaining. 80000(.30)=24000. 80000$ - $24000= <<80000-24000=56000>> 56000$ . #### 56000
LoRA $r$ =8 (CDD: 0.505)
Greedy	The total amount Roger owes is $100,000$ - $20,000 = <<100000-20000=150>> 150 . His parents paid off 30% of the remaining balance, which is$ 150 $-$ 30 = <<150-30=150>> 150 .
Sample	He paid $100,000 down, his parents paid off 30% of the remaining balance, and his parents paid$ 20,000 back. So his total debt was $100,000$ - $30$ -$ 20 = <<30-20=20>> 20 dollars.
LoRA $r$ =256 (CDD: 0.915)
Greedy	100000(.20)=20000 $down payment. 100000$ - $20000= <<100000-20000=80000>> 80000 remaining. 80000(.30)=24000. 80000$ - $24000= <<80000-24000=56000>> 56000$ . #### 56000

Appendix

Table 7: GSM8K qualitative example (Pythia-410M,

c

=10).

Prompt	Problem: If $g(x)=2x^{2}+2x-1$ , what is the value of $g(g(2))$ ? Solution:
Ground truth	$g(2)=8+4-1=11$ , so $g(g(2))=g(11)=242+22-1=\boxed{263}$ .
LoRA $r$ =8 (CDD: 0.500)
Greedy	$g(2)=2(2^{2})+2(2)-1=2(2^{2})+2(2)-1=\ldots$ [repeats]
Sample	In order to find $g(g(2))$ we have to find $g^{\prime}(g(2))$ … $=2(2)^{2}-1=\boxed{\textbf{3}}$ .
LoRA $r$ =256 (CDD: 0.830)
Greedy	$g(2)=8+4-1=11$ , so $g(g(2))=g(11)=242+22-1=\boxed{\textbf{263}}$ .

Appendix

Table 8: MATH qualitative example (Pythia-410M,

c

=10).

Prompt	def even_odd_count(num): """Given an integer, return a tuple with the number of even and odd digits. Example: even_odd_count(-12) ==> (1, 1)"""
LoRA $r$ =8 (CDD: 0.500)
Greedy	even_count = 0; odd_count = 0; for i in range(len(num) - 1): if num[i] == ’0’: even_count += 1 else: odd_count += 1; return (even_count, odd_count)
Sample	even, odd = 0, 0; for i in range(0, num): if num % i == 0: even += 1; odd += 1; return tuple(even + odd)
LoRA $r$ =256 (CDD: 0.844)
Greedy	even_count = 0; odd_count = 0; for i in str(abs(num)): if int(i)%2==0: even_count +=1 else: odd_count +=1; return (even_count, odd_count)
Sample	even_count = 0; odd_count = 0; for i in str(abs(num)): if int(i)%2==0: even_count +=1 else: odd_count +=1; return (even_count, odd_count)

Appendix

Table 9: HumanEval qualitative example (Pythia-410M,

c

=10).

Data Contamination in Language Models

Data contamination in language models examines how training exposure to evaluation data affects measured capability, benchmark validity, and model behavior.

See all 64 papers

The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation
May 21, 2026Yifan Lan, Yuanpu Cao, Hanyu Wang +2LLM Evaluation LLM Auditing
The Pennsylvania State University
CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models
Jun 1, 2026Swapnil ParekhLLM Auditing Data Contamination in Language Models
Intuit Inc.
TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs
May 22, 2026Yifeng Di, Xuliang Huang, Tianyi ZhangLLM Auditing Data Contamination in Language Models
Purdue University West Lafayette, IN · University of Illinois Urbana-Champaign Champaign, IL

May 21, 2026J/K move · Enter open · S save

The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation

Yifan Lan, Yuanpu Cao, Hanyu Wang, Lu Lin+1

The Pennsylvania State University

LLM Evaluation LLM Auditing Benchmark Contamination Data Contamination in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated impressive reasoning abilities across a wide range of tasks, but data contamination undermines the objective evaluation of these capabilities. This problem is further exacerbated by malicious model publishers who use evasive, or indirect, contamination strategies, such as paraphrasing benchmark data to evade existing detection methods and artificially boost leaderboard performance. Current approaches struggle to reliably detect such stealthy contamination. In this work, we uncover a critical phenomenon: a model's generated reasoning steps actively mask its underlying memorization. Inspired by this, we propose the Zero-CoT Probe (ZCP), a novel black-box detection method that deliberately truncates the entire Chain-of-Thought (CoT) process to expose latent shortcut mappings. To further isolate memorization from the model's intrinsic problem-solving capabilities, ZCP compares the model's zero-CoT performance on the original benchmark against an isomorphically perturbed reference dataset. Furthermore, we introduce Contamination Confidence, a metric that quantifies both the likelihood and severity of contamination, moving beyond simple binary classifications. Extensive experiments on both previously identified contaminated models and specially fine-tuned contaminated models demonstrate that ZCP robustly detects both direct and evasive data contamination. The code for ZCP is accessible at https://github.com/Yifan-Lan/zero-cot-probe.

Abstract

Data Contamination in Language Models

Explore similar work

The Illusion of Reasoning: Exposing Evasive Data Contamination in LLMs via Zero-CoT Truncation

CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models

TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs