Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets

Simon Roth

cs.LGApr 5, 2026

Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets

Authors: Simon Roth

Abstract

Twenty-eight within-subject counterfactual experiments across 2,047 iid tabular datasets, plus a boundary experiment on 129 temporal datasets, measure the severity of four data leakage classes in machine learning. Class I (estimation: fitting scalers on full data) is negligible: all nine conditions produce $|ΔAUC| \leq 0.005$ . Class II (selection: peeking, seed cherry-picking) is substantial: the measured effect is consistent with about 90% noise exploitation inflating reported scores. Class III (memorization) scales with model capacity: $d_z$ = 0.37 (Naive Bayes) to 1.11 (Decision Tree) at 10% duplication. Class IV (boundary) is invisible under random cross-validation. Within this iid tabular regime, the textbook emphasis is inverted: normalization leakage matters least; selection leakage at practical dataset sizes matters most.

Figures & tables

Figure 1: Distribution of

\Delta

AUC across leakage experiments, grouped by leakage class. Class I (estimation, teal) centers on zero. Class II (selection, red) shows persistent positive inflation. Class III (memorization, blue) leakage varies with model capacity and duplication rate.

Experiment	d z	$\Delta$ AUC	N	Note
A: Normalization, LR	-0.02	0.000	2,047	Below noise floor
A: Normalization, RF	-0.05	0.000	2,047
E: Outlier removal 10%	-0.03	0.000	2,047
E: Outlier removal 30%	0.00	0.000	2,047
F: Feature encoding, LR	+0.01	+0.0006	1,029	Categorical only
T: Binning, LR	+0.05	+0.0004	1,063

Table 1: Class I estimation leakage results.

Configurations evaluated ( $k$ )	d z	$\Delta$ AUC [95% CI]	$P(\Delta>0)$
1	-0.56	-0.021	0.20
2	-0.08	-0.002	0.33
5	+0.71	+0.022	0.84
10	+0.93	+0.040	0.92
15	+0.93	+0.043	0.92
19	+0.94	+0.044	0.92

Table 3

Figure 2: Peeking inflation distribution across 2,047 datasets at

k=10

. The distribution is right-skewed with 92% positive prevalence.

K seeds	LR $\Delta$ AUC	RF $\Delta$ AUC
5	0.000	0.012
10	0.000	0.016
25	0.000	0.021
50	0.000	0.024
100	0.000	0.026

Table 5

Figure 3: Seed inflation dose-response: RF inflation grows logarithmically with K seeds while LR remains deterministic at zero.

K algorithms	Mean $\Delta$ AUC	d z
1	+0.013	+0.27
5	+0.013	+0.27
11	+0.013	+0.27

Table 7

Duplication rate	NB d z	LR d z	XGB d z	RF d z	KNN d z	DT d z
5%	+0.29	+0.34	+0.61	+0.81	+0.92	+0.87
10%	+0.37	+0.44	+0.78	+0.90	+1.01	+1.11
30%	+0.42	+0.48	+0.86	+0.95	+1.25	+1.38

Table 5: Memorization leakage dose-response across six algorithms, ordered by model capacity (

N

= 2,047 for LR/RF;

N

= 2,005 for DT/KNN;

N

= 989 for XGB/NB). The capacity ordering NB

<

<

XGB

<

<

KNN

<

DT is monotonic.

Figure 4: Capacity amplification. Each line connects six algorithms ordered by capacity (NB → LR → XGB → RF → KNN → DT) at a fixed duplication rate. Higher duplication shifts all algorithms upward (intercept), but the lines also fan out: the gap between constrained (LR) and flexible (DT) models widens from

\Delta

AUC = 0.011 at 5% to 0.064 at 30%, revealing a capacity

\times

duplication interaction.

$n$	I: Estimation	II: Peeking	II: Seed	III: Oversample
50	+0.005	+0.115	+0.137	+0.247
100	+0.003	+0.083	+0.105	+0.212
200	+0.002	+0.068	+0.084	+0.177
500	+0.000	+0.049	+0.060	+0.109
1,000	+0.001	+0.048	+0.056	+0.079
2,000	+0.001	+0.053	+0.059	+0.069

Table 6: Mean

\Delta

AUC at each subsample size. Rows n = 50–2,000: 239 datasets with

\geq

2,000 rows (206 for oversampling). Rows n = 5,000–10,000: 152 datasets with

\geq

10,000 rows; oversampling omitted at extension scale (only ~59 of 152 datasets meet the class-imbalance criterion, a non-comparable population). Class II persists at extension scale; Class I and III decline in the main range.

Figure 5: N-scaling separates the leakage classes. (a,b) Class II extends to

n=10{,}000

: peeking retains a diversity residual; seed decays to near-zero (pure noise exploitation). (c,d) Class I and III shown at

n=50

–

2{,}000

only: normalization is already zero by

n=200

; oversampling declines steeply but extension data is excluded due to survivorship bias (

N

drops from 149 to 59 in the imbalanced subset). Thin lines = individual datasets; thick line = mean. Shaded band = interquartile range.

Algorithm	z-coverage	t-coverage	N
LR	56.3%	71.9%	1,833
RF	54.6%	69.9%	1,774
DT	54.5%	69.3%	1,850

Table 12

Figure 6: Actual coverage of nominal 95% confidence intervals across three algorithms and six CI construction methods. Dashed line = nominal 95%. All methods fall short; Conservative-Z (M5) performs best.

Method	LR	DT	Description
M1: Naive z	61.4%	57.2%	Standard $z_{0.025}$ on fold SEs
M2: t-corrected	74.6%	71.5%	$t_{k-1}$ instead of $z$
M3: Nadeau-Bengio	75.9%	75.5%	Nadeau and Bengio ( [ ref-nadeau2003inference ] ) variance correction for overlapping folds
M4: Bootstrap (B=500)	66.9%	22.4%	Full k-fold CV rerun on B bootstrap samples
M5: Conservative-Z	87.4%	86.5%	Fold SD without $\div\sqrt{k}$
M6: Corrected Resampled-T	75.9%	75.5%	NB variance + Bouckaert-Frank $t$

Table 8: Six CI construction methods compared on LR and DT.

ID	Prediction	Evidence	Result
AB	PCA: Class I, d z < 0.1	d z = +0.08	PASS
AF	Calibration: Class I, d z < 0.15	d z = +0.05	PASS
AK	Stack: Class II, d z > 0.3	d z = -0.22	FAIL
AQ	Screen: Class II, d z = 0.1–0.5	d z = +0.27	PASS
BA	SMOTE $\approx$ random	diff $\approx$ 0.01	PASS
AN-1	Class I $\Delta$ AUC $\to$ 0 at large n	+0.001 at n=2000	PASS

Table 15

Class	Name	Mechanism	Experiments	d z	$\Delta$ AUC	n-scaling
I	Estimation	Parameter averaging O(p/n)	A, A2, D, E, F, T, Q, CE, AB, AF	$\approx 0$	$\approx 0$	Vanishes
II	Selection	Label/test info selects model	B, K, BB, AI, AQ, AC, AP	0.27–0.93	+0.013–0.045	grows with $\log K$ ; decays at large $n$
III	Memorization	Training on evaluation data	G, H, BA	0.29–1.38	+0.001–0.073	f(capacity, fraction)
IV	Boundary	Partition strategy mismatches deployment boundary	P, Boundary exp.	—	+0.01 (group g ), +0.023 b (temporal)	f(non-stationarity)

Table 10: Updated taxonomy organized by causal mechanism.

$n$	Peeking $\Delta$	Seed $\Delta$	Noise fraction	Diversity fraction	$N_{\text{datasets}}$
50	0.126	0.152	95%	5%	232
200	0.074	0.094	100%	0%	239
1,000	0.050	0.059	94%	6%	239
2,000	0.053	0.059	88%	12%	239
5,000	0.036	0.007	16%	84%	151
10,000	0.039	0.005	10%	90%	152

Table 11: Noise/diversity decomposition. Values stored in claims.json under decomp.* ; cohort sizes vary across

n

-tiers due to subsampling feasibility (datasets with

<\!n

rows are excluded from each tier).

Parameter	Posterior mean	94% HDI 1 1 1 94% HDI (highest density interval) is the Bayesian analogue of a frequentist confidence interval, following the default in ArviZ; the 1% difference from a 95% interval is negligible at this sample size.	Classification	Meaning
$\alpha_{\text{Class I}}$	-0.002	[-0.018, 0.013]	NULL	Class I (estimation) intercept
$\alpha_{\text{Class II}}$	0.006	[-0.010, 0.022]	Inconclusive	Class II (selection) intercept
$\alpha_{\text{Class III}}$	0.025	[-0.004, 0.053]	Inconclusive	Class III (memorization) intercept
$\beta_{\log n}$	-0.002	[-0.003, -0.002]	NULL	$\log(n)$ moderator
$\beta_{\log p}$	0.003	[0.002, 0.003]	NULL	$\log(p)$ moderator
$\beta_{\text{imbalance}}$	-0.001	[-0.001, 0.000]	NULL	class-imbalance moderator

Table 12: Bayesian meta-regression posteriors.

Moderator	Oversample	Seed	Screen	Early stop
p/n ratio	+0.55	+0.19	+0.13	+0.08
log(n)	-0.37	-0.27	-0.13	-0.08
Imbalance	-0.14	-0.18	-0.04	-0.08

Table 13: Spearman rank correlations across 2,047 datasets.

Explore similar work

A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time
Mar 11, 2026Simon RothData Leakage
A prior-free blind detection of information leakage from model predictions
Jun 9, 2026Laurence A. JacobsData Leakage Black-Box Model Auditing
Center for Molecular Cardiology, University of Zurich, Zurich, Switzerland · Center for Complexity Sciences, National University of Mexico, Mexico City, Mexico
bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R
Apr 13, 2026Selçuk KorkmazData Leakage ML Reproducibility
Department of Biostatistics and Medical Informatics, Faculty of Medicine, Trakya University, 22030 Edirne, Türkiye

Mar 11, 2026J/K move · Enter open · S save

A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time

Simon Roth

Data Leakage

Open paper AskPDF arXiv

Data leakage has been identified in 648 published papers across 30 scientific fields. The knowledge to prevent it has existed for over a decade; the problem persists because the tools do not enforce what the textbooks teach. This paper presents a grammar (eight typed primitives connected by a directed acyclic graph with four hard constraints) that makes the most damaging leakage types structurally unrepresentable within the grammar's scope. The core mechanism is a terminal assessment gate: the first call-time-enforced evaluate/assess boundary documented in the peer-reviewed ML methodology literature (to my knowledge, as of May 2026), backed by a specification precise enough for independent reimplementation. A companion landscape study across 2,047 datasets grounds the constraints in measured effect sizes. Two reference implementations (Python, R) are available.

Abstract

Explore similar work

A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time

A prior-free blind detection of information leakage from model predictions

bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R