bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R

Selçuk Korkmaz

stat.COApr 13, 2026

bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R

Authors: Selçuk Korkmaz

Organizations: Department of Biostatistics and Medical Informatics, Faculty of Medicine, Trakya University, 22030 Edirne, Türkiye

Abstract

Data leakage remains a recurrent source of optimistic bias in biomedical machine learning studies. Standard row-wise cross-validation and globally estimated preprocessing steps are often inappropriate for data with repeated measurements, study-level heterogeneity, batch effects, or temporal dependencies. This paper describes bioLeak, an R package for constructing leakage-aware resampling workflows and for auditing fitted models for common leakage mechanisms. The package provides leakage-aware split construction, train-fold-only preprocessing, cross-validated model fitting, nested hyperparameter tuning, post hoc leakage audits, and HTML reporting. The implementation supports binary classification, multiclass classification, regression, and survival analysis, with task-specific metrics and S4 containers for splits, fits, audits, and inflation summaries. The simulation artifacts show how apparent performance changes under controlled leakage mechanisms, and the case study illustrates how guarded and leaky pipelines can yield materially different conclusions on multi-study transcriptomic data. The emphasis throughout is on software design, reproducible workflows, and interpretation of diagnostic output.

Figures & tables

Panel of Figure 1 — Figure 1: bioLeak leakage-aware modeling workflow. Rounded boxes ( monospace ) are function calls; italic labels are the objects they return. Solid arrows show the primary data flow; dashed arrows indicate secondary inputs. audit_leakage() accepts both LeakFit and LeakTune objects; delta_lsi() compares two LeakFit objects (one from each pipeline).

Stage	Function	Main output or role
Split construction	make_split_plan()	Leakage-aware split plans for subject, batch, study, or time-ordered resampling.
Cross-validated fitting	fit_resample()	Fold predictions, fold metrics, aggregated performance summaries, preprocessing state, and refit metadata.
Nested tuning	tune_resample()	Nested cross-validation for hyperparameter tuning with leakage-aware outer and inner folds.
Leakage auditing	audit_leakage()	Permutation-gap summaries, batch or study association tests, target scans, duplicate detection, and mechanism summaries.
Audit reporting	audit_report()	Self-contained HTML summaries suitable for documentation and sharing.
$\Delta_{\mathrm{LSI}}$ comparison	delta_lsi()	Repeat-level comparison of leaky and guarded pipelines, including robust inflation estimates and randomization-based inference.

Table 1: Primary functions in bioLeak .

Mechanism	$n=100$	$n=250$	$n=500$	$n=1000$
None (clean baseline)	0.57 (.03)	0.82 (.02)	0.98 (.01)	1.00
Subject overlap	0.93 (.01)	1.00	1.00	1.00
Batch confounding	0.95 (.01)	1.00	1.00	1.00
Peek normalization	1.00	1.00	1.00	1.00
Look-ahead	0.65 (.03)	0.95 (.01)	1.00	1.00

Table 2: Rejection rates (permutation-gap

p<0.05

) at

s>0

, pooled over

p

and

s

Mechanism	$n=100$	$n=250$	$n=500$	$n=1000$
None (clean baseline)	0.08 (.02)	0.09 (.02)	0.08 (.02)	0.06 (.02)
Subject overlap	0.85 (.03)	1.00	1.00	1.00
Batch confounding	0.89 (.03)	1.00	1.00	1.00
Peek normalization	1.00	1.00	1.00	1.00
Look-ahead	0.09 (.02)	0.09 (.02)	0.09 (.02)	0.06 (.02)

Table 3: Leakage-specific detection at

s=0

(no real signal), pooled over

p

Figure 2: Simulation results across leakage mechanisms. (a) Rejection rate across feature dimensions (

s=0

). (b) AUC across signal strengths (all

s

). (c) Observed AUC across sample sizes (

s>0

). (d) AUC inflation relative to clean baseline (

s>0

). Error bars are

95\%

confidence intervals: Wilson for the binomial rejection rate in panel (a); Student-

t

for the AUC panels (b, c, d). Intervals may be too narrow to be visible at highly precise points. Points are horizontally jittered to avoid overlap.

Mode	None	Subj. overlap	Batch conf.	Peek norm.	Look-ahead
Subj. grouped	0.06 (.03)	1.00	0.98 (.02)	1.00	0.04 (.03)
Batch blocked	0.04 (.03)	1.00	0.02 (.02)	1.00	0.02 (.02)
Study LOOCV	0.06 (.03)	1.00	0.98 (.02)	1.00	0.04 (.03)
Time series	0.02 (.02)	1.00	0.98 (.02)	1.00	0.02 (.02)

Table 4: Supplementary split-mode analysis: rejection rates (permutation-gap

p<0.05

) at

(n=500,p=20,s=0)

Figure 3: Multivariate target-scan

p

-values at

s=0

	Guarded	Guarded (no FS)	Naive (no leaks)	Leaky
Split design	Study LOOCV	Study LOOCV	5-fold CV	5-fold CV
Feature selection	t-test top 100	None	None	None
Leak features	No	No	No	Yes
AUC	0.610	0.682	0.717	0.810
SD	0.063	0.079	0.025	0.024
Perm. gap	0.101	0.173	0.208	0.301

Table 5: Case-study summary. The four columns form a decomposition chain: each successive pipeline changes exactly one factor (feature selection, split design, or leakage features), isolating its contribution to the AUC gap. Study-fold

V

is Cramér’s

V

between outer fold assignment and study identity; the multivariate scan

p

is from the target-scan audit.

Figure 4: Case-study summaries. (a) Top univariate target-association scores (rescaled AUC,

|\text{AUC}-0.5|\times 2

) for the leaky pipeline feature matrix; asterisks mark injected synthetic leakage features. (b) Repeat-level inflation scores

\Delta_{r}

(leaky minus guarded AUC) from

\Delta_{\mathrm{LSI}}

with 20 paired repeats; the solid line is the robust Huber estimate

\widehat{\Delta}_{\mathrm{LSI}}

, the dashed line is the arithmetic mean, and the shaded band is the

95\%

BCa bootstrap interval.

Function	Binary	Multiclass	Regression	Survival
make_split_plan	✓	✓	✓	✓
fit_resample	✓	✓	✓	✓
tune_resample	✓	✓	✓	—
audit_leakage	✓	(✓)	(✓)	(✓)
delta_lsi	✓	(✓)	(✓)	(✓)
Target scan	✓	(✓)	(✓)	—

Table 6: Function-level task-type coverage. ✓ = supported, (✓) = supported but not empirically evaluated in this paper, — = not yet implemented.

Explore similar work

fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R
Apr 6, 2026Selcuk Korkmaz, Dincer Goksuluk, Eda KaraismailogluAutomated Machine Learning
Faculty of Medicine, Department of Biostatistics, Trakya University, Edirne 22030, Türkiye · Faculty of Medicine, Department of Biostatistics, Sakarya University, Sakarya 54290, Türkiye · Faculty of Medicine, Department of Biostatistics, University of Health Sciences, Ankara 06010, Türkiye
Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets
Apr 5, 2026Simon RothData Leakage Tabular ML
A prior-free blind detection of information leakage from model predictions
Jun 9, 2026Laurence A. JacobsData Leakage Black-Box Model Auditing
Center for Molecular Cardiology, University of Zurich, Zurich, Switzerland · Center for Complexity Sciences, National University of Mexico, Mexico City, Mexico

Apr 6, 2026J/K move · Enter open · S save

fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R

Selcuk Korkmaz, Dincer Goksuluk, Eda Karaismailoglu

Faculty of Medicine, Department of Biostatistics, Trakya University, Edirne 22030, Türkiye · Faculty of Medicine, Department of Biostatistics, Sakarya University, Sakarya 54290, Türkiye · Faculty of Medicine, Department of Biostatistics, University of Health Sciences, Ankara 06010, Türkiye

Automated Machine Learning

Open paper AskPDF arXiv

Preprocessing leakage arises when scaling, imputation, or other data-dependent transformations are estimated before resampling, inflating apparent performance while remaining hard to detect. We present fastml, an R package that provides a single-call interface for leakage-aware machine learning through guarded resampling, where preprocessing is re-estimated inside each resample and applied to the corresponding assessment data. The package supports grouped and time-ordered resampling, blocks high-risk configurations, audits recipes for external dependencies, and includes sandboxed execution and integrated model explanation. We evaluate fastml with a Monte Carlo simulation contrasting global and fold-local normalization, a usability comparison with tidymodels under matched specifications, and survival benchmarks across datasets of different sizes. The simulation demonstrates that global preprocessing substantially inflates apparent performance relative to guarded resampling. fastml matched held-out performance obtained with tidymodels while reducing workflow orchestration, and it supported consistent benchmarking of multiple survival model classes through a unified interface.

Abstract

Explore similar work

fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R

Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets

A prior-free blind detection of information leakage from model predictions