fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R

Selcuk Korkmaz; Dincer Goksuluk; Eda Karaismailoglu

stat.COApr 6, 2026

fastml: Guarded Resampling Workflows for Safer Automated Machine Learning in R

Authors: Selcuk Korkmaz, Dincer Goksuluk, Eda Karaismailoglu

Organizations: Faculty of Medicine, Department of Biostatistics, Trakya University, Edirne 22030, Türkiye · Faculty of Medicine, Department of Biostatistics, Sakarya University, Sakarya 54290, Türkiye · Faculty of Medicine, Department of Biostatistics, University of Health Sciences, Ankara 06010, Türkiye

Abstract

Preprocessing leakage arises when scaling, imputation, or other data-dependent transformations are estimated before resampling, inflating apparent performance while remaining hard to detect. We present fastml, an R package that provides a single-call interface for leakage-aware machine learning through guarded resampling, where preprocessing is re-estimated inside each resample and applied to the corresponding assessment data. The package supports grouped and time-ordered resampling, blocks high-risk configurations, audits recipes for external dependencies, and includes sandboxed execution and integrated model explanation. We evaluate fastml with a Monte Carlo simulation contrasting global and fold-local normalization, a usability comparison with tidymodels under matched specifications, and survival benchmarks across datasets of different sizes. The simulation demonstrates that global preprocessing substantially inflates apparent performance relative to guarded resampling. fastml matched held-out performance obtained with tidymodels while reducing workflow orchestration, and it supported consistent benchmarking of multiple survival model classes through a unified interface.

Figures & tables

Figure 1: Illustration of a leaky cross-validation workflow caused by global preprocessing.

Slot	Description	Practical guarantee
models	Fitted model objects, indexed by algorithm and engine	Models are stored as fitted objects; downstream predict / plot methods are intended to use stored fits rather than refitting. Reproducibility depends on package versions and environment consistency.
preprocessor	Stored preprocessing object	The stored preprocessor corresponds to the final training split ( train_data ), not to per-fold analysis subsets. Fold-local preprocessing occurs inside resampling when that path is used, but the stored final preprocessor is trained on the training split used for the final fit.
predictions	Stored predictions from evaluation	Predictions stored here are the holdout/test predictions produced during evaluation. Training-set predictions and per-fold resampling predictions are not necessarily stored in the same slot by default.
performance	Aggregated performance metrics	Metrics are computed from evaluation outputs (typically holdout/test predictions). If resampling is used, summary performance reflects the resampling results recorded by the resampling layer.
metric	Primary evaluation metric	Defines the main comparison target used in model reporting within the object.
best_model_name	Identifier of the default reference model	A presentation/default-selection label; it should not be interpreted as changing how metrics were computed.

Table 1: Core components of a fastml object and associated guarantees.

Feature	caret (Legacy)	tidymodels (Modular)	mlr3 (High-Performance)	h2o (AutoML)	fastml (Guarded Facade)
Interface Philosophy	Unified Wrapper	Modular Pipeline	Object-Oriented	Integrated Engine	Guarded Facade
	Function-based syntax (e.g., train() )	Verbose composition (Recipe + Spec + Workflow)	R6 classes (Tasks, Learners, Graphs)	Server-client AutoML platform	Simplified single-function call ( fastml() )
Preprocessing and Leakage Control	Manual	Optional/User-Assembled	Robust / Graph-Based	Automated	Enforced Guarded Resampling
	User must manually prevent leakage; error-prone.	Tools exist (recipes) but isolation is not enforced.	Pipelines define explicit graphs to handle flow.	Internal handling within the H2O engine.	Preprocessing is strictly re-estimated per fold by default.
Survival Analysis Support	Basic	Expanding	Extensive	Standard	Native Hybrid
	Primarily Cox Proportional Hazards	Via censored extension package.	Via mlr3 extensions.	Cox AFT models	Unified interface for Penalized Cox + Native XGBoost AFT & Piecewise Exp.

Table 2: Architectural and functional comparison of major R machine learning frameworks.

Figure 2: Preprocessing placement under grouped cross-validation: (A) ROC AUC distributions for leaky versus guarded workflows; (B) paired ROC AUC showing inflation with global preprocessing.

Dataset	Cox PH (Baseline)	Weibull AFT (Baseline)	fastml (Penalized Cox)	fastml (XGBoost AFT)	fastml (RF Survival)
Lung (N = 228)	0.631 $\pm$ 0.064 (0.530 to 0.716)	0.633 $\pm$ 0.062 (0.535 to 0.716)	0.641 $\pm$ 0.066 (0.528 to 0.739)	0.618 $\pm$ 0.050 (0.521 to 0.700)	0.641 $\pm$ 0.063 (0.517 to 0.734)
Rotterdam (N = 2,982)	0.690 $\pm$ 0.014 (0.669 to 0.713)	0.690 $\pm$ 0.014 (0.669 to 0.712)	0.695 $\pm$ 0.015 (0.673 to 0.718)	0.705 $\pm$ 0.012 (0.690 to 0.724)	0.712 $\pm$ 0.014 (0.688 to 0.730)
Flchain (N = 7,874)	0.794 $\pm$ 0.013 (0.775 to 0.817)	0.794 $\pm$ 0.013 (0.776 to 0.817)	0.793 $\pm$ 0.012 (0.776 to 0.816)	0.795 $\pm$ 0.014 (0.778 to 0.819)	0.799 $\pm$ 0.013 (0.784 to 0.823)

Table 3: Mean standardized Harrell ' s C-index

\pm

SD (95% CI) across data scales from the external benchmark.

Explore similar work

bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R
Apr 13, 2026Selçuk KorkmazData Leakage ML Reproducibility
Department of Biostatistics and Medical Informatics, Faculty of Medicine, Trakya University, 22030 Edirne, Türkiye
A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time
Mar 11, 2026Simon RothData Leakage
Automated Residual Plot Assessment With the R Package autovi and the Shiny Application autovi.web
Jun 23, 2026Weihao Li, Dianne Cook, Emi Tanaka +2Automated Evaluation
Department of Econometrics and Business Statistics, Monash University, Wellington Road, VIC 3800, Australia · Research School of Finance, Actuarial Studies and Statistics, The Australian National University, CBE Building 26C, Kingsley Street, ACT 2600, Australia · Department of Statistics, University of Nebraska, Hardin Hall, 3310 Holdrege St Suite 340, Lincoln, NE 68583, United States

Apr 13, 2026J/K move · Enter open · S save

bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R

Selçuk Korkmaz

Department of Biostatistics and Medical Informatics, Faculty of Medicine, Trakya University, 22030 Edirne, Türkiye

Data Leakage ML Reproducibility Temporal Data Leakage

Open paper AskPDF arXiv

Data leakage remains a recurrent source of optimistic bias in biomedical machine learning studies. Standard row-wise cross-validation and globally estimated preprocessing steps are often inappropriate for data with repeated measurements, study-level heterogeneity, batch effects, or temporal dependencies. This paper describes bioLeak, an R package for constructing leakage-aware resampling workflows and for auditing fitted models for common leakage mechanisms. The package provides leakage-aware split construction, train-fold-only preprocessing, cross-validated model fitting, nested hyperparameter tuning, post hoc leakage audits, and HTML reporting. The implementation supports binary classification, multiclass classification, regression, and survival analysis, with task-specific metrics and S4 containers for splits, fits, audits, and inflation summaries. The simulation artifacts show how apparent performance changes under controlled leakage mechanisms, and the case study illustrates how guarded and leaky pipelines can yield materially different conclusions on multi-study transcriptomic data. The emphasis throughout is on software design, reproducible workflows, and interpretation of diagnostic output.

Abstract

Explore similar work

bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R

A Grammar of Machine Learning Workflows: Rejecting Data Leakage at Call Time

Automated Residual Plot Assessment With the R Package autovi and the Shiny Application autovi.web