Organizations: Faculty of Medicine, Department of Biostatistics, Trakya University, Edirne 22030, Türkiye · Faculty of Medicine, Department of Biostatistics, Sakarya University, Sakarya 54290, Türkiye · Faculty of Medicine, Department of Biostatistics, University of Health Sciences, Ankara 06010, Türkiye
Preprocessing leakage arises when scaling, imputation, or other data-dependent transformations are estimated before resampling, inflating apparent performance while remaining hard to detect. We present fastml, an R package that provides a single-call interface for leakage-aware machine learning through guarded resampling, where preprocessing is re-estimated inside each resample and applied to the corresponding assessment data. The package supports grouped and time-ordered resampling, blocks high-risk configurations, audits recipes for external dependencies, and includes sandboxed execution and integrated model explanation. We evaluate fastml with a Monte Carlo simulation contrasting global and fold-local normalization, a usability comparison with tidymodels under matched specifications, and survival benchmarks across datasets of different sizes. The simulation demonstrates that global preprocessing substantially inflates apparent performance relative to guarded resampling. fastml matched held-out performance obtained with tidymodels while reducing workflow orchestration, and it supported consistent benchmarking of multiple survival model classes through a unified interface.
Figures & tables
Figure 1: Illustration of a leaky cross-validation workflow caused by global preprocessing.
Slot
Description
Practical guarantee
models
Fitted model objects, indexed by algorithm and engine
Models are stored as fitted objects; downstream predict / plot methods are intended to use stored fits rather than refitting. Reproducibility depends on package versions and environment consistency.
preprocessor
Stored preprocessing object
The stored preprocessor corresponds to the final training split ( train_data ), not to per-fold analysis subsets. Fold-local preprocessing occurs inside resampling when that path is used, but the stored final preprocessor is trained on the training split used for the final fit.
predictions
Stored predictions from evaluation
Predictions stored here are the holdout/test predictions produced during evaluation. Training-set predictions and per-fold resampling predictions are not necessarily stored in the same slot by default.
performance
Aggregated performance metrics
Metrics are computed from evaluation outputs (typically holdout/test predictions). If resampling is used, summary performance reflects the resampling results recorded by the resampling layer.
metric
Primary evaluation metric
Defines the main comparison target used in model reporting within the object.
best_model_name
Identifier of the default reference model
A presentation/default-selection label; it should not be interpreted as changing how metrics were computed.
Table 1: Core components of a fastml object and associated guarantees.
Feature
caret (Legacy)
tidymodels (Modular)
mlr3 (High-Performance)
h2o (AutoML)
fastml (Guarded Facade)
Interface Philosophy
Unified Wrapper
Modular Pipeline
Object-Oriented
Integrated Engine
Guarded Facade
Function-based syntax (e.g., train() )
Verbose composition (Recipe + Spec + Workflow)
R6 classes (Tasks, Learners, Graphs)
Server-client AutoML platform
Simplified single-function call ( fastml() )
Preprocessing and Leakage Control
Manual
Optional/User-Assembled
Robust / Graph-Based
Automated
Enforced Guarded Resampling
User must manually prevent leakage; error-prone.
Tools exist (recipes) but isolation is not enforced.
Pipelines define explicit graphs to handle flow.
Internal handling within the H2O engine.
Preprocessing is strictly re-estimated per fold by default.
Table 2: Architectural and functional comparison of major R machine learning frameworks.
Figure 2: Preprocessing placement under grouped cross-validation: (A) ROC AUC distributions for leaky versus guarded workflows; (B) paired ROC AUC showing inflation with global preprocessing.
Dataset
Cox PH (Baseline)
Weibull AFT (Baseline)
fastml (Penalized Cox)
fastml (XGBoost AFT)
fastml (RF Survival)
Lung (N = 228)
0.631 ± 0.064 (0.530 to 0.716)
0.633 ± 0.062 (0.535 to 0.716)
0.641 ± 0.066 (0.528 to 0.739)
0.618 ± 0.050 (0.521 to 0.700)
0.641 ± 0.063 (0.517 to 0.734)
Rotterdam (N = 2,982)
0.690 ± 0.014 (0.669 to 0.713)
0.690 ± 0.014 (0.669 to 0.712)
0.695 ± 0.015 (0.673 to 0.718)
0.705 ± 0.012 (0.690 to 0.724)
0.712 ± 0.014 (0.688 to 0.730)
Flchain (N = 7,874)
0.794 ± 0.013 (0.775 to 0.817)
0.794 ± 0.013 (0.776 to 0.817)
0.793 ± 0.012 (0.776 to 0.816)
0.795 ± 0.014 (0.778 to 0.819)
0.799 ± 0.013 (0.784 to 0.823)
Table 3: Mean standardized Harrell ' s C-index ± SD (95% CI) across data scales from the external benchmark.
Data leakage remains a recurrent source of optimistic bias in biomedical machine learning studies. Standard row-wise cross-validation and globally estimated preprocessing steps are often inappropriate for data with repeated measurements, study-level heterogeneity, batch effects, or temporal dependencies. This paper describes bioLeak, an R package for constructing leakage-aware resampling workflows and for auditing fitted models for common leakage mechanisms. The package provides leakage-aware split construction, train-fold-only preprocessing, cross-validated model fitting, nested hyperparameter tuning, post hoc leakage audits, and HTML reporting. The implementation supports binary classification, multiclass classification, regression, and survival analysis, with task-specific metrics and S4 containers for splits, fits, audits, and inflation summaries. The simulation artifacts show how apparent performance changes under controlled leakage mechanisms, and the case study illustrates how guarded and leaky pipelines can yield materially different conclusions on multi-study transcriptomic data. The emphasis throughout is on software design, reproducible workflows, and interpretation of diagnostic output.
Selçuk Korkmaz
Department of Biostatistics and Medical Informatics, Faculty of Medicine, Trakya University, 22030 Edirne, Türkiye
Data leakage has been identified in 648 published papers across 30 scientific fields. The knowledge to prevent it has existed for over a decade; the problem persists because the tools do not enforce what the textbooks teach. This paper presents a grammar (eight typed primitives connected by a directed acyclic graph with four hard constraints) that makes the most damaging leakage types structurally unrepresentable within the grammar's scope. The core mechanism is a terminal assessment gate: the first call-time-enforced evaluate/assess boundary documented in the peer-reviewed ML methodology literature (to my knowledge, as of May 2026), backed by a specification precise enough for independent reimplementation. A companion landscape study across 2,047 datasets grounds the constraints in measured effect sizes. Two reference implementations (Python, R) are available.
Visual assessment of residual plots is a common approach for diagnosing linear models, but it relies on manual evaluation, which does not scale well and can lead to inconsistent decisions across analysts. The lineup protocol, which embeds the observed plot among null plots, can reduce subjectivity but requires even more human effort. In today's data-driven world, such tasks are well suited for automation. We present a new R package that uses a computer vision model to automate the evaluation of residual plots. An accompanying Shiny application is provided for ease of use. Given a sample of residuals, the model predicts a visual signal strength (VSS) and offers supporting information to help analysts assess model fit.
Weihao Li, Dianne Cook, Emi Tanaka +2
Department of Econometrics and Business Statistics, Monash University, Wellington Road, VIC 3800, Australia · Research School of Finance, Actuarial Studies and Statistics, The Australian National University, CBE Building 26C, Kingsley Street, ACT 2600, Australia · Department of Statistics, University of Nebraska, Hardin Hall, 3310 Holdrege St Suite 340, Lincoln, NE 68583, United States