When to Retrain after Drift: A Data-Only Test of Post-Drift Data Size Sufficiency

Ren Fujiwara; Yasuko Matsubara; Yasushi Sakurai

cs.LGMar 9, 2026

When to Retrain after Drift: A Data-Only Test of Post-Drift Data Size Sufficiency

Authors: Ren Fujiwara, Yasuko Matsubara, Yasushi Sakurai

Organizations: SANKEN, The University of Osaka, Japan

Abstract

Sudden concept drift makes previously trained predictors unreliable, yet deciding when to retrain and what post-drift data size is sufficient is rarely addressed. We propose CALIPER - a detector- and model-agnostic, data-only test that estimates the post-drift data size required for stable retraining. CALIPER exploits state dependence in streams generated by dynamical systems: we run a single-pass weighted local regression over the post-drift window and track a one-step proxy error as a function of a locality parameter $θ$ . When an effective sample size gate is satisfied, a monotonically non-increasing trend in this error with increasing a locality parameter indicates that the data size is sufficiently informative for retraining. We also provide a theoretical analysis of our method, and we show that the algorithm has a low per-update time and memory. Across datasets from four heterogeneous domains, three learner families, and two detectors, CALIPER consistently matches or exceeds the best fixed data size for retraining while incurring negligible overhead and often outperforming incremental updates. CALIPER closes the gap between drift detection and data-sufficient adaptation in streaming learning.

Figures & tables

Figure 1: Overview of CALIPER . (a) Unlike window-based detectors (e.g., ADWIN/KSWIN) that only indicate if/when drift occurs, CALIPER estimates how much post-drift data are needed for stable retraining. (b) State dependence: for dynamical systems

(\mathbf{x}_{t+1}=f(\mathbf{x}_{t}))

, nearby states exhibit similar one-step transitions; thus data sufficiency reduces to testing whether the post-drift window exhibits adequate state dependence. (c) Pipeline: a locality parameter

\theta

reweights nearby samples in weighted local regression; when the proxy error is monotonically non-increasing as

\theta

increases and the neighborhood is sufficiently populated, retraining is triggered. (d) Results: star markers denote CALIPER’s estimated data sizes that yield low post-drift errors across heterogeneous learners (Kernel Ridge Regression (KRR), MLP, Transformer). CALIPER selects the optimal post-drift data size—i.e., the point at which retraining would be stable—without any retraining.

Figure 2: CALIPER , a model-agnostic framework for dynamically estimating the data size required for retraining after sudden concept drift in a data stream: (i) Window normalization and split: after a drift alarm, the post-drift segment is normalized and partitioned into reference pairs

(\mathbf{X}_{h},\mathbf{Y}_{h})

and a query

(\mathbf{x}_{q},\mathbf{y}_{q})

. (ii) ESS check: kernel weights

\mathbf{w}_{\theta}=\exp(-\theta\times\mathbf{r})

at the largest

\theta

define an effective neighborhood; proceed only if

ESS\geq C\times(d+1)

. (iii) Weighted local regression: for each

\theta

on a fixed grid, solve the weighted normal equations to obtain

\hat{\mathbf{y}}_{\theta}

and compute a proxy prediction error. (iv) Test and trigger: a monotonic non-increase of the error as

\theta

increases, sustained for consecutive updates, indicates (a proxy for) sufficient local regularity/state dependence and triggers retraining. The rightmost panel illustrates the monotone trend of error versus

\theta

Figure 3: Performance of CALIPER on four datasets (MoCap, TEP, Automobile, Dysts) and three model families (KRR, MLP, Transformer). We compare fixed data sizes (128/512/2048; blue) with CALIPER (orange, “CALIPER”). Each panel reports MSE (left, “-i”) and MAE (right, “-ii”) as a function of the retraining data size after each drift detected by ADWIN (circles) or KSWIN (squares). CALIPER matches or exceeds the best fixed data size without per-dataset tuning; notably, the data size selected by CALIPER typically aligns with the dataset-specific optimal fixed data size. See Tables 3 and 2 for full results in Appendix G .

Figure 4: Per time step and average wall clock time on Dysts sequences #1 and #2 for ADWIN/KSWIN with CALIPER and fixed-size buffers (128/512/2048); curves are flat with low means, and occasional spikes reflect retraining rather than CALIPER .

Model	Detector	MoCap		TEP		Automobile		Dysts
Model	Detector	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MLP	ADWIN	7.106	1.387	1.790	0.704	2.090	0.892	0.432	0.460
	KSWIN	7.449	1.436	1.760	0.699	2.081	0.889	0.467	0.468
	(Incremental)	412.6	8.462	3.699	0.879	2.406	0.894	71.75	1.524
Transformer	ADWIN	10.96	1.744	1.818	0.698	1.948	0.883	0.549	0.530
	KSWIN	11.35	1.810	1.808	0.690	2.070	0.881	0.501	0.505

Table 1: Adaptation after drift. CALIPER-triggered retraining vs. Incremental updating. MSE/MAE averaged over (1,15,30) with the past sequence length is 30; detectors ADWIN/KSWIN; models KRR/MLP/Transformer. Lower is better; best bold, second underlined. Tables 3 and 2 in the Appendix for the full results.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The average per time step wall-clock time (bar chart) for MoCap, TEP, Automobile, and Dysts under ADWIN/KSWIN with CALIPER vs. fixed data size (128/512/2048) using KRR/MLP/Transformer; CALIPER matches fixed data size baselines.

Figure 6: Hyperparameter sensitivity of CALIPER. Test MAE as a function of the ESS multiplier

C\in\{2,3,4\}

(left panels) and the locality parameter

\theta_{\max}\in\{12,14,16,18,20\}

(right panels) on the four datasets and for both ADWIN and KSWIN. The y-axis is on the same absolute MAE scale as in the main results. The curves are nearly flat on three of the datasets, and only large values of

\theta_{\max}

on the MoCap dataset cause a noticeable degradation, consistent with the role of

\theta_{\max}

as a locality parameter.

Dataset	Model	$l_{s}$	CALIPER		Fix (128)		Fix (512)		Fix (2048)		Incremental
Dataset	Model	$l_{s}$	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MoCap	KRR	1	6.952	1.146	7.607	1.339	12.369	1.719	8.074	1.228	16.003	2.201
		15	7.691	1.480	8.934	1.561	15.165	2.230	10.010	1.802	19.552	2.538
		30	9.431	1.716	9.879	1.650	17.725	2.395	19.373	2.565	18.946	2.524
		Avg	8.025	1.447	8.807	1.516	15.086	2.114	12.486	1.865	18.167	2.421
	ExtraTrees	1	7.822	1.455	15.295	2.025	11.100	1.804	9.541	1.626	35.622	3.337

Appendix

Table 2: Full results with ADWIN. For each dataset and model family, we report MSE/MAE at prediction horizons (1, 15, 30) and their average. We compare CALIPER (ours) with fixed retraining windows of 128, 512, and 2048 steps (“Fix (·)”) and with an online Incremental baseline (no explicit window retraining). The past sequence length is 30. Datasets span four heterogeneous domains (TEP, MoCap, Automobile, Dysts). Lower is better; best per column in bold (second best underlined, if applicable).

Dataset	Model	$l_{s}$	CALIPER		Fix (128)		Fix (512)		Fix (2048)		Incremental
Dataset	Model	$l_{s}$	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MoCap	KRR	1	5.979	1.060	7.726	1.312	10.824	1.592	8.052	1.215	16.003	2.201
		15	7.690	1.428	9.842	1.593	12.718	1.978	9.483	1.726	19.552	2.538
		30	9.471	1.656	10.853	1.759	14.109	2.154	16.875	2.434	18.946	2.524
		Avg	7.713	1.381	9.474	1.554	12.550	1.908	11.470	1.792	18.167	2.421
	ExtraTrees	1	7.348	1.374	13.249	1.867	10.291	1.750	9.456	1.599	35.622	3.337

Appendix

Table 3: Full results with KSWIN. For each dataset and model family, we report MSE/MAE at prediction horizons (1, 15, 30) and their average. We compare CALIPER (ours) with fixed retraining windows of 128, 512, and 2048 steps (“Fix (·)”) and with an online Incremental baseline (no explicit window retraining). The past sequence length is 30. Datasets span four heterogeneous domains (TEP, MoCap, Automobile, Dysts). Lower is better; best per column in bold (second best underlined, if applicable).

Concept Drift

Concept drift is change over time in the relationship between data and target outcomes, which can erode the accuracy of deployed models.

See all 90 papers

Aug 19, 2026

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.

Sawan Dasari

Drifting Models Concept Drift Streaming Algorithms Incremental Learning

Ask PDF

Sep 8, 2026

SCCM : Stream Cruise Control Method for Automated Drift Detection and Adaptation

Real-world datasets often exhibit evolving distributions, known as concept drift. Ignoring drift degrades predictive performance, while reliance on fixed hyperparameters further limits model adaptability under changing conditions. Adaptive learning addresses this challenge by continuously updating models online, allowing them to incrementally adjust and remain effective as data distributions evolve. This paper presents the Stream Cruise Control Method (SCCM), a comprehensive framework for drift detection and adaptation in online regression. SCCM enables automated adaptation through early-response, pre-update drift detection, drift magnitude quantification, KPI-window-based thresholding for local false-alarm mitigation, dynamic hyperparameter tuning, and model recalibration. SCCM also adopts an in-memory design for real-time adaptability, unlike purely reactive methods that typically activate adaptation only after performance degradation is observed. By using dynamic thresholding and remaining agnostic to data distributions, SCCM supports KPI-based monitoring across varying data streams, including high-dimensional and large-scale settings. SCCM is integrated with four online regression models and evaluated on 18 synthetic datasets covering abrupt, incremental, and alternating gradual drift, together with eight real-world datasets. The evaluation uses both R2 and MSE and compares against eight detector--adaptation baselines. Results show improved predictive performance and effective drift handling across the evaluated online regression settings.

Mohammad Abu-Shaira, Weishi Shi

Department of Computer Science and Engineering, University of North Texas, 1155 Union Circle #311366, Denton, 76203, Texas, USA

Concept Drift Adaptive Learning

Ask PDF

Jun 18, 2026

Learner-based Concept Drift Detection: Analysis and Evaluation

Machine learning algorithms deployed for evolving streaming environments must handle the non-stationary data distributions, commonly referred to as concept drift. The presence of concept drift poses a major challenge for many real-world applications because it can severely degrade their predictive performance, hindering their ability to support robust decision-making. Consequently, the timely and efficient detection of drift events is critical for sustaining high accuracy over time. This study examines theoretically the concept drift characteristics and numerous drift detection algorithms across several categories. Furthermore, we evaluate their performance on both synthetic and real-world datasets exhibiting diverse streaming scenarios and drift characteristics, such as abrupt and gradual changes. This study aims to enhance understanding of the complex notion of concept drift characteristics and behavior of drift detectors, along with their applicability to diverse contexts.

Md Moman Ul Haque Khan, Samira Sadaoui

Department of Computer Science, University of Regina, Canada

Concept Drift

Ask PDF

Abstract

Concept Drift

Explore similar work

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

SCCM : Stream Cruise Control Method for Automated Drift Detection and Adaptation

Learner-based Concept Drift Detection: Analysis and Evaluation