When to Retrain after Drift: A Data-Only Test of Post-Drift Data Size Sufficiency

Ren Fujiwara; Yasuko Matsubara; Yasushi Sakurai

cs.LGMar 9, 2026

When to Retrain after Drift: A Data-Only Test of Post-Drift Data Size Sufficiency

Authors: Ren Fujiwara, Yasuko Matsubara, Yasushi Sakurai

Organizations: SANKEN, The University of Osaka, Japan

Abstract

Sudden concept drift makes previously trained predictors unreliable, yet deciding when to retrain and what post-drift data size is sufficient is rarely addressed. We propose CALIPER - a detector- and model-agnostic, data-only test that estimates the post-drift data size required for stable retraining. CALIPER exploits state dependence in streams generated by dynamical systems: we run a single-pass weighted local regression over the post-drift window and track a one-step proxy error as a function of a locality parameter $θ$ . When an effective sample size gate is satisfied, a monotonically non-increasing trend in this error with increasing a locality parameter indicates that the data size is sufficiently informative for retraining. We also provide a theoretical analysis of our method, and we show that the algorithm has a low per-update time and memory. Across datasets from four heterogeneous domains, three learner families, and two detectors, CALIPER consistently matches or exceeds the best fixed data size for retraining while incurring negligible overhead and often outperforming incremental updates. CALIPER closes the gap between drift detection and data-sufficient adaptation in streaming learning.

Figures & tables

Figure 1: Overview of CALIPER . (a) Unlike window-based detectors (e.g., ADWIN/KSWIN) that only indicate if/when drift occurs, CALIPER estimates how much post-drift data are needed for stable retraining. (b) State dependence: for dynamical systems

(\mathbf{x}_{t+1}=f(\mathbf{x}_{t}))

, nearby states exhibit similar one-step transitions; thus data sufficiency reduces to testing whether the post-drift window exhibits adequate state dependence. (c) Pipeline: a locality parameter

\theta

reweights nearby samples in weighted local regression; when the proxy error is monotonically non-increasing as

\theta

increases and the neighborhood is sufficiently populated, retraining is triggered. (d) Results: star markers denote CALIPER’s estimated data sizes that yield low post-drift errors across heterogeneous learners (Kernel Ridge Regression (KRR), MLP, Transformer). CALIPER selects the optimal post-drift data size—i.e., the point at which retraining would be stable—without any retraining.

Figure 2: CALIPER , a model-agnostic framework for dynamically estimating the data size required for retraining after sudden concept drift in a data stream: (i) Window normalization and split: after a drift alarm, the post-drift segment is normalized and partitioned into reference pairs

(\mathbf{X}_{h},\mathbf{Y}_{h})

and a query

(\mathbf{x}_{q},\mathbf{y}_{q})

. (ii) ESS check: kernel weights

\mathbf{w}_{\theta}=\exp(-\theta\times\mathbf{r})

at the largest

\theta

define an effective neighborhood; proceed only if

ESS\geq C\times(d+1)

. (iii) Weighted local regression: for each

\theta

on a fixed grid, solve the weighted normal equations to obtain

\hat{\mathbf{y}}_{\theta}

and compute a proxy prediction error. (iv) Test and trigger: a monotonic non-increase of the error as

\theta

increases, sustained for consecutive updates, indicates (a proxy for) sufficient local regularity/state dependence and triggers retraining. The rightmost panel illustrates the monotone trend of error versus

\theta

Figure 3: Performance of CALIPER on four datasets (MoCap, TEP, Automobile, Dysts) and three model families (KRR, MLP, Transformer). We compare fixed data sizes (128/512/2048; blue) with CALIPER (orange, “CALIPER”). Each panel reports MSE (left, “-i”) and MAE (right, “-ii”) as a function of the retraining data size after each drift detected by ADWIN (circles) or KSWIN (squares). CALIPER matches or exceeds the best fixed data size without per-dataset tuning; notably, the data size selected by CALIPER typically aligns with the dataset-specific optimal fixed data size. See Tables 3 and 2 for full results in Appendix G .

Figure 4: Per time step and average wall clock time on Dysts sequences #1 and #2 for ADWIN/KSWIN with CALIPER and fixed-size buffers (128/512/2048); curves are flat with low means, and occasional spikes reflect retraining rather than CALIPER .

Model	Detector	MoCap		TEP		Automobile		Dysts
Model	Detector	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MLP	ADWIN	7.106	1.387	1.790	0.704	2.090	0.892	0.432	0.460
	KSWIN	7.449	1.436	1.760	0.699	2.081	0.889	0.467	0.468
	(Incremental)	412.6	8.462	3.699	0.879	2.406	0.894	71.75	1.524
Transformer	ADWIN	10.96	1.744	1.818	0.698	1.948	0.883	0.549	0.530
	KSWIN	11.35	1.810	1.808	0.690	2.070	0.881	0.501	0.505

Table 1: Adaptation after drift. CALIPER-triggered retraining vs. Incremental updating. MSE/MAE averaged over (1,15,30) with the past sequence length is 30; detectors ADWIN/KSWIN; models KRR/MLP/Transformer. Lower is better; best bold, second underlined. Tables 3 and 2 in the Appendix for the full results.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The average per time step wall-clock time (bar chart) for MoCap, TEP, Automobile, and Dysts under ADWIN/KSWIN with CALIPER vs. fixed data size (128/512/2048) using KRR/MLP/Transformer; CALIPER matches fixed data size baselines.

Figure 6: Hyperparameter sensitivity of CALIPER. Test MAE as a function of the ESS multiplier

C\in\{2,3,4\}

(left panels) and the locality parameter

\theta_{\max}\in\{12,14,16,18,20\}

(right panels) on the four datasets and for both ADWIN and KSWIN. The y-axis is on the same absolute MAE scale as in the main results. The curves are nearly flat on three of the datasets, and only large values of

\theta_{\max}

on the MoCap dataset cause a noticeable degradation, consistent with the role of

\theta_{\max}

as a locality parameter.

Dataset	Model	$l_{s}$	CALIPER		Fix (128)		Fix (512)		Fix (2048)		Incremental
Dataset	Model	$l_{s}$	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MoCap	KRR	1	6.952	1.146	7.607	1.339	12.369	1.719	8.074	1.228	16.003	2.201
		15	7.691	1.480	8.934	1.561	15.165	2.230	10.010	1.802	19.552	2.538
		30	9.431	1.716	9.879	1.650	17.725	2.395	19.373	2.565	18.946	2.524
		Avg	8.025	1.447	8.807	1.516	15.086	2.114	12.486	1.865	18.167	2.421
	ExtraTrees	1	7.822	1.455	15.295	2.025	11.100	1.804	9.541	1.626	35.622	3.337

Appendix

Table 2: Full results with ADWIN. For each dataset and model family, we report MSE/MAE at prediction horizons (1, 15, 30) and their average. We compare CALIPER (ours) with fixed retraining windows of 128, 512, and 2048 steps (“Fix (·)”) and with an online Incremental baseline (no explicit window retraining). The past sequence length is 30. Datasets span four heterogeneous domains (TEP, MoCap, Automobile, Dysts). Lower is better; best per column in bold (second best underlined, if applicable).

Dataset	Model	$l_{s}$	CALIPER		Fix (128)		Fix (512)		Fix (2048)		Incremental
Dataset	Model	$l_{s}$	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
MoCap	KRR	1	5.979	1.060	7.726	1.312	10.824	1.592	8.052	1.215	16.003	2.201
		15	7.690	1.428	9.842	1.593	12.718	1.978	9.483	1.726	19.552	2.538
		30	9.471	1.656	10.853	1.759	14.109	2.154	16.875	2.434	18.946	2.524
		Avg	7.713	1.381	9.474	1.554	12.550	1.908	11.470	1.792	18.167	2.421
	ExtraTrees	1	7.348	1.374	13.249	1.867	10.291	1.750	9.456	1.599	35.622	3.337

Appendix

Table 3: Full results with KSWIN. For each dataset and model family, we report MSE/MAE at prediction horizons (1, 15, 30) and their average. We compare CALIPER (ours) with fixed retraining windows of 128, 512, and 2048 steps (“Fix (·)”) and with an online Incremental baseline (no explicit window retraining). The past sequence length is 30. Datasets span four heterogeneous domains (TEP, MoCap, Automobile, Dysts). Lower is better; best per column in bold (second best underlined, if applicable).

Concept Drift

Concept drift is change over time in the relationship between data and target outcomes, which can erode the accuracy of deployed models.

See all 90 papers

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
Aug 19, 2026Sawan DasariDrifting Models Concept Drift
SCCM : Stream Cruise Control Method for Automated Drift Detection and Adaptation
Sep 8, 2026Mohammad Abu-Shaira, Weishi ShiConcept Drift Adaptive Learning
Department of Computer Science and Engineering, University of North Texas, 1155 Union Circle #311366, Denton, 76203, Texas, USA
Learner-based Concept Drift Detection: Analysis and Evaluation
Jun 18, 2026Md Moman Ul Haque Khan, Samira SadaouiConcept Drift
Department of Computer Science, University of Regina, Canada

Aug 19, 2026J/K move · Enter open · S save

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

Sawan Dasari

Drifting Models Concept Drift Streaming Algorithms Incremental Learning

Open paper AskPDF arXiv

Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.

Abstract

Concept Drift

Explore similar work

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

SCCM : Stream Cruise Control Method for Automated Drift Detection and Adaptation

Learner-based Concept Drift Detection: Analysis and Evaluation