cs.LGNov 1, 2026

Retrain, Alert, or Rollback? A Three-Response Decision Framework for Production ML Under Concept Drift

Authors: Ali Sadeghian, Alireza Shafaei, Hamid Hadibeik, Babak Hossein Khalaj

Organizations: Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran. · Independent Researcher, San Jose, CA, USA.

Abstract

When a deployed model degrades under concept drift, an operator can retrain it, raise an alert and monitor, or roll back. Yet most drift-adaptation methods consider only retraining. We present RAR (Retrain-Alert-Rollback), a state machine that gives each response an explicit trigger and cost. Retraining is gated on drift detection, label availability, and measured degradation. Every retrained model must pass a two-stage canary before promotion, and corruption-triggered and quality-triggered rollbacks enter different hold states. Under a prospectively specified protocol we compare RAR with six baselines, including a cost-aware retraining policy, on 13 streams: eight real public streams and five semi-synthetic streams derived from them. RAR does not dominate. At low retraining cost, retrain-heavy baselines lead, and no policy's rank differs significantly from RAR's. At higher cost, RAR beats detector-triggered retraining, periodic retraining, and no response simultaneously on four streams, each within a bounded cost window. Its advantage over detector-triggered retraining is consistent with a latency-feasibility condition stated in advance. Against the cost-aware policy, results are heterogeneous. RAR retrains before true drift onsets one eighth to one third as often as detector-triggered retraining, but at lower net utility on most semi-synthetic streams. On H2's designated corruption stream (a registered follow-up), rollbacks fire yet RAR loses to retraining baselines. At ρ=1ρ= 1, removing the canary, and thus rollback, raises utility on 11 of 13 streams. On a post-hoc follow-up stream, cause-specific hold release resumes adaptation where a single cooldown does not. We provide the harness, configurations, and per-cell results for reproduction.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 19, 2026

When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints

Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.
Nov 6, 2026

Multi-objective drift-aware MLOps agent for model retraining

Deployed ML models face non-stationary data and delayed labels, yet most retraining strategies rely on fixed schedules or brittle drift thresholds that ignore operational cost. In this setting, retraining is a sequential decision problem rather than a one-time threshold choice. We present DARE (Drift-Aware Retraining Engine), an offline, event-driven retraining framework that separates ingestion, monitoring, policy decisions, training, and logging through explicit schemas and deterministic replay. We compare DARE against practical heuristic retraining baselines, including fixed-interval, threshold-persistence, and action-rate-matched fixed scheduling policies. On Airline, the locked LinUCB policy achieves a higher mean F1 while substantially reducing intervention frequency relative to heuristic and action-rate-matched baselines. Additional runtime-calibrated evaluation shows that these gains persist when measured using the cumulative update wall-clock time. These results suggest that conservative sequential retraining policies can improve performance while reducing unnecessary operational overhead under delayed feedback.
Mar 9, 2026

When to Retrain after Drift: A Data-Only Test of Post-Drift Data Size Sufficiency

Sudden concept drift makes previously trained predictors unreliable, yet deciding when to retrain and what post-drift data size is sufficient is rarely addressed. We propose CALIPER - a detector- and model-agnostic, data-only test that estimates the post-drift data size required for stable retraining. CALIPER exploits state dependence in streams generated by dynamical systems: we run a single-pass weighted local regression over the post-drift window and track a one-step proxy error as a function of a locality parameter θθ. When an effective sample size gate is satisfied, a monotonically non-increasing trend in this error with increasing a locality parameter indicates that the data size is sufficiently informative for retraining. We also provide a theoretical analysis of our method, and we show that the algorithm has a low per-update time and memory. Across datasets from four heterogeneous domains, three learner families, and two detectors, CALIPER consistently matches or exceeds the best fixed data size for retraining while incurring negligible overhead and often outperforming incremental updates. CALIPER closes the gap between drift detection and data-sufficient adaptation in streaming learning.