Machine learning models deployed in non-stationary environments are exposed to temporal distribution shift, which can erode predictive reliability over time. While common mitigation strategies such as periodic retraining and recalibration aim to preserve performance, they typically focus on average metrics evaluated at isolated time points and do not explicitly model how reliability evolves during deployment. We propose a deployment-centric framework that treats reliability as a dynamic state composed of discrimination and calibration. The trajectory of this state across sequential evaluation windows induces a measurable notion of volatility, allowing deployment adaptation to be formulated as a multi-objective control problem that balances reliability stability against cumulative intervention cost. Within this framework, we define a family of state-dependent intervention policies and empirically characterize the resulting cost-volatility Pareto frontier. Experiments on a large-scale, temporally indexed credit-risk dataset (1.35M loans, 2007-2018) show that selective, drift-triggered interventions can achieve smoother reliability trajectories than continuous rolling retraining while substantially reducing operational cost. These findings position deployment reliability under temporal shift as a controllable multi-objective system and highlight the role of policy design in shaping stability-cost trade-offs in high-stakes tabular applications.
Figures & tables
Policy
Trigger
Update Action
P0 (Static)
None
NoOp
P1 (Periodic recalibration)
Every window
Recalibrate (isotonic, using t−1 data)
P2 (Rolling retraining)
Every window
Retrain (rolling W=3 years)
P3 (DTRC)
Drift & reliability thresholds
State-dependent (Retrain / Recalibrate / Both)
P4 (MORC)
Threshold sweep
Pareto-based policy selection
Table 1: Deployment policy family evaluated in this study.
Policy
Hyperparameters
P0 (Static)
No updates; model trained once on pre-2010 data
P1 (Periodic recalib.)
Recalibration interval Δt=1 year (isotonic, previous-year cohort)
P2 (Rolling retrain)
Rolling retraining window W=3 years
P3 (DTRC)
θd1=0.0342,θd2=0.0517,θC=0.0937,θA=0.5663
P4 (MORC)
Grid of (θd1,θd2) pairs (69 candidates, 23 distinct operating points)
Table 2: Policy hyperparameters. DTRC thresholds are instantiated using the MORC knee configuration.
Statistic
Value
Total samples
1,347,681
Date range
2007-06 to 2018-12
Number of features
10
Target variable
Default (binary)
Default rate
19.98%
Evaluation windows ( T )
9 (annual, 2010–2018)
Table 3: Dataset summary and temporal evaluation structure.
Policy
Mean AUC
Mean ECE
Mean Brier
VL1
VL1−
Cost
P0 (Static)
0.608470
0.035015
0.146761
0.032064
0.019486
0
P1 (Periodic recalibration)
0.608128
0.025923
0.144873
0.042180
0.024017
9
P2 (Rolling retrain)
0.661221
0.024565
0.140004
0.030862
0.013754
45
P3 (DTRC)
0.641033
0.031933
0.142921
0.028620
0.016649
12
Table 4: Global policy comparison aggregated over the full deployment horizon. Lower is better for ECE, Brier score, volatility ( VL1 , VL1− ), and cost. Cost is computed under the weighted scheme (5 per retrain, 1 per recalibration, 0 for NoOp).
Figure 1: ROC AUC across deployment windows. Rolling retraining yields the highest average discrimination but exhibits noticeable fluctuations. DTRC maintains intermediate discrimination while avoiding extreme swings.
Figure 2: Expected Calibration Error (ECE) across deployment windows. Periodic recalibration reduces calibration error on average but introduces oscillatory behaviour across windows; DTRC stabilizes calibration without continuous updates.
Figure 3: Drift signal over time summarizing distributional change between evaluation windows. Periods of elevated drift (e.g., mid-horizon years) coincide with higher reliability volatility under static deployment. DTRC activates selectively during these intervals, smoothing subsequent trajectory evolution.
Figure 4: Empirical Pareto frontier in cost–volatility space for MORC policies. Points represent distinct threshold configurations; the highlighted point corresponds to the knee configuration used for DTRC.
Label
Mean AUC
Mean ECE
Mean Brier
VL1
VL1−
Cost
Retrains / Recalibs
P4 (Min-cost)
0.641010
0.026333
0.142472
0.034894
0.016649
11
1 / 1
MORC-Knee
0.641033
0.031933
0.142921
0.028620
0.016649
12
1 / 2
Table 5: Representative MORC policies along the cost–volatility frontier. Cost is computed under the weighted scheme.
Policy
VL1 (95% CI)
VL1− (95% CI)
P0
[0.033, 0.078]
[0.015, 0.041]
P1
[0.036, 0.118]
[0.017, 0.059]
P2
[0.028, 0.066]
[0.013, 0.034]
P3
[0.026, 0.069]
[0.013, 0.035]
Table 6: Block bootstrap 95% confidence intervals for volatility metrics.
Hyperparameter
Value
n_estimators
200
max_depth
4
learning_rate
0.05
subsample
0.9
colsample_bytree
0.9
min_child_weight
5
Table 7: XGBoost hyperparameters used in all experiments.
Temporal distribution shift studies how changing data over time affects model accuracy and reliability, and how systems can detect and adapt to those changes.
Evaluating robustness under temporal distribution shift remains an open challenge. Existing metrics quantify the average decline in performance, but fail to capture how models adapt to evolving data. As a result, temporal degradation is often misinterpreted: when accuracy declines, it is unclear whether the model is failing to adapt or whether the data itself has become inherently more challenging to learn. In this work, we propose three complementary metrics to distinguish adaptation from intrinsic difficulty in the data. Together, these metrics provide a dynamic and interpretable view of model behavior under temporal distribution shift. Results show that our metrics uncover adaptation patterns hidden by existing analysis, offering a richer understanding of temporal robustness in evolving environments.
Lorenzo Iovine, Giacomo Ziffer, Emanuele Della Valle
Politecnico di Milano, DEIB, Via Giuseppe Ponzio, 34, 20133 Milan, Italy
Production machine learning systems degrade under concept drift, yet practitioners have little principled guidance on when to retrain. Retraining is costly, retraining budgets are finite, and a retrained model does not take effect instantly: training and deployment latency leave a stale model serving predictions while the data continues to move. We present a controlled empirical study of three practical model-refresh policies (periodic retraining, error-threshold triggering, and statistical drift-triggered retraining with ADWIN) against a no-retrain baseline, evaluated under a unified system model that makes retraining budgets and training-plus-deployment latency explicit. Across 3,933 experiment runs spanning three drift regimes, three budget levels, up to five latency levels, three datasets, and two learning modes, we find that the single most consequential design decision is not the retraining policy but whether the deployed model learns incrementally. With per-sample incremental updates, and for the linear online learner with immediate labels studied here, no policy differs from the no-retrain baseline by a practically significant margin in any of 54 paired comparisons, even at extreme latency. Without incremental updates, policy choice separates outcomes by 15-55 percentage points of post-drift accuracy, and simple periodic retraining significantly outperforms both reactive policies under abrupt and gradual drift, while reactive policies retain an advantage only under recurring drift. We document systematic failure modes of reactive policies and a latency-budget queueing interaction that silently halves effective retraining budgets, and release the full simulator, dataset pipelines, and per-run artifacts for reproducibility.
Reinforcement Learning (RL) has received increasing attention and adoption in real-world use cases. Most of these systems follow a train-then-fix paradigm, where trained agents do not learn while interacting with the world until performance degrades and retraining becomes necessary. In this position paper, we argue that deploying an agent that is incapable of optimality, but receives an evaluative reward signal, is inherently a continual RL problem. We identify four sources of non-stationarity after deployment that necessitate never-ending learning, and highlight why the best deployed agents never stop adapting. We analyze successful examples of continual RL in the real world, and present the community with the advantages and measures to move away from the current train-then-fix paradigm.
Parnian Behdin, Kevin Roice, Golnaz Mesbahi
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.