Multi-objective drift-aware MLOps agent for model retraining
Organizations: Computer Science California State University, Fullerton Fullerton, United States of America
Abstract
Deployed ML models face non-stationary data and delayed labels, yet most retraining strategies rely on fixed schedules or brittle drift thresholds that ignore operational cost. In this setting, retraining is a sequential decision problem rather than a one-time threshold choice. We present DARE (Drift-Aware Retraining Engine), an offline, event-driven retraining framework that separates ingestion, monitoring, policy decisions, training, and logging through explicit schemas and deterministic replay. We compare DARE against practical heuristic retraining baselines, including fixed-interval, threshold-persistence, and action-rate-matched fixed scheduling policies. On Airline, the locked LinUCB policy achieves a higher mean F1 while substantially reducing intervention frequency relative to heuristic and action-rate-matched baselines. Additional runtime-calibrated evaluation shows that these gains persist when measured using the cumulative update wall-clock time. These results suggest that conservative sequential retraining policies can improve performance while reducing unnecessary operational overhead under delayed feedback.
Figures & tables
| Policy | Runtime(s) | F1 | Precision | Recall | Action rate | Balanced accuracy |
|---|---|---|---|---|---|---|
| No retrain | 0.0000 0.0000 | 0.0000 0.0000 | 0.0000 0.0000 | 0.0000 0.0000 | 0.0000 0.0000 | 0.0000 0.0000 |
| Fixed (N=10) | 2.2936 1.3266 | 0.1724 0.0114 | 0.1309 0.0076 | 0.4800 0.0431 | 0.0991 0.0000 | 0.5089 0.0033 |
| Fixed matched (N=64) | 0.3870 0.2118 | 0.1742 0.0278 | 0.1258 0.0182 | 0.4990 0.0901 | 0.0154 0.0000 | 0.5026 0.0053 |
| Threshold persistence (1.8,3) | 1.8466 0.2802 | 0.1547 0.0543 | 0.1167 0.0290 | 0.4465 0.1993 | 0.1145 0.0000 | 0.5002 0.0041 |
| LinUCB (locked) | 0.2687 0.0859 | 0.2065 0.0254 | 0.1596 0.0179 | 0.6012 0.1310 | 0.0157 0.0036 | 0.5057 0.0059 |