Cost-sensitive retraining via posterior learning debt
Organizations: Finance Data Science & Strategy, Airbnb Inc., San Francisco, CA, USA
Abstract
Deployed prediction systems are often retrained on fixed calendars, even when model staleness and retraining burden vary over time. This short communication formulates retraining for Bayesian prediction systems as a cost-sensitive predictive-regret decision. The central monitoring state is posterior learning debt, defined as the Kullback--Leibler divergence from a reference shadow posterior to the deployed frozen posterior. In the decision layer, a retraining cost is compared with the expected one-period predictive regret of waiting. A continuous-severity version retrains when calibrated expected regret exceeds the retraining cost, while the familiar two-state excess-loss rule is a special case. The empirical study is an exact-state proof-of-concept in a synthetic conjugate simulation with warm-started deployed and shadow normal-inverse-gamma posteriors, separate update, monitoring, and evaluation batches, lagged deployment actions, expanded baseline grids, and score-unit sensitivity. Under the primary 75th-percentile score-unit scaling, an age-adjusted debt-threshold policy improves on tuned calendar retraining in all 72 non-stable scenario cells and on tuned CUSUM in 58 of 72 cells, with mean relative objectives 0.677 and 0.975, respectively. Debt-utility and hybrid-utility policies also improve strongly over tuned calendar retraining, but they do not dominate tuned CUSUM. Median and mean score-unit sensitivities show the same main calendar result, while the CUSUM comparison remains policy-dependent. The contribution is a transparent decision layer for deployed Bayesian prediction systems, not a universal replacement for drift detection.
Figures & tables
| Policy | Benchmark | Wins | Mean rel. [95% CI] | Median rel. [IQR] |
|---|---|---|---|---|
| Age-adjusted debt threshold | Tuned calendar | 72/72 | 0.677 [0.674, 0.681] | 0.670 [0.641, 0.705] |
| Age-adjusted debt threshold | Tuned CUSUM | 58/72 | 0.975 [0.973, 0.977] | 0.964 [0.945, 0.986] |
| Debt utility | Tuned calendar | 67/72 | 0.801 [0.795, 0.809] | 0.787 [0.721, 0.881] |
| Debt utility | Tuned CUSUM | 6/72 | 1.158 [1.150, 1.168] | 1.133 [1.037, 1.225] |
| Hybrid utility | Tuned calendar | 69/72 | 0.795 [0.790, 0.802] | 0.809 [0.701, 0.893] |
| Hybrid utility | Tuned CUSUM | 2/72 | 1.148 [1.141, 1.157] | 1.089 [1.049, 1.220] |
Bayesian Decision Theory
Bayesian decision theory studies how to choose actions under uncertainty by combining probabilistic beliefs with utilities or losses.
See all 32 papers