Decision-Relevant Forecast Calibration for Predictive Control
Organizations: Department of Electrical and Systems Engineering · Department of Computer and Information Science
Abstract
Calibration evidence for a predictive controller need not remain valid when operating-context frequencies change. We develop an audit and certification procedure for a declared family of context-density shifts, with a frozen predictor/controller and unchanged conditional outcome law. Coverage requires calibrated tests to span density-weighted selected-plan sensitivities. A residual Gram matrix yields the minimum augmentation rank and an optimal test budget for uncovered geometry. This optimum need not minimize bias over probability-feasible conditional means. For finite contexts, an asymmetric absolute-loss dual and directional linear programs enclose the continuous worst-shift feasible bias. Intersecting simultaneous moment intervals with the conditional-mean domain gives a further certificate and a linear program for selecting an operating shift after validation. In a designed 240-context study, two spectral directions remove at least 80.8% of the feasible worst-case bias. For a separate misspecified logistic forecaster, matched sensitivity and generic corrections reduce expected cost by 2.590% and 3.239%, with the generic method winning all 200 refits. On 100,000 new episodes, both corrections certify absolute evaluation bias below the illustrative tolerance 0.04 throughout a declared shift interval, while the base certifies no shift. This outcome is highly predictable under the known synthetic law used for design. The probability-feasible certificate does not improve on the moment rectangle in this sample. The contribution is reusable evaluation and audit capability with explicit geometric and statistical limits, rather than recalibration superiority. Evidence uses synthetic finite contexts, known frequencies and committed plans; evaluation bias, oracle regret, constraint risk and feedback deployment remain distinct.
Figures & tables
| Symbol | Meaning |
|---|---|
| Random history, realized history, and history space. | |
| Prediction horizon; episode length; number of independent calibration episodes. | |
| Random forecast and its realized value. | |
| Future utility statistic and forecast residual. | |
| Candidate conditional mean of ; fixes the law of . | |
| Selected-plan utility sensitivity; certified vector tests. |
| Quantity | Value |
|---|---|
| Horizon and step | , |
| Input and state threshold | , |
| Wind modes | , |
| Reference and weights | ; |
| Initial position | |
| Initial velocity |
| Method | Excess over oracle | Risk (%) | Observed risk (%) | |
|---|---|---|---|---|
| nominal | 2.820303 | 0.003600 | 5.615 | 5.650 |
| histogram | 2.820488 | 0.003785 | 5.680 | 5.600 |
| regime | 2.819798 | 0.003094 | 5.623 | 5.700 |
| sensitivity | 2.816942 | 0.000239 | 5.880 | 5.950 |
| oracle | 2.816704 | 0.000000 | 5.622 | 5.550 |
| robust | 3.132998 | 0.316295 | 4.221 | 4.100 |
| Test class | Columns | Rank | representation error | |
|---|---|---|---|---|
| Forecast-group indicators | 5 | 5 | ||
| Group-by-regime indicators | 10 | 10 | ||
| Context monomials, degree | 10 | 10 | ||
| Active-set indicators | 4 | 4 | ||
| Active sets monomials, degree | 40 | 35 | ||
| Groups plus nominal sensitivity | 6 | 6 |
| Control | Cost difference | interval | Nominal risk (%) | Sensitivity risk (%) |
|---|---|---|---|---|
| zero | 0.000492 | 5.576 | 5.405 | |
| orthogonal | 0.000823 | 5.857 | 5.857 | |
| easy | -0.000242 | 0.000 | 0.000 |
| Cell | Difference | interval | Adjusted | |||
|---|---|---|---|---|---|---|
| 1 | 1.6 | 0.25 | 0.60 | -0.001560 | ||
| 2 | 1.6 | 0.25 | 0.70 | -0.001553 | ||
| 3 | 1.6 | 0.35 | 0.60 | -0.002089 | ||
| 4 | 1.6 | 0.35 | 0.70 | -0.002015 | ||
| 5 | 1.6 | 0.45 | 0.60 | -0.002102 | ||
| 6 | 1.6 | 0.45 | 0.70 | -0.001868 |
| Case/method | Mean cost difference | Cost SD | Mean risk difference | Risk SD | Risk |
|---|---|---|---|---|---|
| histogram | 0.000369 | 0.000253 | 0.049 | 0.176 | 95/73/32 |
| regime | -0.000232 | 0.000355 | -0.082 | 0.171 | 65/135/0 |
| sensitivity | -0.002851 | 0.000328 | 0.016 | 0.247 | 115/84/1 |
| zero control | 0.000669 | 0.000312 | 0.020 | 0.161 | 80/86/34 |
| orthogonal control | 0.000670 | 0.000340 | -0.003 | 0.131 | 78/86/36 |
| easy control | -0.000276 | 0.000070 | 0.000 | 0.000 | 0/0/200 |
| Cell | Mean cost difference | Cost SD | Mean risk difference (pp) | Risk |
|---|---|---|---|---|
| 1 | -0.001520 | 0.000185 | 0.000 | 0/0/200 |
| 2 | -0.001476 | 0.000200 | -0.181 | 1/195/4 |
| 3 | -0.001989 | 0.000317 | -0.050 | 5/139/56 |
| 4 | -0.001705 | 0.000296 | -0.119 | 57/107/36 |
| 5 | -0.002556 | 0.000388 | 0.138 | 128/70/2 |
| 6 | -0.002051 | 0.000434 | -0.341 | 2/168/30 |
| Construction | Absolute-bias upper bound | Signed utility-bias interval |
|---|---|---|
| Six tests, original Hoeffding | — | |
| Direct test, symmetric Hoeffding | ||
| Direct test, full-range Hoeffding | ||
| Direct test, empirical Bernstein |
| Quantity | Source-only law | Structured law |
|---|---|---|
| Exact source bias | ||
| Exact slope | ||
| Exact target bias | ||
| Estimated target bias | ||
| Simultaneous interval at | ||
| Upper bound on |
| Added directions | Total test rank | Worst uncovered transfer norm |
| 0 | 6 | |
| 1 | 7 | |
| 2 | 8 | |
| 3 | 9 |
| Pipeline | Uniform upper bound | |||
|---|---|---|---|---|
| Base | ||||
| Sensitivity | ||||
| Generic context |
| Extra moments | Added directions | Uncovered norm | Feasible-bias enclosure |
|---|---|---|---|
| 0 | None | 0.7393895 | |
| 1 | First spectral | 0.7007588 | |
| 2 | First two spectral | 0.1351792 | |
| 3 | All three spectral | ||
| 1 | Position product | 0.7330117 | |
| 1 | Velocity product | 0.7072554 |
| Method | Cost | Gap closed (%) | Full-plan risk (%) | |
|---|---|---|---|---|
| Trained base | 3.019914 | 0.0 | 6.210 | 0.123344 |
| Global intercept | 3.019670 | 0.2 | 6.485 | 0.126232 |
| Sensitivity, rank 10 | 2.941704 | 77.4 | 5.356 | |
| Generic context, rank 10 | 2.922113 | 96.8 | 6.051 | |
| Oracle | 2.918838 | 100.0 | 5.666 | 0 |
| Method | Mean cost diff. | Cost SD | Central 95% range | Cost |
|---|---|---|---|---|
| Global intercept | 0.000414 | 142/58 | ||
| Sensitivity | 0.000423 | 200/0 | ||
| Generic context | 0.000428 | 200/0 |
| Pipeline | Uniform bound | Certified set | |||
|---|---|---|---|---|---|
| Base | |||||
| Sensitivity | |||||
| Generic context |
| Pipeline | |||
|---|---|---|---|
| Base | |||
| Sensitivity | |||
| Generic context |
| Episodes | Detections | Frequency | Monte Carlo SE |
|---|---|---|---|
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Purpose | Seed or entropy | Use and indexing |
|---|---|---|
| Original calibration | context/outcome pairs per case. | |
| Original confirmation | episodes for the primary and each control; per grid cell. | |
| Conditional bootstrap | paired episode-bootstrap replicates for the primary and each control; per grid cell. One set of multinomial weights is reused across methods and endpoints within a case. | |
| Exploratory calibration refits | , key | fits per case, , each using new pairs. Evaluation is exact enumeration, with no confirmation or bootstrap draws. |
| Purpose | Entropy and key | Use |
|---|---|---|
| Transfer implementation check | Exploratory prototype inspected before the final transfer protocol; excluded from reported validation data. The retained protocol does not specify the child key or full draw record. | |
| Final constructed transfer validation | , keys , | episodes for each of the source-only and structured laws, in that order; no bootstrap. |
| Base-forecaster training | , key | pairs for the single logistic base fit. |
| Learned correction fits | , keys | for the original -pair correction fit; for the additional calibration refits. The base fit stays fixed. |
| Learned confirmation | , key | episodes shared by all frozen learned pipelines. |
| Learned conditional bootstrap | , key | multinomial episode-bootstrap replicates, with weights shared across methods and reported endpoints. |
Model Calibration
Model calibration aligns predicted confidence or model outputs with observed outcomes and measurements to support more reliable decisions.
See all 243 papers