eess.SYNov 16, 2026

Decision-Relevant Forecast Calibration for Predictive Control

Authors: Victor M. Preciado

Organizations: Department of Electrical and Systems Engineering · Department of Computer and Information Science

Abstract

Calibration evidence for a predictive controller need not remain valid when operating-context frequencies change. We develop an audit and certification procedure for a declared family of context-density shifts, with a frozen predictor/controller and unchanged conditional outcome law. Coverage requires calibrated tests to span density-weighted selected-plan sensitivities. A residual Gram matrix yields the minimum augmentation rank and an optimal test budget for uncovered L2L_2 geometry. This optimum need not minimize bias over probability-feasible conditional means. For finite contexts, an asymmetric absolute-loss dual and directional linear programs enclose the continuous worst-shift feasible bias. Intersecting simultaneous moment intervals with the conditional-mean domain gives a further certificate and a linear program for selecting an operating shift after validation. In a designed 240-context study, two spectral directions remove at least 80.8% of the feasible worst-case bias. For a separate misspecified logistic forecaster, matched sensitivity and generic corrections reduce expected cost by 2.590% and 3.239%, with the generic method winning all 200 refits. On 100,000 new episodes, both corrections certify absolute evaluation bias below the illustrative tolerance 0.04 throughout a declared shift interval, while the base certifies no shift. This outcome is highly predictable under the known synthetic law used for design. The probability-feasible certificate does not improve on the moment rectangle in this sample. The contribution is reusable evaluation and audit capability with explicit geometric and statistical limits, rather than recalibration superiority. Evidence uses synthetic finite contexts, known frequencies and committed plans; evaluation bias, oracle regret, constraint risk and feedback deployment remain distinct.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 29, 2026

Recipes for Calibration Checks in Safety-Critical Applications

Safety-critical prediction systems, such as autonomous vehicles, weather forecasters, and medical monitors, commonly rely on probabilistic forecasters. These forecasters make predictions about possible future outcomes, and their quality and robustness needs to be validated and certified. Often, only accuracy -- the mean of the predictions -- is evaluated against true outcomes. However, for safety-critical scenarios and decision making under uncertainty, the full distributional properties of the forecasts should be checked: do the observed prediction errors actually follow the forecasted probability distributions? To this end, we introduce a framework for calibration checks: statistical tests that validate distributional properties of forecasts when measured over many samples. In order to support ease-of-use in real-world operations, these checks produce a single accept/reject decision for data collected from a forecaster. This contrasts typical calibration calculations which produce one or multiple continuous calibration scores and require expertise to implement in a validation workflow. We further support operationalization by introducing modifications to calibration testing that (a) reject only overconfident predictions, allowing for pessimistic or cautious predictions in safety-critical settings, and (b) tolerate small, operationally acceptable deviations even for large numbers of validation samples. We organize the calibration checking process into a modular pipeline comprising four steps: (i) the data model, (ii) the chosen metric, (iii) the hypothesis formulation, and (iv) the testing procedure. Each step consists of independently swappable components, thereby supporting a large variety of possible use-cases and trade-offs. We demonstrate the applicability of the framework on two complementary example problems, weather forecasting and robot pose estimation.
May 26, 2026

Proper Calibeating

The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of proper\textit{proper} scoring rules (for which the best forecast is the true distribution) and define proper-calibration\textit{proper-calibration} and proper-calibeating\textit{proper-calibeating} by requiring the errors to converge to zero uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper-calibration, whereas calibeating need not imply proper-calibeating. Second, we show how to guarantee proper-calibeating and proper-multicalibeating. Finally, we demonstrate the equivalence between proper-calibration and universal no regret when best replying to forecasts in decision-making under uncertainty.
Jul 13, 2026

Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters

Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing such forecasters with dependent, potentially nonstationary observations. We first formalize notions of conditional quantile calibration relative to the contextual features available to the auditor, showing how coarser information can limit detection. We then identify detectable alternative classes and, for contextual bets linear in the features, derive finite-time detection guarantees without an i.i.d. assumption. The resulting evidence processes provide interpretable, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.