Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration

Jakob Benjamin Wessel; Sam Allen

stat.MLOct 7, 2026

Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration

Authors: Jakob Benjamin Wessel, Sam Allen

Organizations: Institute for Meteorology University of Leipzig · Institute of Statistics Karlsruhe Institute of Technology

Abstract

Calibration is an essential requirement for probabilistic predictions to be useful for decision making. While state-of-the-art prediction methods often yield miscalibrated predictive distributions, several post-hoc recalibration schemes have been proposed to generate calibrated predictions. However, popular recalibration schemes can conceal miscalibration in specific regions of the outcome space. Since particular outcomes, such as extreme events, often matter most for decision making, probabilistic predictions should be calibrated when evaluation is restricted to these outcomes. Hence, in this paper, we introduce outcome-conditional recalibration, a post-hoc method to recalibrate probabilistic predictions on user-defined regions of the outcome space. The method is simple, easy to implement, and can be applied to arbitrary predictive distributions. It works by applying the quantile recalibration approach of Kuleshov et al. (2018) to forecast conditional distributions, before rescaling these conditional distributions so that forecast event probabilities match empirical occurrence frequencies. This produces valid and continuous predictive distributions that are calibrated within each region of interest. Across regression benchmarks, we demonstrate that existing recalibration schemes do not necessarily yield calibrated predictions when interest is on particular outcomes, and that our approach improves outcome-conditional calibration relative to existing conditional and unconditional recalibration methods, while retaining competitive calibration overall. In an application to day-ahead electricity price forecasting, the approach substantially improves calibration when predicting negative prices, at negligible cost to forecast accuracy.

Figures & tables

Figure 1: Outcome-conditional (CPIT/subset) vs overall calibration error averaged over all datasets and models for each of the recalibration methods. The vertical and horizontal bars indicate standard error across seeds, datasets and base methods.

Method	CRPS	$\mathrm{cal}_{\mathrm{all}}$	$\mathrm{cal}_{g_{0}}$	$\mathrm{occ}_{g_{0}}$
Raw	2.65	0.0009	0.0611	0.939
KFE18	2.67	0.0008	0.0636	0.973
KD22	2.78	0.0006	0.1290	1.141
Ours: OC	2.70	0.0004	0.0051	1.050
Ours: OC + QR	2.71	0.0010	0.0082	1.041

Table 1: Rolling (time-adaptive, 182-day) recalibration of the DDNN-JSU, German day-ahead prices, 2019–2020. CRPS in EUR/MWh;

\mathrm{cal}

is the calibration error (lower is better);

g_{0}

is the negative-price group;

\mathrm{occ}_{g_{0}}

is its occurrence ratio (1 = calibrated). Best in each column in bold.

Figure 2: Rolling (182-day) recalibration of the DDNN-JSU, German day-ahead prices, 2019–2020. Left: empirical CDF of PIT values. Middle: quantile-quantile plot of CPIT values for the negative price group against reference values. Right: occurrence ratio of expected over realised events. For the left and middle panels, a calibrated forecast would lie on the 1-1 line, whereas on the right, the occurrence ratio should be close to one.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Uncal.		Kuleshov et al. ’18		Kuleshov–Deshpande ’22		Ours: OC		Ours: OC + QR
	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$
airfoil	$2.60\pm 0.47$	$1.93\pm 0.34$	$3.10\pm 0.58$	$2.18\pm 0.37$	$1.89\pm 0.69$	$1.19\pm 0.28$	$\mathbf{1.39\pm 0.35}$	$\mathbf{0.54\pm 0.30}$	$1.66\pm 0.29$	$0.65\pm 0.32$
ames	$7.28\pm 0.96$	$2.56\pm 0.24$	$6.16\pm 0.90$	$2.19\pm 0.28$	$1.40\pm 0.60$	$0.62\pm 0.21$	$\mathbf{1.06\pm 0.41}$	$\mathbf{0.43\pm 0.16}$	$2.79\pm 0.83$	$1.02\pm 0.24$
bank	$32.26\pm 1.32$	$3.74\pm 0.16$	$30.32\pm 1.58$	$3.73\pm 0.16$	$1.67\pm 1.03$	$0.44\pm 0.14$	$\mathbf{0.43\pm 0.14}$	$\mathbf{0.27\pm 0.09}$	$1.31\pm 0.46$	$0.34\pm 0.12$
elevators	$26.56\pm 0.56$	$4.79\pm 0.15$	$19.05\pm 0.60$	$3.03\pm 0.16$	$13.82\pm 0.92$	$0.54\pm 0.21$	$\mathbf{0.20\pm 0.06}$	$\mathbf{0.28\pm 0.09}$	$0.42\pm 0.12$	$0.66\pm 0.13$
energy	$29.09\pm 5.59$	$3.79\pm 0.46$	$27.02\pm 6.43$	$3.78\pm 0.47$	$9.16\pm 3.68$	$1.22\pm 0.49$	$\mathbf{3.46\pm 1.03}$	$\mathbf{0.91\pm 0.20}$	$5.26\pm 1.90$	$1.12\pm 0.40$

Appendix

Table 2: Outcome-conditional (CPIT/subset) calibration error and occurrence error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Dataset	Uncal.	Kuleshov et al. ’18	Kuleshov–Deshpande ’22	Ours: OC	Ours: OC + QR
airfoil	$0.37\pm 0.21$	$\mathbf{0.31\pm 0.13}$	$0.55\pm 0.45$	$0.70\pm 0.38$	$0.36\pm 0.28$
ames	$3.19\pm 0.74$	$0.58\pm 0.28$	$0.55\pm 0.41$	$1.97\pm 0.45$	$\mathbf{0.42\pm 0.22}$
bank	$1.16\pm 0.46$	$0.15\pm 0.19$	$0.41\pm 0.29$	$1.18\pm 0.33$	$\mathbf{0.15\pm 0.20}$
elevators	$2.69\pm 0.38$	$\mathbf{0.07\pm 0.06}$	$0.18\pm 0.18$	$0.26\pm 0.10$	$0.07\pm 0.05$
energy	$3.39\pm 1.58$	$0.96\pm 0.59$	$2.08\pm 1.21$	$1.98\pm 1.00$	$\mathbf{0.50\pm 0.37}$
kinematics	$0.75\pm 0.26$	$\mathbf{0.08\pm 0.05}$	$0.18\pm 0.09$	$0.90\pm 0.34$	$0.08\pm 0.05$

Appendix

Table 3: Overall calibration error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Dataset	Uncal.		Kuleshov et al. ’18		Kuleshov–Deshpande ’22		Ours: OC		Ours: OC + QR
	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$
airfoil	$2.62\pm 0.56$	$1.95\pm 0.40$	$3.20\pm 0.98$	$2.24\pm 0.49$	$3.16\pm 1.30$	$1.49\pm 0.35$	$\mathbf{2.10\pm 0.31}$	$\mathbf{0.75\pm 0.37}$	$2.29\pm 0.51$	$0.85\pm 0.38$
ames	$7.69\pm 1.09$	$2.59\pm 0.20$	$6.68\pm 1.26$	$2.37\pm 0.24$	$2.05\pm 0.79$	$0.76\pm 0.20$	$\mathbf{1.81\pm 0.72}$	$\mathbf{0.52\pm 0.20}$	$2.66\pm 1.01$	$0.84\pm 0.31$
bank	$32.19\pm 1.34$	$3.75\pm 0.18$	$30.45\pm 1.72$	$3.73\pm 0.17$	$2.93\pm 1.84$	$0.55\pm 0.18$	$\mathbf{0.69\pm 0.33}$	$\mathbf{0.30\pm 0.11}$	$1.64\pm 0.80$	$0.37\pm 0.13$
elevators	$26.56\pm 0.65$	$4.79\pm 0.14$	$18.77\pm 0.74$	$3.05\pm 0.23$	$14.37\pm 0.95$	$0.66\pm 0.26$	$\mathbf{0.49\pm 0.22}$	$\mathbf{0.41\pm 0.16}$	$0.83\pm 0.38$	$0.80\pm 0.22$
energy	$29.13\pm 5.55$	$3.83\pm 0.43$	$26.12\pm 7.46$	$3.82\pm 0.47$	$11.42\pm 5.93$	$1.65\pm 0.73$	$\mathbf{6.36\pm 2.49}$	$\mathbf{1.37\pm 0.58}$	$9.04\pm 3.54$	$1.81\pm 0.77$

Appendix

Table 4: Outcome-conditional calibration (CPIT/subset) error and occurrence error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Dataset	Uncal.	Kuleshov et al. ’18	Kuleshov–Deshpande ’22	Ours: OC	Ours: OC + QR
airfoil	$\mathbf{0.37\pm 0.23}$	$0.51\pm 0.38$	$1.30\pm 0.70$	$0.95\pm 0.66$	$0.60\pm 0.62$
ames	$2.74\pm 1.02$	$\mathbf{0.49\pm 0.31}$	$0.66\pm 0.99$	$1.80\pm 0.47$	$0.50\pm 0.39$
bank	$1.14\pm 0.45$	$\mathbf{0.16\pm 0.19}$	$0.58\pm 0.36$	$1.17\pm 0.42$	$0.21\pm 0.28$
elevators	$2.71\pm 0.37$	$0.11\pm 0.10$	$0.14\pm 0.10$	$0.29\pm 0.11$	$\mathbf{0.11\pm 0.10}$
energy	$3.39\pm 1.66$	$1.45\pm 1.56$	$2.84\pm 1.40$	$2.71\pm 1.40$	$\mathbf{1.33\pm 0.99}$
kinematics	$0.71\pm 0.26$	$\mathbf{0.14\pm 0.10}$	$0.20\pm 0.14$	$1.03\pm 0.40$	$0.16\pm 0.09$

Appendix

Table 5: Overall calibration error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Method	Cal-all	$G{=}5$	union16	$G{=}5$ shift	$G{=}10$	$G{=}20$	tail	rand	mean
Uncalibrated	$11.8$	$198.4$	$178.4$	$235.1$	$158.9$	$133.8$	$252.4$	$217.3$	$196.3$
KFE18	$2.2$	$177.7$	$163.9$	$218.0$	$147.5$	$124.5$	$225.9$	$195.5$	$179.0$
KD22	$4.9$	$20.7$	$37.8$	$36.6$	$44.7$	$71.7$	$\mathbf{118.7}$	$48.0$	$54.0$
Ours: OC $G{=}5$	$7.9$	$\mathbf{8.7}$	$32.0$	$31.3$	$41.5$	$80.2$	$154.7$	$42.3$	$55.8$
Ours: OC, $G{=}16$	$10.1$	$13.2$	$\mathbf{19.0}$	$\mathbf{15.4}$	$\mathbf{26.3}$	$56.0$	$132.8$	$37.9$	$\mathbf{42.9}$
Ours: OC, $G{=}20$	$10.0$	$13.2$	$20.2$	$16.2$	$26.9$	$\mathbf{54.1}$	$132.7$	$40.3$	$43.4$

Appendix

Table 6: Evaluation of recalibration schemes when calibration is assessed on different partitions to those used for training the outcome-conditional recalibration scheme. The recalibration is trained using

G=5,16,20

partitions, all defined using evenly spaced quantiles of the outcome distribution. Calibration error is then calculated for the following partitions: “Cal-all” is the standard (partition-free) calibration error;

G=5,10,20

evenly spaced partitions;

G=5

shift sets quantile breaks at

[0.1,0.3,0.5,0.7]

; union16 pairs adjacent atoms of the

16

-cell partition; tail is a partition at quantile breakpoints

0.5

0.9

0.95

and

0.99

quantiles; rand averages three partitions with 4 break levels drawn uniformly per seed; mean averages the seven partition columns. Results are aggregated over all datasets with five random seeds, with Bayesian ridge as the base model. All calibration errors have been scaled by

10^{3}

to aid readability.

Explore similar work

Jun 8, 2026cs.LG

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting

As renewable energy integration increases market volatility, probabilistic electricity price forecasting has become essential for effective risk management. However, current-proper-scoring rules often prioritize forecast sharpness at the expense of calibration, leading to overconfident and statistically unreliable uncertainty estimates. This work highlights the critical gap between theoretical scoring and practical calibration, demonstrating that models can become mere proxies for deterministic forecasts when reliability is neglected. We conclude that future research must shift toward calibration-aware objectives and architectures to ensure the distributional integrity of energy market forecasts.

Jan Niklas Lettner, Hadeer El Ashhab, Benjamin Schäfer

Institute for Automation and Applied Informatics

Electricity Price Forecasting Probabilistic Forecasting Challenges

Ask PDF

Jul 22, 2026stat.ML

Optimal Recalibration of an Online Predictor

We study the problem of recalibrating an online predictor [KE17, OKS24]: given an arbitrary "hint" sequence of forecasts, the learner must output new predictions that are calibrated while incurring small excess error relative to the original forecasts, under a proper loss. We give an online algorithm that achieves

(\varepsilon, \varepsilon^2)

-recalibration for Lipschitz proper losses in

T \approx \varepsilon^{-3}

rounds, using an imbalanced extension of the recent simultaneous Blackwell approachability reduction framework of [HTY26]. We show that this tradeoff is optimal by proving a matching lower bound for recalibrating against the squared loss. We also prove a companion

\mathcal{K}_2

-recalibration theorem that obtains the same tradeoffs up to a logarithmic factor. As our main application, we show how our recalibration algorithms can be combined with the online refinement method of [FH23] to obtain simultaneous

\varepsilon

-calibration and

\varepsilon^2

-calibeating for smooth proper losses at the same asymptotic rate, improving upon prior works that achieved these properties separately or with a worse

\varepsilon

dependence. In particular, the

\mathcal{K}_2

variant answers a question of [CHJL26] on simultaneously achieving near-optimal calibeating and calibration rates. We also derive extensions to settings with multiple hint sequences. Finally, we empirically evaluate our algorithms on a classification dataset undergoing distribution shift.

Lunjia Hu, Kevin Tian, Chutong Yang

Northeastern University · University of Texas at Austin

Recalibration Learning-Augmented Algorithms Prediction Error

Ask PDF

May 26, 2026econ.TH

Proper Calibeating

The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of

\textit{proper}

scoring rules (for which the best forecast is the true distribution) and define

\textit{proper-calibration}

and

\textit{proper-calibeating}

by requiring the errors to converge to zero uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper-calibration, whereas calibeating need not imply proper-calibeating. Second, we show how to guarantee proper-calibeating and proper-multicalibeating. Finally, we demonstrate the equivalence between proper-calibration and universal no regret when best replying to forecasts in decision-making under uncertainty.

Dean P. Foster, Sergiu Hart

Department of Statistics, Wharton, University of Pennsylvania, Philadelphia, and Amazon, New York. · Institute of Mathematics, Department of Economics, and Federmann Center for the Study of Rationality, The Hebrew University of Jerusalem.

Calibrated Uncertainty Forecast Uncertainty Distributions

Ask PDF