Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration

Jakob Benjamin Wessel; Sam Allen

stat.MLOct 7, 2026

Towards Calibrated Probabilistic Forecasts for Events of Interest via Outcome-Conditional Recalibration

Authors: Jakob Benjamin Wessel, Sam Allen

Organizations: Institute for Meteorology University of Leipzig · Institute of Statistics Karlsruhe Institute of Technology

Abstract

Calibration is an essential requirement for probabilistic predictions to be useful for decision making. While state-of-the-art prediction methods often yield miscalibrated predictive distributions, several post-hoc recalibration schemes have been proposed to generate calibrated predictions. However, popular recalibration schemes can conceal miscalibration in specific regions of the outcome space. Since particular outcomes, such as extreme events, often matter most for decision making, probabilistic predictions should be calibrated when evaluation is restricted to these outcomes. Hence, in this paper, we introduce outcome-conditional recalibration, a post-hoc method to recalibrate probabilistic predictions on user-defined regions of the outcome space. The method is simple, easy to implement, and can be applied to arbitrary predictive distributions. It works by applying the quantile recalibration approach of Kuleshov et al. (2018) to forecast conditional distributions, before rescaling these conditional distributions so that forecast event probabilities match empirical occurrence frequencies. This produces valid and continuous predictive distributions that are calibrated within each region of interest. Across regression benchmarks, we demonstrate that existing recalibration schemes do not necessarily yield calibrated predictions when interest is on particular outcomes, and that our approach improves outcome-conditional calibration relative to existing conditional and unconditional recalibration methods, while retaining competitive calibration overall. In an application to day-ahead electricity price forecasting, the approach substantially improves calibration when predicting negative prices, at negligible cost to forecast accuracy.

Figures & tables

Figure 1: Outcome-conditional (CPIT/subset) vs overall calibration error averaged over all datasets and models for each of the recalibration methods. The vertical and horizontal bars indicate standard error across seeds, datasets and base methods.

Method	CRPS	$\mathrm{cal}_{\mathrm{all}}$	$\mathrm{cal}_{g_{0}}$	$\mathrm{occ}_{g_{0}}$
Raw	2.65	0.0009	0.0611	0.939
KFE18	2.67	0.0008	0.0636	0.973
KD22	2.78	0.0006	0.1290	1.141
Ours: OC	2.70	0.0004	0.0051	1.050
Ours: OC + QR	2.71	0.0010	0.0082	1.041

Table 1: Rolling (time-adaptive, 182-day) recalibration of the DDNN-JSU, German day-ahead prices, 2019–2020. CRPS in EUR/MWh;

\mathrm{cal}

is the calibration error (lower is better);

g_{0}

is the negative-price group;

\mathrm{occ}_{g_{0}}

is its occurrence ratio (1 = calibrated). Best in each column in bold.

Figure 2: Rolling (182-day) recalibration of the DDNN-JSU, German day-ahead prices, 2019–2020. Left: empirical CDF of PIT values. Middle: quantile-quantile plot of CPIT values for the negative price group against reference values. Right: occurrence ratio of expected over realised events. For the left and middle panels, a calibrated forecast would lie on the 1-1 line, whereas on the right, the occurrence ratio should be close to one.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Uncal.		Kuleshov et al. ’18		Kuleshov–Deshpande ’22		Ours: OC		Ours: OC + QR
	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$
airfoil	$2.60\pm 0.47$	$1.93\pm 0.34$	$3.10\pm 0.58$	$2.18\pm 0.37$	$1.89\pm 0.69$	$1.19\pm 0.28$	$\mathbf{1.39\pm 0.35}$	$\mathbf{0.54\pm 0.30}$	$1.66\pm 0.29$	$0.65\pm 0.32$
ames	$7.28\pm 0.96$	$2.56\pm 0.24$	$6.16\pm 0.90$	$2.19\pm 0.28$	$1.40\pm 0.60$	$0.62\pm 0.21$	$\mathbf{1.06\pm 0.41}$	$\mathbf{0.43\pm 0.16}$	$2.79\pm 0.83$	$1.02\pm 0.24$
bank	$32.26\pm 1.32$	$3.74\pm 0.16$	$30.32\pm 1.58$	$3.73\pm 0.16$	$1.67\pm 1.03$	$0.44\pm 0.14$	$\mathbf{0.43\pm 0.14}$	$\mathbf{0.27\pm 0.09}$	$1.31\pm 0.46$	$0.34\pm 0.12$
elevators	$26.56\pm 0.56$	$4.79\pm 0.15$	$19.05\pm 0.60$	$3.03\pm 0.16$	$13.82\pm 0.92$	$0.54\pm 0.21$	$\mathbf{0.20\pm 0.06}$	$\mathbf{0.28\pm 0.09}$	$0.42\pm 0.12$	$0.66\pm 0.13$
energy	$29.09\pm 5.59$	$3.79\pm 0.46$	$27.02\pm 6.43$	$3.78\pm 0.47$	$9.16\pm 3.68$	$1.22\pm 0.49$	$\mathbf{3.46\pm 1.03}$	$\mathbf{0.91\pm 0.20}$	$5.26\pm 1.90$	$1.12\pm 0.40$

Appendix

Table 2: Outcome-conditional (CPIT/subset) calibration error and occurrence error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Dataset	Uncal.	Kuleshov et al. ’18	Kuleshov–Deshpande ’22	Ours: OC	Ours: OC + QR
airfoil	$0.37\pm 0.21$	$\mathbf{0.31\pm 0.13}$	$0.55\pm 0.45$	$0.70\pm 0.38$	$0.36\pm 0.28$
ames	$3.19\pm 0.74$	$0.58\pm 0.28$	$0.55\pm 0.41$	$1.97\pm 0.45$	$\mathbf{0.42\pm 0.22}$
bank	$1.16\pm 0.46$	$0.15\pm 0.19$	$0.41\pm 0.29$	$1.18\pm 0.33$	$\mathbf{0.15\pm 0.20}$
elevators	$2.69\pm 0.38$	$\mathbf{0.07\pm 0.06}$	$0.18\pm 0.18$	$0.26\pm 0.10$	$0.07\pm 0.05$
energy	$3.39\pm 1.58$	$0.96\pm 0.59$	$2.08\pm 1.21$	$1.98\pm 1.00$	$\mathbf{0.50\pm 0.37}$
kinematics	$0.75\pm 0.26$	$\mathbf{0.08\pm 0.05}$	$0.18\pm 0.09$	$0.90\pm 0.34$	$0.08\pm 0.05$

Appendix

Table 3: Overall calibration error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Dataset	Uncal.		Kuleshov et al. ’18		Kuleshov–Deshpande ’22		Ours: OC		Ours: OC + QR
	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$	Cal-sub. $\downarrow$	Occ-sub. $\downarrow$
airfoil	$2.62\pm 0.56$	$1.95\pm 0.40$	$3.20\pm 0.98$	$2.24\pm 0.49$	$3.16\pm 1.30$	$1.49\pm 0.35$	$\mathbf{2.10\pm 0.31}$	$\mathbf{0.75\pm 0.37}$	$2.29\pm 0.51$	$0.85\pm 0.38$
ames	$7.69\pm 1.09$	$2.59\pm 0.20$	$6.68\pm 1.26$	$2.37\pm 0.24$	$2.05\pm 0.79$	$0.76\pm 0.20$	$\mathbf{1.81\pm 0.72}$	$\mathbf{0.52\pm 0.20}$	$2.66\pm 1.01$	$0.84\pm 0.31$
bank	$32.19\pm 1.34$	$3.75\pm 0.18$	$30.45\pm 1.72$	$3.73\pm 0.17$	$2.93\pm 1.84$	$0.55\pm 0.18$	$\mathbf{0.69\pm 0.33}$	$\mathbf{0.30\pm 0.11}$	$1.64\pm 0.80$	$0.37\pm 0.13$
elevators	$26.56\pm 0.65$	$4.79\pm 0.14$	$18.77\pm 0.74$	$3.05\pm 0.23$	$14.37\pm 0.95$	$0.66\pm 0.26$	$\mathbf{0.49\pm 0.22}$	$\mathbf{0.41\pm 0.16}$	$0.83\pm 0.38$	$0.80\pm 0.22$
energy	$29.13\pm 5.55$	$3.83\pm 0.43$	$26.12\pm 7.46$	$3.82\pm 0.47$	$11.42\pm 5.93$	$1.65\pm 0.73$	$\mathbf{6.36\pm 2.49}$	$\mathbf{1.37\pm 0.58}$	$9.04\pm 3.54$	$1.81\pm 0.77$

Appendix

Table 4: Outcome-conditional calibration (CPIT/subset) error and occurrence error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Dataset	Uncal.	Kuleshov et al. ’18	Kuleshov–Deshpande ’22	Ours: OC	Ours: OC + QR
airfoil	$\mathbf{0.37\pm 0.23}$	$0.51\pm 0.38$	$1.30\pm 0.70$	$0.95\pm 0.66$	$0.60\pm 0.62$
ames	$2.74\pm 1.02$	$\mathbf{0.49\pm 0.31}$	$0.66\pm 0.99$	$1.80\pm 0.47$	$0.50\pm 0.39$
bank	$1.14\pm 0.45$	$\mathbf{0.16\pm 0.19}$	$0.58\pm 0.36$	$1.17\pm 0.42$	$0.21\pm 0.28$
elevators	$2.71\pm 0.37$	$0.11\pm 0.10$	$0.14\pm 0.10$	$0.29\pm 0.11$	$\mathbf{0.11\pm 0.10}$
energy	$3.39\pm 1.66$	$1.45\pm 1.56$	$2.84\pm 1.40$	$2.71\pm 1.40$	$\mathbf{1.33\pm 0.99}$
kinematics	$0.71\pm 0.26$	$\mathbf{0.14\pm 0.10}$	$0.20\pm 0.14$	$1.03\pm 0.40$	$0.16\pm 0.09$

Appendix

Table 5: Overall calibration error across datasets and methods. Values report the mean

\pm

sample standard deviation over ten seeds, scaled by

10^{2}

. Lower is better. Bold indicates the lowest mean among methods for a fixed base model, dataset, and metric.

Method	Cal-all	$G{=}5$	union16	$G{=}5$ shift	$G{=}10$	$G{=}20$	tail	rand	mean
Uncalibrated	$11.8$	$198.4$	$178.4$	$235.1$	$158.9$	$133.8$	$252.4$	$217.3$	$196.3$
KFE18	$2.2$	$177.7$	$163.9$	$218.0$	$147.5$	$124.5$	$225.9$	$195.5$	$179.0$
KD22	$4.9$	$20.7$	$37.8$	$36.6$	$44.7$	$71.7$	$\mathbf{118.7}$	$48.0$	$54.0$
Ours: OC $G{=}5$	$7.9$	$\mathbf{8.7}$	$32.0$	$31.3$	$41.5$	$80.2$	$154.7$	$42.3$	$55.8$
Ours: OC, $G{=}16$	$10.1$	$13.2$	$\mathbf{19.0}$	$\mathbf{15.4}$	$\mathbf{26.3}$	$56.0$	$132.8$	$37.9$	$\mathbf{42.9}$
Ours: OC, $G{=}20$	$10.0$	$13.2$	$20.2$	$16.2$	$26.9$	$\mathbf{54.1}$	$132.7$	$40.3$	$43.4$

Appendix

Table 6: Evaluation of recalibration schemes when calibration is assessed on different partitions to those used for training the outcome-conditional recalibration scheme. The recalibration is trained using

G=5,16,20

partitions, all defined using evenly spaced quantiles of the outcome distribution. Calibration error is then calculated for the following partitions: “Cal-all” is the standard (partition-free) calibration error;

G=5,10,20

evenly spaced partitions;

G=5

shift sets quantile breaks at

[0.1,0.3,0.5,0.7]

; union16 pairs adjacent atoms of the

16

-cell partition; tail is a partition at quantile breakpoints

0.5

0.9

0.95

and

0.99

quantiles; rand averages three partitions with 4 break levels drawn uniformly per seed; mean averages the seven partition columns. Results are aggregated over all datasets with five random seeds, with Bayesian ridge as the base model. All calibration errors have been scaled by

10^{3}

to aid readability.

Explore similar work

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting
Jun 8, 2026Jan Niklas Lettner, Hadeer El Ashhab, Benjamin SchäferElectricity Price Forecasting Probabilistic Forecasting
Institute for Automation and Applied Informatics
Optimal Recalibration of an Online Predictor
Jul 22, 2026Lunjia Hu, Kevin Tian, Chutong YangRecalibration Learning-Augmented Algorithms
Northeastern University · University of Texas at Austin
Proper Calibeating
May 26, 2026Dean P. Foster, Sergiu HartCalibrated Uncertainty Forecast
Department of Statistics, Wharton, University of Pennsylvania, Philadelphia, and Amazon, New York. · Institute of Mathematics, Department of Economics, and Federmann Center for the Study of Rationality, The Hebrew University of Jerusalem.

Jun 8, 2026, cs.LGJ/K move · Enter open · S save

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting

Jan Niklas Lettner, Hadeer El Ashhab, Benjamin Schäfer

Institute for Automation and Applied Informatics

Electricity Price Forecasting Probabilistic Forecasting Challenges

Open paper AskPDF arXiv

As renewable energy integration increases market volatility, probabilistic electricity price forecasting has become essential for effective risk management. However, current-proper-scoring rules often prioritize forecast sharpness at the expense of calibration, leading to overconfident and statistically unreliable uncertainty estimates. This work highlights the critical gap between theoretical scoring and practical calibration, demonstrating that models can become mere proxies for deterministic forecasts when reliability is neglected. We conclude that future research must shift toward calibration-aware objectives and architectures to ensure the distributional integrity of energy market forecasts.

Abstract

Explore similar work

Investigating Calibration Challenges in Probabilistic Electricity Price Forecasting

Optimal Recalibration of an Online Predictor

Proper Calibeating