Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

Arthur Charpentier; Agathe Fernandes Machado

cs.LGMar 16, 2026

Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

Authors: Arthur Charpentier, Agathe Fernandes Machado

Organizations: UQAM, Montréal, Canada · Kyoto University, Japan

Abstract

Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level $\mathcal A$ , the expected loss of an $\mathcal A$ -measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels $\mathcal A\subseteq\mathcal B$ , a chain decomposition quantifies the information gain from $\mathcal A$ to $\mathcal B$ . Applied to classification with features $\boldsymbol{X}$ and score $S=s(\boldsymbol{X})$ , this yields a three-term identity: miscalibration, a {\em grouping} term measuring information loss from $\boldsymbol{X}$ to $S$ , and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.

Figures & tables

Panel of Figure 1 — Figure 1: Left: learned monotone calibration maps $\widehat{g}(\cdot)$ for the two base models ( $\widehat{s}_{1}$ in green, $\widehat{s}_{2}$ in blue) and for their average $\widehat{s}_{\mathrm{ens}}$ (red), shown here for $\rho=0$ . The grey diagonal corresponds to perfect calibration. Right: local calibration score as a function of $\rho$ for the two base models (green/blue) and their average (red), computed on an independent test split.

Figure 2: Reducible Brier components on test data. Bars show reliability $\widehat{\mathbb{E}}[(S-\widehat{C})^{2}]$ and grouping $\widehat{\mathbb{E}}[(\widehat{C}-Q)^{2}]$ before and after monotone recalibration, for three scores: a logistic model on $X_{1}$ (left), a logistic model on $(X_{1},X_{2})$ (middle), and a quantized score (right). Recalibration eliminates reliability but leaves grouping essentially unchanged.

Model	Brier			LogLoss			LCS
	raw	recal	$\Delta$	raw	recal	$\Delta$
GLM	0.19	0.18	0.01	0.73	0.55	0.19	0.012
RF	0.17	0.17	-0.00	0.52	0.63	-0.11	0.014
Average	0.17	0.18	-0.01	0.52	0.75	-0.24	0.013
Stacking	0.17	0.18	-0.00	0.52	0.63	-0.11	0.006

Table 1: GermanCredit : test-set performance before/after monotone recalibration. Gains are defined as

\Delta=\text{raw}-\text{recal}

(positive is better).

Figure 3: GermanCredit : monotone-smoothed calibration curves on the test split. Green: raw score; red: after monotone post-hoc recalibration fit on a separate calibration split. The diagonal indicates perfect calibration.

Figure 4: GermanCredit : summary metrics on the test split. Left: miscalibration proxy $\mathrm{LCS}=\widehat{\mathbb{E}}(S-\widehat{C}(S))^{2}$ . Middle/right: proper losses (Brier and log-loss) before vs. after recalibration.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: True conditional probability surface $Q(x_{1},x_{2})$ for the synthetic model, on the left hand side. Then 3 scatterplots of simulated data, with $\rho=-0.7$ (middle left), $\rho=0$ (middle right), and $\rho=0.7$ (right).

Method	Log-loss (raw)	Log-loss (recal.)	Rel log
Average	0.496 $\pm$ 0.031	0.601 $\pm$ 0.093	0.028 $\pm$ 0.014
GLM	0.565 $\pm$ 0.084	0.640 $\pm$ 0.124	0.033 $\pm$ 0.026
RF	0.501 $\pm$ 0.027	0.549 $\pm$ 0.071	0.040 $\pm$ 0.014
Stacking	0.504 $\pm$ 0.037	0.589 $\pm$ 0.085	0.021 $\pm$ 0.008

Appendix

Table 2: GermanCredit robustness over repeated splits (mean

\pm

sd over

R=100

seeds). Proper-loss reliability uses the fitted calibrator

\widehat{C}

trained on the calibration split and evaluated on the test split.

Method	$\Delta$ Log-loss (recal.)	Win%	$p$ (Holm)	$\Delta$ Rel log	Win%	$p$ (Holm)
GLM	0.0281 $\pm$ 0.1091	34%	0.996	0.0141 $\pm$ 0.0388	39%	1
RF	-0.0184 $\pm$ 0.0963	60%	0.056	0.0134 $\pm$ 0.0140	15%	1
Stacking	0.0020 $\pm$ 0.0753	55%	0.949	-0.0055 $\pm$ 0.0109	64%	$<10^{-4}$

Appendix

Table 3: Paired comparison against Average (computed per split/seed).

\Delta

denotes (method

-

reference) so negative values indicate an improvement over the reference. We report win-rate and one-sided Wilcoxon signed-rank tests (

H_{1}

: method

<

reference), with Holm correction across methods.

Panel of Figure 6 — Figure 6: Plain-logistic model fitted on simulated data $\{(x_{1,i},x_{2,i},y_{i})\}$ , on the left, and two estimations $\hat{g}$ , using an isotonic regression (in green), a $\mathcal{C}^{2}$ smoothing mapping (in red) and a monotone $\mathcal{C}^{2}$ smoothing mapping (in blue), on the middle left. On the right, evolution of $h\mapsto$ integrated calibration index ( $\ell_{1}$ ) on the middle right, and local calibration score ( $\ell_{2}$ ) for $\hat{g}_{h}$ on the right, as a function of the bandwidth parameter $h>0$ , for three values of $k$ , and using 5-fold cross validation

Probability Calibration

Probability calibration measures and improves whether predicted probabilities match observed outcome frequencies, supporting reliable decisions under uncertainty.

See all 131 papers

Calibeating for general proper losses: A Bregman divergence approach
May 17, 2026Maximilian Fichtl, Cristóbal Guzmán, Nishant A. MehtaFunctional Bregman Divergences Proper Scoring Rules
Independent researcher · Institute for Mathematical and Computational Engineering, Faculty of Mathematics and School of Engineering, Pontificia Universidad Católica de Chile · Department of Computer Science, University of Victoria
Calibrating conditional risk
Apr 22, 2026Andrey Vasilyev, Yikai Wang, Xiaocheng Li +1
Imperial College Business School, Imperial College London · Department of Statistics and Operations Research, UNC-Chapel Hill
Proper Calibeating
May 26, 2026Dean P. Foster, Sergiu HartProper Scoring Rules Uncertainty Calibration
Department of Statistics, Wharton, University of Pennsylvania, Philadelphia, and Amazon, New York. · Institute of Mathematics, Department of Economics, and Federmann Center for the Study of Rationality, The Hebrew University of Jerusalem.

May 17, 2026J/K move · Enter open · S save

Calibeating for general proper losses: A Bregman divergence approach

Maximilian Fichtl, Cristóbal Guzmán, Nishant A. Mehta

Independent researcher · Institute for Mathematical and Computational Engineering, Faculty of Mathematics and School of Engineering, Pontificia Universidad Católica de Chile · Department of Computer Science, University of Victoria

Functional Bregman Divergences Proper Scoring Rules Regret Minimization

Open paper AskPDF arXiv

This work introduces a general framework for calibeating based on regret minimization. As compared to Foster and Hart's seminal calibeating work which had specialized treatments of Brier score (squared loss) and log loss, we consider a large family of proper losses that includes

α

-Tsallis losses (for

α\in [1, 2]

) and Lipschitz losses. Our results for Tsallis losses also hold for an unscaled version of Tsallis loss that recovers log loss. Our analysis is oriented around the Bregman divergence view of a proper loss. Technically, our results for the family of Tsallis losses that we consider are U-calibration results, simultaneously obtaining logarithmic regret for all losses in this family while having a weaker dependence on the dimension compared to previous results. Of potential independent interest, we also show a new regret equality for the regret of Be The Regularized Leader. This regret equality holds for general proper losses and itself is based on two results related to online updating formulas for the generalized variance, the latter being a previously introduced generalization of variance based on Bregman divergences.

Abstract

Probability Calibration

Explore similar work

Calibeating for general proper losses: A Bregman divergence approach

Calibrating conditional risk

Proper Calibeating