Calibration is a conditional property that depends on the information retained by a predictor. We develop decomposition identities for arbitrary proper losses that make this dependence explicit. At any information level A, the expected loss of an A-measurable predictor splits into a proper-regret (reliability) term and a conditional entropy (residual uncertainty) term. For nested levels A⊆B, a chain decomposition quantifies the information gain from A to B. Applied to classification with features X and score S=s(X), this yields a three-term identity: miscalibration, a {\em grouping} term measuring information loss from X to S, and irreducible uncertainty at the feature level. We leverage the framework to analyze post-hoc recalibration, aggregation of calibrated models, and stagewise/boosting constructions, with explicit forms for Brier and log-loss.
Figures & tables
Figure 1: Left: learned monotone calibration maps g(⋅) for the two base models ( s1 in green, s2 in blue) and for their average sens (red), shown here for ρ=0 . The grey diagonal corresponds to perfect calibration. Right: local calibration score as a function of ρ for the two base models (green/blue) and their average (red), computed on an independent test split.
Figure 2: Reducible Brier components on test data. Bars show reliability E[(S−C)2] and grouping E[(C−Q)2] before and after monotone recalibration, for three scores: a logistic model on X1 (left), a logistic model on (X1,X2) (middle), and a quantized score (right). Recalibration eliminates reliability but leaves grouping essentially unchanged.
Model
Brier
LogLoss
LCS
raw
recal
Δ
raw
recal
Δ
GLM
0.19
0.18
0.01
0.73
0.55
0.19
0.012
RF
0.17
0.17
-0.00
0.52
0.63
-0.11
0.014
Average
0.17
0.18
-0.01
0.52
0.75
-0.24
0.013
Stacking
0.17
0.18
-0.00
0.52
0.63
-0.11
0.006
Table 1: GermanCredit : test-set performance before/after monotone recalibration. Gains are defined as Δ=raw−recal (positive is better).
Figure 3: GermanCredit : monotone-smoothed calibration curves on the test split. Green: raw score; red: after monotone post-hoc recalibration fit on a separate calibration split. The diagonal indicates perfect calibration.
Figure 4: GermanCredit : summary metrics on the test split. Left: miscalibration proxy LCS=E(S−C(S))2 . Middle/right: proper losses (Brier and log-loss) before vs. after recalibration.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: True conditional probability surface Q(x1,x2) for the synthetic model, on the left hand side. Then 3 scatterplots of simulated data, with ρ=−0.7 (middle left), ρ=0 (middle right), and ρ=0.7 (right).
Method
Log-loss (raw)
Log-loss (recal.)
Rel log
Average
0.496 ± 0.031
0.601 ± 0.093
0.028 ± 0.014
GLM
0.565 ± 0.084
0.640 ± 0.124
0.033 ± 0.026
RF
0.501 ± 0.027
0.549 ± 0.071
0.040 ± 0.014
Stacking
0.504 ± 0.037
0.589 ± 0.085
0.021 ± 0.008
Appendix
Table 2: GermanCredit robustness over repeated splits (mean ± sd over R=100 seeds). Proper-loss reliability uses the fitted calibrator C trained on the calibration split and evaluated on the test split.
Method
Δ Log-loss (recal.)
Win%
p (Holm)
Δ Rel log
Win%
p (Holm)
GLM
0.0281 ± 0.1091
34%
0.996
0.0141 ± 0.0388
39%
1
RF
-0.0184 ± 0.0963
60%
0.056
0.0134 ± 0.0140
15%
1
Stacking
0.0020 ± 0.0753
55%
0.949
-0.0055 ± 0.0109
64%
<10−4
Appendix
Table 3: Paired comparison against Average (computed per split/seed). Δ denotes (method − reference) so negative values indicate an improvement over the reference. We report win-rate and one-sided Wilcoxon signed-rank tests ( H1 : method < reference), with Holm correction across methods.
Figure 6: Plain-logistic model fitted on simulated data {(x1,i,x2,i,yi)} , on the left, and two estimations g^ , using an isotonic regression (in green), a C2 smoothing mapping (in red) and a monotone C2 smoothing mapping (in blue), on the middle left. On the right, evolution of h↦ integrated calibration index ( ℓ1 ) on the middle right, and local calibration score ( ℓ2 ) for g^h on the right, as a function of the bandwidth parameter h>0 , for three values of k , and using 5-fold cross validation
Probability calibration measures and improves whether predicted probabilities match observed outcome frequencies, supporting reliable decisions under uncertainty.
This work introduces a general framework for calibeating based on regret minimization. As compared to Foster and Hart's seminal calibeating work which had specialized treatments of Brier score (squared loss) and log loss, we consider a large family of proper losses that includes α-Tsallis losses (for α∈[1,2]) and Lipschitz losses. Our results for Tsallis losses also hold for an unscaled version of Tsallis loss that recovers log loss. Our analysis is oriented around the Bregman divergence view of a proper loss. Technically, our results for the family of Tsallis losses that we consider are U-calibration results, simultaneously obtaining logarithmic regret for all losses in this family while having a weaker dependence on the dimension compared to previous results. Of potential independent interest, we also show a new regret equality for the regret of Be The Regularized Leader. This regret equality holds for general proper losses and itself is based on two results related to online updating formulas for the generalized variance, the latter being a previously introduced generalization of variance based on Bregman divergences.
Maximilian Fichtl, Cristóbal Guzmán, Nishant A. Mehta
Independent researcher · Institute for Mathematical and Computational Engineering, Faculty of Mathematics and School of Engineering, Pontificia Universidad Católica de Chile · Department of Computer Science, University of Victoria
We introduce and study the problem of calibrating conditional risk, which involves estimating the expected loss of a prediction model conditional on input features. We analyze this problem in both classification and regression settings and show that it is fundamentally equivalent to a standard regression task. For classification settings, we further establish a connection between conditional risk calibration and individual/conditional probability calibration, and develop theoretical insights for the performance metric. This reveals that while conditional risk calibration is related to existing uncertainty quantification problems, it remains a distinct and standalone machine learning problem. Empirically, we validate our theoretical findings and demonstrate the practical implications of conditional risk calibration in the learning to defer (L2D) framework. Our systematic experiments provide both qualitative and quantitative assessments, offering guidance for future research in uncertainty-aware decision-making.
Andrey Vasilyev, Yikai Wang, Xiaocheng Li +1
Imperial College Business School, Imperial College London · Department of Statistics and Operations Research, UNC-Chapel Hill
The classic concept of "calibrated forecasts" and its more recent refinement, "calibeating," are defined with respect to the standard quadratic scoring rule. We extend these notions to the class of proper scoring rules (for which the best forecast is the true distribution) and define proper-calibration and proper-calibeating by requiring the errors to converge to zero uniformly over all bounded proper scoring rules. We first establish that calibration always implies proper-calibration, whereas calibeating need not imply proper-calibeating. Second, we show how to guarantee proper-calibeating and proper-multicalibeating. Finally, we demonstrate the equivalence between proper-calibration and universal no regret when best replying to forecasts in decision-making under uncertainty.
Dean P. Foster, Sergiu Hart
Department of Statistics, Wharton, University of Pennsylvania, Philadelphia, and Amazon, New York. · Institute of Mathematics, Department of Economics, and Federmann Center for the Study of Rationality, The Hebrew University of Jerusalem.