V-ECE: Estimating General Expected Calibration Errors

Eugène Berta; Sacha Braun; Francis Bach; Michael I. Jordan; David Holzmüller

stat.MLFeb 27, 2026

V-ECE: Estimating General Expected Calibration Errors

Authors: Eugène Berta, Sacha Braun, Francis Bach, Michael I. Jordan, David Holzmüller

Organizations: INRIA · Ecole Normale Sup´erieure, PSL Research University · UC Berkeley

Abstract

In probabilistic classification, calibration error (CE) measures the average divergence of predicted probabilities $f(X)$ from $\mathbb{P}(Y|f(X))$ , the true class distribution for that predicted probability. While being a useful diagnostic tool, it is hard to estimate: popular binning-based estimators are often inconsistent and scale poorly beyond two classes. Recent work rewrites the CE as the excess risk of a model compared to the best recalibration of its own predictions, measured with a proper loss. However, this only works for Bregman-divergence-based calibration errors like the squared error, excluding the more popular $L_1$ -distance-based CE. We show that using prediction-dependent proper scores can alleviate this restriction, allowing us to estimate CEs with general convex divergences, including $L_p$ distances with closed-form losses in the binary and multiclass settings. To estimate the excess risk, we introduce a more accurate recalibrator that fits a residual to temperature scaling with gradient boosting. The resulting variational estimator, V-ECE, needs no bins or clusters and lower-bounds the true calibration error in expectation. On a benchmark of semi-synthetic tasks built from real classifiers, with known true CE, V-ECE is among the most accurate binary estimators for every calibration error and significantly outperforms all multiclass estimators. Our results are accompanied by additional theory on $L_p$ CE, estimator bias, and over- or under-confidence estimation.

Figures & tables

Figure 1: The

L_{1}

calibration error and how we estimate it , for an over-confident binary model. (a) The

L_{1}

CE is the orange area between

\mathbb{E}[Y|f(X)]

and the diagonal

f(X)

, weighted by the distribution of

f(X)

. (b) Re-expressing the

L_{1}

CE as a risk difference: writing

R_{\ell_{f}}(g\circ f)=\mathbb{E}[\ell_{f(X)}(g\circ f(X),Y)]

for the risk under the prediction-dependent score of Corollary 2 , the largest drop achievable over all recalibration functions is exactly the

L_{1}

CE. Any

\hat{g}

therefore gives a lower bound.

Metric	Loss employed	CE estimated
Cross-entropy	$-\sum_{i}Y_{i}\log(z_{i})$	$\mathbb{E}[\mathrm{KL}(C\\|f)]$
Brier score	$\\|z-Y\\|_{2}^{2}$	$\mathbb{E}[\\|f-C\\|_{2}^{2}]$
Ours ( $\mathbf{L_{1}}$ )	$\mathds{1}_{z\neq f}\langle\sign(z\!-\!f),f\!-\!Y\rangle$	$\mathbb{E}[\\|f-C\\|_{1}]$
Ours ( $\mathbf{L_{2}}$ )	$\mathds{1}_{z\neq f}\langle\frac{z-f}{\\|z-f\\|_{2}},f\!-\!Y\rangle$	$\mathbb{E}[\\|f\!-\!C\\|_{2}]$
Ours (cvx $D_{f}$ )	$-D_{f}(z)\!-\!\langle Y\!-\!z,\nabla D_{f}(z)\rangle$	$\mathbb{E}[D_{f}(C)]$

Table 1: Losses and the calibration error they estimate , writing

f

for

f(X)

. The estimate is the risk of

f

minus the risk of

\hat{g}\circ f

under the loss of the second column. The first two rows are fixed proper losses and give Bregman divergences; the next two are prediction-dependent and give

L_{p}

distances, which no fixed proper loss can produce ( Proposition 5 ); and the last row works for any convex

D_{f}

that satisfies the conditions of Theorem 1 .

Figure 2: Calibration error recovered by each recalibration function. For each classifier and calibration error, recalibration functions are ranked by their cross-validated estimate (rank 1 for the largest, i.e. the tightest lower bound), and ranks are averaged over classifiers. The vertical line is the average rank of a method no better than the others.

Figure 3: Accuracy of calibration error estimators on the semi-synthetic benchmark , where the true calibration error of each classifier is known. Normalized absolute error : on each classifier and label draw, the absolute error of an estimator divided by the largest absolute error among the estimators, averaged over draws and classifiers (lower is better; values depend on the pool of estimators, so they are not comparable between the binary (top) and multiclass (bottom) panels). % above truth : share of draws on which the estimate exceeds the true calibration error. Estimators are sorted by their normalized absolute error on the squared calibration error; Debiased-ECE only defines

\mathrm{CE}_{f,(\cdot)^{2}}

, and Smooth-ECE, which only defines

\mathrm{CE}_{f,|\cdot|}

, is listed last. Exact values, NMAE and runtimes are in Tables F.2 and F.3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: The divergence of a proper loss as a gap under a supporting line. (a) For a fixed proper loss with entropy

H

, the divergence

d_{\ell}(f,C)

is the vertical gap at

z=C

between the tangent to

H

z=f

and the graph of

H

. Where

H

is twice differentiable this gap closes quadratically, and for the Brier entropy

H(z)=z(1-z)

it is exactly

(f-C)^{2}

. (b) The prediction-dependent entropy

H_{f}(z)=-|z-f|

has a kink at

z=f

. The supporting line with slope

0

is horizontal, and the gap at

z=C

|f-C|

. (c) The two divergences as functions of

C

: linear against quadratic near the prediction.

Figure A.2: What the estimator returns as a function of the value reported by

\hat{g}

, at a fixed prediction

f

with

C>f

. (a) With the

L_{1}

score the value is a step: it equals the true

|f-C|

for every

z

on the same side of

f

C

, and

-|f-C|

on the other side. (b) With the Brier score the value is a parabola peaked at

z=C

, so only the exact conditional probability attains

(f-C)^{2}

\widehat{\mathrm{CE}}^{(j)}=-\cfrac{1}{|\mathcal{I}_{\text{val}}^{(j)}|}\sum_{i\in\mathcal{I}_{\text{val}}^{(j)}}\ell_{f(X_{i})}\big(g^{(j)}(f(X_{i})),Y_{i}\big),\quad\ell_{u}(z,Y)=-D_{u}(z)-\langle Y-z,\nabla D_{u}(z)\rangle.

Appendix

Algorithm 1 Computing

\widehat{\mathrm{CE}}

with a prediction-dependent proper score.

Binary			Multiclass
Dataset	$n_{\mathrm{cal}}$	$n$	Dataset	$n_{\mathrm{cal}}$	$n$	$k$
2dplanes	36,691	4,077	artificial-characters	9,196	1,022	10
adult	43,957	4,885	connect-4	60,801	6,756	3
ailerons	12,375	1,375	Diabetes130US	91,589	10,177	3
Amazon_employee_access	29,492	3,277	dilbert	9,000	1,000	5
APSFailure	68,400	7,600	eye_movements	9,842	1,094	3

Appendix

Table E.1: TabRepo datasets used in the experiments , with the sizes of the calibration split (

n_{\mathrm{cal}}

) and of the test split (

n

), and the number of classes

k

of the multiclass datasets. Each dataset contributes eight classifiers, one per model family.

Estimator	Recalibration $\hat{g}$ / construction	Fitted	Functional	$k=2$	$k>2$
Uniform-ECE	15 bins, equal width	in-sample	binned distance	✓
Quantile-ECE	15 bins, equal mass	in-sample	binned distance	✓
Debiased-ECE	15 bins, equal width	in-sample	debiased binned, $(\cdot)^{2}$ only	✓
Smooth-ECE	Gaussian smoothing	in-sample	smoothed, $\|\cdot\|$ only	✓
Isotonic-ECE	isotonic regression	in-sample	plug-in distance	✓
KDE-ECE	Beta / Dirichlet kernel	leave-one-out	plug-in distance	✓	✓

Appendix

Table E.2: Estimators of the semi-synthetic benchmark , grouped by family: plug-in estimators (ECE), variational estimators with an in-sample

\hat{g}

(IS), and variational estimators with an out-of-sample

\hat{g}

, i.e. V-ECE with different recalibration functions. V-ECE (Default), in bold, is the configuration we recommend. Plug-in distance :

\frac{1}{n}\sum_{i}d(f(X_{i}),\hat{g}(f(X_{i})))

. Binned distance :

\sum_{b}\frac{n_{b}}{n}d(\bar{f}_{b},\bar{y}_{b})

, comparing the average prediction

\bar{f}_{b}

and the frequency

\bar{y}_{b}

of class 1 in each bin. Variational : the risk of

f

minus the risk of

\hat{g}\circ f

, with the loss of Table 1 that corresponds to the calibration error. The last two columns indicate whether each estimator is run in the binary (

k=2

) and multiclass (

k>2

) benchmarks: the binned, isotonic, smoothing, quadratic-scaling and Beta-calibration estimators are only defined for binary predictions, and SMS is only used in the multiclass case.

		Average rank $\downarrow$			Largest estimate (%) $\uparrow$			Time
	Recalibrator	$L_{1}$	$L_{2}$	squared	$L_{1}$	$L_{2}$	squared	[ms/1k]
Binary	WS-CatBoost	3.15	–	2.82	19	–	31	1634
	Beta	3.29	–	3.06	12	–	21	3.1
	Quadratic	3.32	–	3.19	14	–	13	3.5
	Isotonic	3.52	–	3.51	17	–	18	5.5
	Uniform	3.67	–	4.02	23	–	11	0.44

Appendix

Table F.1: Recalibration functions inside the variational estimator ( Section 5.1 ), on 149 binary and 150 multiclass TabRepo classifiers with real labels. Average rank : rank of the estimate among the recalibration functions (1 for the largest), averaged over classifiers. Largest estimate : share of classifiers on which the recalibration function returns the largest estimate.

L_{1}

stands for

\mathrm{CE}_{f,|\cdot|}

(binary) and

\mathrm{CE}_{f,\|\cdot\|_{1}}

(multiclass),

L_{2}

for

\mathrm{CE}_{f,\|\cdot\|_{2}}

, and squared for

\mathrm{CE}_{f,(\cdot)^{2}}

(binary) and

\mathrm{CE}_{f,\|\cdot\|_{2}^{2}}

(multiclass). Time : wall-clock time of the 5-fold estimate in milliseconds per 1,000 samples. Best values in bold.

	$\mathrm{CE}_{\|\cdot\|}$				$\mathrm{CE}_{(\cdot)^{2}}$				Time
Estimator	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	[ms/1k]
Debiased-ECE	–	–	–	–	0.09 †	25	0.97	45	0.06
V-ECE (Beta)	0.20	22	0.84	24	0.12 †	38	0.48	16	3.4
V-ECE (Quadratic)	0.20	22	0.82	19	0.12 †	39	0.50	15	3.6
V-ECE (Default)	0.21	22	0.84	27	0.11	25	0.48	16	1619
V-ECE (Isotonic)	0.25	25	0.74	19	0.15	37	0.21	9	5.8

Appendix

Table F.2: Accuracy of the binary estimators on 176 semi-synthetic classifiers. NAE : normalized absolute error ( Figure 3 ). NMAE :

\sum|\widehat{\mathrm{CE}}-\mathrm{CE}|/\sum\mathrm{CE}

, in %. Ratio : median of

\widehat{\mathrm{CE}}/\mathrm{CE}

. Over : share of draws with

\widehat{\mathrm{CE}}>\mathrm{CE}

, in %. All statistics are computed on each of the three label draws of every classifier ( Section E.6 ). Time : milliseconds per 1,000 samples for one estimate.

\dagger

: A dagger marks the estimators whose absolute errors are not significantly different from those of V-ECE (Default) (Wilcoxon signed-rank test, 5%); all other estimators are significantly less accurate than V-ECE (Default), except V-ECE (Beta), which is significantly more accurate for

\mathrm{CE}_{f,|\cdot|}

. Estimates are clipped at 0 after aggregating the folds, so a median ratio of 0 means that the estimate is clipped on most draws. Best values in bold.

	$\mathrm{CE}_{\\|\cdot\\|_{1}}$				$\mathrm{CE}_{\\|\cdot\\|_{2}}$				$\mathrm{CE}_{\\|\cdot\\|_{2}^{2}}$				Time
Estimator	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	[ms/1k]
V-ECE (Default)	0.27	26	0.70	2	0.26	26	0.70	2	0.15	44	0.34	0	2473
V-ECE (SMS)	0.32	37	0.64	2	0.31	37	0.65	2	0.17	58	0.25	2	102
V-ECE (TS)	0.36	44	0.63	2	0.36	44	0.61	2	0.20	63	0.12	1	12.8
V-ECE (KDE)	0.42	44	0.46	1	0.40	43	0.47	1	0.29	88	0.00	0	46.1
KDE-ECE	0.91	111	2.31	98	0.92	116	2.37	98	0.96	294	7.58	98	43.9

Appendix

Table F.3: Accuracy of the multiclass estimators on 168 semi-synthetic classifiers, with the columns of Table F.2 . V-ECE (Default) is significantly more accurate than every other estimator for the three calibration errors (Wilcoxon signed-rank test, 5%).

Figure F.1: Estimate divided by the true calibration error on each semi-synthetic classifier and label draw, against the true calibration error (log scale), for

\mathrm{CE}_{f,|\cdot|}

(binary, top) and

\mathrm{CE}_{f,\|\cdot\|_{2}}

(multiclass, bottom). The horizontal line marks an exact estimate. Triangles are ratios outside the plotted range, drawn on its boundary.

Figure F.2: Calibration error estimators as tools for comparing models. Within each dataset, the eight models are ranked by their true and by their estimated calibration error. Spearman correlation : rank correlation between the two orderings, computed on each label draw and averaged over draws and datasets (error bars: one standard error over datasets). % pairs ordered : share of the 28 pairs of models ordered as by the truth, a tied pair counting as half (50% is chance).

	$\mathrm{CE}_{\|\cdot\|}$			$\mathrm{CE}_{(\cdot)^{2}}$
Estimator	Spearman $\uparrow$	pairs (%)	top-1 (%)	Spearman $\uparrow$	pairs (%)	top-1 (%)
Smooth-ECE	0.78 $\pm$ 0.06	84	45	–	–	–
Isotonic-ECE	0.79 $\pm$ 0.06	85	59	0.72 $\pm$ 0.05	80	50
Isotonic-IS	0.74 $\pm$ 0.07	81	27	0.70 $\pm$ 0.04	79	45
Uniform-ECE	0.79 $\pm$ 0.05	85	59	0.65 $\pm$ 0.05	77	32
V-ECE (Default)	0.66 $\pm$ 0.08	78	32	0.70 $\pm$ 0.06	79	38

Appendix

Table F.4: Ordering of the eight models of each binary dataset (22 datasets). Spearman : rank correlation between the estimated and the true ordering, computed on each label draw and averaged over draws and datasets,

\pm

one standard error over datasets. Pairs : share of the 28 pairs of models ordered as by the truth, a tied pair counting as half. Top-1 : share of draws on which the model with the smallest estimate is the best-calibrated one, shared among tied models (chance: 12.5%). Best values in bold.

	$\mathrm{CE}_{\\|\cdot\\|_{1}}$			$\mathrm{CE}_{\\|\cdot\\|_{2}}$			$\mathrm{CE}_{\\|\cdot\\|_{2}^{2}}$
Estimator	Spearman $\uparrow$	pairs (%)	top-1 (%)	Spearman $\uparrow$	pairs (%)	top-1 (%)	Spearman $\uparrow$	pairs (%)	top-1 (%)
V-ECE (Default)	0.89 $\pm$ 0.03	89	57	0.88 $\pm$ 0.02	89	52	0.83 $\pm$ 0.04	86	60
V-ECE (KDE)	0.82 $\pm$ 0.03	86	52	0.82 $\pm$ 0.04	86	45	0.56 $\pm$ 0.08	59	14
V-ECE (SMS)	0.76 $\pm$ 0.06	84	43	0.76 $\pm$ 0.06	83	48	0.66 $\pm$ 0.07	78	36
KDE-ECE	0.73 $\pm$ 0.05	81	43	0.68 $\pm$ 0.06	79	43	0.41 $\pm$ 0.10	67	14
V-ECE (TS)	0.65 $\pm$ 0.07	77	24	0.64 $\pm$ 0.07	77	19	0.51 $\pm$ 0.08	71	21

Appendix

Table F.5: Ordering of the eight models of each multiclass dataset (21 datasets), with the columns of Table F.4 .

Panel of Figure 17 — Figure G.1: Separating over- and under-confidence. Top: the three simulated scenarios, showing $C=\mathbb{E}[Y\mid f(X)]$ against the prediction $f(X)$ (points) and the diagonal of perfect calibration (dashed). Predictions are over-confident (left), under-confident (middle), or a mix of both (right). Bottom: the corresponding V-ECE estimates, clipped at $0$ and averaged over 10 independent runs.

Explore similar work

Sep 14, 2026stat.ME

A Ranking Approach for Measuring Calibration

When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that

Y=1

) exactly matches the forecasted probability

f(X)

. In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability

f(X)

. Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.

Anirban Chatterjee, Rina Foygel Barber

Department of Mathematics and Statistics Boston University · Department of Statistics University of Chicago

Probability Calibration Model Calibration

Ask PDF

May 21, 2025stat.ML

Adaptive Cumulative Mass Calibration with Conformal Prediction

Reliable probability estimates by classifiers are essential in high-risk applications. In practice, however, predicted probabilities are often miscalibrated, and many existing post-hoc calibration methods typically lack guarantees that a specific notion of calibration is achieved after the correction procedure is applied. We introduce a set-based perspective on calibration through the notion of cumulative mass calibration and the corresponding error measures. We propose a new calibration procedure based on conformal prediction that forms cumulative probabilities with guaranteed marginal coverage. We introduce an adaptive temperature scaling algorithm, with the temperature tuned for each input to satisfy the conformal coverage constraint. As we show, this procedure can be efficiently implemented. Across image classification tasks, particularly in settings with many classes, our method improves newly introduced calibration error measures (CMCE and

α

-CMCE) and standard metrics (such as ECE, cw-ECE, MCE) over the existing baselines.

Daniil Kazantsev, Eric Moulines, Maxim Panov +2

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · École pour l’informatique et les techniques avancées (EPITA), Paris, France

Post-Hoc Calibration Conformal Prediction

Ask PDF

Jun 9, 2026cs.LG

Can we trust our models? Epistemic calibration in second-order classification

Uncertainty estimation is critical for deploying machine learning models in high-stakes settings. However, classical calibration only assesses the reliability of predicted probabilities and does not evaluate whether epistemic uncertainty estimates are themselves trustworthy. This limitation is particularly relevant for second-order classification models. We introduce epistemic calibration, a principled criterion that measures whether reported epistemic uncertainty faithfully reflects the dispersion of model predictions around the ground truth. We show that epistemic calibration is a strictly stronger notion than classical calibration and captures failure modes invisible to standard metrics. We relate this work to the existing literature through an impossibility theorem that holds under the epistemic calibration hypothesis. To operationalize this concept, we propose the Expected Epistemic Calibration Error (EECE), which we prove to be a consistent estimator of a True Epistemic Calibration Error (TECE). Experiments across a broad range of uncertainty quantification methods show that epistemic calibration is a coherent and meaningful criterion and reveal substantial differences across methods, despite similar predictive performance.

Arthur Hoarau

Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France

Uncertainty Quantification Uncertainty Calibration Epistemic Uncertainty

Ask PDF