V-ECE: Estimating General Expected Calibration Errors

Eugène Berta; Sacha Braun; Francis Bach; Michael I. Jordan; David Holzmüller

stat.MLFeb 27, 2026

V-ECE: Estimating General Expected Calibration Errors

Authors: Eugène Berta, Sacha Braun, Francis Bach, Michael I. Jordan, David Holzmüller

Organizations: INRIA · Ecole Normale Sup´erieure, PSL Research University · UC Berkeley

Abstract

In probabilistic classification, calibration error (CE) measures the average divergence of predicted probabilities $f(X)$ from $\mathbb{P}(Y|f(X))$ , the true class distribution for that predicted probability. While being a useful diagnostic tool, it is hard to estimate: popular binning-based estimators are often inconsistent and scale poorly beyond two classes. Recent work rewrites the CE as the excess risk of a model compared to the best recalibration of its own predictions, measured with a proper loss. However, this only works for Bregman-divergence-based calibration errors like the squared error, excluding the more popular $L_1$ -distance-based CE. We show that using prediction-dependent proper scores can alleviate this restriction, allowing us to estimate CEs with general convex divergences, including $L_p$ distances with closed-form losses in the binary and multiclass settings. To estimate the excess risk, we introduce a more accurate recalibrator that fits a residual to temperature scaling with gradient boosting. The resulting variational estimator, V-ECE, needs no bins or clusters and lower-bounds the true calibration error in expectation. On a benchmark of semi-synthetic tasks built from real classifiers, with known true CE, V-ECE is among the most accurate binary estimators for every calibration error and significantly outperforms all multiclass estimators. Our results are accompanied by additional theory on $L_p$ CE, estimator bias, and over- or under-confidence estimation.

Figures & tables

Figure 1: The

L_{1}

calibration error and how we estimate it , for an over-confident binary model. (a) The

L_{1}

CE is the orange area between

\mathbb{E}[Y|f(X)]

and the diagonal

f(X)

, weighted by the distribution of

f(X)

. (b) Re-expressing the

L_{1}

CE as a risk difference: writing

R_{\ell_{f}}(g\circ f)=\mathbb{E}[\ell_{f(X)}(g\circ f(X),Y)]

for the risk under the prediction-dependent score of Corollary 2 , the largest drop achievable over all recalibration functions is exactly the

L_{1}

CE. Any

\hat{g}

therefore gives a lower bound.

Metric	Loss employed	CE estimated
Cross-entropy	$-\sum_{i}Y_{i}\log(z_{i})$	$\mathbb{E}[\mathrm{KL}(C\\|f)]$
Brier score	$\\|z-Y\\|_{2}^{2}$	$\mathbb{E}[\\|f-C\\|_{2}^{2}]$
Ours ( $\mathbf{L_{1}}$ )	$\mathds{1}_{z\neq f}\langle\sign(z\!-\!f),f\!-\!Y\rangle$	$\mathbb{E}[\\|f-C\\|_{1}]$
Ours ( $\mathbf{L_{2}}$ )	$\mathds{1}_{z\neq f}\langle\frac{z-f}{\\|z-f\\|_{2}},f\!-\!Y\rangle$	$\mathbb{E}[\\|f\!-\!C\\|_{2}]$
Ours (cvx $D_{f}$ )	$-D_{f}(z)\!-\!\langle Y\!-\!z,\nabla D_{f}(z)\rangle$	$\mathbb{E}[D_{f}(C)]$

Table 1: Losses and the calibration error they estimate , writing

f

for

f(X)

. The estimate is the risk of

f

minus the risk of

\hat{g}\circ f

under the loss of the second column. The first two rows are fixed proper losses and give Bregman divergences; the next two are prediction-dependent and give

L_{p}

distances, which no fixed proper loss can produce ( Proposition 5 ); and the last row works for any convex

D_{f}

that satisfies the conditions of Theorem 1 .

Figure 2: Calibration error recovered by each recalibration function. For each classifier and calibration error, recalibration functions are ranked by their cross-validated estimate (rank 1 for the largest, i.e. the tightest lower bound), and ranks are averaged over classifiers. The vertical line is the average rank of a method no better than the others.

Figure 3: Accuracy of calibration error estimators on the semi-synthetic benchmark , where the true calibration error of each classifier is known. Normalized absolute error : on each classifier and label draw, the absolute error of an estimator divided by the largest absolute error among the estimators, averaged over draws and classifiers (lower is better; values depend on the pool of estimators, so they are not comparable between the binary (top) and multiclass (bottom) panels). % above truth : share of draws on which the estimate exceeds the true calibration error. Estimators are sorted by their normalized absolute error on the squared calibration error; Debiased-ECE only defines

\mathrm{CE}_{f,(\cdot)^{2}}

, and Smooth-ECE, which only defines

\mathrm{CE}_{f,|\cdot|}

, is listed last. Exact values, NMAE and runtimes are in Tables F.2 and F.3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: The divergence of a proper loss as a gap under a supporting line. (a) For a fixed proper loss with entropy

H

, the divergence

d_{\ell}(f,C)

is the vertical gap at

z=C

between the tangent to

H

z=f

and the graph of

H

. Where

H

is twice differentiable this gap closes quadratically, and for the Brier entropy

H(z)=z(1-z)

it is exactly

(f-C)^{2}

. (b) The prediction-dependent entropy

H_{f}(z)=-|z-f|

has a kink at

z=f

. The supporting line with slope

0

is horizontal, and the gap at

z=C

|f-C|

. (c) The two divergences as functions of

C

: linear against quadratic near the prediction.

Figure A.2: What the estimator returns as a function of the value reported by

\hat{g}

, at a fixed prediction

f

with

C>f

. (a) With the

L_{1}

score the value is a step: it equals the true

|f-C|

for every

z

on the same side of

f

C

, and

-|f-C|

on the other side. (b) With the Brier score the value is a parabola peaked at

z=C

, so only the exact conditional probability attains

(f-C)^{2}

\widehat{\mathrm{CE}}^{(j)}=-\cfrac{1}{|\mathcal{I}_{\text{val}}^{(j)}|}\sum_{i\in\mathcal{I}_{\text{val}}^{(j)}}\ell_{f(X_{i})}\big(g^{(j)}(f(X_{i})),Y_{i}\big),\quad\ell_{u}(z,Y)=-D_{u}(z)-\langle Y-z,\nabla D_{u}(z)\rangle.

Appendix

Algorithm 1 Computing

\widehat{\mathrm{CE}}

with a prediction-dependent proper score.

Binary			Multiclass
Dataset	$n_{\mathrm{cal}}$	$n$	Dataset	$n_{\mathrm{cal}}$	$n$	$k$
2dplanes	36,691	4,077	artificial-characters	9,196	1,022	10
adult	43,957	4,885	connect-4	60,801	6,756	3
ailerons	12,375	1,375	Diabetes130US	91,589	10,177	3
Amazon_employee_access	29,492	3,277	dilbert	9,000	1,000	5
APSFailure	68,400	7,600	eye_movements	9,842	1,094	3

Appendix

Table E.1: TabRepo datasets used in the experiments , with the sizes of the calibration split (

n_{\mathrm{cal}}

) and of the test split (

n

), and the number of classes

k

of the multiclass datasets. Each dataset contributes eight classifiers, one per model family.

Estimator	Recalibration $\hat{g}$ / construction	Fitted	Functional	$k=2$	$k>2$
Uniform-ECE	15 bins, equal width	in-sample	binned distance	✓
Quantile-ECE	15 bins, equal mass	in-sample	binned distance	✓
Debiased-ECE	15 bins, equal width	in-sample	debiased binned, $(\cdot)^{2}$ only	✓
Smooth-ECE	Gaussian smoothing	in-sample	smoothed, $\|\cdot\|$ only	✓
Isotonic-ECE	isotonic regression	in-sample	plug-in distance	✓
KDE-ECE	Beta / Dirichlet kernel	leave-one-out	plug-in distance	✓	✓

Appendix

Table E.2: Estimators of the semi-synthetic benchmark , grouped by family: plug-in estimators (ECE), variational estimators with an in-sample

\hat{g}

(IS), and variational estimators with an out-of-sample

\hat{g}

, i.e. V-ECE with different recalibration functions. V-ECE (Default), in bold, is the configuration we recommend. Plug-in distance :

\frac{1}{n}\sum_{i}d(f(X_{i}),\hat{g}(f(X_{i})))

. Binned distance :

\sum_{b}\frac{n_{b}}{n}d(\bar{f}_{b},\bar{y}_{b})

, comparing the average prediction

\bar{f}_{b}

and the frequency

\bar{y}_{b}

of class 1 in each bin. Variational : the risk of

f

minus the risk of

\hat{g}\circ f

, with the loss of Table 1 that corresponds to the calibration error. The last two columns indicate whether each estimator is run in the binary (

k=2

) and multiclass (

k>2

) benchmarks: the binned, isotonic, smoothing, quadratic-scaling and Beta-calibration estimators are only defined for binary predictions, and SMS is only used in the multiclass case.

		Average rank $\downarrow$			Largest estimate (%) $\uparrow$			Time
	Recalibrator	$L_{1}$	$L_{2}$	squared	$L_{1}$	$L_{2}$	squared	[ms/1k]
Binary	WS-CatBoost	3.15	–	2.82	19	–	31	1634
	Beta	3.29	–	3.06	12	–	21	3.1
	Quadratic	3.32	–	3.19	14	–	13	3.5
	Isotonic	3.52	–	3.51	17	–	18	5.5
	Uniform	3.67	–	4.02	23	–	11	0.44

Appendix

Table F.1: Recalibration functions inside the variational estimator ( Section 5.1 ), on 149 binary and 150 multiclass TabRepo classifiers with real labels. Average rank : rank of the estimate among the recalibration functions (1 for the largest), averaged over classifiers. Largest estimate : share of classifiers on which the recalibration function returns the largest estimate.

L_{1}

stands for

\mathrm{CE}_{f,|\cdot|}

(binary) and

\mathrm{CE}_{f,\|\cdot\|_{1}}

(multiclass),

L_{2}

for

\mathrm{CE}_{f,\|\cdot\|_{2}}

, and squared for

\mathrm{CE}_{f,(\cdot)^{2}}

(binary) and

\mathrm{CE}_{f,\|\cdot\|_{2}^{2}}

(multiclass). Time : wall-clock time of the 5-fold estimate in milliseconds per 1,000 samples. Best values in bold.

	$\mathrm{CE}_{\|\cdot\|}$				$\mathrm{CE}_{(\cdot)^{2}}$				Time
Estimator	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	[ms/1k]
Debiased-ECE	–	–	–	–	0.09 †	25	0.97	45	0.06
V-ECE (Beta)	0.20	22	0.84	24	0.12 †	38	0.48	16	3.4
V-ECE (Quadratic)	0.20	22	0.82	19	0.12 †	39	0.50	15	3.6
V-ECE (Default)	0.21	22	0.84	27	0.11	25	0.48	16	1619
V-ECE (Isotonic)	0.25	25	0.74	19	0.15	37	0.21	9	5.8

Appendix

Table F.2: Accuracy of the binary estimators on 176 semi-synthetic classifiers. NAE : normalized absolute error ( Figure 3 ). NMAE :

\sum|\widehat{\mathrm{CE}}-\mathrm{CE}|/\sum\mathrm{CE}

, in %. Ratio : median of

\widehat{\mathrm{CE}}/\mathrm{CE}

. Over : share of draws with

\widehat{\mathrm{CE}}>\mathrm{CE}

, in %. All statistics are computed on each of the three label draws of every classifier ( Section E.6 ). Time : milliseconds per 1,000 samples for one estimate.

\dagger

: A dagger marks the estimators whose absolute errors are not significantly different from those of V-ECE (Default) (Wilcoxon signed-rank test, 5%); all other estimators are significantly less accurate than V-ECE (Default), except V-ECE (Beta), which is significantly more accurate for

\mathrm{CE}_{f,|\cdot|}

. Estimates are clipped at 0 after aggregating the folds, so a median ratio of 0 means that the estimate is clipped on most draws. Best values in bold.

	$\mathrm{CE}_{\\|\cdot\\|_{1}}$				$\mathrm{CE}_{\\|\cdot\\|_{2}}$				$\mathrm{CE}_{\\|\cdot\\|_{2}^{2}}$				Time
Estimator	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	NAE $\downarrow$	NMAE $\downarrow$	ratio	over	[ms/1k]
V-ECE (Default)	0.27	26	0.70	2	0.26	26	0.70	2	0.15	44	0.34	0	2473
V-ECE (SMS)	0.32	37	0.64	2	0.31	37	0.65	2	0.17	58	0.25	2	102
V-ECE (TS)	0.36	44	0.63	2	0.36	44	0.61	2	0.20	63	0.12	1	12.8
V-ECE (KDE)	0.42	44	0.46	1	0.40	43	0.47	1	0.29	88	0.00	0	46.1
KDE-ECE	0.91	111	2.31	98	0.92	116	2.37	98	0.96	294	7.58	98	43.9

Appendix

Table F.3: Accuracy of the multiclass estimators on 168 semi-synthetic classifiers, with the columns of Table F.2 . V-ECE (Default) is significantly more accurate than every other estimator for the three calibration errors (Wilcoxon signed-rank test, 5%).

Figure F.1: Estimate divided by the true calibration error on each semi-synthetic classifier and label draw, against the true calibration error (log scale), for

\mathrm{CE}_{f,|\cdot|}

(binary, top) and

\mathrm{CE}_{f,\|\cdot\|_{2}}

(multiclass, bottom). The horizontal line marks an exact estimate. Triangles are ratios outside the plotted range, drawn on its boundary.

Figure F.2: Calibration error estimators as tools for comparing models. Within each dataset, the eight models are ranked by their true and by their estimated calibration error. Spearman correlation : rank correlation between the two orderings, computed on each label draw and averaged over draws and datasets (error bars: one standard error over datasets). % pairs ordered : share of the 28 pairs of models ordered as by the truth, a tied pair counting as half (50% is chance).

	$\mathrm{CE}_{\|\cdot\|}$			$\mathrm{CE}_{(\cdot)^{2}}$
Estimator	Spearman $\uparrow$	pairs (%)	top-1 (%)	Spearman $\uparrow$	pairs (%)	top-1 (%)
Smooth-ECE	0.78 $\pm$ 0.06	84	45	–	–	–
Isotonic-ECE	0.79 $\pm$ 0.06	85	59	0.72 $\pm$ 0.05	80	50
Isotonic-IS	0.74 $\pm$ 0.07	81	27	0.70 $\pm$ 0.04	79	45
Uniform-ECE	0.79 $\pm$ 0.05	85	59	0.65 $\pm$ 0.05	77	32
V-ECE (Default)	0.66 $\pm$ 0.08	78	32	0.70 $\pm$ 0.06	79	38

Appendix

Table F.4: Ordering of the eight models of each binary dataset (22 datasets). Spearman : rank correlation between the estimated and the true ordering, computed on each label draw and averaged over draws and datasets,

\pm

one standard error over datasets. Pairs : share of the 28 pairs of models ordered as by the truth, a tied pair counting as half. Top-1 : share of draws on which the model with the smallest estimate is the best-calibrated one, shared among tied models (chance: 12.5%). Best values in bold.

	$\mathrm{CE}_{\\|\cdot\\|_{1}}$			$\mathrm{CE}_{\\|\cdot\\|_{2}}$			$\mathrm{CE}_{\\|\cdot\\|_{2}^{2}}$
Estimator	Spearman $\uparrow$	pairs (%)	top-1 (%)	Spearman $\uparrow$	pairs (%)	top-1 (%)	Spearman $\uparrow$	pairs (%)	top-1 (%)
V-ECE (Default)	0.89 $\pm$ 0.03	89	57	0.88 $\pm$ 0.02	89	52	0.83 $\pm$ 0.04	86	60
V-ECE (KDE)	0.82 $\pm$ 0.03	86	52	0.82 $\pm$ 0.04	86	45	0.56 $\pm$ 0.08	59	14
V-ECE (SMS)	0.76 $\pm$ 0.06	84	43	0.76 $\pm$ 0.06	83	48	0.66 $\pm$ 0.07	78	36
KDE-ECE	0.73 $\pm$ 0.05	81	43	0.68 $\pm$ 0.06	79	43	0.41 $\pm$ 0.10	67	14
V-ECE (TS)	0.65 $\pm$ 0.07	77	24	0.64 $\pm$ 0.07	77	19	0.51 $\pm$ 0.08	71	21

Appendix

Table F.5: Ordering of the eight models of each multiclass dataset (21 datasets), with the columns of Table F.4 .

Panel of Figure 17 — Figure G.1: Separating over- and under-confidence. Top: the three simulated scenarios, showing $C=\mathbb{E}[Y\mid f(X)]$ against the prediction $f(X)$ (points) and the diagonal of perfect calibration (dashed). Predictions are over-confident (left), under-confident (middle), or a mix of both (right). Bottom: the corresponding V-ECE estimates, clipped at $0$ and averaged over 10 independent runs.

Explore similar work

A Ranking Approach for Measuring Calibration
Sep 14, 2026Anirban Chatterjee, Rina Foygel BarberProbability Calibration Model Calibration
Department of Mathematics and Statistics Boston University · Department of Statistics University of Chicago
Adaptive Cumulative Mass Calibration with Conformal Prediction
May 21, 2025Daniil Kazantsev, Eric Moulines, Maxim Panov +2Post-Hoc Calibration Conformal Prediction
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · École pour l’informatique et les techniques avancées (EPITA), Paris, France
Can we trust our models? Epistemic calibration in second-order classification
Jun 9, 2026Arthur HoarauUncertainty Quantification Uncertainty Calibration
Université de Lorraine, CentraleSupélec · Loria, CNRS, Metz, France

Sep 14, 2026, stat.MEJ/K move · Enter open · S save

A Ranking Approach for Measuring Calibration

Anirban Chatterjee, Rina Foygel Barber

Department of Mathematics and Statistics Boston University · Department of Statistics University of Chicago

Probability Calibration Model Calibration

Open paper AskPDF arXiv

When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that

Y=1

) exactly matches the forecasted probability

f(X)

. In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability

f(X)

. Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.

Abstract

Explore similar work

A Ranking Approach for Measuring Calibration

Adaptive Cumulative Mass Calibration with Conformal Prediction

Can we trust our models? Epistemic calibration in second-order classification