Hierarchical Utility Calibration for Structured Multiclass Decisions

Futoshi Futami; Jerry Huang; Ichiro Takeuchi

stat.MLSep 29, 2026

Hierarchical Utility Calibration for Structured Multiclass Decisions

Authors: Futoshi Futami, Jerry Huang, Ichiro Takeuchi

Organizations: The University of Osaka · The University of Tokyo · RIKEN Center for Advanced Intelligence Project (AIP) · Mila – Quebec AI Institute ·

Abstract

In multiclass probabilistic prediction, Utility Calibration (UC), which focuses auditing on specified utilities, has recently received attention as a way to guarantee downstream decisions while controlling computational and sample requirements. At the same time, some multiclass problems have meaningful label hierarchies that play important roles in medicine and image classification, yet how UC evaluates utility within a hierarchy remains insufficiently understood. We show that the difference between realized utility and predicted mean utility admits an exact decomposition into a sum of contributions from the internal nodes of the label tree. This decomposition shows that positive and negative contributions from different nodes can cancel, and that even when UC is small, the utility errors remaining in parts of the hierarchy need not be small. To address this problem, we propose Hierarchical Utility Calibration (HUC), which evaluates each node contribution before summation while retaining the same target utility, subgroup, and predicted-utility interval. We further provide finite-sample evaluation over all predicted-utility intervals and propose HUC-Boost, which updates only violated internal nodes, with theoretical guarantees for both.

Figures & tables

Panel of Figure 1 — Figure 1: A four-leaf tree with predicted and true conditional branch probabilities at each node.

Panel of Figure 2 — Figure 2: Toy tree and population UC/HUC after UC- and HUC-Boost with $n$ calibration labels.

	iNaturalist					SUPPORT2
Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.489 $\pm$ .090	.900 $\pm$ .108	.272 $\pm$ .161	.125 $\pm$ .059	0.0 $\pm$ 0.0	.499 $\pm$ .199	.676 $\pm$ .083	.167 $\pm$ .178	.148 $\pm$ .142	0.0 $\pm$ 0.0
UC-B	.462 $\pm$ .088	.914 $\pm$ .066	.045 $\pm$ .018	.036 $\pm$ .011	25.4 $\pm$ 24.1	.575 $\pm$ .051	.681 $\pm$ .079	.039 $\pm$ .014	.050 $\pm$ .031	33.9 $\pm$ 39.4
HUC-B	.463 $\pm$ .090	.922 $\pm$ .060	.075 $\pm$ .032	.034 $\pm$ .009	145.9 $\pm$ 180.5	.575 $\pm$ .045	.698 $\pm$ .060	.036 $\pm$ .010	.035 $\pm$ .009	45.2 $\pm$ 46.9
Temp.	.489 $\pm$ .090	.890 $\pm$ .110	.130 $\pm$ .114	.092 $\pm$ .076	0.0 $\pm$ 0.0	.499 $\pm$ .199	.686 $\pm$ .072	.077 $\pm$ .069	.076 $\pm$ .069	0.0 $\pm$ 0.0
Temp. $\to$ HUC	.470 $\pm$ .088	.930 $\pm$ .050	.047 $\pm$ .014	.029 $\pm$ .008	43.1 $\pm$ 25.9	.587 $\pm$ .032	.703 $\pm$ .057	.032 $\pm$ .006	.032 $\pm$ .006	20.4 $\pm$ 22.6

Table 1: Independent-test results: mean

\pm

sample standard deviation over six base predictors and five splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Appendix	Content and proof locations
A : Formal definitions and tree properties	Defines the probability space, label-tree quantities, subtree mean utilities, and the path-product representation.
B : Binary classification and trees	Develops the binary-tree calculations and path decomposition. Appendix B.4 directly proves Eq. ( 5 ) and the UC identity in Section 2.2 .
C : General multiway trees	Appendix C.1 proves Eq. ( 7 ); Appendix C.2 proves Theorem 3.1 , Eqs. ( 9 ) and ( 8 ), and the mean-consistency decomposition; Appendix C.3 proves orthogonality, uniqueness, and the Parseval identity.
D : Basic properties of HUC	Appendix D.1 proves Proposition 3.3 , Eq. ( 14 ), and the implication from canonical calibration. Appendices D.2 – D.4 give counterexamples concerning conditional branches, utility impact, and node-specific audit populations.
E : Proof of HUC-Boost	Appendices E.1 and E.2 analyze Algorithm 1 and prove Theorem 3.5 ; Appendix E.3 gives the ordinary-leaf-Brier comparison; Appendix E.4 proves the path-product results.
F : Finite-sample evaluation	Appendix F.1 proves Theorem 3.4 ; Appendix F.3 derives the exact tie-preserving scan; Appendix F.4 derives the exact UC–HUC operation counts.

Appendix

Table 4

Computational stage	UC	HUC
1) Precomputation of the required statistics	$O(n\|{\mathcal{U}}\|K)$	$O\!\left(n\|{\mathcal{U}}\|(K+\|E_{{\mathcal{T}}}\|)\right)$
2) Sorting the predicted utility scores	$O(n\|{\mathcal{U}}\|\log n)$	$O(n\|{\mathcal{U}}\|\log n)$
3) Maximization over all intervals	$O(n\|{\mathcal{C}}\|\|{\mathcal{U}}\|)$	$O\!\left(n\|{\mathcal{C}}\|\sum_{u\in{\mathcal{U}}}D_{u}\right)$

Appendix

Table 2: Stage-wise operation counts for exact empirical UC and HUC auditing on the same sample, for general dense utilities. Model inference is excluded; each total is the sum of the three stages.

Panel of Figure 3 — Figure 3: The eight-leaf binary tree used in the Toy experiments.

Method	Transformation	Parameters learned on calibration data
Temperature	$g_{\mathrm{Temp},T}(p)=\operatorname{softmax}(\log p/T)$	scalar $T>0$
Vector	$g_{\mathrm{Vec},a,b_{\mathrm{V}}}(p)=\operatorname{softmax}(a\odot\log p+b_{\mathrm{V}})$	$a,b_{\mathrm{V}}\in\mathbb{R}^{K}$
Dirichlet	$g_{\mathrm{Dir},W,b_{\mathrm{D}}}(p)=\operatorname{softmax}(W\log p+b_{\mathrm{D}})$	$W\in\mathbb{R}^{K\times K}$ , $b_{\mathrm{D}}\in\mathbb{R}^{K}$

Appendix

Table 3: Whole-distribution recalibration baselines and the parameters learned on calibration data.

Panel of Figure 4 — Figure 4: The biological label tree for the selected iNaturalist species. Species labels abbreviate the genus, which is written at their parent node. Every line denotes a parent–child edge.

Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.4886 $\pm$ .0895	.9001 $\pm$ .1082	.2721 $\pm$ .1615	.1251 $\pm$ .0595	0.00 $\pm$ 0.00
UC-B	.4618 $\pm$ .0877	.9138 $\pm$ .0664	.0452 $\pm$ .0177	.0361 $\pm$ .0108	25.43 $\pm$ 24.07
HUC-B	.4633 $\pm$ .0896	.9223 $\pm$ .0599	.0748 $\pm$ .0317	.0345 $\pm$ .0090	145.93 $\pm$ 180.47
HUC-B (0.05)	.4853 $\pm$ .0888	.8971 $\pm$ .1106	.1837 $\pm$ .1767	.0815 $\pm$ .0725	329.23 $\pm$ 270.41
Temp.	.4886 $\pm$ .0895	.8902 $\pm$ .1097	.1303 $\pm$ .1143	.0920 $\pm$ .0756	0.00 $\pm$ 0.00
Temp. $\to$ HUC	.4702 $\pm$ .0878	.9301 $\pm$ .0501	.0473 $\pm$ .0135	.0289 $\pm$ .0081	43.07 $\pm$ 25.92

Appendix

Table 4: Complete iNaturalist results for the five rank-wise correctness utilities. Values are mean

\pm

sample standard deviation over six base predictors and five splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

	LR					RF
Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.5436 $\pm$ .0222	.9522 $\pm$ .0028	.2439 $\pm$ .0136	.1225 $\pm$ .0066	0.00 $\pm$ 0.00	.5187 $\pm$ .0081	.9509 $\pm$ .0027	.2957 $\pm$ .0089	.1246 $\pm$ .0089	0.00 $\pm$ 0.00
UC-B	.4786 $\pm$ .0206	.9531 $\pm$ .0031	.0721 $\pm$ .0135	.0536 $\pm$ .0043	9.80 $\pm$ 3.42	.5178 $\pm$ .0069	.9550 $\pm$ .0027	.0299 $\pm$ .0037	.0328 $\pm$ .0067	18.80 $\pm$ 7.46
Temp.	.5436 $\pm$ .0222	.9537 $\pm$ .0030	.0704 $\pm$ .0072	.0574 $\pm$ .0073	0.00 $\pm$ 0.00	.5187 $\pm$ .0081	.9572 $\pm$ .0022	.0285 $\pm$ .0043	.0249 $\pm$ .0049	0.00 $\pm$ 0.00
Vec.	.5460 $\pm$ .0192	.9556 $\pm$ .0028	.0639 $\pm$ .0082	.0379 $\pm$ .0060	0.00 $\pm$ 0.00	.5220 $\pm$ .0155	.9573 $\pm$ .0021	.0390 $\pm$ .0184	.0215 $\pm$ .0053	0.00 $\pm$ 0.00
Dir.	.5055 $\pm$ .0115	.9521 $\pm$ .0028	.1810 $\pm$ .0101	.0891 $\pm$ .0098	0.00 $\pm$ 0.00	.2984 $\pm$ .0157	.8933 $\pm$ .0067	.1638 $\pm$ .0076	.0769 $\pm$ .0067	0.00 $\pm$ 0.00

Appendix

Table 5: iNaturalist (all utilities): baseline methods by base predictor. Values are mean

\pm

sample standard deviation over five data splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

	LR					RF
Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
HUC-B	.5094 $\pm$ .0049	.9550 $\pm$ .0027	.0876 $\pm$ .0050	.0354 $\pm$ .0048	41.80 $\pm$ 7.09	.5034 $\pm$ .0094	.9513 $\pm$ .0030	.0429 $\pm$ .0116	.0286 $\pm$ .0034	79.20 $\pm$ 21.49
HUC-B (0.05)	.5286 $\pm$ .0144	.9559 $\pm$ .0031	.1124 $\pm$ .0198	.0446 $\pm$ .0081	400.20 $\pm$ 62.97	.5196 $\pm$ .0063	.9544 $\pm$ .0021	.0410 $\pm$ .0081	.0206 $\pm$ .0033	385.20 $\pm$ 70.12
Temp. $\to$ HUC	.5181 $\pm$ .0093	.9560 $\pm$ .0025	.0584 $\pm$ .0114	.0336 $\pm$ .0090	46.00 $\pm$ 31.02	.5070 $\pm$ .0116	.9564 $\pm$ .0028	.0315 $\pm$ .0072	.0204 $\pm$ .0007	29.60 $\pm$ 22.13
Vec. $\to$ HUC	.5406 $\pm$ .0104	.9566 $\pm$ .0028	.0551 $\pm$ .0109	.0349 $\pm$ .0055	17.80 $\pm$ 14.86	.5226 $\pm$ .0122	.9571 $\pm$ .0022	.0298 $\pm$ .0024	.0232 $\pm$ .0073	12.00 $\pm$ 12.90
Dir. $\to$ HUC	.5055 $\pm$ .0115	.9521 $\pm$ .0028	.1810 $\pm$ .0101	.0891 $\pm$ .0098	0.00 $\pm$ 0.00	.2984 $\pm$ .0157	.8933 $\pm$ .0067	.1638 $\pm$ .0076	.0769 $\pm$ .0067	0.00 $\pm$ 0.00

Appendix

Table 6: iNaturalist (all utilities): HUC-Boost results by base predictor. Values are mean

\pm

sample standard deviation over five data splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Panel of Figure 5 — Figure 5: SUPPORT2 outcome tree. The leaves are: $y_{1}$ , no moderate-or-greater limitation; $y_{2}$ , assistance with daily activities; $y_{3}$ , major functional limitation; $y_{4}$ , intubation or coma; $y_{5}$ , death within two months. The hierarchy and utilities are synthetic, not clinical recommendations.

Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.4990 $\pm$ .1994	.6763 $\pm$ .0828	.1671 $\pm$ .1777	.1481 $\pm$ .1422	0.00 $\pm$ 0.00
UC-B	.5746 $\pm$ .0506	.6813 $\pm$ .0789	.0394 $\pm$ .0139	.0504 $\pm$ .0313	33.90 $\pm$ 39.40
HUC-B	.5754 $\pm$ .0453	.6985 $\pm$ .0600	.0364 $\pm$ .0101	.0351 $\pm$ .0089	45.20 $\pm$ 46.89
HUC-B (0.05)	.5023 $\pm$ .1967	.6844 $\pm$ .0787	.1466 $\pm$ .1828	.1270 $\pm$ .1444	198.80 $\pm$ 148.38
Temp.	.4990 $\pm$ .1994	.6865 $\pm$ .0722	.0769 $\pm$ .0692	.0757 $\pm$ .0693	0.00 $\pm$ 0.00
Temp. $\to$ HUC	.5872 $\pm$ .0319	.7031 $\pm$ .0568	.0324 $\pm$ .0064	.0323 $\pm$ .0062	20.40 $\pm$ 22.61

Appendix

Table 7: Complete SUPPORT2 results for the nine-utility class

{\mathcal{U}}^{(3)}

. Values are mean

\pm

sample standard deviation over six base predictors and five data splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

	LR					RF
Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.6088 $\pm$ .0125	.7621 $\pm$ .0218	.0385 $\pm$ .0075	.0385 $\pm$ .0075	0.00 $\pm$ 0.00	.6152 $\pm$ .0053	.7462 $\pm$ .0289	.0473 $\pm$ .0083	.0473 $\pm$ .0083	0.00 $\pm$ 0.00
UC-B	.6088 $\pm$ .0120	.7614 $\pm$ .0210	.0255 $\pm$ .0019	.0255 $\pm$ .0019	3.80 $\pm$ 2.17	.6152 $\pm$ .0128	.7436 $\pm$ .0289	.0350 $\pm$ .0104	.0350 $\pm$ .0104	7.60 $\pm$ 10.41
Temp.	.6088 $\pm$ .0125	.7621 $\pm$ .0216	.0284 $\pm$ .0055	.0268 $\pm$ .0067	0.00 $\pm$ 0.00	.6152 $\pm$ .0053	.7442 $\pm$ .0284	.0407 $\pm$ .0083	.0407 $\pm$ .0083	0.00 $\pm$ 0.00
Vec.	.6094 $\pm$ .0119	.7633 $\pm$ .0217	.0269 $\pm$ .0035	.0269 $\pm$ .0035	0.00 $\pm$ 0.00	.6140 $\pm$ .0038	.7286 $\pm$ .0394	.0306 $\pm$ .0071	.0306 $\pm$ .0071	0.00 $\pm$ 0.00
Dir.	.6100 $\pm$ .0120	.7677 $\pm$ .0182	.0251 $\pm$ .0028	.0251 $\pm$ .0028	0.00 $\pm$ 0.00	.6124 $\pm$ .0087	.7475 $\pm$ .0133	.0310 $\pm$ .0095	.0310 $\pm$ .0095	0.00 $\pm$ 0.00

Appendix

Table 8: SUPPORT2 (

{\mathcal{U}}^{(3)}

): baseline methods by base predictor. Values are mean

\pm

sample standard deviation over five data splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

	LR					RF
Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
HUC-B	.6086 $\pm$ .0121	.7580 $\pm$ .0208	.0282 $\pm$ .0019	.0271 $\pm$ .0017	12.80 $\pm$ 7.16	.6100 $\pm$ .0166	.7459 $\pm$ .0397	.0342 $\pm$ .0108	.0342 $\pm$ .0107	19.20 $\pm$ 12.77
HUC-B (0.05)	.6112 $\pm$ .0138	.7608 $\pm$ .0197	.0285 $\pm$ .0034	.0285 $\pm$ .0034	72.80 $\pm$ 54.21	.6156 $\pm$ .0109	.7485 $\pm$ .0314	.0332 $\pm$ .0095	.0332 $\pm$ .0095	120.00 $\pm$ 90.77
Temp. $\to$ HUC	.6098 $\pm$ .0117	.7597 $\pm$ .0200	.0303 $\pm$ .0052	.0303 $\pm$ .0052	9.60 $\pm$ 15.13	.6158 $\pm$ .0097	.7499 $\pm$ .0330	.0306 $\pm$ .0070	.0318 $\pm$ .0067	16.80 $\pm$ 17.30
Vec. $\to$ HUC	.6088 $\pm$ .0112	.7555 $\pm$ .0230	.0296 $\pm$ .0062	.0278 $\pm$ .0038	8.00 $\pm$ 10.20	.6162 $\pm$ .0117	.7281 $\pm$ .0297	.0305 $\pm$ .0077	.0301 $\pm$ .0082	16.80 $\pm$ 12.46
Dir. $\to$ HUC	.6098 $\pm$ .0118	.7641 $\pm$ .0171	.0256 $\pm$ .0033	.0256 $\pm$ .0033	0.80 $\pm$ 1.79	.6138 $\pm$ .0097	.7267 $\pm$ .0230	.0314 $\pm$ .0071	.0316 $\pm$ .0071	20.00 $\pm$ 12.33

Appendix

Table 9: SUPPORT2 (

{\mathcal{U}}^{(3)}

): HUC-Boost results by base predictor. Values are mean

\pm

sample standard deviation over five data splits. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Target class	Utilities	Relevant nodes	HUC error after correction			Updates
			UC-B	HUC-B	HUC-B (0.05)	UC-B	HUC-B	HUC-B (0.05)
$\mathcal{U}^{(1)}$	3	1	.0333 $\pm$ .0083	.0343 $\pm$ .0081	.1264 $\pm$ .1451	11.57 $\pm$ 11.41	11.90 $\pm$ 11.79	47.77 $\pm$ 36.81
$\mathcal{U}^{(2)}$	6	3	.0557 $\pm$ .0431	.0346 $\pm$ .0081	.1268 $\pm$ .1448	29.30 $\pm$ 34.74	32.70 $\pm$ 35.19	138.60 $\pm$ 108.89
$\mathcal{U}^{(3)}$	9	4	.0504 $\pm$ .0313	.0351 $\pm$ .0089	.1270 $\pm$ .1444	33.90 $\pm$ 39.40	45.20 $\pm$ 46.89	198.80 $\pm$ 148.38

Appendix

Table 10: Supplementary SUPPORT2 utility-class expansion. Each correction is fitted separately from Baseline for each target class. HUC-B uses

\alpha=\Gamma/\Lambda

; HUC-B (0.05) uses constant step magnitude

0.05

. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Panel of Figure 6 — Figure 6: MASSIVE label tree. Dots omit scenario and intent nodes. The intents below $s_{j}$ are ${\mathcal{Y}}_{s_{j}}=\{y_{j,1},\ldots,y_{j,k_{j}}\}$ , with $\sum_{j=1}^{18}k_{j}=60$ ; each intent belongs to exactly one scenario.

Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.8222 $\pm$ .0025	.9913 $\pm$ .0008	.1380 $\pm$ .0039	.1190 $\pm$ .0035	0.00 $\pm$ 0.00
UC-B	.8223 $\pm$ .0026	.9899 $\pm$ .0009	.0216 $\pm$ .0077	.0142 $\pm$ .0043	44.60 $\pm$ 33.25
HUC-B	.8197 $\pm$ .0036	.9891 $\pm$ .0009	.0302 $\pm$ .0078	.0170 $\pm$ .0065	251.60 $\pm$ 50.97
HUC-B (0.05)	.8223 $\pm$ .0027	.9897 $\pm$ .0009	.0213 $\pm$ .0083	.0140 $\pm$ .0071	395.80 $\pm$ 66.34
Temp.	.8222 $\pm$ .0025	.9906 $\pm$ .0007	.0335 $\pm$ .0075	.0189 $\pm$ .0040	0.00 $\pm$ 0.00
Temp. $\to$ HUC	.8206 $\pm$ .0029	.9904 $\pm$ .0008	.0371 $\pm$ .0057	.0181 $\pm$ .0048	109.40 $\pm$ 110.03

Appendix

Table 11: MASSIVE results. Values are mean

\pm

sample standard deviation over five data splits of the TF–IDF logistic-regression predictor. HUC-B (0.05) uses constant step magnitude

0.05

. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Panel of Figure 7 — Figure 7: NSL-KDD label tree. The five leaves are normal traffic and the four attack families.

Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.9504 $\pm$ .1004	.9577 $\pm$ .0653	.0471 $\pm$ .1011	.0452 $\pm$ .0980	0.00 $\pm$ 0.00
UC-B	.9816 $\pm$ .0244	.9687 $\pm$ .0444	.0039 $\pm$ .0034	.0035 $\pm$ .0031	62.07 $\pm$ 39.19
HUC-B	.9801 $\pm$ .0283	.9672 $\pm$ .0514	.0039 $\pm$ .0031	.0033 $\pm$ .0026	92.37 $\pm$ 69.38
HUC-B (0.05)	.9505 $\pm$ .1004	.9621 $\pm$ .0605	.0448 $\pm$ .1014	.0432 $\pm$ .0983	85.37 $\pm$ 75.29
Temp.	.9504 $\pm$ .1004	.9619 $\pm$ .0563	.0285 $\pm$ .0555	.0275 $\pm$ .0552	0.00 $\pm$ 0.00
Temp. $\to$ HUC	.9838 $\pm$ .0195	.9766 $\pm$ .0363	.0036 $\pm$ .0028	.0032 $\pm$ .0026	64.43 $\pm$ 66.43

Appendix

Table 12: NSL-KDD results. Values are mean

\pm

sample standard deviation over six base predictors and five data splits. HUC-B (0.05) uses constant step magnitude

0.05

. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Panel of Figure 8 — Figure 8: N-BaIoT label tree. Identically named attack types under different families are different leaves; for example, Gafgyt scan and Mirai scan are distinct labels.

Method	Acc. $\uparrow$	AUC $\uparrow$	UC $\downarrow$	HUC $\downarrow$	Updates
Baseline	.7852 $\pm$ .1560	.9703 $\pm$ .0382	.1021 $\pm$ .0998	.0672 $\pm$ .0633	0.00 $\pm$ 0.00
UC-B	.7960 $\pm$ .1262	.9720 $\pm$ .0300	.0338 $\pm$ .0221	.0324 $\pm$ .0197	31.50 $\pm$ 30.98
HUC-B	.7987 $\pm$ .1254	.9736 $\pm$ .0307	.0357 $\pm$ .0212	.0278 $\pm$ .0186	35.27 $\pm$ 28.58
HUC-B (0.05)	.7960 $\pm$ .1281	.9749 $\pm$ .0283	.0415 $\pm$ .0332	.0325 $\pm$ .0316	137.33 $\pm$ 80.47
Temp.	.7852 $\pm$ .1560	.9711 $\pm$ .0396	.0415 $\pm$ .0354	.0391 $\pm$ .0364	0.00 $\pm$ 0.00
Temp. $\to$ HUC	.8010 $\pm$ .1264	.9764 $\pm$ .0265	.0326 $\pm$ .0239	.0265 $\pm$ .0189	24.80 $\pm$ 33.13

Appendix

Table 13: N-BaIoT results. Values are mean

\pm

sample standard deviation over six base predictors and five data splits. HUC-B (0.05) uses constant step magnitude

0.05

. UC-Boost and HUC-Boost are abbreviated as UC-B and HUC-B, respectively.

Probability Calibration

Probability calibration measures and improves whether predicted probabilities match observed outcome frequencies, supporting reliable decisions under uncertainty.

See all 134 papers

How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification
Sep 22, 2026Sophie Henning, Georg Hofmann, Alexander Schulte +2Multi-Label Classification Language Model Calibration
TU Munich · Munich Center for Machine Learning (MCML) · LMU Munich +3
CalArena: A Large-Scale Post-Hoc Calibration Benchmark
May 28, 2026Eugène Berta, David Holzmüller, Francis Bach +1Post-Hoc Calibration Probability Calibration
Inria - Ecole Normale Supérieure PSL Research University · Inria
Adaptive Cumulative Mass Calibration with Conformal Prediction
May 21, 2025Daniil Kazantsev, Eric Moulines, Maxim Panov +2Post-Hoc Calibration Conformal Prediction
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · École pour l’informatique et les techniques avancées (EPITA), Paris, France

Sep 22, 2026J/K move · Enter open · S save

How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification

Sophie Henning, Georg Hofmann, Alexander Schulte, Alexander Fraser+1

TU Munich · Munich Center for Machine Learning (MCML) · LMU Munich+3

Multi-Label Classification Language Model Calibration Model Calibration Classification

Open paper AskPDF arXiv

A key factor in deciding whether to trust an automatic prediction is its confidence score, which should be calibrated to match the actual probability of the prediction being correct. Most confidence calibration metrics target binary or multi-class tasks, while multi-label calibration remains largely underexplored. Multi-label classification tasks, such as assigning medical codes to clinical notes or determining news topics, are usually dominated by a large number of negatives, i.e., labels that do not apply. We show that existing binning schemes to compute label-wise expected calibration error either underestimate the error, simply reflect label frequency, or suffer from many bins with very few instances. To achieve trustworthy label-wise calibration errors, we propose a new binning scheme that gives equal weight to positive and negative label assignments. Our empirical study demonstrates that in contrast to existing binning schemes, our new scheme results in meaningful estimates of calibration error in hierarchical and in extreme multi-label classification. We also show that calibrating confidence scores of large language models for multi-label predictions is an open challenge. Our detailed analysis lays the foundation for further research by providing a solid evaluation metric for measuring calibration in multi-label classification.

Abstract

Probability Calibration

Explore similar work

How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Adaptive Cumulative Mass Calibration with Conformal Prediction