Signal-Routed Temperature Scaling: Low-Capacity Risk-Conditioned Calibration for Small Validation Budgets

Wenhao Liang; Liangwei Nathan Zheng; Lin Yue; Wei Emma Zhang; Mingyu Guo; Olaf Maennel; Weitong Chen

cs.LGSep 30, 2026

Signal-Routed Temperature Scaling: Low-Capacity Risk-Conditioned Calibration for Small Validation Budgets

Authors: Wenhao Liang, Liangwei Nathan Zheng, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen

Organizations: Adelaide University, Adelaide, Australia

Abstract

When a classifier is recalibrated from only a few thousand held-out examples, the capacity of the calibration map becomes a statistical design choice rather than a purely architectural one: a scalar map can underfit structured residual miscalibration, while a highly adaptive map can be hard to estimate reliably from so small a split. We disentangle the calibration objective from adaptive capacity and propose signal-routed temperature scaling (SRTS-BCE), a 10-parameter, argmax-preserving calibrator that cross-fits a correctness-risk score over six logit statistics and fits one top-label-BCE temperature per $K=3$ risk groups, recovering TvA-TS as its $K=1$ limit. On fine-tuned CIFAR-100 / ViT-B/16, SRTS-BCE reduces $\mathrm{ECE}_{15}$ from 1.65 (scalar TvA-TS) to 0.96, matching the higher-capacity SMART+BCE head (0.95) at the full calibration budget. The two regimes separate as the budget shrinks: at $n=250$ SRTS-BCE beats SMART+BCE on all three CIFAR-100 backbones (the seed-to-draw hierarchical interval excludes zero), whereas the flagship comparison against the scalar remains directional. A protocol-frozen Tiny-ImageNet follow-up reproduces the small-budget separation and exhibits a budget-dependent ranking reversal on Swin-T; matched routing and map controls show that the effect is tied neither to the learned router nor to discrete grouping. Together the results identify post-hoc calibrator capacity as a finite-sample design choice whose preferred level shifts with the amount of available calibration data.

Figures & tables

Figure 1: Problem, capacity spectrum, method, and small-budget evidence. (1) A small validation set can leave a fine-tuned classifier miscalibrated (illustrative predictions, not benchmark samples). (2) Calibration maps span a capacity spectrum: a single shared temperature (TvA-TS) can underfit and a sample-adaptive head carries higher estimation variance at small $n$ ; SRTS-BCE takes a low-capacity middle ground with three risk-conditioned group temperatures. (3) Six logit statistics yield five-fold cross-fitted out-of-fold risk scores that define three equal-frequency risk tertiles ( $T_{1}$ / $T_{2}$ / $T_{3}$ ), each fit with one top-label-BCE temperature; the router is refit on the full split for deployment, and each test sample is rerouted from its own logits with thresholds and temperatures fixed. (4) Measured data (unlike the schematic panels 1–3): mean $\mathrm{ECE}_{15}$ vs. validation set size $n$ on CIFAR-100 / ViT-B/16 for TvA-TS, SMART+BCE, and SRTS-BCE (five seeds $\times$ $20$ draws; protocol in Results); SMART+BCE loses its held-out advantage at the smallest budget $n{=}250$ .

Method	Params	NLL	TopBCE	$\mathrm{ECE}_{15}$	AdaECE 15	smECE	eAURC
CE / NoTS	0	$0.4142\pm 0.0204$	$0.2633\pm 0.0078$	$9.69\pm 1.10$	$9.65\pm 1.15$	$9.63\pm 1.14$	$0.0165\pm 0.0041$
TS-NLL	1	$0.3309\pm 0.0308$	$0.2050\pm 0.0143$	$1.94\pm 0.65$	$2.10\pm 0.62$	$1.97\pm 0.52$	$0.0139\pm 0.0032$
HTS-NLL	2	$0.3311\pm 0.0310$	$0.2054\pm 0.0142$	$2.07\pm 0.51$	$2.26\pm 0.34$	$2.13\pm 0.38$	$0.0140\pm 0.0038$
SRTS-NLL	10	$\mathbf{0.3271\pm 0.0275}$	$0.1994\pm 0.0102$	$1.79\pm 0.62$	$1.77\pm 0.61$	$1.81\pm 0.55$	$\mathbf{0.0113\pm 0.0011}$
TvA-TS ( $K{=}1$ )	1	$0.3321\pm 0.0296$	$0.2041\pm 0.0135$	$1.65\pm 0.44$	$1.77\pm 0.69$	$1.71\pm 0.45$	$0.0142\pm 0.0033$
HTS-BCE	2	$0.3325\pm 0.0304$	$0.2041\pm 0.0135$	$1.63\pm 0.25$	$1.51\pm 0.31$	$1.58\pm 0.22$	$0.0146\pm 0.0039$

Table 1: RQ1: objective and capacity on ViT-B/16 / CIFAR-100 (%, mean

\pm

std over five seeds; top-1

91.00\pm 0.46

; fit on the validation split, evaluated on held-out test; metric definitions, including TopBCE, in App. A ). Params: fitted scalars. TvA-TS

\equiv

TS-BCE (

K{=}1

); HTS ( Balanya et al. 2024 ) is fitted with its original NLL objective and with this paper’s top-label BCE (Setup, Baselines). Bold : best per column among deployable calibrators. SMART+BCE has the lowest clean

\mathrm{ECE}_{15}

point estimate (

0.9483

vs.

0.9559

); the paired difference is unresolved (hierarchical CI

[-0.19,+0.12]

). Full metric set and further adaptive-head rows: Tab. 53 .

Figure 2: Small-budget stability (RQ1b). Calibrators are fit on

20

stratified calibration resamples per training seed and evaluated on the fixed test set. (a) CIFAR-100 / ViT-B/16 mean

\mathrm{ECE}_{15}

vs. budget (

\pm

std bands): the low-capacity maps (SRTS-BCE, Margin-K3) degrade gracefully; SMART+BCE rises above the scalar at

250

. (b) win rate against TvA-TS: SRTS-BCE stays high (

81\%

250

) while SMART+BCE falls to parity (

47\%

). (c) cross-backbone

\Delta\mathrm{ECE}_{15}

250

(thick: seed

\to

draw

95\%

CI; thin: seed-cluster interval): vs. SMART+BCE zero is excluded on all three backbones; vs. TvA-TS on DeiT-S and Swin-T, with flagship ViT-B/16 directional (interval crosses zero). Full tables, the Tiny-ImageNet/DeiT-S and Swin-T replications, and the Tiny-ImageNet/ViT-B/16 boundary: App. E .

Figure 3: Routing-rule ablation (

K{=}3

, BCE objective; five-seed ViT-B/16 / CIFAR-100). Conditioning on a genuine reliability signal matters: risk and margin beat confidence-only and random partitions, but are not distinguished from each other (risk lower on

\mathrm{ECE}_{15}

/eAURC, margin on AdaECE; paired CI includes zero); under NLL every rule collapses to one band (App. B ).

Method	C100	Tiny	IN-100	C100-C
TvA-TS ( $K{=}1$ )	$1.65{\pm}0.44$	$1.26{\pm}0.18$	$1.75{\pm}0.15$	$5.72{\pm}1.13$
Margin-K3	$0.97{\pm}0.32$	$0.99{\pm}0.16$	$1.83{\pm}0.30$	$4.70{\pm}0.78$
SMART+BCE	$0.95{\pm}0.29$	$0.97{\pm}0.22$	$1.71{\pm}0.26$	$4.75{\pm}0.79$
LinearRiskTemp	$1.29{\pm}0.48$	$1.35{\pm}0.08$	$1.74{\pm}0.26$	$4.89{\pm}0.76$
PWLinear-3	$1.01{\pm}0.32$	$1.26{\pm}0.11$	$1.86{\pm}0.26$	$4.58{\pm}0.86$
SRTS-BCE (Ours)	$0.96{\pm}0.32$	$1.11{\pm}0.15$	$1.84{\pm}0.22$	$4.63{\pm}0.84$

Table 2: RQ2: matched-parameterisation comparison (

\mathrm{ECE}_{15}

, mean

{\pm}

std; columns are ViT-B/16 except IN-100, which is DeiT-S; C100-C: per-seed means over the 95 corruption cells, deployable protocol). Within each training seed all methods share the same fixed calibration split; all use the top-label BCE objective. Top block: conditioning signal varies. Bottom block: same OOF risk score, objective, and fitting discipline; map family varies (PWLinear-3 exactly matches SRTS-BCE’s fitted router and parameter count). (Seeds: five on C100, three on Tiny and the protocol-frozen IN-100 subset. Full method set, five-seed IN-100, and three-tier CIs: Tabs. 13 , 36 , § C .)

Figure 4: CIFAR-100-C shift under deployable rerouting. (a) Among the plotted methods, SRTS-BCE has the lowest mean

\mathrm{ECE}_{15}

point estimate at every severity (

\pm

std; CIs in text). (b) corrupted examples shift into higher-risk groups; dashed: clean-test high-risk share (

32.7\%

). Legend abbreviations: SMART denotes SMART+BCE; SRTS denotes SRTS-BCE.

Method	Params	Fit (s)	Inference ( $\mu$ s/sample)
TS-NLL	1	0.019	1.19
TvA-TS	1	0.030	0.82
SRTS-BCE $K{=}3$	10	0.137	7.00
SMART+BCE	49	4.256	1.44
SATS-Logit	13,057	5.066	1.55

Table 3: Calibrator cost (CPU, 4 threads; fit on the validation split, inference on

10{,}000

test; 3-seed mean).

NLL objective
Routing rule	NLL	Brier	$\mathrm{ECE}_{15}$	eAURC
SRTS-NLL K=3 risk	$0.3217\pm 0.0274$	$0.1333\pm 0.0070$	$1.7058\pm 0.6483$	$0.0115\pm 0.0016$
K=3 confidence	$0.3243\pm 0.0302$	$0.1340\pm 0.0074$	$1.8149\pm 0.6923$	$0.0131\pm 0.0033$
K=3 random equal-frequency	$0.3241\pm 0.0296$	$0.1342\pm 0.0076$	$1.8181\pm 0.7090$	$0.0134\pm 0.0033$
K=3 shuffled risk-score	$0.3239\pm 0.0297$	$0.1342\pm 0.0076$	$1.7970\pm 0.7138$	$0.0132\pm 0.0034$
BCE objective

Table 4: Objective-conditional routing ablation (full table). Modern-recipe ViT-B/16 / CIFAR-100, mean

\pm

std;

K{=}3

fixed, only the routing rule varies. The BCE block is over five seeds (matching the main paper’s routing-rule ablation figure, Fig. 3, which plots these BCE routing rules); the NLL block is an archived three-seed diagnostic. NLL objective : deployable routing variants collapse to one ECE band. BCE objective : risk routing beats confidence and random (risk

<

confidence

<

shuffled

\approx

random on ECE). SMART+BCE uses the SMART logit-gap architecture ( Guo et al. 2026 ) retrained with top-label BCE. The oracle-risk row routes on the label-informed true-label risk

1{-}p_{y}

(not a binary correct/incorrect partition); it uses labels and is diagnostic only, not an upper bound.

Routing rule	NLL	Brier	$\mathrm{ECE}_{15}$	eAURC
Brier objective
TvA-Brier $\equiv$ K=1 Brier	$0.3276\pm 0.0228$	$0.1332\pm 0.0058$	$1.6830\pm 0.6586$	$0.0137\pm 0.0030$
SRTS-Brier K=3 risk	$0.3233\pm 0.0211$	$0.1326\pm 0.0054$	$\mathbf{1.0518\pm 0.3480}$	$\mathbf{0.0114\pm 0.0014}$
K=3 confidence	$0.3270\pm 0.0234$	$0.1331\pm 0.0057$	$1.1723\pm 0.2711$	$0.0135\pm 0.0027$
K=3 random equal-frequency	$0.3282\pm 0.0229$	$0.1333\pm 0.0059$	$1.6811\pm 0.5510$	$0.0139\pm 0.0031$

Table 5: Objective-agnostic routing check:

K{=}3

group-wise temperature calibration under the top-label Brier objective. Same modern-recipe ViT-B/16 / CIFAR-100 base as Table 4 , mean

\pm

std over seeds 0/1/2. Top-label Brier (

\mathcal{L}=\overline{(q_{i}-b_{i})^{2}}

) is a proper scoring rule distinct from both BCE and NLL. OOF risk routing again outperforms confidence and random routing by

0.12

–

0.63

pp ECE, replicating the ordering from the BCE objective and confirming that the routing benefit is a property of the OOF risk signal rather than an artefact of the BCE loss function. TvA-Brier (

K{=}1

) attains ECE

\approx

random (

1.68

vs.

1.68

), consistent with random routing degenerating to near-scalar behaviour.

Method	NLL	$\mathrm{ECE}_{15}$
SRTS-Mixed $K{=}3$ risk, $\lambda=0.1$	0.3217	1.6617
SRTS-Mixed $K{=}3$ risk, $\lambda=0.3$	0.3218	1.5023
SRTS-Mixed $K{=}3$ risk, $\lambda=1$	0.3222	1.1884
SRTS-Mixed $K{=}3$ risk, $\lambda=3$	0.3231	0.9698
SRTS-Mixed $K{=}3$ risk, $\lambda=10$	0.3241	0.8667

Table 6: Mixed-objective SRTS diagnostic (three-seed mean;

\mathcal{L}=\mathcal{L}_{\mathrm{NLL}}+\lambda\mathcal{L}_{\mathrm{BCE}}

). These are ablations, not deployed-method rows; as

\lambda

grows the mixed fit approaches the pure-BCE configuration (three-seed value

0.83{\pm}0.28

; the five-seed value is

0.96{\pm}0.32

Method (best-val cell)	val NLL	test $\mathrm{ECE}_{15}$	test NLL
Reference: scalar TS-NLL	—	$1.7950\pm 0.6977$	$0.3239\pm 0.0297$
Reference: SRTS-NLL $K{=}3$ (ablation)	—	$1.7058\pm 0.5294$	$0.3217\pm 0.0274$
Reference: SRTS-BCE $K{=}3$ risk (main)	—	$0.8276\pm 0.2791$	$0.3246\pm 0.0206$
Reference: SMART+BCE	—	$0.8573\pm 0.2871$	$0.3245\pm 0.0217$
SATS-Logit (best-val)	$0.2151\pm 0.0074$	$5.7416\pm 0.4385$	$0.6917\pm 0.0726$
SATS-Features (best-val)	$0.2099\pm 0.0042$	$6.4659\pm 0.4906$	$0.8418\pm 0.2136$

Table 7: Adaptive-head fairness sweep as a capacity-control diagnostic on modern-recipe ViT-B/16 / CIFAR-100, three seeds. For each unrestricted adaptive-head family, a

48

-cell grid (head width

\in\{

linear

,16,32,128\}\times

weight decay

\in\{0,10^{-4},10^{-3},10^{-2}\}\times

learning rate

\in\{10^{-3},10^{-4},10^{-5}\}

) is fit with early stopping on val NLL (patience 20). The cell with the lowest validation NLL is selected per (method, seed) — no test-set tuning. Across all nine (method, seed) combinations the val-only selection picks the largest head (width

128

, learning rate

10^{-3}

). The constrained and SMART+BCE rows are references from the objective-aligned comparison, not cells in this NLL sweep. Notes. All

9/9

best-val cells select head width

128

and learning rate

10^{-3}

. Weight decay variation is essentially ignored by the val-NLL criterion because lower-

\ell_{2}

cells reach lower val NLL by over-fitting. SATS-Features and LTS-feature share the same feature-input MLP topology in our implementation and select the same cell, so their best-val rows are identical.

Regime / backbone	SATS-Logit	LTS-feat. / Feat-SATS / UTempH	Range
C100 / ViT-B/16	$6.17{\pm}0.31$	$6.71{\pm}0.44$	$6.2$ – $6.7$
C100 / DeiT-S	$7.02{\pm}1.05$	$8.31{\pm}0.65$	$7.0$ – $8.3$
C100 / Swin-T	$7.24{\pm}0.49$	$8.46{\pm}0.44$	$7.2$ – $8.5$
TinyIN / ViT-B/16	$7.40{\pm}0.35$	$7.05{\pm}0.46$	$7.1$ – $7.4$
TinyIN / DeiT-S	$9.75{\pm}0.16$	$10.41{\pm}0.07$	$9.8$ – $10.4$
TinyIN / Swin-T	$10.06{\pm}0.30$	$11.02{\pm}0.15$	$10.1$ – $11.0$

Table 8: Unrestricted adaptive-head

\mathrm{ECE}_{15}

failure summary (%, mean

\pm

std over three seeds). All four methods are NLL-trained MLP temperature heads with varying input representations (logits or CLS features). Every unrestricted head produces

\mathrm{ECE}_{15}\geq 6\%

across all six backbone

\times

dataset settings, far exceeding both scalar TS (

1

–

3\%

) and the constrained adaptive calibrators SRTS-BCE and SMART+BCE (approximately

1.25\%

or below), motivating the deliberate capacity constraint in SRTS-BCE (

K+(D{+}1)=10

parameters) and SMART+BCE (

49

parameters). Full per-method tables are in § G .

Constrained scalar / grouped baselines
Method	Top-1	ECE 15	NLL	eAURC
TS-NLL	$91.17$	$1.80{\pm}0.57$	$0.324{\pm}0.024$	$0.01324{\pm}0.0028$
TvA-TS ( $K{=}1$ BCE)	$91.17$	$1.54{\pm}0.42$	$0.325{\pm}0.023$	$0.01349{\pm}0.0029$
SRTS-BCE $K{=}3$	$91.17$	$0.83{\pm}0.28$	$0.325{\pm}0.021$	$0.01147{\pm}0.0013$
SMART+BCE	$91.17$	$0.86{\pm}0.29$	$0.325{\pm}0.022$	$0.01187{\pm}0.0014$
BCE-trained unrestricted adaptive heads

Table 9: BCE-trained unrestricted adaptive-head fairness check. SATS-Logit-BCE and Feature-SATS-BCE are trained and selected under the same top-label BCE objective as SRTS-BCE and SMART+BCE, using an 80/20 Cal-Fit/Cal-Select split and a 36-config hyperparameter sweep (hidden

\in\{16,32,128\}

, lr

\in\{10^{-3},10^{-4},10^{-5}\}

, weight-decay

\in\{0,10^{-4},10^{-3},10^{-2}\}

); the best Cal-Select configuration is refit on the full calibration split. Both MLP variants share the same input

\to

hidden

\to

GELU

\to

\to

softplus architecture; they differ only in input dimension (100 logits vs. 768 features). ECE

{}_{15}\geq 8

confirms the failure mode is not specific to the NLL objective.

Method	Top-1	NLL	Brier	ECE 15	AdaECE 15	AUROC	AURC	eAURC
CE / NoTS	53.71	1.7179	0.5975	3.1788	3.2246	0.8089	0.2246	0.0956
CE + TS-NLL	53.71	1.7065	0.5963	2.3578	2.3451	0.8101	0.2241	0.0950
CE + LTS scalar	53.71	1.7063	0.5962	2.1861	2.2012	0.8100	0.2241	0.0950
CE + SATS logits	53.71	2.2788	0.6483	12.4806	12.5604	0.7481	0.2689	0.1399
CE + LTS feature	53.71	2.5166	0.6584	13.5758	13.6473	0.7401	0.2746	0.1455
CE + Feature-SATS 1 1 1 Feature-SATS uses the same feature-input SATS fitting path as the frozen-feature variant in our implementation and therefore reduces to one row.	53.71	2.5185	0.6585	13.5917	13.6543	0.7400	0.2746	0.1455

Table 10: Controlled single-checkpoint diagnostic under NLL fitting (CIFAR-100, ViT-B/16). Exposes the validation-overfit failure of unrestricted sample-adaptive heads under matched checkpoint and protocol; failure-mode evidence, not the five-seed headline performance estimate reported in the main paper’s ID-clean summary table. SRTS-NLL logits/features are NLL-objective ablations. All rows are post-hoc calibrators applied to the same CE base checkpoint and therefore preserve top-1 accuracy. Bold : best per column; underline : second-best.

Figure 5: Recovery case study: NLL-objective SRTS applied to a severely miscalibrated Unrestricted TempHead. Diagnostic figure generated with the SRTS-NLL ablation variant. Single-run point estimates: pre-

\mathrm{ECE}_{15}=29.38\to

post

1.40

; pre-NLL

4.253\to

post

1.281

; pre-HCFP@

0.90

23.11\to

post

0.83

(§ A ). Recovery uses only logit signals — no further training, no feature signals, and no access to the miscalibrated head’s learned

T(x)

. This plot documents an NLL-objective diagnostic only; it is not evidence of a BCE-objective recovery run.

Figure 6: Signal-routing ablation (NLL-objective diagnostic). Logits-only routing is the reliable default; feature-input routing achieves higher val risk-AUROC but regresses on test

\mathrm{ECE}_{15}

(controlled setting:

2.62

vs.

2.14

), consistent with small-val feature-routing overfit. Routing on a Stop-Grad TempHead’s learned

T(x)

does not improve over scalar TS; the signal collapses to near-global scaling. The signal-input choice (logits vs. features) applies equally to SRTS-NLL and SRTS-BCE; the main text’s SRTS-BCE uses the same logits-only routing because of this diagnostic. x-axis bases: CE+Brier (CE+Brier baseline), UTempH (Unrestricted TempHead), SG-TempH (Stop-Grad TempHead).

Base	$K$	ECE 15	NLL	HCFP@0.9	$T_{g}$ range	$T_{\max}/T_{\min}$
CE	1	1.06	1.236	0.97	[1.62, 1.62]	1.00 $\times$
CE	2	1.02	1.235	1.05	[1.60, 1.63]	1.02 $\times$
CE	3	1.03	1.235	1.06	[1.58, 1.67]	1.06 $\times$
CE	5	1.25	1.236	1.27	[1.54, 1.71]	1.11 $\times$
CE	10	0.89	1.237	1.22	[1.46, 1.81]	1.24 $\times$
Unrestricted TempHead	1	1.48	1.282	0.75	[5.61, 5.61]	1.00 $\times$

Table 11:

K

-group ablation for SRTS-NLL (NLL-objective ablation) on three matched single-checkpoint bases (CE, Unrestricted TempHead, Stop-Grad TempHead).

K=1

reduces to global scalar TS-NLL by construction. Numbers are percent except the group-temperature spread

T_{g}

range /

T_{\max}/T_{\min}

. The per-base

\mathrm{ECE}_{15}

band across

K\in\{1,2,3,5,10\}

is at most

\approx 0.6

percentage points, while the per-group temperature spread

T_{\max}/T_{\min}

grows monotonically with

K

; we read this as evidence that larger

K

expands the parameter budget without delivering matching calibration improvement on this protocol.

Base	Variant	ECE 15	NLL	HCFP@0.9	AUROC ${}_{\text{corr}}$
CE	raw	14.27 [13.50,15.05]	1.441 [1.393,1.487]	5.67 [5.23,6.14]	0.850 [0.843,0.858]
CE	SRTS-NLL (logits)	1.47 [0.95,2.05]	1.235 [1.202,1.267]	1.05 [0.85,1.27]	0.854 [0.846,0.861]
CE	SRTS-NLL (features)	3.82 [3.12,4.52]	1.244 [1.213,1.277]	1.89 [1.62,2.17]	0.853 [0.845,0.861]
CE	SRTS-NLL (temphead)	3.65 [2.93,4.35]	1.243 [1.212,1.276]	1.86 [1.60,2.14]	0.854 [0.846,0.861]
Unrestricted TempHead	raw	29.39 [28.48,30.29]	4.250 [4.095,4.413]	23.11 [22.31,23.97]	0.825 [0.817,0.832]
Unrestricted TempHead	SRTS-NLL (logits)	1.71 [1.12,2.38]	1.281 [1.248,1.315]	0.83 [0.66,1.02]	0.845 [0.837,0.852]

Table 12: SRTS-NLL single-checkpoint bootstrap 95% CIs on CIFAR-100 test (NLL-objective ablation;

B{=}1000

resamples; the fitted routing is held fixed). Numbers are percent.

Panel of Figure 7 — Figure 7: ID clean results over three modern-recipe ViT seeds (NLL-objective rows). Bars show mean $\pm$ standard deviation over seeds 0/1/2 (top-1 unchanged within each reported post-hoc seed; 3-seed mean $91.17\pm 0.46$ ); the SATS-Logit AURC/eAURC bars show the 3-seed mean without whiskers (per-seed spread not archived). Under the NLL objective, SRTS-NLL $K{=}1$ recovers TS-NLL and $K{=}3$ adds only modest gain ( $-0.09$ pp). Under the BCE objective (see the main text’s five-seed ID clean summary table and Table 4 ; this NLL-objective figure is an archived three-seed diagnostic), SRTS-BCE $K{=}3$ risk attains $\mathrm{ECE}_{15}=0.96$ (five seeds), beating TvA-TS ( $K{=}1$ BCE) by $0.70$ pp. SATS-Logit remains substantially worse across seeds, illustrating the validation-overfit failure mode of unrestricted adaptive temperature heads.

Method	C100/ViT	Tiny/ViT	IN-100/DeiT	C100-C/ViT
TvA-TS ( $K{=}1$ )	$1.65{\pm}0.44$	$1.26{\pm}0.18$	$1.75{\pm}0.15$	$5.72{\pm}1.13$
QaTS-NLL	$1.90{\pm}0.61$	$1.33{\pm}0.24$	$2.56{\pm}0.20$	$6.51{\pm}0.57$
QaTS-BCE	$1.35{\pm}0.31$	$1.11{\pm}0.06$	$2.03{\pm}0.20$	$5.18{\pm}0.68$
Margin-K3+BCE	$0.97{\pm}0.32$	$0.99{\pm}0.16$	$1.83{\pm}0.30$	$4.70{\pm}0.78$
LinearRiskTemp+BCE	$1.29{\pm}0.48$	$1.35{\pm}0.08$	$1.74{\pm}0.26$	$4.89{\pm}0.76$
PWLinear-3+BCE	$1.01{\pm}0.32$	$1.26{\pm}0.11$	$1.86{\pm}0.26$	$4.58{\pm}0.86$

Table 13: Matched-capacity comparison, full method set (

\mathrm{ECE}_{15}

, mean

{\pm}

std; C100/ViT five seeds, others three; C100-C as in the main RQ2 table). LinearRiskTemp and SplineRiskTemp use the same router and fitting discipline; the spline’s natural cubic basis with the three frozen knots has three coefficients and converges to a near-linear solution on every setting.

vs. comparator	point	cell	corr.-type	hierarchical
TvA-TS ( $K{=}1$ )	$-1.09$	$[-1.28,-0.88]$	$[-1.64,-0.49]$	$[-2.02,-0.36]$
QaTS-NLL	$-1.88$	$[-2.11,-1.64]$	$[-2.56,-1.15]$	$[-2.90,-0.88]$
QaTS-BCE	$-0.55$	$[-0.65,-0.44]$	$[-0.83,-0.25]$	$[-1.04,-0.13]$
Margin-K3+BCE	$-0.07$	$[-0.09,-0.04]$	$[-0.11,-0.03]$	$[-0.21,+0.02]$
LinearRiskTemp+BCE	$-0.26$	$[-0.32,-0.20]$	$[-0.36,-0.15]$	$[-0.43,-0.06]$
PWLinear-3+BCE	$+0.05$	$[+0.03,+0.08]$	$[+0.01,+0.10]$	$[-0.01,+0.14]$

Table 14: CIFAR-100-C three-tier bootstraps for

\Delta=

SRTS-BCE

-

comparator (

\mathrm{ECE}_{15}

; negative favours SRTS-BCE;

B{=}20{,}000

). Tiers: paired cell-level; corruption-type cluster; full seed

\times

corruption

\times

severity hierarchical (primary).

			C100-C	SRTS $-$ HTS at budget $250$
Variant	Clean ECE 15	C100-C ECE 15	SRTS $-$ HTS (hier.)	ViT-B/16	DeiT-S	Swin-T
HTS-NLL	$2.07{\pm}0.51$	$6.96{\pm}0.30$	$[-3.70,-1.12]$	$[-0.95,-0.40]$	$[-1.58,-1.24]$	$[-1.74,-1.07]$
HTS-BCE	$1.63{\pm}0.25$	$5.41{\pm}0.54$	$[-1.47,-0.22]$	$[-0.60,-0.13]$	$[-0.68,-0.20]$	$[-0.86,-0.38]$

Table 15: HTS comparison (equation-faithful reimplementation of the entropy-conditioned two-parameter map of Balanya et al. (2024) , fitted with its original NLL objective and with this paper’s top-label BCE). Clean: CIFAR-100/ViT-B/16

\mathrm{ECE}_{15}

, five seeds. C100-C: deployable protocol, three seeds, with the three-tier hierarchical CI for SRTS-BCE

-

HTS (negative favours SRTS-BCE). Budget

250

: seed-level hierarchical CIs of SRTS-BCE

-

HTS on the identical resampling draws. Bold : interval excludes zero.

Method	$\Phi_{\text{TopBCE}}$	$\Phi_{\text{NLL}}$	$\Phi_{\text{Brier}}$
TS-NLL	$0.0577\pm 0.0057$	$0.0828\pm 0.0078$	$0.0100\pm 0.0026$
TvA-TS	$0.0587\pm 0.0053$	$0.0814\pm 0.0069$	$0.0106\pm 0.0026$
SRTS-NLL	$0.0613\pm 0.0027$	$\mathbf{0.0850\pm 0.0056}$	$0.0108\pm 0.0020$
SMART+BCE	$0.0628\pm 0.0029$	$0.0822\pm 0.0063$	$0.0113\pm 0.0020$
PWLinear-3	$\mathbf{0.0643\pm 0.0023}$	$0.0822\pm 0.0049$	$\mathbf{0.0116\pm 0.0018}$
SRTS-BCE	$0.0637\pm 0.0020$	$0.0821\pm 0.0049$	$\mathbf{0.0116\pm 0.0018}$

Table 16: Proper-score improvements over the uncalibrated model (

\Phi_{\ell}=\ell(\text{NoTS})-\ell(\text{calibrated})

; higher is better; ViT-B/16 / CIFAR-100, per-seed differences, mean

\pm

std over three seeds). All calibrators recover an essentially identical NLL and Brier improvement; the conditioned maps add only

{\approx}0.005

TopBCE over the scalars and are mutually indistinguishable.

Figure 8: Motivating diagnostics for risk-conditioned scaling (clean CIFAR-100 / ViT-B/16; five seeds). (a) At matched scalar-TvA confidence, empirical accuracy differs across OOF-risk groups, exposing residual heterogeneity hidden by a global confidence value. (b) Scalar TvA-TS leaves the largest residual in the high-risk group, while the fitted group temperatures differ across risk levels. (c) The five-seed

K

-ablation illustrates the capacity trade-off:

K{=}1

underfits,

K{=}3

captures most of the gain, and larger

K

provides no stable additional benefit. These diagnostics motivate risk conditioning but do not establish a unique advantage for the six-signal router or the piecewise-constant map.

Routing signal	ECE 15	NLL	Brier	eAURC
Confidence-only	1.29 $\pm$ 0.43	0.3323 $\pm$ 0.0294	0.1357 $\pm$ 0.0077	0.0139 $\pm$ 0.0030
Logit-margin-only	0.97 $\pm$ 0.31	0.3302 $\pm$ 0.0264	0.1349 $\pm$ 0.0071	0.0120 $\pm$ 0.0013
Prob-margin-only	1.17 $\pm$ 0.44	0.3321 $\pm$ 0.0287	0.1355 $\pm$ 0.0076	0.0139 $\pm$ 0.0030
Confidence + logit-margin	0.98 $\pm$ 0.31	0.3305 $\pm$ 0.0269	0.1350 $\pm$ 0.0072	0.0122 $\pm$ 0.0016
Full OOF risk (six signals)	0.96 $\pm$ 0.32	0.3297 $\pm$ 0.0257	0.1348 $\pm$ 0.0070	0.0113 $\pm$ 0.0011
Random equal-frequency	1.68 $\pm$ 0.43	0.3322 $\pm$ 0.0297	0.1362 $\pm$ 0.0081	0.0142 $\pm$ 0.0032

Table 17: Routing-signal ablation on clean CIFAR-100 / ViT-B/16 (top-label BCE,

K{=}3

, five-seed mean

\pm

std). All rows share the identical

K{=}3

group-wise BCE temperature fit and differ only in the routing signal. Lower is better for all metrics.

Figure 9: OOF risk vs. confidence (clean CIFAR-100 / ViT-B/16, seed 0). (a) The OOF risk score is tightly coupled to confidence; (b) it is a monotone reliability ordering with only a modest incremental edge over confidence at ranking errors (AUROC

0.900

vs.

0.889

). We do not claim a strong risk-beyond-confidence separation from this view; the incremental value of the full six-signal router is better quantified by the routing-signal ablation (Table 17 ).

	Clean CIFAR-100		CIFAR-100-C
Grouping rule ( $K{=}3$ )	ECE 15	eAURC	ECE 15	eAURC
Margin tertiles	0.837 $\pm$ 0.247	0.0117 $\pm$ 0.0013	4.695 $\pm$ 0.778	0.0438 $\pm$ 0.0042
SRTS risk tertiles	0.817 $\pm$ 0.285	0.0114 $\pm$ 0.0013	4.626 $\pm$ 0.839	0.0431 $\pm$ 0.0045
Shuffled risk	1.538 $\pm$ 0.471	0.0136 $\pm$ 0.0029	5.713 $\pm$ 1.173	0.0458 $\pm$ 0.0054
Oracle correctness	0.249 $\pm$ 0.006	0.0000 $\pm$ 0.0000	0.514 $\pm$ 0.007	0.0000 $\pm$ 0.0000

Table 18: Margin is a strong routing signal: fixed-budget routing ablation (ViT-B/16, 3 seeds). Clean held-out and CIFAR-100-C corruption-test

\mathrm{ECE}_{15}

/eAURC (mean

\pm

std over seeds); lower is better. Objective, validation split, and

K{=}3

budget are held fixed and only the grouping rule varies (protocol, statistical test, and within-table-comparison scope in the text). The oracle-correctness row routes by the true correct/incorrect label (label-leaking, non-deployable, diagnostic reference only); its near-zero eAURC is an artifact of using correctness labels for routing.

Margin tertile	Calibrator	ECE 15	NLL	AURC	eAURC
low	TvA-TS	8.668	1.365	0.2105	0.1242
	Margin-K3	6.327	1.355	0.2120	0.1258
	SRTS-K3	6.297	1.355	0.2103	0.1241
	SMART+BCE	6.433	1.355	0.2122	0.1260
mid	TvA-TS	3.135	0.370	0.0482	0.0450
	Margin-K3	3.424	0.379	0.0473	0.0441

Table 19: Matched-margin CIFAR-100-C risk (deployable rerouting, 3 seeds). These are corruption-test metrics only, computed per corruption/severity cell within a logit-margin tertile and averaged over the full 19 corruptions

\times

5 severities for each seed, then over three seeds. Margin-tertile thresholds are fixed on clean validation. All calibrators, including the diagnostic Margin-K3 temperatures and SRTS-K3 temperatures/router, are fit or refit on clean validation only and applied unchanged to CIFAR-100-C; SRTS uses deployable rerouting from each corrupted sample’s own logits. No oracle row or corrupted labels are used. In the high-margin region margin routing is competitive on

\mathrm{ECE}_{15}

; in the low-margin region the full six-signal router has slightly lower point-estimate ECE 15 and eAURC than margin routing. These within-tertile diagnostics are descriptive and do not establish router superiority; they are not replacements for the main-paper aggregate metrics. Lower is better.

Figure 10: High-margin CIFAR-100-C aggregate diagnostic. Within the high-margin tertile, margin routing matches SRTS on

\mathrm{ECE}_{15}

, while SRTS retains the lower eAURC. This 3-seed corruption-test diagnostic averages the full

19\times 5

CIFAR-100-C protocol inside the high-margin subset: margin tertiles are fixed on clean validation, temperatures/routers are fit on clean validation only, and SRTS uses deployable rerouting from each corrupted sample’s own logits. No corrupted labels or oracle routing are used; the panel is intended for within-tertile mechanism comparison, not as a replacement for the aggregate main-paper metrics.

Routing target	ECE 15	AdaECE 15	smECE	NLL	Brier	eAURC
TvA-TS / TS-BCE ( $K{=}1$ )	$1.537\pm 0.417$	$1.596\pm 0.673$	$1.562\pm 0.417$	$0.3253$	$0.1335$	$0.0135$
OOF wrongness-risk (SRTS-BCE)	$\mathbf{0.814\pm 0.264}$	$0.957\pm 0.134$	$1.073\pm 0.151$	$0.3247$	$0.1326$	$0.0115$
Margin-tertile ( $K{=}3$ )	$0.836\pm 0.256$	$\mathbf{0.823\pm 0.216}$	$\mathbf{1.037\pm 0.179}$	$0.3246$	$0.1326$	$0.0117$
Signed-residual ( $K{=}3$ )	$1.093\pm 0.148$	$0.968\pm 0.306$	$1.140\pm 0.145$	$\mathbf{0.3201}$	$\mathbf{0.1322}$	$\mathbf{0.0099}$
Absolute-residual ( $K{=}3$ )	$0.862\pm 0.276$	$0.905\pm 0.249$	$1.072\pm 0.195$	$0.3241$	$0.1327$	$0.0112$

Table 20: Alternative low-capacity routing targets (clean CIFAR-100 / ViT-B/16, 3 seeds;

K{=}3

, top-label BCE). All routed rows fix the objective, the

K{=}3

budget, and the clean-validation split, and vary only the routing target; TvA-TS (

K{=}1

) is the no-routing reference. The OOF wrongness-risk router (the deployed SRTS-BCE design) is best on the primary

\mathrm{ECE}_{15}

metric; margin-tertile routing is highly competitive and better on AdaECE 15 /smECE; signed-residual routing improves NLL/Brier/eAURC but worsens

\mathrm{ECE}_{15}

; absolute-residual routing is close but does not improve

\mathrm{ECE}_{15}

over wrongness-risk. This is a metric trade-off under the present small-validation ID clean setting, not a universal ranking. Bold marks the best (lowest) value per column. Lower is better.

Figure 11: Per-group temperatures learned by SRTS variants (NLL-objective diagnostic). Under NLL fitting, SRTS-NLL group temperatures cluster tightly around the global-TS value. Under BCE fitting (Table 4 , § B ), the per-group temperature spread is substantially larger (range

0.6

–

0.9

on seed 0 vs. TvA-TS single

T\approx 0.76

), which is consistent with the BCE-objective

K{=}3

risk router achieving the

0.73

pp gain over TvA-TS.

Figure 12: Risk-score density by variant: correct vs. wrong predictions. Logit-derived signals separate correct from wrong predictions on the validation out-of-fold split. Under NLL fitting, stronger separation does not translate to lower test-side

\mathrm{ECE}_{15}

(§ B , NLL block); under BCE fitting, the same separation is exploited to achieve the

0.73

pp improvement over TvA-TS (§ B , BCE block).

Figure 13: Bootstrap

95\%

CIs across signal-routing variants (NLL-objective diagnostic;

B{=}1000

, values as in Table 12 ). On NLL the three signal variants are indistinguishable (near-complete CI overlap), whereas on

\mathrm{ECE}_{15}

and HCFP@

0.90

the logit-input variant is clearly preferable (CIs disjoint from the feature/temphead variants on the CE and Stop-Grad bases); logit-input routing is the chosen default for the main paper’s SRTS-BCE method on these overfitting-safety grounds.

Method	$10\%$	$25\%$	$50\%$	$100\%$
TvA-TS	1.81	1.65	1.56	1.54
SRTS-BCE $K{=}3$ risk	1.45	1.18	0.93	0.81
SMART+BCE	1.95	1.41	1.00	0.83

Table 21: BCE-objective validation-size sensitivity on ViT-B/16 / CIFAR-100. Mean

\mathrm{ECE}_{15}

over the three backbone seeds and five calibration-subsamples per seed. SRTS-BCE has the lowest absolute ECE at every tested fraction (

10

–

100\%

), including at

10\%

(

1.45

vs. TvA-TS

1.81

, SMART+BCE

1.95

). However, the absolute ECE gap relative to TvA-TS shrinks from

0.73

pp at

100\%

0.36

pp at

10\%

, because SRTS-BCE’s ECE worsens by

0.64

pp as budget falls from

100\%

10\%

while TvA-TS worsens by only

0.27

pp. TvA-TS is the most rate-stable single-parameter fallback at extreme budgets.

Figure 14: Validation-budget trade-off on CIFAR-100 / ViT-B/16 (companion plot to Table 21 ). SRTS-BCE has the lowest ECE at the smallest validation budget and converges with SMART+BCE at the full budget, while unrestricted adaptive heads remain poorly calibrated across budgets. This supports the fixed low-capacity routing design under small validation budgets.

Budget	Method	Mean	SD	Paired CI vs TvA	Win
250	TvA-TS	$1.96$	$0.72$	—	—
	Margin-K3	$1.55$	$0.64$	$[-0.53,-0.28]$	77%
	SMART+BCE	$2.03$	$0.72$	$[-0.06,+0.23]$	47%
	SRTS-BCE	$1.60$	$0.70$	$[-0.48,-0.23]$	81%
625	TvA-TS	$1.81$	$0.62$	—	—
	Margin-K3	$1.24$	$0.53$	$[-0.66,-0.47]$	86%

Table 22: Calibration-budget resampling on CIFAR-100 / ViT-B/16 (

\mathrm{ECE}_{15}

%). Each calibrator is fit on

20

stratified calibration subsamples per training seed (five seeds; the

2500

row is the full split, no resampling variance). SD : standard deviation over calibration resamples. CI : draw-level bootstrap

95\%

interval of the paired mean difference vs. TvA-TS (negative favours the method); descriptive only — seed-level inference is in Table 23 . Win : fraction of resamples with lower

\mathrm{ECE}_{15}

than TvA-TS.

	Budget $250$		Budget $2500$
Cell	SRTS $-$ TvA	SRTS $-$ SMART	SRTS $-$ TvA	SRTS $-$ SMART
C100 / ViT-B/16 (5 seeds)	$[-0.72,+0.01]$	$[-0.59,-0.28]$	$[-0.84,-0.55]$	$[-0.03,+0.06]$
C100 / DeiT-S (3 seeds)	$[-1.94,-1.17]$	$[-1.34,-0.30]$	$[-2.88,-1.79]$	$[-0.29,-0.15]$
C100 / Swin-T (3 seeds)	$[-1.30,-0.94]$	$[-0.59,-0.25]$	$[-2.11,-1.54]$	$[-0.39,+0.14]$
Aggregate (backbone $\to$ seed $\to$ draw; secondary)	$[-1.57,-0.39]$	$[-0.87,-0.35]$	$[-2.42,-0.72]$	$[-0.24,+0.02]$

Table 23: Hierarchical (training-seed

\to

calibration-draw)

95\%

CIs for the budget experiment (

\Delta\,\mathrm{ECE}_{15}

, negative favours SRTS-BCE;

B{=}20{,}000

). Draws within a training seed share the model and test set, so seeds are the top-level resampling unit within each backbone; the aggregate row adds a backbone level (backbone

\to

seed

\to

draw) and is a secondary summary — the per-backbone intervals are the primary replication evidence. Bold : interval excludes zero.

	AdaECE 15		smECE
Cell	SRTS $-$ TvA	SRTS $-$ SMART	SRTS $-$ TvA	SRTS $-$ SMART
ViT-B/16	$[-0.71,+0.07]$	$[-0.54,-0.22]$	$[-0.31,+0.27]$	$[-0.70,-0.34]$
DeiT-S	$[-2.11,-1.30]$	$[-1.23,-0.27]$	$[-1.20,-0.73]$	$[-1.39,-0.42]$
Swin-T	$[-1.38,-0.90]$	$[-0.57,-0.19]$	$[-0.92,-0.43]$	$[-0.73,-0.32]$

Table 24: Metric sensitivity of the

250

-sample result (seed-level hierarchical

95\%

CIs of

\Delta

; negative favours SRTS-BCE; same draws as the

\mathrm{ECE}_{15}

analysis, official smECE estimator). Bold : interval excludes zero.

	ViT-B/16		DeiT-S		Swin-T
Map	$\Delta$ CI	win	$\Delta$ CI	win	$\Delta$ CI	win
PWLinear-3	$[-0.19,+0.06]$	$73\%$	$[-0.56,+0.03]$	$93\%$	$[-0.16,+0.07]$	$98\%$
LinearRiskTemp	$[-0.48,-0.12]$	$57\%$	$[-1.26,-0.48]$	$83\%$	$[-0.79,-0.52]$	$90\%$
SplineRiskTemp	$[-0.37,-0.18]$	$57\%$	$[-0.83,-0.11]$	$88\%$	$[-0.43,-0.06]$	$92\%$

Table 25: Small-budget matched continuous-map test (

n{=}250

; the exact calibration draws, training seeds, router pipeline, optimizer, knots, initialisation, and bounds of the budget-resampling protocol; frozen before execution). Per backbone: seed

\to

draw hierarchical

95\%

CI of

\Delta\mathrm{ECE}_{15}=

SRTS-BCE

-

map (negative favours SRTS-BCE; bold: CI excludes zero) and the map’s win rate against TvA-TS. SRTS-BCE vs. the matched three-anchor PWLinear-3 is statistically unresolved on all three backbones; the linear and spline parameterizations of the same risk score are significantly worse.

Backbone	$n$	TvA-TS	SMART+BCE	SRTS-BCE	SRTS $-$ TvA CI	SRTS $-$ SMART CI	win
DeiT-S	$250$	$3.08$	$2.51$	$2.01$	$[-1.35,-0.80]$	$[-0.71,-0.28]$	$95\%$
DeiT-S	$625$	$3.05$	$1.74$	$1.53$	$[-1.92,-1.22]$	$[-0.36,-0.07]$	$100\%$
DeiT-S	$1250$	$3.06$	$1.48$	$1.28$	$[-2.11,-1.55]$	$[-0.30,-0.10]$	$100\%$
DeiT-S	$2500$	$3.06$	$1.16$	$1.12$	$[-2.31,-1.73]$	$[-0.14,+0.05]$	$100\%$
Swin-T	$250$	$2.85$	$2.95$	$2.03$	$[-1.02,-0.60]$	$[-1.27,-0.58]$	$92\%$
Swin-T	$625$	$2.71$	$1.73$	$1.56$	$[-1.35,-0.87]$	$[-0.27,-0.07]$	$98\%$

Table 26: Protocol-frozen cross-dataset small-budget follow-up (Tiny-ImageNet) (200 classes; three training seeds

\times

20 draws per budget). Mean test

\mathrm{ECE}_{15}

(%) over seeds

\times

draws per method, hierarchical seed

\to

draw

95\%

CIs of

\Delta\mathrm{ECE}_{15}

(negative favours SRTS-BCE; bold: CI excludes zero), and SRTS-BCE’s draw-paired win rate against TvA-TS. Cells, budgets, draws, methods, metrics, inference, and outcome interpretation were frozen before execution; no method, metric, optimizer, or draw selection changed after inspection. On Swin-T the higher-capacity SMART+BCE is significantly better at the largest evaluated

2{,}500

-sample budget and the ordering reverses as the budget shrinks. Tiny-ImageNet/ViT-B/16 remains the boundary result reported in the budget-resampling section.

Backbone	Comparison	$\Delta$ AdaECE 15	$\Delta$ smECE
DeiT-S	SRTS $-$ TvA-TS	$[-1.45,-0.82]$	$[-0.92,-0.44]$
DeiT-S	SRTS $-$ SMART	$[-0.65,-0.25]$	$[-0.84,-0.35]$
Swin-T	SRTS $-$ TvA-TS	$[-1.28,-0.70]$	$[-0.67,-0.21]$
Swin-T	SRTS $-$ SMART	$[-1.19,-0.52]$	$[-1.45,-0.69]$

Table 27: Metric sensitivity of the Tiny-ImageNet replication (

n{=}250

; the exact Table 26 predictions, seeds, and draws; protocol frozen before execution, run once). Seed

\to

draw hierarchical

95\%

CIs of

\Delta

AdaECE 15 and

\Delta

smECE (official estimator); SMART denotes SMART+BCE; negative favours SRTS-BCE; bold: CI excludes zero. All four comparisons are supported under both alternative estimators.

Diagnostic	Full budget	$n=250$ ( $220$ fits)
Group temps $T_{1..3}$ , med. [IQR]	$.63\,[.63,.65]$ / $.73\,[.73,.74]$ / $.84\,[.82,.87]$	$.17\,[.13,.73]$ / $.76\,[.69,.80]$ / $.89\,[.83,.94]$
Router thresholds (mean $\pm$ std)	$0.008{\pm}0.001$ / $0.042{\pm}0.007$	—
Bound saturation	$0\%$	$0.0\%$
Scalar fallback	$0\%$	$0.0\%$
Risk ordering monotone	$100\%$	$93\%$
Temperature ordering monotone	$100\%$	$64\%$

Table 28: Auditability diagnostics for SRTS-BCE. Full budget: CIFAR-100/ViT-B/16, five seeds.

n{=}250

: all

220

calibration draws across the three CIFAR-100 backbones. Saturation: fitted group temperatures at the

[0.05,20]

bounds. Fallback: groups below the

50

-sample floor (reverting to the global scalar). Risk-monotone: fits whose empirical group error rates increase with the routed risk index.

NLL objective (ablation)
Method	Top-1	NLL	Brier	$\mathrm{ECE}_{15}$	AURC	eAURC
CE / NoTS	$78.09{\pm}0.76$	$0.9450{\pm}0.0494$	$0.3191{\pm}0.0122$	$9.1334{\pm}0.5508$	$0.0869{\pm}0.0100$	$0.0487{\pm}0.0061$
TS-NLL	$78.09{\pm}0.76$	$0.9094{\pm}0.0642$	$0.3190{\pm}0.0153$	$6.7545{\pm}0.9060$	$0.0836{\pm}0.0094$	$0.0454{\pm}0.0055$
SRTS-NLL $K{=}3$ (ablation)	$78.09{\pm}0.76$	$0.9068{\pm}0.0613$	$0.3178{\pm}0.0143$	$7.0772{\pm}0.9234$	$0.0766{\pm}0.0066$	$0.0384{\pm}0.0025$
BCE objective
TvA-TS $\equiv$ TS-BCE $K{=}1$	$78.09{\pm}0.76$	$0.9004{\pm}0.0644$	$0.3163{\pm}0.0156$	$5.7158{\pm}1.1275$	$0.0840{\pm}0.0094$	$0.0457{\pm}0.0055$

Table 29: CIFAR-100-C corruption-shift calibration under deployable routing (3-seed mean

\pm

standard deviation over seeds 0/1/2; each per-seed value is the mean over 95 corruption/severity cells). Calibrators are fit only on clean CIFAR-100 validation data and applied unchanged under shift. For SRTS-BCE, each corrupted sample is routed using its own corrupted logits through the clean-validation-fitted risk router; no clean-test routing group is reused. SRTS-NLL is included as an NLL-objective ablation. Among the headline calibrators in this table, SRTS-BCE has the lowest ECE under the deployable BCE protocol and retains a small eAURC advantage over SMART+BCE (the matched PWLinear-3 diagnostic attains a slightly lower point estimate, § C ); the SRTS–SMART ECE margin is small: cell-level and corruption-type bootstraps exclude zero, while the full seed

\times

corruption

\times

severity hierarchical bootstrap (§ J ) marginally includes zero; we treat the gain as directional rather than two-sided significant.

Figure 15: Corruption risk-flow diagnostic (clean

\to

severity-5). Flow of samples between OOF risk tertiles from clean CIFAR-100 to severity-5 corruption (seed 0, averaged over 19 corruptions): the high-risk share grows from

{\approx}32\%

(clean) to

{\approx}62\%

(severity 5). This diagram uses the clean/corrupted image pairing to trace per-sample movement and is therefore a mechanism diagnostic only; it is not used for any deployable evaluation , which reroutes each corrupted input from its own logits without any clean pairing.

Figure 16: CIFAR-100-C 3-seed aggregate calibration metrics (deployable BCE comparison). Mean

\pm

variability over seed

\times

severity aggregates of the deployable corruption evaluation (each corrupted sample rerouted from its own logits). The panel compares the methods of the final deployable protocol: TvA-TS (

K{=}1

BCE), SMART+BCE, and SRTS-BCE

K{=}3

alongside NoTS/TS-NLL references.

Figure 17: Per-corruption

\Delta\mathrm{ECE}_{15}

of SRTS-BCE

K{=}3

minus TvA-TS on CIFAR-100-C. Negative values indicate that deployable risk routing improves over scalar BCE on that corruption; positive values indicate a local regression. This view exposes the heterogeneous corruption profile that is averaged in Table 29 .

Figure 18: CIFAR-100-C

\mathrm{ECE}_{15}

by severity (deployable BCE comparison). SRTS-BCE and SMART+BCE track below TvA-TS across severity, while TS-NLL degrades fastest at high severity. These are the same clean-validation-fitted calibrators applied unchanged to corrupted inputs, with SRTS rerouting each corrupted sample from its own logits.

Clean CIFAR-100
Method	HCFP@.90	HCFP@.95	Risk@80	Risk@90	Risk@95
TvA-TS	$22.94$	$12.00$	$2.38$	$4.59$	$6.47$
SMART+BCE	$15.41$	$9.56$	$2.40$	$4.60$	$6.49$
SRTS-BCE	$15.37$	$7.95$	$2.42$	$4.58$	$6.45$
CIFAR-100-C (rerouting)
TvA-TS	$15.43$	$7.97$	$12.06$	$16.65$	$19.24$

Table 30: Deployment-facing selective metrics (%; five-seed means on both clean CIFAR-100 and CIFAR-100-C), from cached calibrated probabilities. HCFP.

t

: fraction of wrong predictions with confidence

{\geq}t

(confidently wrong). R@

c

: selective error at coverage

c

(error among the most-confident

c

fraction). Lower is better throughout.

Clean CIFAR-100 (five seeds)
	$t=0.80$			$t=0.90$			$t=0.95$
Method	Cov	Risk	Gap	Cov	Risk	Gap	Cov	Risk	Gap
NoTS	$72.3$	$1.97$	$-8.86{\pm}0.47$	$32.5$	$1.09$	$-6.15{\pm}0.41$	$5.5$	$2.02$	$-1.94{\pm}1.76$
TvA-TS	$87.6$	$3.87$	$+0.35{\pm}0.57$	$81.0$	$2.55$	$-0.09{\pm}0.42$	$70.8$	$1.53$	$-0.50{\pm}0.18$
HTS-BCE	$87.2$	$3.74$	$+0.31{\pm}0.53$	$80.4$	$2.48$	$-0.03{\pm}0.55$	$70.3$	$1.57$	$-0.30{\pm}0.51$
SMART+BCE	$84.4$	$3.14$	$+0.19{\pm}0.59$	$76.0$	$1.83$	$+0.17{\pm}0.48$	$68.1$	$1.27$	$+0.24{\pm}0.37$

Table 31: Confidence-threshold deployment at pre-specified thresholds

t\in\{0.80,0.90,0.95\}

(fixed before inspecting results; existing predictions only). Cov : retained coverage (%). Risk : error among retained (%). Gap : mean retained confidence

-

retained accuracy (pp; positive

=

overconfident),

\pm

std over seeds (five clean, three CIFAR-100-C).

Clean CIFAR-100 (five seeds)
		@cov $(t{=}0.80)$		@cov $(t{=}0.90)$		@cov $(t{=}0.95)$
Method	AURC	Risk	Gap	Risk	Gap	Risk	Gap
NoTS	$20.73$	$3.98$	$-9.66$	$2.74$	$-9.41$	$1.85$	$-8.83$
TvA-TS	$18.38$	$3.87$	$+0.35$	$2.55$	$-0.09$	$1.53$	$-0.50$
HTS-BCE	$18.80$	$3.87$	$+0.35$	$2.57$	$-0.00$	$1.60$	$-0.30$
SMART+BCE	$16.49$	$3.89$	$+0.18$	$2.54$	$+0.20$	$1.44$	$+0.21$

Table 32: Matched-coverage control for the threshold comparison. Because different calibrators retain different fractions at a fixed confidence threshold, every method is evaluated at the same reference coverages — TvA-TS’s coverage at the pre-specified thresholds

\{0.80,0.90,0.95\}

(no operating point chosen after inspecting results; top-

k

retained set with

k=\operatorname{round}(cN)

, the nearest attainable coverage). AURC : area under the full risk–coverage curve (

\times 10^{3}

, lower is better). Risk / Gap : retained error (%) and nominal-vs-realized confidence gap (pp) at the matched coverage.

Figure 19: Per-corruption routing gain vs corruption severity (CIFAR-100-C, three seeds). Each point is one of the

19

corruptions:

x

is its NoTS

\mathrm{ECE}_{15}

(how badly the uncalibrated model is miscalibrated under that corruption),

y

is the SRTS-BCE routing gain over scalar TvA-TS in

\mathrm{ECE}_{15}

points. The clean-fitted router helps on milder corruptions and degrades on the most severe ones (Spearman

\rho=-0.68

p=0.001

; Theil–Sen trend). Points above the dashed line favour routing.

Method	ResNet-50	RegNetY-1.6GF
NoTS	$14.39{\pm}0.75$	$15.24{\pm}0.82$
TS-NLL	$19.95{\pm}0.68$	$12.77{\pm}0.62$
TvA-TS	$18.16{\pm}1.10$	$11.60{\pm}0.84$
SMART+BCE	$14.02{\pm}1.16$	$\mathbf{9.08{\pm}0.86}$
SRTS-BCE $K{=}3$	$\mathbf{13.72{\pm}0.79}$	$9.44{\pm}0.78$
SRTS $-$ SMART $\Delta$ (cell-level CI)	$-0.31$ $[-0.37,-0.25]$	$+0.37$ $[+0.32,+0.41]$

Table 33: ConvNet CIFAR-100-C deployable rerouting (

\mathrm{ECE}_{15}

, %). Values are averaged over 95 corruption/severity cells per seed and then reported as mean

\pm

std over three seeds; lower is better. The SRTS

-

SMART interval is a paired cell-level bootstrap conditional on the three trained seeds, not a seed-hierarchical interval.

Method	top-1 (%)	NLL	ECE 15 (%)	AdaECE 15 (%)	AURC
NoTS	65.69 $\pm$ 12.15	1.504 $\pm$ 0.717	2.60 $\pm$ 1.79	2.59 $\pm$ 1.79	0.1356 $\pm$ 0.0858
TS-NLL	65.69 $\pm$ 12.15	1.503 $\pm$ 0.718	2.96 $\pm$ 1.89	2.94 $\pm$ 1.90	0.1355 $\pm$ 0.0858
SRTS-NLL K=1 (ablation)	65.69 $\pm$ 12.15	1.503 $\pm$ 0.718	2.96 $\pm$ 1.89	2.94 $\pm$ 1.90	0.1355 $\pm$ 0.0858
SRTS-NLL K=3 (ablation)	65.69 $\pm$ 12.15	1.503 $\pm$ 0.720	3.15 $\pm$ 2.26	3.14 $\pm$ 2.26	0.1358 $\pm$ 0.0858
SATS-Logit	65.69 $\pm$ 12.15	3.176 $\pm$ 1.644	18.47 $\pm$ 6.90	18.25 $\pm$ 6.94	0.1755 $\pm$ 0.0996

Table 34: ImageNet-C 75-cell calibration summary (NLL-objective diagnostic only) (15 canonical Hendrycks main corruptions

\times

5 severities, 50,000 images per cell; single seed; pretrained ViT-B/16). All four calibrators are fit ONCE on the 5,000-sample stratified-first-5-per-class split of clean ImageNet-1K val and applied unchanged to each (corruption, severity) cell (no refit on corruption test data). TS-NLL and SRTS-NLL

K{=}1

are identical to 4 decimal places on every cell. Selective-classification cost (AURC) is included to expose the SATS val-overfit signature, which can amplify under shift even when held-out ECE looks acceptable on clean data.

Figure 20: ImageNet-C

\mathrm{ECE}_{15}

vs severity (mean over 15 corruptions; deployable BCE comparison). On this strongly pretrained base, the BCE-objective variants are tightly clustered and do not reproduce the CIFAR-100-C routing gain, consistent with the text: TvA-TS remains a very strong scalar baseline and SRTS-BCE/SMART+BCE offer no aggregate ECE improvement on ImageNet-C.

Method	NLL	Brier	$\mathrm{ECE}_{15}$	AURC
NoTS	$1.504\pm 0.000$	$0.453\pm 0.000$	$2.599\pm 0.000$	$0.1356\pm 0.0000$
TS-NLL	$1.504\pm 0.001$	$0.453\pm 0.000$	$3.169\pm 0.260$	$0.1355\pm 0.0000$
SRTS-NLL $K{=}1$ (ablation)	$1.504\pm 0.001$	$0.453\pm 0.000$	$3.169\pm 0.260$	$0.1355\pm 0.0000$
SRTS-NLL $K{=}3$ (ablation)	$1.504\pm 0.001$	$0.454\pm 0.000$	$3.414\pm 0.320$	$0.1357\pm 0.0001$
SATS-Logit	$3.026\pm 0.703$	$0.529\pm 0.016$	$17.729\pm 2.126$	$0.1695\pm 0.0079$

Table 35: ImageNet-C calibration-subset resampling under the NLL objective on the pretrained ViT-B/16 ImageNet-1K base (

B=10

resamples) . Reported values are the mean

\pm

standard deviation across resamples of the per-resample 75-cell mean. This is an NLL-objective ablation: TS-NLL and SRTS-NLL

K{=}3

do not improve

\mathrm{ECE}_{15}

over NoTS on this well-calibrated base, and

K{=}3

does not beat

K{=}1

. BCE-objective results are reported in § F .

Method	ViT-B/16	DeiT-S	Swin-T
NoTS	$3.97{\pm}0.24$	$4.40{\pm}0.24$	$3.14{\pm}0.25$
TS-NLL	$1.63{\pm}0.18$	$2.58{\pm}0.21$	$1.91{\pm}0.23$
TvA-TS	$1.58{\pm}0.24$	$1.83{\pm}0.16$	$1.70{\pm}0.28$
SMART+BCE	$1.55{\pm}0.18$	$1.74{\pm}0.20$	$1.67{\pm}0.23$
SRTS-BCE	$1.64{\pm}0.27$	$2.00{\pm}0.29$	$1.72{\pm}0.20$

Table 36: ImageNet-100 fine-tuned calibration, three backbones (

\mathrm{ECE}_{15}

%, mean

\pm

std; DeiT-S over five seeds, ViT-B/16 and Swin-T over three; each calibrator fit on a

{\approx}2{,}500

-sample clean-validation split and evaluated on the held-out IN-100 test half, all with the canonical post-hoc pipeline). An ImageNet-scale boundary diagnostic : on clean IN-100 the scalar, SMART+BCE and SRTS-BCE calibrators land within

{\approx}0.5

pp of one another (

1.5

–

2.0

), so risk conditioning yields no routing advantage over scalar TvA-TS at this fine-tuned-but-large scale, and SMART+BCE is stable and competitive across seeds. Ordering among the sub-

2

-pp calibrators is backbone-dependent and within seed variance. This table reports clean IN-100; corruption shift is covered on CIFAR-100-C and ImageNet-C.

Backbone	top-1	NoTS	TS-NLL	TvA-TS	SMART+BCE	SRTS-BCE
ResNet-50	$77.0$	$14.65$	$3.83$	$2.02$	$\mathbf{1.30}$	$1.34$
EfficientNet-B0	$73.4$	$6.82$	$2.54$	$\mathbf{0.93}$	$0.97$	$1.00$
Swin-B	$83.4$	$49.14$	$2.45$	$1.32$	$1.03$	$\mathbf{0.91}$
ViT-B/16	$84.4$	$1.02$	$1.17$	$0.96$	$\mathbf{0.47}$	$0.62$

Table 37: Pretrained ImageNet-1K clean calibration (

\mathrm{ECE}_{15}

%, eval-only off-the-shelf backbones;

5

/class fits each calibrator, remaining

45{,}000

val images are test; canonical pipeline). On these well-trained ImageNet heads the calibration residual is small (NoTS

\mathrm{ECE}_{15}

from

1.0

49

; the large Swin-B value reflects systematic under -confidence — mean confidence

34\%

against top-1

83\%

, so the

49

-point gap is the confidence–accuracy deficit, which a single temperature removes, TS-NLL

2.45

), and the low-capacity BCE calibrators give a small clean improvement over scalar TvA-TS on three of four backbones — consistent with a modest structured residual even on pretrained heads. This clean gain does not transfer to ImageNet-C shift (§ F ), where scalar TvA-TS is best: the pretrained boundary is specific to distribution shift, not to clean in-distribution calibration. Bold : best deployable calibrator per row.

Method	ECE 15	eAURC	NLL	Sev. 4–5 ECE
NoTS	$2.60{\pm}0.00$	$0.0565{\pm}0.0000$	—	$3.72{\pm}0.00$
TS-NLL	$2.99{\pm}0.26$	$0.0565{\pm}0.0013$	—	$4.17{\pm}0.36$
TvA-TS ( $K{=}1$ )	$\mathbf{2.47}{\pm}0.08$	$0.0566{\pm}0.0005$	—	$\mathbf{3.48}{\pm}0.12$
SRTS-BCE $K{=}3$	$2.56{\pm}0.23$	$0.0568{\pm}0.0027$	—	$3.86{\pm}0.34$
SMART+BCE	$2.56{\pm}0.26$	$0.0572{\pm}0.0028$	—	$3.91{\pm}0.40$

Table 38: BCE-objective ImageNet-C evaluation. Calibrators fitted on a 5,000-sample clean ImageNet-1K calibration subset (5 per class;

B{=}10

resamples), applied unchanged to 75 ImageNet-C cells (15 corruptions

\times

5 severities). Values are mean

\pm

std across resamples. On this pretrained backbone setting, TvA-TS (

K{=}1

BCE) achieves the lowest ECE 15 ; SRTS-BCE and SMART+BCE offer no improvement over TvA-TS and are comparable to NoTS on ECE. TS-NLL worsens ECE relative to NoTS, indicating the pretrained model is already well-calibrated under NLL and a scalar NLL temperature slightly miscalibrates it further. The BCE routing benefit observed on fine-tuned CIFAR-100 is absent on ImageNet-C; see the accompanying discussion.

Shift	Best	Outcome
CIFAR-100-C (synthetic, fine-tuned)	SRTS-BCE	$4.63$ vs. SMART+BCE $4.75$ ; matched PWLinear-3 $4.58$ lower, unresolved; hier. CI incl. 0
ImageNet-R/-A (natural, pretrained)	NoTS	every clean-fitted calibrator worsens ECE and TopBCE
ImageNet-V2 (natural, fine-tuned)	BCE family	objective transfers (CIs excl. 0); routing vs. scalar unresolved

Table 39: Distribution-shift scope (

\mathrm{ECE}_{15}

; best headline calibrator per regime). The pretrained-vs-fine-tuned split on natural shift is the boundary: calibration transfers under fine-tuning, not on an already-calibrated pretrained head.

	ImageNet-R		ImageNet-A
Method	$\mathrm{ECE}_{15}$	TopBCE	$\mathrm{ECE}_{15}$	TopBCE
NoTS	$\mathbf{5.57{\pm}0.00}$	$\mathbf{0.395{\pm}0.000}$	$\mathbf{22.89{\pm}0.00}$	$\mathbf{0.712{\pm}0.000}$
TS-NLL	$10.23{\pm}0.46$	$0.430{\pm}0.005$	$28.09{\pm}1.05$	$0.818{\pm}0.024$
TvA-TS	$8.55{\pm}0.38$	$0.415{\pm}0.003$	$26.32{\pm}0.89$	$0.779{\pm}0.018$
QaTS-NLL	$9.91{\pm}0.02$	$0.428{\pm}0.002$	$28.09{\pm}1.05$	$0.818{\pm}0.024$
QaTS-BCE	$7.05{\pm}2.03$	$0.410{\pm}0.006$	$26.32{\pm}0.89$	$0.779{\pm}0.018$

Table 40: Natural-shift boundary: clean-ImageNet-fit calibration on ImageNet-R and ImageNet-A (pretrained ViT-B/16; label space restricted to each variant’s 200 classes; calibrators fit on a clean five-per-class validation draw,

{\approx}1{,}000

samples, applied unchanged to the shifted test set; mean

{\pm}

std over three calibration draws). Every clean-fitted calibrator worsens both metrics relative to the uncalibrated model. Bold : best per column.

Regime	Best method	Support
Clean C100, 3 ViT backbones	SRTS-BCE 3/3	ViT cell bootstrap-tied
Clean C100, 2 ConvNets	split 1/1	gaps $\leq 0.13$ pp
Clean CIFAR-10, 5 backbones	SMART-SoftECE on $\mathrm{ECE}_{15}$	near-saturated; SRTS-BCE best AdaECE 15 5/5
Clean Tiny-IN, 3 backbones	SMART+BCE 2/3	point estimates
Clean IN-100, 3 fine-tuned backbones	tie (all methods)	no routing advantage; within seed variance ( $1.5$ – $2.0$ )
C100-C, deployable rerouting	SRTS-BCE	directional; hierarch. CI incl. 0

Table 41: Explicit per-regime win/loss map (primary metric

\mathrm{ECE}_{15}

; the flagship CIFAR-100/ViT-B/16 cell uses five seeds, other multi-run regimes three unless stated otherwise). “Support” qualifies the winning margin (row groups: clean, shift, boundary). Where the honest outcome is a tie or a loss, the map reports it as such; per-dataset evidence is in the subsections below.

Backbone	Method	Top-1	NLL	Brier	$\mathrm{ECE}_{15}$	AURC	eAURC
DeiT-S	NoTS	$88.76{\pm}0.63$	$0.523{\pm}0.005$	$0.178{\pm}0.008$	$8.53{\pm}1.11$	$0.0328{\pm}0.0022$	$0.0262{\pm}0.0029$
	TS-NLL	$88.76{\pm}0.63$	$0.458{\pm}0.020$	$0.174{\pm}0.006$	$2.90{\pm}0.19$	$0.0296{\pm}0.0026$	$0.0230{\pm}0.0033$
	SRTS-NLL $K{=}1$ (ablation)	$88.76{\pm}0.63$	$0.458{\pm}0.020$	$0.174{\pm}0.006$	$2.90{\pm}0.19$	$0.0296{\pm}0.0026$	$0.0230{\pm}0.0033$
	SRTS-NLL $K{=}3$ (ablation)	$88.76{\pm}0.63$	$0.447{\pm}0.017$	$0.170{\pm}0.007$	$2.65{\pm}0.12$	$0.0221{\pm}0.0004$	$0.0156{\pm}0.0011$
	SATS-Logit	$88.76{\pm}0.63$	$1.116{\pm}0.342$	$0.191{\pm}0.014$	$7.02{\pm}1.05$	$0.0393{\pm}0.0031$	$0.0327{\pm}0.0023$
	LTS-feature	$88.76{\pm}0.63$	$2.048{\pm}0.262$	$0.199{\pm}0.012$	$8.30{\pm}0.65$	$0.0449{\pm}0.0034$	$0.0383{\pm}0.0027$

Table 42: Full three-seed DeiT-S and Swin-T CIFAR-100 post-hoc calibration tables (mean

\pm

standard deviation over seeds 0/1/2). NLL-objective SRTS rows are reported as ablations; the BCE-objective confirmation rows include TvA-TS, SRTS-BCE

K{=}3

risk, and SMART+BCE. SRTS-BCE wins ECE on both backbones. Top-1 was empirically unchanged for every post-hoc row in the seed-level audit.

Regime / backbone	Rows checked	Missing post-hoc rows
CIFAR-100 / ViT-B/16	21 + TvA aggregate	TvA-TS from companion 3-seed evaluation
CIFAR-100 / DeiT-S	24	0
CIFAR-100 / Swin-T	24	0
Tiny-ImageNet / ViT-B/16	21	Later BCE rows in Table 49

Table 43: Cross-backbone top-1 invariance audit. Each post-hoc row is compared against its corresponding NoTS seed before aggregation. No reported post-hoc row changed top-1 in the seed-level metrics. Tiny-ImageNet BCE rows, including TvA-TS, are in Table 49 .

Backbone	Method	Top-1	NLL	Brier	$\mathrm{ECE}_{15}$	AURC	eAURC
DeiT-S	NoTS	$84.27{\pm}0.07$	$0.717{\pm}0.005$	$0.234{\pm}0.001$	$9.10{\pm}0.34$	$0.0381{\pm}0.0014$	$0.0250{\pm}0.0015$
	TS-NLL	$84.27{\pm}0.07$	$0.638{\pm}0.003$	$0.228{\pm}0.002$	$3.27{\pm}0.17$	$0.0355{\pm}0.0014$	$0.0224{\pm}0.0015$
	SRTS-NLL $K{=}1$ (ablation)	$84.27{\pm}0.07$	$0.638{\pm}0.003$	$0.228{\pm}0.002$	$3.27{\pm}0.17$	$0.0355{\pm}0.0014$	$0.0224{\pm}0.0015$
	SRTS-NLL $K{=}3$ (ablation)	$84.27{\pm}0.07$	$0.632{\pm}0.003$	$0.226{\pm}0.001$	$3.10{\pm}0.13$	$0.0315{\pm}0.0003$	$0.0184{\pm}0.0004$
	SATS-Logit	$84.27{\pm}0.07$	$1.509{\pm}0.089$	$0.262{\pm}0.001$	$9.75{\pm}0.16$	$0.0532{\pm}0.0017$	$0.0401{\pm}0.0017$
	LTS-feature	$84.27{\pm}0.07$	$1.852{\pm}0.058$	$0.265{\pm}0.002$	$10.40{\pm}0.07$	$0.0542{\pm}0.0023$	$0.0411{\pm}0.0023$

Table 44: Full three-seed DeiT-S and Swin-T Tiny-ImageNet post-hoc calibration tables (mean

\pm

standard deviation over seeds 0/1/2). NLL-objective SRTS rows are reported as ablations; the BCE-objective confirmation rows include TvA-TS, SRTS-BCE

K{=}3

risk, and SMART+BCE (per-backbone ordering discussed in the text). Top-1 was empirically unchanged for every post-hoc row in the seed-level audit.

Regime / backbone	NoTS	TS-NLL	TvA-TS	SMART-SoftECE	SRTS-BCE
C10 / ViT-B/16	$10.96$	$0.55{\pm}0.07/0.73{\pm}0.16$	$0.52{\pm}0.03/0.73{\pm}0.15$	$\mathbf{0.35{\pm}0.13}/1.47{\pm}0.47$	$0.47{\pm}0.05/\mathbf{0.58{\pm}0.08}$
C10 / DeiT-S	$10.81$	$0.49{\pm}0.04/0.85{\pm}0.16$	$0.46{\pm}0.04/0.85{\pm}0.17$	$\mathbf{0.24{\pm}0.07}/1.34{\pm}0.43$	$0.41{\pm}0.07/\mathbf{0.70{\pm}0.11}$
C10 / Swin-T	$11.30$	$0.44{\pm}0.05/0.68{\pm}0.09$	$0.39{\pm}0.08/0.67{\pm}0.09$	$\mathbf{0.19{\pm}0.03}/1.37{\pm}0.22$	$0.35{\pm}0.11/\mathbf{0.53{\pm}0.13}$
C10 / ResNet-50	$20.23$	$0.63{\pm}0.03/1.17{\pm}0.08$	$0.60{\pm}0.04/1.18{\pm}0.09$	$\mathbf{0.32{\pm}0.13}/1.37{\pm}0.75$	$0.60{\pm}0.04/\mathbf{0.99{\pm}0.11}$
C10 / RegNetY-1.6GF	$18.52$	$0.51{\pm}0.05/0.72{\pm}0.02$	$0.53{\pm}0.02/0.72{\pm}0.01$	$\mathbf{0.40{\pm}0.07}/1.28{\pm}0.79$	$0.42{\pm}0.06/\mathbf{0.57{\pm}0.03}$
C100 / ViT-B/16	$9.69$	$1.94{\pm}0.65/2.10{\pm}0.62$	$1.65{\pm}0.44/1.77{\pm}0.69$	$1.21{\pm}0.28/1.45{\pm}0.51$	$\mathbf{0.96{\pm}0.32}/\mathbf{0.96{\pm}0.33}$

Table 45: Empirical scope map (fixed

K{=}3

cross-regime evaluation). Cells:

\mathrm{ECE}_{15}

/ AdaECE 15 (

\pm

std; CIFAR-100/ViT-B/16 and IN-100/DeiT-S five seeds, other cells three; IN-100 rows computed with the canonical post-hoc pipeline, SMART-SoftECE column three-seed); SRTS-BCE uses fixed

K{=}3

everywhere, no test-set or per-cell

K

selection. SMART-SoftECE: the SMART head ( Guo et al. 2026 ) with its own training objective (SMART+BCE: the main paper’s RQ1 table, § F , and § G here). Bold: lower of SMART-SoftECE/SRTS-BCE per metric (C100 and Tiny-IN blocks); best over all methods (diagnostic C10 and IN-100 blocks, § F ; C10: SMART-SoftECE wins

\mathrm{ECE}_{15}

, worst AdaECE 15 — metric dependence).

Regime / backbone	Metric	NoTS	TS-NLL	TvA-TS	SMART+BCE	SRTS-BCE	Adaptive range
C100 / ViT-B/16	ECE 15	$9.69{\pm}1.10$	$1.94{\pm}0.65$	$1.65{\pm}0.44$	$\mathbf{0.95{\pm}0.29}$	$0.96{\pm}0.32$	$6.15$ – $6.71$
	AdaECE 15	$9.65{\pm}1.15$	$2.10{\pm}0.62$	$1.77{\pm}0.69$	$1.02{\pm}0.29$	$\mathbf{0.96{\pm}0.33}$	—
	smECE	$9.63{\pm}1.14$	$1.97{\pm}0.52$	$1.71{\pm}0.45$	$1.16{\pm}0.22$	$\mathbf{1.13{\pm}0.22}$	—
C100 / DeiT-S	ECE 15	$8.53{\pm}0.90$	$2.91{\pm}0.16$	$3.37{\pm}0.40$	$1.17{\pm}0.01$	$\mathbf{0.94{\pm}0.07}$	$7.02$ – $8.31$
	AdaECE 15	$8.45{\pm}0.97$	$3.74{\pm}0.24$	$3.52{\pm}0.38$	$1.23{\pm}0.12$	$\mathbf{1.05{\pm}0.23}$	—
	smECE	$8.33{\pm}1.07$	$2.94{\pm}0.13$	$2.83{\pm}0.19$	$1.24{\pm}0.04$	$\mathbf{1.14{\pm}0.06}$	—

Table 46: Full cross-backbone calibration metrics under fixed deployed configurations. The CIFAR-100/ViT-B/16 cell is five-seed; other cells are three-seed. SRTS-BCE uses the single

K{=}3

, fixed in advance, for every cell, with no

K

-selection or test-set selection; SMART+BCE and the scalar anchors (TS-NLL, TvA-TS) use the same fit-on-validation, evaluate-on-test protocol. A separate

K

-sweep diagnostic (§ G ) is not used for any main-text claim. Bold on ECE 15 marks the lower of SMART+BCE/SRTS-BCE; bold on AdaECE 15 /smECE marks the true minimum over all five methods.

Regime / backbone	Metric	NoTS	TS-NLL	TvA-TS	SMART+BCE	SRTS-BCE
CIFAR-100 / ViT-B/16	ECE 15	$9.69\pm 1.10$	$1.94\pm 0.65$	$1.65\pm 0.44$	$0.95\pm 0.29$	$\mathbf{0.90\pm 0.36}$
	AdaECE 15	$9.65\pm 1.15$	$2.10\pm 0.62$	$1.77\pm 0.69$	$0.99\pm 0.27$	$\mathbf{0.96\pm 0.27}$
	smECE	$9.63\pm 1.14$	$1.97\pm 0.52$	$1.71\pm 0.45$	$1.16\pm 0.22$	$\mathbf{1.16\pm 0.21}$
CIFAR-100 / DeiT-S	ECE 15	$8.53\pm 0.90$	$2.91\pm 0.16$	$3.37\pm 0.40$	$1.17\pm 0.01$	$\mathbf{0.86\pm 0.20}$
	AdaECE 15	$8.45\pm 0.97$	$3.74\pm 0.24$	$3.52\pm 0.38$	$1.23\pm 0.12$	$\mathbf{1.05\pm 0.23}$
	smECE	$8.33\pm 1.07$	$2.94\pm 0.13$	$2.83\pm 0.19$	$1.24\pm 0.04$	$\mathbf{1.12\pm 0.07}$

Table 47: Oracle best-

K

diagnostic (not used for main claims). Cross-backbone and cross-dataset calibration (%, mean

\pm

std; CIFAR-100/ViT-B/16 over five seeds, other cells three; ddof=0). All calibrators are fit on validation and evaluated on the held-out test set. Here SRTS-BCE reports the oracle best-

K

cell over

K\in\{1,2,3,5,10,15,20\}

, i.e. the lowest-ECE

K

selected with knowledge of the test metric. This is a diagnostic upper bound on what risk routing can achieve and is not the deployed configuration: every main-text claim uses the single

K{=}3

fixed in advance (the fixed-

K{=}3

cross-backbone summary in the main paper). Bold marks the lowest value among all five deployable post-hoc calibrators. Unrestricted adaptive temperature heads are omitted; their ECE 15 remains

{\geq}6\%

across all settings (§ B ).

Method	top-1 (%)	NLL	ECE 15 (%)	AdaECE 15 (%)	smECE (%)	AURC
NoTS	98.50	0.1466	9.89	9.86	10.05	0.0017
TS-NLL ( $T{=}0.527$ )	98.50	0.0533	0.51	0.65	0.71	0.0013
SRTS-NLL $K{=}1$ (ablation)	98.50	0.0533	0.51	0.65	0.71	0.0013
SRTS-NLL $K{=}3$ (ablation)	98.50	0.0588	0.49	0.55	0.81	0.0014
SATS-Logit	98.50	0.0835	0.79	0.74	0.86	0.0032

Table 48: CIFAR-10 modern-recipe seed-0 calibration (NLL-objective diagnostic only) (10,000-sample held-out test split). The CE+MixUp+label-smoothing fine-tune is strongly overconfident (NoTS ECE

{}_{15}=9.89\,\%

, scalar fitted

T=0.527<1

). TS-NLL and SRTS-NLL

K{=}1

are identical to 4 decimal places on every metric, confirming the

K{=}1

short-circuit at a third independent dataset (after clean CIFAR-100 and clean ImageNet-1K).

K{=}3

over

K{=}1

on clean C10 is a wash: small ECE 15 gain (

-0.014

absolute), small NLL loss (

+0.005

). SATS-Logit exhibits partial val-overfit: clean ECE 15 is still much better than NoTS (

0.79\,\%

) but worse than TS-NLL, and AURC degrades to

\approx 2\times

NoTS — the same shape documented on CIFAR-100 / ImageNet-1K val-fits.

BCE objective
Method	Top-1	NLL	Brier	$\mathrm{ECE}_{15}$	AURC	eAURC
NoTS	$87.46\pm 0.31$	$0.588\pm 0.014$	$0.197\pm 0.004$	$10.04\pm 1.09$	$0.0256\pm 0.0012$	$0.0174\pm 0.0012$
TS-NLL	$87.46\pm 0.31$	$0.505\pm 0.022$	$0.184\pm 0.005$	$1.32\pm 0.29$	$0.0236\pm 0.0012$	$0.0153\pm 0.0010$
SRTS-NLL $K{=}1$ (ablation)	$87.46\pm 0.31$	$0.505\pm 0.022$	$0.184\pm 0.005$	$1.32\pm 0.29$	$0.0236\pm 0.0012$	$0.0153\pm 0.0010$
SRTS-NLL $K{=}3$ (ablation)	$87.46\pm 0.31$	$0.503\pm 0.023$	$0.184\pm 0.005$	$1.34\pm 0.29$	$0.0223\pm 0.0013$	$0.0140\pm 0.0010$
TvA-TS $\equiv$ TS-BCE $K{=}1$	$87.46\pm 0.31$	$0.508\pm 0.017$	$0.184\pm 0.004$	$1.26\pm 0.18$	$0.0238\pm 0.0009$	$0.0156\pm 0.0008$

Table 49: Tiny-ImageNet / ViT-B/16 three-seed clean calibration (mean

\pm

standard deviation over seeds 0/1/2). The NLL block reports the NLL-objective diagnostic rows; the BCE block reports the top-label-BCE calibrators under the identical protocol. All post-hoc methods are fit on Tiny-ImageNet validation data and applied unchanged to held-out test data. SMART+BCE leads SRTS-BCE on ECE in this setting, while SRTS-BCE leads it on eAURC.

Regime / backbone	$K^{\star}$ (seeds 0,1,2)	SRTS-BCE-Select ECE 15	Fixed $K{=}3$ ECE 15
CIFAR-100 / ViT-B/16	10, 2, 5	$1.72\pm 0.55$	$0.83\pm 0.28$
CIFAR-100 / DeiT-S	1, 3, 3	$2.71\pm 0.12$	$0.94\pm 0.07$
CIFAR-100 / Swin-T	3, 2, 5	$2.77\pm 0.31$	$1.04\pm 0.27$
Tiny-ImageNet / ViT-B/16	1, 1, 1	$1.26\pm 0.18$	$1.12\pm 0.17$
Tiny-ImageNet / DeiT-S	10, 1, 3	$2.98\pm 0.23$	$1.09\pm 0.28$
Tiny-ImageNet / Swin-T	1, 1, 1	$2.71\pm 0.07$	$1.24\pm 0.11$

Table 50: Cross-backbone SRTS-BCE-Select (%, mean

\pm

std over three seeds).

K^{\star}

is selected per-cell by Cal-Select ECE 15 from

\mathcal{K}=\{1,2,3,5,10\}

, then refitted on the full calibration split. Columns show the selected

K^{\star}

per seed, the test-set ECE 15 of SRTS-BCE-Select, and the fixed

K{=}3

reference from the main paper’s cross-backbone summary table (selection protocol and the too-small-split conclusion are in the text).

Regime	picks	$K{=}1$	$K{=}3$	selected	best-of-2
C100/ViT	1/3/3	1.54	0.83	1.08	0.83
C100/DeiT-S	3/3/3	3.37	0.94	0.94	0.94
C100/Swin-T	3/3/3	2.87	1.04	1.04	1.04
C100/ResNet-50	3/3/3	2.83	1.16	1.16	1.16
C100/RegNetY	3/3/3	1.54	1.01	1.01	1.01
C10/ViT	1/1/1	0.52	0.47	0.52	0.47

Table 51: Validation-only capacity selection with the one-standard-error rule (per-regime mean test

\mathrm{ECE}_{15}

over seeds; “picks” = selected

K

per seed; “oracle” = the per-seed

\min(K{=}1,K{=}3)

averaged over seeds, the selector’s own two-candidate choice; C100/ViT and all IN-100 rows here use the cached three-seed subset, not the five-seed canonical aggregate). The rule uses only each cell’s clean calibration split (repeated cross-validated top-label-BCE gain; select

K{=}3

iff the mean gain exceeds one standard error) and has no cross-regime fitted parameters. Mean test-ECE excess over the per-seed best-of-2: selector

0.09

, always-

K{=}3

0.18

, always-

K{=}1

0.59

Regime	$B/\bar{\sigma}^{2}$	floor $\tfrac{K-1}{n}$	random	ECE outcome
CIFAR-100 / ViT-B/16	$2.63$	$0.80$	$0.99$	SRTS wins
CIFAR-100 / ResNet-56	$1.32$	$2.00$	$3.30$	mixed (ConvNet)
ImageNet-1K / ResNet-50	$4.25$	$0.40$	$0.35$	SRTS loses
ImageNet-1K / Swin-B	$3.94$	$0.40$	$0.38$	SRTS loses

Table 52: Empirical routing signal-to-noise

B/\bar{\sigma}^{2}

across regimes (per-sample top-label-BCE gradient at the pooled optimum

T^{\star}

;

K{=}3

risk partition; values

\times 10^{-3}

). The theory (§ I ) says grouping lowers BCE iff

B/\bar{\sigma}^{2}>(K{-}1)/n

. The risk router clears the floor by

3

–

12\times

the random control in the ViT and ImageNet regimes (the fitted-objective condition holds); a random partition sits at the floor, as predicted. This is a mechanism diagnostic, not an ECE predictor : the ImageNet-pretrained cells clear the BCE floor yet SRTS loses in ECE, because

B/\bar{\sigma}^{2}

governs the BCE gain and the ECE outcome additionally depends on the BCE

\to

ECE gap and finite-sample noise.

Objective	Method	NLL	$\mathrm{ECE}_{15}$	$\Delta_{\text{TvA}}$	AdaECE 15	CW-ECE 15	smECE	eAURC
—	CE / NoTS	$0.4142\pm 0.0204$	$9.69\pm 1.10$	$+8.04$	$9.65\pm 1.15$	$0.27\pm 0.01$	$9.63\pm 1.14$	$0.0165\pm 0.0041$
NLL	TS-NLL	$0.3309\pm 0.0308$	$1.94\pm 0.65$	$+0.29$	$2.10\pm 0.62$	$0.12\pm 0.01$	$1.97\pm 0.52$	$0.0139\pm 0.0032$
NLL	SATS-Logit	$0.8128\pm 0.0391$	$6.04\pm 0.23$	$+4.39$	$5.39\pm 0.23$	$0.15\pm 0.01$	$7.47\pm 0.16$	$0.0281\pm 0.0035$
NLL	LTS-feature	$2.17\pm 1.46$	$6.71\pm 0.36$	$+5.17$	$6.26\pm 0.32$	$0.15\pm 0.01$	$8.54\pm 0.37$	$0.0296\pm 0.0079$
NLL	Feature-SATS †	$2.18\pm 1.48$	$6.71\pm 0.36$	$+5.17$	$6.27\pm 0.32$	$0.15\pm 0.01$	$8.54\pm 0.37$	$0.0296\pm 0.0078$
BCE	TvA-TS ( $K{=}1$ )	$0.3321\pm 0.0296$	$1.65\pm 0.44$	$0$	$1.77\pm 0.69$	$0.13\pm 0.01$	$1.71\pm 0.45$	$0.0142\pm 0.0033$

Table 53: Full clean metric set: ViT-B/16 / CIFAR-100 post-hoc calibration (%, mean

\pm

std; deployable rows five seeds, top-1

91.00\pm 0.46

; SATS-Logit five seeds; LTS-feature/Feature-SATS archived three seeds; fit on the validation split, evaluated on held-out test; metric definitions: § A ). Row groups by capacity: scalar baselines, unrestricted adaptive heads (the failure mode), BCE peers, SRTS routing. TvA-TS

\equiv

TS-BCE (

K{=}1

); SMART-SoftECE is the SMART head with its own objective; SMART+BCE swaps it for top-label BCE.

\Delta_{\text{TvA}}

\mathrm{ECE}_{15}

relative to scalar TvA-TS. Bold : best per column among deployable calibrators. † Frozen-feature UTempH gives identical three-seed metrics.

Method (argmax-changing)	Top-1	NLL	$\mathrm{ECE}_{15}$	eAURC
Ref: scalar TS-NLL (argmax-preserving)	$91.17\pm 0.46$	$0.3239\pm 0.0297$	$1.7950\pm 0.6977$	$0.0132\pm 0.0035$
Ref: SRTS-NLL $K{=}3$ (ablation)	$91.17\pm 0.46$	$0.3217\pm 0.0274$	$1.7058\pm 0.5294$	$0.0115\pm 0.0016$
Ref: TvA-TS (argmax-preserving)	$91.17\pm 0.46$	$0.3253\pm 0.0283$	$1.5370\pm 0.4171$	$0.0135\pm 0.0035$
Ref: SRTS-BCE $K{=}3$ risk (main)	$91.17\pm 0.46$	$0.3246\pm 0.0206$	$\mathbf{0.8276\pm 0.2791}$	$\mathbf{0.0114\pm 0.0013}$
Ref: SMART+BCE	$91.17\pm 0.46$	$0.3245\pm 0.0217$	$0.8573\pm 0.2871$	$0.0119\pm 0.0014$
Vector Scaling (per-class T, $\ell_{2}$ )	$91.24\pm 0.34$	$0.3259\pm 0.0303$	$2.3327\pm 0.4743$	$0.0145\pm 0.0031$

Table 54: Argmax-changing classical baselines on the modern-recipe ViT-B/16 / CIFAR-100 base (three seeds). Vector and Dirichlet scaling change the predicted class and sit in a different capacity regime from the constrained argmax-preserving family. Under the same

\approx 2{,}500

-sample validation budget and val-NLL-based hyperparameter selection, both are worse on

\mathrm{ECE}_{15}

than SRTS-BCE, SMART+BCE, TvA-TS, TS-NLL, and the SRTS-NLL ablation. Reference rows use the archived three-seed diagnostic subset for protocol matching.

Explore similar work

Jun 19, 2026cs.CV

Quantile Adaptive Temperature Scaling for Confidence Calibration

Deep neural networks often produce poorly calibrated confidence estimates, overstating their certainty even when predictions are incorrect. Temperature Scaling remains the most widely used posthoc calibration method due to its simplicity and effectiveness, yet its global, uniform rescaling of logits fails to correct the highly heterogeneous structure of miscalibration observed across the confidence spectrum. In particular, the largest correctness confidence discrepancies arise in different quantile regions depending on the setting, low confidence predictions, where uncertainty matters most, tend to exhibit the largest correctness confidence discrepancies, which standard TS leaves largely unaddressed. We introduce Quantile Adaptive Temperature Scaling (QaTS), a simple and efficient post hoc calibration method that adapts the temperature as a function of a predictions empirical confidence quantile. By mapping confidences into the quantile space, QaTS normalizes the calibration problem, makes the structure of miscalibration explicit and enables a monotone temperature function that adapts across quantiles while leaving well calibrated high confidence predictions largely unchanged. preserving high confidence behavior. This quantile aware formulation aligns naturally with a reparameterized Expected Calibration Error (ECE) objective and yields a sample wise temperature that is robust across a variety of challenging scenarios, such as class imbalance and distributional shifts. Across a broad range of datasets, architectures, evaluation scenarios and diverse tasks, QaTS consistently, and substantially, outperforms state of the art post hoc calibration methods, delivering more reliable and trustworthy confidence estimates without modifying model predictions.

Omprakash Chakraborty, Leo Fillioux, Ismail Ben Ayed +1

1 ÉTS Montréal, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, CDSU, IHU PRISM, Gif-sur-Yvette

Confidence Estimation Confidence Calibration Recalibration

Ask PDF

May 21, 2025stat.ML

Adaptive Cumulative Mass Calibration with Conformal Prediction

Reliable probability estimates by classifiers are essential in high-risk applications. In practice, however, predicted probabilities are often miscalibrated, and many existing post-hoc calibration methods typically lack guarantees that a specific notion of calibration is achieved after the correction procedure is applied. We introduce a set-based perspective on calibration through the notion of cumulative mass calibration and the corresponding error measures. We propose a new calibration procedure based on conformal prediction that forms cumulative probabilities with guaranteed marginal coverage. We introduce an adaptive temperature scaling algorithm, with the temperature tuned for each input to satisfy the conformal coverage constraint. As we show, this procedure can be efficiently implemented. Across image classification tasks, particularly in settings with many classes, our method improves newly introduced calibration error measures (CMCE and

α

-CMCE) and standard metrics (such as ECE, cw-ECE, MCE) over the existing baselines.

Daniil Kazantsev, Eric Moulines, Maxim Panov +2

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · École pour l’informatique et les techniques avancées (EPITA), Paris, France

Calibrated Uncertainty Recalibration

Ask PDF

Jul 13, 2026cs.LG

Condition-Stratified Robustness Analysis of Post-Hoc Calibration Methods for Probabilistic Classifiers

Post-hoc calibration is widely adopted to correct probability estimates from trained classifiers, yet most evaluations report aggregate performance without testing whether that performance holds across distinct operating conditions within a single dataset. We present a pre-registered, condition-stratified robustness analysis comparing temperature scaling (TEMP) and isotonic regression (ISO) across four controlled conditions (C1--C4). Four hypothesis groups are evaluated: discrimination deltas with Holm-corrected multiplicity control (H1), Brier score differences (H2), calibration slope outcomes (H3), and AUROC differences under best-condition setups (H4). TEMP-minus-ISO discrimination deltas remain small across all conditions (-0.0155 to 0.0139), with Holm-adjusted p-values of 0.9895 everywhere. TEMP Brier differences are consistently negative (C1: -0.0002 through C4: -0.0074), while ISO shows sign reversals. TEMP calibration slopes stay closer to unity in every condition (range 0.7597--0.9493) than ISO slopes (0.1364--0.2726). AUROC differences shift from near zero in C1 (-0.0004) to positive in C4 (0.0264). These results establish that in-dataset robustness is condition-dependent and metric-specific. No claim of external transportability is made.

Gurdeep Singh Virdee

Fergana State Technical University, Uzbekistan

Post-Hoc Calibrated Uncertainty Robustness Verification Classifier

Ask PDF