Auditing Routing Entropy as an Uncertainty Signal in Attention-Residual Transformers

Wenhao Liang; Lin Yue; Wei Emma Zhang; Mingyu Guo; Olaf Maennel; Weitong Chen

cs.AIOct 1, 2026

Auditing Routing Entropy as an Uncertainty Signal in Attention-Residual Transformers

Authors: Wenhao Liang, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen

Organizations: Adelaide University, Adelaide, Australia

Abstract

Dynamic architectures leave a per-example routing trace beside each prediction, and diffuse routing is easy to read as a sign that the prediction is unreliable. We audit that reading for routing entropy in Attention-Residual (AR) variants of Swin-Tiny and DeiT-Small, trained from scratch on CIFAR-10/100 with a soft-binned calibration auxiliary loss, asking whether the trace carries information about correctness beyond what the model's own confidence already reveals. Three checks probe this increment: does a routing signal appear at fixed confidence, does it replicate across training seeds, and can a held-out predictor exploit it against output-only and shuffled-trace controls? A sensitivity audit then injects effects of known size and measures the fraction of each that the probes recover. No test in the fixed 30-test binned family survives multiplicity correction, and neither the nominal hit nor a borderline result recurs in its sibling seeds. Across 24 paired runs a scalar routing probe yields no pooled improvement in routing-stratified calibration, and an entropy-profile probe predicts correctness better than the same probe given shuffled profiles yet worse than a confidence-only predictor in both binary log-loss and Brier score: a gain over shuffled traces does not become a gain over the output. Conditioning on the complete logit vector leaves the corresponding comparison unresolved. The audit bounds how far these non-detections can be read: at an injected effect of 0.010 nats the profile probe recovers 24-59% of the oracle gain, and a reference-preserving correction probe recovers 8% and 23% in the two CIFAR-100 settings, below the threshold we fixed for applying it to real labels. The results establish control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.

Figures & tables

Figure 1: From a routing trace to a testable uncertainty claim. (a) An AR vision transformer exposes top-class confidence

c(x)

and, at every layer, routing weights over earlier states; their normalised entropies

H_{l}\in[0,1]

form the trace

R(x)

(input and values illustrative; the token index

n

of Eq. ( 3 ) is suppressed). (b) The target: at equal confidence, does

R

change the probability of a correct prediction? (schematic data). (c) The three checks and the audit with the observed results: (1) within-bin permutation

p

-values of the 30-test family, with one nominal hit and one borderline result at

p\approx 0.05

(§ 4.1 ); (2) the hit is not reproduced by seeds 1–2 and no test passes

0.05/30

; (3) Profile minus shuffled-profile and minus confidence-only binary log-loss with 95% intervals (

\times 10^{-3}

nats; positive is worse for the profile; § 4.3 ); (A) recovery of a

0.010

-nat injected effect by the reference-preserving correction probe on the two CIFAR-100 settings (§ 4.4 ).

Figure 2: Within-bin permutation

p

-values (5,000 permutations) for the fixed 30-test

r_{\rm agg}

family. Rows are backbone–dataset/AR cells; filled points are replicated Swin/DeiT runs and open points single-seed ViT runs. The star marks the nominal hit and the diamond the borderline run, whose

p

lies within Monte Carlo error of

0.05

; vertical lines mark

0.05

and the Bonferroni threshold

0.05/30

Figure 3: Scalar probe minus NW(

c

) in worst-tertile ECE for each of the 24 runs, grouped by backbone–dataset setting (circles Block-AR, squares Full-AR). The pooled diamond and bar show the mean and the setting-clustered 95% interval; no setting-specific interval is inferred.

Figure 4: Entropy-profile comparisons across 24 paired runs. Profile minus comparator in binary log-loss (

10^{-3}

nats) and Brier score (

10^{-3}

); negative values favour the profile. Points are pooled means (record-weighted and equal-setting means coincide) with 95% hierarchical-bootstrap intervals over the four settings. The two comparators support different conclusions.

Setting	gain	E2	E3	E2	E3	(E3)
	Oracle	Routing increment $G_{\rm route}$		Candidate gain $G_{\rm cand}$		Recovery
Swin / C-100	0	$+0.00$	$-0.02$ $[-0.06,+0.02]$	$-7.86$	$-0.09$	–
Swin / C-100	0.005	$-0.02$	$+0.28$ $[+0.21,+0.34]$	$-6.67$	$+0.19$	5.5%
Swin / C-100	0.010	$-0.02$	$+0.81$ $[+0.72,+0.91]$	$-5.71$	$+0.74$	8.1%
DeiT / C-100	0	$+0.06$	$-0.02$ $[-0.06,+0.01]$	$-7.71$	$-0.05$	–
DeiT / C-100	0.005	$-0.04$	$+0.77$ $[+0.66,+0.89]$	$-5.07$	$+0.72$	15.5%

Table 1: Injected-signal simulations for the complete-output probe (E2) and the reference-preserving correction probe (E3) on the two CIFAR-100 settings, 20 replicates per condition. Unlike Fig. 4 and Tables 10 and 13 , entries are gains in expected-label binary log-loss (

10^{-3}

nats per example), so positive values favour the routing arm. The routing increment

G_{\rm route}=L(A_{\rm sel})-L(B_{\rm sel})

compares the selected output-only and routing arms under the same budget and is the quantity that measures routing (95% Monte Carlo intervals for E3); recovery divides the E3 increment by the exact oracle gain. The candidate gain

G_{\rm cand}=L(\mathrm{REF})-L(B_{\rm raw})

compares the unselected routing candidate with the output-only ridge reference; it includes baseline predictive improvement and is not recovered routing information. All labels are simulated; on the real CIFAR-100 labels every E2 selection returned the reference (§ 4.4 ).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Max gap	Wt. gap	$95\%$ CI (Wt.)	Bins	Min/ $q_{25}$ /Med	Null $q_{.975}$	Perm. $p$
Block-AR	$0.3455$	$0.016$	$[0.012,\,0.030]$	$12/15$	$11/86/143$	$0.379$	$\mathbf{0.0382}$

Appendix

Table 2: Single-run diagnostic on Swin-Tiny CIFAR-10 seed 0. Samples are split by

r_{\mathrm{agg}}

tertile and compared within 15 confidence bins; Bins and Min/

q_{25}

/Med encode shared-bin support. Wt. CI uses 5,000 bootstrap resamples, while Null

q_{.975}

and

p

use 5,000 within-bin routing permutations. This is the nominal hit of the 30-test family, which also contains one borderline result; definitions, the borderline run and caveats are in App. G .

Method	ECE $\downarrow$	AdaECE $\downarrow$	NLL $\downarrow$	Brier $\downarrow$	Worst-tert. ECE $\downarrow$	$\Delta$ Acc@ $1$ (%)
No calibration	$0.1514\,{\scriptstyle\pm 0.0028}$	$0.1514\,{\scriptstyle\pm 0.0028}$	$0.427$	$0.175$	$0.1581\,{\scriptstyle\pm 0.0049}$	$0.00\,{\scriptstyle\pm 0.00}$
Temp. Scaling (TS)	$0.0111\,{\scriptstyle\pm 0.0008}$	$0.0123\,{\scriptstyle\pm 0.0004}$	$0.295$	$0.146$	$0.0198\,{\scriptstyle\pm 0.0020}$	$0.00\,{\scriptstyle\pm 0.00}$
Ensemble TS	$0.0111\,{\scriptstyle\pm 0.0008}$	$0.0123\,{\scriptstyle\pm 0.0004}$	$0.295$	$0.146$	$0.0198\,{\scriptstyle\pm 0.0020}$	$0.00\,{\scriptstyle\pm 0.00}$
Vector Scaling	$0.0094\,{\scriptstyle\pm 0.0022}$	$0.0106\,{\scriptstyle\pm 0.0025}$	$\mathbf{0.288}$	$\mathbf{0.142}$	$0.0169\,{\scriptstyle\pm 0.0022}$	$+0.24\,{\scriptstyle\pm 0.16}$
Classwise TS	$0.0098\,{\scriptstyle\pm 0.0010}$	$0.0115\,{\scriptstyle\pm 0.0004}$	$0.294$	$0.145$	$0.0182\,{\scriptstyle\pm 0.0009}$	$0.00\,{\scriptstyle\pm 0.00}$
Parametric TS	$0.0113\,{\scriptstyle\pm 0.0010}$	$\underline{0.0094\,{\scriptstyle\pm 0.0005}}$	$0.298$	$0.146$	$0.0179\,{\scriptstyle\pm 0.0007}$	$0.00\,{\scriptstyle\pm 0.00}$

Appendix

Table 3: Descriptive 14-method benchmark on the Block-AR

b=2

retrain family (Swin-Tiny, CIFAR-10; 5,000-example test half; three training seeds). ECE-style metrics are mean

\pm

1

std; NLL and Brier are seed means, with paired deltas in Tab. 5 . Bold and underline mark the first two descriptive means; worst-tertile ECE uses

r_{\mathrm{std}}

Figure 5: Reliability diagrams on Block-AR for four substrates (rows: Swin-Tiny / DeiT-Small

\times

CIFAR-10 / CIFAR-100) under four calibrators (columns). Bars show empirical accuracy in each of

15

equal-width confidence bins; dashed line is perfect calibration; red shading marks over-confidence (predicted

>

accuracy) and blue marks under-confidence, both relative to the bin’s mean confidence. Per-panel global ECE is annotated in the upper-left corner. All cells use a seed-

0

=

299

checkpoint with the 50/50 cal/test split (seed

42

; the plotted half has

n=5{,}000

). On these uncalibrated panels the shading is predominantly blue. All four rows are seed-0 runs of the 24-run design, with calibrators refitted on their cached logits. The figure is descriptive and is not an input to any primary result.

Figure 6: Calibration-set-size sensitivity for the Swin-Tiny/CIFAR-10 Block-AR (

b{=}2

) seed-0 run. Curves show mean

\pm

1 std across 5 calibration subsamples per size with the 5,000-example test split held fixed.

Method	MCE $\downarrow$	Classwise ECE $\downarrow$	SmoothECE $\downarrow$
No calibration	$0.2441\,{\scriptstyle\pm 0.0019}$	$0.0325\,{\scriptstyle\pm 0.0010}$	$0.1512\,{\scriptstyle\pm 0.0035}$
Temp. Scaling (TS)	$0.1406\,{\scriptstyle\pm 0.0828}$	$0.0065\,{\scriptstyle\pm 0.0001}$	$0.0136\,{\scriptstyle\pm 0.0002}$
Ensemble TS	$0.1406\,{\scriptstyle\pm 0.0828}$	$0.0065\,{\scriptstyle\pm 0.0001}$	$0.0136\,{\scriptstyle\pm 0.0002}$
Vector Scaling	$0.0997\,{\scriptstyle\pm 0.0628}$	$0.0052\,{\scriptstyle\pm 0.0001}$	$0.0128\,{\scriptstyle\pm 0.0007}$
Classwise TS	$0.1046\,{\scriptstyle\pm 0.0214}$	$0.0061\,{\scriptstyle\pm 0.0002}$	$0.0136\,{\scriptstyle\pm 0.0003}$
Parametric TS	$0.0930\,{\scriptstyle\pm 0.0605}$	$0.0065\,{\scriptstyle\pm 0.0002}$	$0.0142\,{\scriptstyle\pm 0.0007}$

Appendix

Table 4: Additional calibration metrics on the same Block-AR

b=2

Swin-Tiny/CIFAR-10 family as Table 3 . Every entry is the mean

\pm

one sample standard deviation over training seeds 0, 1, and 2, using their fixed 5,000-example evaluation halves. MCE is the largest accuracy–confidence gap among 15 equal-width bins with at least five examples. Classwise ECE averages the ten one-versus-rest ECEs. SmoothECE is the kernel-smoothed metric. Rankings are descriptive and are not used for inference.

Method	$\Delta$ NLL vs raw $\downarrow$	$\Delta$ Brier vs raw $\downarrow$
Temp. Scaling (TS)	$-0.1320\,{\scriptstyle\pm 0.0050}$	$-0.0298\,{\scriptstyle\pm 0.0016}$
Ensemble TS	$-0.1320\,{\scriptstyle\pm 0.0050}$	$-0.0299\,{\scriptstyle\pm 0.0016}$
Vector Scaling	$\mathbf{-0.1387\,{\scriptstyle\pm 0.0051}}$	$\mathbf{-0.0331\,{\scriptstyle\pm 0.0019}}$
Classwise TS	$-0.1328\,{\scriptstyle\pm 0.0049}$	$-0.0304\,{\scriptstyle\pm 0.0016}$
Parametric TS	$-0.1272\,{\scriptstyle\pm 0.0044}$	$-0.0284\,{\scriptstyle\pm 0.0011}$
Histogram Binning	$-0.1029\,{\scriptstyle\pm 0.0068}$	$-0.0259\,{\scriptstyle\pm 0.0014}$

Appendix

Table 5: Paired proper-score changes relative to the uncalibrated model (Block-AR,

b=2

, Swin-Tiny, CIFAR-10, ep

=

299

; complement to Tab. 3 ). For each training seed

s

, we compute

\Delta\mathrm{NLL}_{m,s}=\mathrm{NLL}_{m,s}-\mathrm{NLL}_{\mathrm{raw},s}

and analogously for Brier, then report mean

\pm

1

std over seeds. This paired view isolates each calibrator’s within-seed change. The raw per-seed NLLs are

0.42519

0.43120

, and

0.42348

(sample std

0.00406

); the paired

\Delta

-stds are method-dependent and are not claimed to be uniformly smaller. Bold = most-negative (best) mean; underlined =

2

nd-best; rank markers use unrounded means.

Setting	Variant	Seed	$\Delta L$ , $r_{\rm agg}$	$\Delta L$ , profile	$p_{50}$	$p_{100}$	$p_{200}$	$p_{400}$
Swin/C-10	Block	0	$+0.11\;[-0.01,+0.24]$	$+0.34\;[-0.50,+1.22]$	0.500	0.625	0.653	0.798
Swin/C-10	Block	1	$+0.00\;[-0.10,+0.10]$	$-0.63\;[-1.60,+0.45]$	0.682	0.254	0.421	0.913
Swin/C-10	Block	2	$+0.01\;[-0.11,+0.13]$	$+0.16\;[-0.54,+0.86]$	0.774	0.623	0.491	0.083
Swin/C-10	Full	0	$-0.12\;[-0.53,+0.29]$	$-0.02\;[-1.27,+1.28]$	0.135	0.331	0.206	0.063
Swin/C-10	Full	1	$-0.30\;[-0.77,+0.17]$	$+0.42\;[-0.94,+1.75]$	0.434	0.740	0.329	0.345
Swin/C-10	Full	2	$-0.14\;[-0.48,+0.19]$	$-1.41\;[-2.81,+0.09]$	0.831	0.523	0.296	0.679

Appendix

Table 6: Continuous-confidence check on all 24 replicated runs (post-hoc extension of the original four representative runs).

\Delta L

is augmented minus confidence-only held-out binary log-loss (

10^{-3}

nats; negative favours routing) for five-fold cross-fitted logistic models on

\operatorname{logit}(c)

plus

r_{\rm agg}

or plus the whole profile, with 95% paired example-bootstrap intervals (2,000 draws); bold intervals exclude zero. Caliper columns give permutation

p

-values (1,000 permutations, smallest attainable

p=0.001

) for the association between

r_{\rm agg}

and correctness within consecutive confidence-sorted windows of 50, 100, 200 and 400 examples.

Statistic	Block-AR
Max gap (point)	$0.3455$
$95\%$ bootstrap CI	$[0.109,\,0.678]$
Weighted integrated gap (point)	$0.016$
$95\%$ bootstrap CI	$[0.012,\,0.030]$
Within-bin permutation $p$	$\mathbf{0.0382}$
Shared bins (of $15$ )	$12$

Appendix

Table 7: Routing-conditional max-gap robustness for Swin-Tiny / CIFAR-10 Block-AR seed 0. Intervals use 5000 bootstrap resamples; the permutation

p

-value uses 5000 within-bin permutations.

Variant	Seed	Acc@1	Max gap	Wt. gap	Wt. CI	Bins	Min/ $q_{25}$ /Med	Perm. $p$	Reject?
Block-AR ( $b\!=\!2$ )	$0$	$0.903$	$0.3455$	$0.016$	$[0.012,\,0.030]$	$12/15$	$11/86/143$	$\mathbf{0.0382}$	yes
Block-AR ( $b\!=\!2$ )	$1$	$0.901$	$0.1325$	$0.011$	$[0.011,\,0.028]$	$12/15$	$18/117/165$	$0.4771$	no
Block-AR ( $b\!=\!2$ )	$2$	$0.902$	$0.1000$	$0.017$	$[0.013,\,0.032]$	$12/15$	$25/134/171$	$0.6989$	no

Appendix

Table 8: Per-seed Swin-Tiny / CIFAR-10 diagnostic statistics for the three-run Block-AR comparison in Fig. 2 . All rows use the same 10,000-example test set and diagnostic protocol; Wt. CI is a percentile-bootstrap interval.

Mode	Multiplier / selection	Global ECE	Worst-tertile ECE	NLL
$0.5\!\times\!$ Scott	$0.50$	$0.0182\!\pm\!0.0039$	$0.0288\!\pm\!0.0080$	$0.4589\!\pm\!0.2307$
$1\!\times\!$ Scott	$1.00$ (paper default)	$0.0105\!\pm\!0.0041$	$0.0235\!\pm\!0.0074$	$0.4348\!\pm\!0.2343$
$2\!\times\!$ Scott	$2.00$	$0.0159\!\pm\!0.0085$	$0.0261\!\pm\!0.0102$	$0.4336\!\pm\!0.2398$
CV-NLL	$5$ -fold cal. NLL	$0.0129\!\pm\!0.0034$	$0.0244\!\pm\!0.0067$	$0.4317\!\pm\!0.2361$
Oracle-ECE	held-out global ECE	$0.0103\!\pm\!0.0037$	$0.0232\!\pm\!0.0069$	$0.4362\!\pm\!0.2372$

Appendix

Table 9: Bandwidth sensitivity for AR-CondCal. Pooled mean

\,\pm\,

std across the available eight CIFAR-10 AR checkpoints described above.

	Log-loss		Brier
Setting	$-$ Confidence	$-$ Shuffled	$-$ Confidence	$-$ Shuffled
Swin / CIFAR-10	$+1.243$	$-0.446$	$+0.296$	$-0.092$
Swin / CIFAR-100	$+0.664$	$-0.164$	$+0.219$	$-0.013$
DeiT / CIFAR-10	$+2.826$	$-0.425$	$+0.636$	$-0.073$
DeiT / CIFAR-100	$+0.394$	$-0.160$	$+0.114$	$-0.064$
Pooled (24 runs)	$+1.282$	$-0.299$	$+0.317$	$-0.061$

Appendix

Table 10: Entropy-profile probe: profile minus comparator in binary log-loss (

10^{-3}

nats) and Brier score (

10^{-3}

); negative values favour the profile. Setting rows are means over six runs; the pooled row is the mean over 24 runs with a 95% hierarchical-bootstrap interval (settings, then runs; 5,000 draws). Record-weighted and equal-setting pooling coincide in this balanced design. Four settings support only procedure-specific intervals.

Setting	gain	$n$	$p<0.05$	$p<0.05/30$	Gain	Recovery
	Oracle	Diagnostic ( $r_{\rm agg}$ )			NW probe ( $r_{\rm std}$ )
Swin / CIFAR-10	0	1000	0.042	0.001	$-5.24\;[-5.78,-4.70]$	–
Swin / CIFAR-10	0.0005	200	0.095	0.000	$-4.41\;[-4.83,-3.99]$	$-883$ %
Swin / CIFAR-10	0.002	200	0.170	0.010	$-2.88\;[-3.43,-2.34]$	$-144$ %
Swin / CIFAR-10	0.005	200	0.545	0.035	$+0.14\;[-0.30,+0.58]$	$3$ %
Swin / CIFAR-10	0.010	200	0.930	0.185	$+5.37\;[+5.03,+5.71]$	$54$ %

Appendix

Table 11: Scalar sensitivity audit on the Block-AR seed-0 run of each setting, at every injected strength (oracle binary log-loss gain, nats per example). Diagnostic : rejection rate of the binned

r_{\rm agg}

test (9,999 permutations per simulated dataset) under a teacher built from within-block ranks of

r_{\rm agg}

, at

\alpha=0.05

and at the family threshold

0.05/30

, recomputed from per-replicate

p

-values;

n

simulated datasets. NW probe : expected-label gain of NW

(c,r_{\rm std})

over NW

(c)

(

10^{-3}

nats; positive favours routing) under a teacher built from ranks of

r_{\rm std}

, with 95% Monte Carlo intervals over 50 replicates, and recovery

=

gain

/

oracle gain. Rejection rates and recovery are different quantities and the two teachers differ.

	Oracle gain $0.005$		Oracle gain $0.010$
Setting	Recovered	Fraction	Recovered	Fraction
Swin / CIFAR-10	$+0.001599$	$32.0\%$	$+0.005932$	$59.3\%$
Swin / CIFAR-100	$+0.001307$	$26.1\%$	$+0.005170$	$51.7\%$
DeiT / CIFAR-10	$+0.000285$	$5.7\%$	$+0.003290$	$32.9\%$
DeiT / CIFAR-100	$-0.000540$	$-10.8\%$	$+0.002436$	$24.4\%$

Appendix

Table 12: Profile expected-label binary log-loss recovery under the fixed interaction teacher: 20 repeats per setting at each strength. Positive gain is confidence-only loss minus Profile loss, in nats/example. Percentages divide recovery by the exact oracle log-loss gain.

	Profile $-$ output-only			Profile $-$ permuted profile
Setting	Selected	MLP	Ridge	MLP	Ridge
Swin / CIFAR-10	$+0.26$	$+1.40$	$+0.07$	$-0.82$	$-0.29$
Swin / CIFAR-100	$+0.00$	$+0.06$	$+0.29$	$-0.90$	$-0.07$
DeiT / CIFAR-10	$+3.21$	$+3.91$	$+0.12$	$-0.41$	$-0.50$
DeiT / CIFAR-100	$+0.00$	$+0.14$	$+0.35$	$-1.38$	$-0.34$
$K{=}10$ runs ( $n{=}12$ )	$+1.73$	$+2.66$	$+0.09$	$-0.62$	$-0.40$

Appendix

Table 13: Complete-output comparison on the real correctness labels: profile minus comparator in binary log-loss (

10^{-3}

nats); negative values favour the profile. Selected applies the selection rule, under which an arm falls back to the shared ridge output-only reference when inner validation does not support its MLP; on the

K{=}100

runs every selection fell back, so that column is exactly zero there. MLP and Ridge compare like learners without selection. Pooled rows as in Table 10 .

	Grid $\{10^{-4},10^{-3},10^{-2}\}$	Grid $\{10^{-4},\ldots,1\}$
Profile $-$ output-only, selected	$+0.87\;[-0.05,+2.46]$	$+0.35\;[-0.18,+1.20]$
Profile $-$ output-only, MLP	$+1.38\;[+0.05,+3.09]$	$+0.75\;[+0.25,+1.46]$
Profile $-$ output-only, ridge	$+0.21\;[-0.03,+0.46]$	$+0.21\;[-0.03,+0.46]$
Profile $-$ permuted, MLP	$-0.88\;[-1.52,-0.23]$	$-0.78\;[-1.19,-0.39]$
Profile $-$ permuted, ridge	$-0.30\;[-0.58,-0.03]$	$-0.30\;[-0.58,-0.03]$
Output-only MLP $-$ reference	$+2.45\;[-2.77,+7.68]$	$-0.14\;[-3.46,+3.02]$

Appendix

Table 14: Post-hoc weight-decay grid sensitivity for the complete-output comparison (24 runs refitted; only the MLP weight-decay grid changes, identically for all arms). Binary log-loss differences (

10^{-3}

nats; negative favours the first-named predictor) with pooled 95% hierarchical-bootstrap intervals. Ridge rows do not depend on the grid.

Dataset	Variant	Max gap	Permutation $p$
CIFAR-10	Block-AR	$0.1255$	$0.421$
CIFAR-10	Full-AR	$0.2222$	$0.511$
CIFAR-100	Block-AR	$0.0936$	$0.360$
CIFAR-100	Full-AR	$0.0672$	$0.827$
Tiny-ImageNet	Block-AR	$0.0898$	$0.449$
Tiny-ImageNet	Full-AR	$0.0984$	$0.318$

Appendix

Table 15: Compact ViT-B/16 routing diagnostic. Every row is one seed and is descriptive only. Max gap and

p

use the same fixed

r_{\mathrm{agg}}

bin-conditional diagnostic as the 30-test family.

Setting	Variant	$n$	Max gap	Min $p$ , $p<0.05$
Sw-T / C-10	Block-AR ( $b=2$ )	$3$	$0.1927\,{\scriptstyle\pm 0.1334}$	$0.0382$ , $1/3$
Sw-T / C-10	Full-AR	$3$	$0.1336\,{\scriptstyle\pm 0.0418}$	$0.5469$ , $0/3$
DeiT-S / C-10	Block-AR ( $b=1$ )	$3$	$0.1222\,{\scriptstyle\pm 0.0276}$	$0.2647$ , $0/3$
DeiT-S / C-10	Full-AR	$3$	$0.1152\,{\scriptstyle\pm 0.0154}$	$0.4951$ , $0/3$
DeiT-S / C-100	Block-AR ( $b=1$ )	$3$	$0.0941\,{\scriptstyle\pm 0.0208}$	$0.4519$ , $0/3$
DeiT-S / C-100	Full-AR	$3$	$0.0912\,{\scriptstyle\pm 0.0285}$	$0.0494^{\lx@sectionsign}$ , $1/3^{\lx@sectionsign}$

Appendix

Table 16: The 30-test diagnostic family aggregated by setting and variant. Replicated rows report mean

\pm

standard deviation of the maximum gap over three seeds, the smallest seed-level permutation

p

(5,000 permutations) and the number of seeds with

p<0.05

; ViT-B/16 rows (

\ast

) are single runs.

Figure 7: Layer-wise routing entropy after confidence matching. For a representative Swin-Tiny/CIFAR-10 Block-AR run (

b=2

, seed

0

), correct and incorrect predictions are matched

1{:}1

without replacement within fixed confidence bands with

|\Delta c|\leq 0.005

. Columns show the stored non-degenerate AR layers and colours show mean normalised routing entropy on a shared

[0,1]

scale. Row labels report sample counts and mean confidence; the low, medium, and high bands contain

530

362

, and

27

pairs, respectively. Correct and incorrect profiles are visually similar in the better-supported bands. High-confidence rows contain fewer than

50

pairs and are hatched and labelled as low support. This visualisation is descriptive and is not a conditional-independence test.

Figure 8: Layer-wise routing entropy differences after confidence matching. For the seed-

0

Block-AR runs of the four replicated backbone–dataset settings, correct and incorrect predictions are matched

1{:}1

without replacement within fixed confidence bands (

c<0.50

0.50\leq c<0.80

c\geq 0.80

;

|\Delta c|\leq 0.005

). Each cell shows the difference in mean per-layer routing entropy, incorrect minus correct, using a shared zero-centred colour scale: positive values indicate higher routing entropy for incorrect predictions. The better-supported low- and medium-confidence groups show small, setting-dependent differences, with no visually common depth-localised pattern across the four settings. Rows with fewer than

50

pairs are hatched and labelled as low support. Depth indices follow each architecture’s stored non-degenerate AR layers. This figure describes feature geometry in the matched subsets and is not a conditional-independence test; the replicated diagnostics in Sec. 4.1 provide the inferential evidence.

Figure 9: Routing-feature geometry on the Swin-Tiny/CIFAR-10 Block-AR (

b{=}2

) seed-0 run (accuracy

0.9026

;

9{,}026

correct and

974

incorrect of

10{,}000

), with the Full-AR seed-0 run for the depth comparison in (a) . Every quantity is computed from the routing caches of these runs. (a) Mean per-layer entropy at

8

versus

16

AR sub-layers. (b) Raw depth standard deviation

r_{\mathrm{std}}

, not rescaled, for correct versus incorrect predictions. (c) Marginal rank association between

r_{\mathrm{std}}

and the per-sample absolute residual

|\mathrm{conf}-\mathrm{correct}|

. That residual is not conditional correctness and not a population calibration error, and this panel is not a test of conditional independence.

CIFAR-10 (Swin-Tiny seed- $0$ , $b\!=\!2$ for Block-AR)
Feature	Block-AR ECE $\downarrow$	Full-AR ECE $\downarrow$
Confidence only	0.0131	0.0232
Predictive entropy	0.0092	0.0199
Aggregate routing entropy $r_{\mathrm{agg}}$	0.0099	0.0241
Last-layer routing entropy $H_{L}$	0.0075	0.0191
Routing concentration $1\!-\!r_{\mathrm{agg}}$	0.0099	0.0241

Appendix

Table 17: Feature ablation across auxiliary

1

-D summaries and controls (Swin-Tiny seed-

0

). Confidence-only uses the genuinely

1

-D

\mathrm{NW}(c)

control; every auxiliary-feature row uses the corresponding

2

-D

\mathrm{NW}(c,f)

estimator under the same split and kernel convention. Confidence and predictive entropy are non-routing controls;

1-r_{\mathrm{agg}}

is redundant with

r_{\mathrm{agg}}

. Across these point estimates, no routing-derived scalar consistently outperforms the controls, and the best routing summary changes across substrates. The table is descriptive: it neither establishes cross-seed equivalence nor rules out other scalar projections.

Setting	AR variant	seed 0	seed 1	seed 2
DeiT-Small / CIFAR-10	Block-AR	0.6947	0.9055	0.9011
DeiT-Small / CIFAR-10	Full-AR	0.8972	0.9180	0.9103
DeiT-Small / CIFAR-100	Block-AR	0.6842	0.7013	0.6880
DeiT-Small / CIFAR-100	Full-AR	0.7207	0.7205	0.7058
Swin-Tiny / CIFAR-10	Block-AR	0.9026	0.9012	0.9024
Swin-Tiny / CIFAR-10	Full-AR	0.7444	0.9242	0.9239

Appendix

Table 18: Test accuracy of every run in the 24-run design (full 10,000-example test set). Two lower-accuracy seed-0 runs (Swin-Tiny Full-AR, 0.744, and DeiT-Small Block-AR, 0.695) are retained under the fixed inclusion rule rather than excluded post hoc by realised accuracy; both reached epoch 299 but did not follow the standard schedule (App. O ).

Explore similar work

May 11, 2026cs.CV

Probing Routing-Conditional Calibration in Attention-Residual Transformers

Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly accompany predictions with sample-specific internal routing traces and pair them with claims of calibration-relevant uncertainty. We ask a basic question: do these traces provide stable routing-specific evidence for post-hoc calibration beyond confidence? We study this in Attention-Residual transformers (Kimi Team, 2026) through a matched-confidence diagnostic suite that stratifies examples by routing-derived state, compares subgroup gaps against within-bin routing-permutation nulls, and evaluates matched post-hoc probes differing only in their auxiliary feature. Across our completed AR runs, scalar routing summaries do not provide stable evidence of routing-conditional miscalibration: weighted gaps remain small or seed-sensitive, and only

1

30

within-bin permutation tests rejects the conditional-null at

α=0.05

(only on one seed; not stable across seeds in that cell). AR-CondCal, a minimal

2

-D Nadaraya--Watson probe on confidence and routing-depth variance, lies within the seed-variance band of matched confidence-only and predictive-entropy controls and does not reliably improve worst-routing-tertile ECE; bandwidth-sensitivity checks (Scott multiples, CV-NLL, global-ECE oracle) do not change this. A full-vector MLP over

(c, H_1, \ldots, H_L)

can appear to improve over a linear confidence baseline, but the apparent gain disappears once a capacity-matched confidence-only MLP is included as a control, and shuffled routing profiles achieve comparable performance. Apparent routing-aware calibration gains in this AR setting should not be read as internal-state calibration until matched-confidence, bandwidth, capacity, and permutation controls rule out common confounds.

Wenhao Liang, Lin Yue, Wei Emma Zhang +4

Adelaide University · Australian Institute for Machine Learning (AIML), Adelaide University · The University of Queensland

Post-Hoc Transformer Architectures

Ask PDF

Sep 30, 2026cs.AI

Routing Probes Can Improve Without New Information: An Exact-Null Audit of Uncertainty Beyond Model Outputs

Routing signals of modern vision transformers -- expert gates, attention-residual weights and halting scores -- often improve probes that predict whether the model is correct, and the improvement is commonly read as evidence that routing carries information about errors beyond the model's outputs. We test this inference directly: keeping real output-routing pairs, we redraw correctness labels from a frozen output-only generator fitted on disjoint data, so that routing is uninformative by construction. Under this exact label null, a width-matched MLP comparison still reports a routing gain in 51.3% of confidence-only evaluations (308/600), while a linear comparison reports none. Holding each training trajectory fixed on a six-model panel and selecting the checkpoint by validation log loss instead of validation accuracy removes the detections (50/120 to 0/120, and 83/120 to 0/120 in an independently implemented probe), identifying accuracy-based checkpoint selection as the cause; across all output views the raw detection rate falls from 27.5% (528/1,920) to zero observed detections. The repaired comparison is not sensitive, detecting an implanted signal of about 0.005 nats in 0/20 replicates in each of two matched settings, whereas a conditional permutation test built on an estimated routing law detects it in 11/20 and 10/20 and rejects rarely under the null. On real correctness labels, the conditional analysis yields model-relative evidence in five DeiT attention-residual families; in four it persists under two specified variants of the conditional law, and no family passes an additional noise criterion. Fitting a better probe and testing for incremental information are different problems, and each needs its own validation.

Wenhao Liang, Lin Yue, Wei Emma Zhang +3

Detection Performance Model Checkpoints Ground Truth Model Auditing Multilayer Perceptrons Uncertainty

Ask PDF

Jun 11, 2026cs.LG

When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals

Block Attention Residuals (Block AttnRes) by replace fixed additive residuals with a learned softmax over earlier depth-source representations, surfacing cross-layer routing as an inspectable tensor in the forward pass. This is a tempting interpretability target: information flow normally inferred indirectly is now directly observable. We ask whether such exposure suffices for mechanistic interpretation. We probe two same-scale (

0.6

B) Block AttnRes checkpoints under identical routing-ablation interventions: a vanilla Qwen3 inference-wrapped through a deterministic recency-bias schedule that the codebase admits as a routing-equivalent loading path, and a Block AttnRes Qwen3 trained from scratch with routing as part of optimisation. The wrapped baseline's routing weights are content-independent and reproduce the schedule's analytic prediction. The trained AttnRes checkpoint instead exhibits three localised routing motifs: an embedding-source pathway through early-layer MLP, a current-state pathway through early-layer attention and MLP, and an older-history pathway through late-layer attention. Beyond this stratification, we find a sharp dissociation between average routing mass and causal importance: in both sublayers, the largest mass slice is not the largest causal contribution, and one source family carries appreciable mass with no detectable causal role under intervention. Architectural exposure of routing is therefore necessary but not sufficient for mechanistic interpretation: structured depth routing emerges only when routing has been part of training, and even then, descriptive routing summaries should be treated as candidate hypotheses to be tested by causal interventions, not as evidence of mechanism in their own right.

Aydin Javadov

ETH Zurich

Model Checkpoints

Ask PDF