Routing Probes Can Improve Without New Information: An Exact-Null Audit of Uncertainty Beyond Model Outputs

Wenhao Liang; Lin Yue; Wei Emma Zhang; Mingyu Guo; Olaf Maennel; Weitong Chen

cs.AISep 30, 2026

Routing Probes Can Improve Without New Information: An Exact-Null Audit of Uncertainty Beyond Model Outputs

Authors: Wenhao Liang, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen

Abstract

Routing signals of modern vision transformers -- expert gates, attention-residual weights and halting scores -- often improve probes that predict whether the model is correct, and the improvement is commonly read as evidence that routing carries information about errors beyond the model's outputs. We test this inference directly: keeping real output-routing pairs, we redraw correctness labels from a frozen output-only generator fitted on disjoint data, so that routing is uninformative by construction. Under this exact label null, a width-matched MLP comparison still reports a routing gain in 51.3% of confidence-only evaluations (308/600), while a linear comparison reports none. Holding each training trajectory fixed on a six-model panel and selecting the checkpoint by validation log loss instead of validation accuracy removes the detections (50/120 to 0/120, and 83/120 to 0/120 in an independently implemented probe), identifying accuracy-based checkpoint selection as the cause; across all output views the raw detection rate falls from 27.5% (528/1,920) to zero observed detections. The repaired comparison is not sensitive, detecting an implanted signal of about 0.005 nats in 0/20 replicates in each of two matched settings, whereas a conditional permutation test built on an estimated routing law detects it in 11/20 and 10/20 and rejects rarely under the null. On real correctness labels, the conditional analysis yields model-relative evidence in five DeiT attention-residual families; in four it persists under two specified variants of the conditional law, and no family passes an additional noise criterion. Fitting a better probe and testing for incremental information are different problems, and each needs its own validation.

Figures & tables

Figure 1: Setup of the audit. In one forward pass a trained vision transformer $f$ maps an input $x$ to an output view $O_{j}$ ( $j=0,\dots,3$ ) and to $R\in\mathbb{R}^{L_{f}}$ , its per-layer routing entropies. A logistic generator $g(O_{0})$ is fitted to real correctness on half $A$ and frozen; on the disjoint half $B_{\rm test}$ , $T^{*}_{i}\sim\mathrm{Bernoulli}\{g(O_{0,i})\}$ . Every view determines $O_{0}$ and $R$ is not used for the labels, so $T^{*}\perp R\mid O_{j},A$ . Probes with the same architecture and input width are cross-fitted on $B_{\rm test}$ to predict $T^{*}$ from $(O_{j},0)$ and from $(O_{j},R)$ and scored by held-out log loss or Brier score $\widehat{L}_{S}$ ; the RAW gain $\widehat{\Delta}_{\rm raw}$ of Eq. ( 2 ) is positive when routing lowers the loss. The families differ in the reference for the probe on $(O_{j},R)$ : the probe on $(O_{j},0)$ (RAW); the same probe refitted after a uniform (GLOBAL) or output-matched (MATCHED) permutation $\pi$ of routing rows; or refits on 79 permutations drawn with weights $\prod_{i}\hat{q}_{j}(R_{\pi(i)}\mid O_{j,i})$ , with $\hat{q}_{j}$ fitted on $A$ (CPT, Eqs. ( 3 )–( 4 )). The input image is illustrative.

Comparator	$n$	$n_{00}$	$n_{01}$	$n_{10}$	$n_{11}$	acc (%)	nll (%)
RAW	1920	1392	0	528	0	27.50	0.00
GLOBAL	1920	1598	14	295	13	16.04	1.41
MATCHED	1920	1617	10	277	16	15.26	1.35
CPT	1920	1872	18	26	4	1.56	1.15

Table 1: All 1,920 common-generator MLP evaluations under both selection policies, paired on identical labels, splits, permutation assignments and bootstrap draws.

n_{ab}

counts evaluations flagged by the accuracy policy (

a

) and by the log-loss policy (

b

); the accuracy column remains the behaviour of the originally defined procedure. Linear results are unchanged, because the linear probe has no validation-based selection, and are not shown. Zero detections in 1,920 dependent evaluations should not be read as a zero population detection probability. Per-view rows: Table 14 .

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Requirement (§ 5 )	This study
Fit and select both probes by the score you report	Log loss and Brier co-primary, out-of-fold (5-fold $\times$ 2 repeats); probes dimension-matched by zero-padding, which does not match fitting (Hazard I, App. F ). The original MLP selected checkpoints by validation accuracy; the corrected log-loss policy and its benchmark-wide re-analysis are in App. K .
Use a conditional reference	Estimated- $q$ CPT with 79 references (App. M ); GLOBAL and post-hoc MATCHED shuffles are assignment controls and the Gaussian block a representation control (Hazard II); standalone shuffle detections in five families, none meeting the frozen screen.
State what the output view contains	Four views $O_{0}$ – $O_{3}$ ; sorted views omit class identity; routing representation stated (entropy profile; raw $\alpha$ on 12 runs, App. D ).
Report operating characteristics for the rule you actually use	Common-generator null rates (App. L ), independent-split power at $O_{1}$ and the matched panel (App. M , K ); three seeds per family (five for one); output positive control 22/22; family-rule power of R1/R2 not established.

Appendix

Table 2: The audit against the four requirements of § 5 : what was done for each, and the inferential limit that remains.

Mechanism	Families	Runs	Positive-control gain	Max naive gain	Frozen full pass
			(range)	(max)	(families)
Attention-residual block	9	27	$+0.0037$ to $+0.0149$	$+0.0561$	0 / 9
Attention-residual full	9	29	$+0.0037$ to $+0.0189$	$+0.0337$	0 / 9
MoE top-1 gate	2	6	$+0.0063$ to $+0.0107$	$+0.0693$	0 / 2
Adaptive-depth halting	2	6	$+0.0088$ to $+0.0088$	$+0.0511$	0 / 2
Total	22	68	22/22 detected	up to $+0.069$	0 / 22

Appendix

Table 3: Positive controls pass, but no family meets the frozen full criterion. Panel A summarizes 22 families and 68 training runs; Panel B separately reports two public pretrained checkpoints. Gains are held-out log-loss improvements in nats.

Figure 2: Large probe gains need not survive routing-specific controls. All 22 families use the same MLP and

O_{0}

; points are mean held-out log-loss contrasts across training seeds, with positive values favouring real routing. Families are ordered by mechanism, backbone and dataset, independently of outcomes. The full screen requires both scores and seed replication: 0/22 families pass across all views and both probes, which does not establish absence of incremental information.

Statistic	Cell	Probe	Reference gain of the synthetic signal (nats)
			0 (null)	0.0005	0.001	0.002	0.005	0.01
CPT	DeiT/C-100/AR-block	linear	4/200	14/200	32/200	86/200	191/200	200/200
CPT	DeiT/C-100/AR-block	mlp	1/50	2/50	2/50	6/50	19/50	46/50
CPT	Swin/C-10/AR-block	linear	1/200	25/200	51/200	129/200	200/200	200/200
CPT	Swin/C-10/AR-block	mlp	0/50	0/50	1/50	3/50	8/50	21/50
CPT $\wedge$ noise	DeiT/C-100/AR-block	linear	1/200	1/200	4/200	22/200	133/200	198/200

Appendix

Table 4: Independent-split operating characteristics at view

O_{1}

\hat{q}

is fitted on 5,000 examples and every statistic is evaluated on the 5,000 disjoint examples. MLP rows use the original accuracy-based stopping and selection; linear rows involve no validation-based selection. Columns are the reference gain of the synthetic routing signal, a model-based quantity that is not an observed conditional-reference contrast. Rows are per-replicate single tests, not the cross-seed R1 or R2 family rules; the linear rows use 200 replicates and the MLP rows 50. The second block is the stored conjunction, CPT

\wedge

noise on the same test rows: a per-replicate quantity, not the cross-seed R2 rule.

Mechanism	Backbone	Data	Runs	Pos. ctrl	$O_{0}$	$O_{1}$	$O_{2}$	$O_{3}$
AR-block	Swin-T	C-10	3	$+0.0037$	$+0.0004$	$+0.0161$	$-0.0003$	$-0.0000$
AR-full	Swin-T	C-10	3	$+0.0037$	$+0.0010$	$+0.0152$	$+0.0048$	$-0.0003$
AR-block	Swin-T	C-100	3	$+0.0075$	$+0.0008$	$+0.0001$	$+0.0006$	$+0.0019$
AR-full	Swin-T	C-100	3	$+0.0100$	$+0.0006$	$-0.0004$	$-0.0001$	$-0.0007$
AR-block	DeiT-S	C-10	3	$+0.0050$	$+0.0008$	$-0.0003$	$+0.0092$	$-0.0000$
AR-full	DeiT-S	C-10	3	$+0.0064$	$+0.0007$	$+0.0000$	$-0.0002$	$-0.0001$

Appendix

Table 5: No family meets the original full criterion despite positive controls in every row. All 22 families are retained, with held-out log-loss improvements in nats.

Figure 3: Two control hazards motivate separate representation and assignment checks. A: a single Swin/AR-block/C10 seed-0 MLP compares each appended block with its own width-matched padded baseline (routing width 8, output-extra width 4); intervals are paired 95% bootstrap intervals. B: token-averaged routing weights are compared with noise and GLOBAL controls; labels count positive log-loss intervals among three seeds, not complete two-score passes. Both panels are descriptive; neither identifies optimisation error as the sole cause.

Benchmark	Label generator	Evaluation size	Exact null?	Primary role
Historical full-sample (App. G )	fold-specific $\hat{\eta}_{F_{i}}(O_{0,i})$ , seed-11 folds	10,000 examples; 6 cells $\times$ 200 draws	no (conditional on fold)	historical null counts, Table 7
Historical fold-specific 5k (App. M.4 )	same recipe on the 5k test half	5,000 examples; 1,920 label records	no	historical CPT calibration, Tables 21 – 22
Common-generator (App. L )	one frozen $g(O_{0})$ fitted on half A	5,000 examples; 600 label vectors, 3,840 evaluations	yes, given $g$	primary null validity; Table 1

Appendix

Table 6: The three label-null benchmarks. Only the common-generator benchmark draws labels from a single frozen function of

O_{0}

, which is what makes its null exact conditional on that generator.

	Linear		MLP
comparison	count	rate	count	rate
RAW	$0/1200$	$0.000$	$448/1200$	$0.373$
GLOBAL	$10/1200$	$0.008$	$165/1200$	$0.138$
MATCHED	$5/1200$	$0.004$	$149/1200$	$0.124$
NOISE	$3/1200$	$0.003$	$477/1200$	$0.398$
joint (RAW $\wedge$ GLOBAL $\wedge$ NOISE)	$0/1200$	$0.000$	$35/1200$	$0.029$

Appendix

Table 7: Historical full-sample detections concentrate in the nonlinear probe. Counts at

O_{0}

compare the original fold-specific label construction.

Figure 4: Historical component-level power does not establish CPT family-rule power. RAW and GLOBAL curves show two routing widths and three sample sizes from the original synthetic benchmark, with 200 repeats per point; the full sample-size grid remains in App. I . The independent-split linear CPT endpoint is separate (App. M ); neither component curves nor that endpoint establish conjunction power.

Family	Probe	View	GLOBAL detected		Matched	Matched	Reps	Fails
			audit	second	contrast	detected
AR-block/DeiT-S/C-100	Linear	O0	yes	yes	$+0.0017$	no	11/20	raw+noise
AR-block/DeiT-S/C-100	MLP	O0	yes	no	$+0.0015$	no	0/20	raw
AR-block/DeiT-S/C-10	MLP	O2	yes	no	$+0.0036$	no	1/20	noise
AR-full/DeiT-S/C-10	MLP	O2	no	no	$+0.0009$	yes	1/20	noise
MoE gate/DeiT-S/C-10	MLP	O1	yes	no	$+0.0039$	no	9/20	raw

Appendix

Table 8: No matched-shuffle candidate meets the full criterion. All combinations flagged by the original GLOBAL draw, independent GLOBAL draw or MATCHED control are retained.

View	MATCHED shuffle		GLOBAL shuffle		Unmatched
	median	p95	median	p95	rate
$O_{0}$	0.004	2.191	0.956	2.760	0.000
$O_{1}$	0.744	4.828	2.695	5.277	0.000
$O_{2}$	4.656	23.870	10.172	26.510	0.000
$O_{3}$	8.393	21.561	12.131	23.150	0.000

Appendix

Table 9: MATCHED assignments are more local than GLOBAL assignments at every output view. Distances are summarised across all 22 families.

Arm	Flags	Gain	vs $g$ (nats)		Selected ep.	Run
		(nats)	output	$+$ R	out / $+$ R	ep.
A: accuracy stopping and selection (stock)	57/120	+0.0169	+0.0365	+0.0196	11 / 19	27
B: log-loss stopping and selection	0/120	-0.0071	+0.0008	+0.0079	35 / 40	45
C acc : 100 epochs, accuracy checkpoint	50/120	+0.0119	+0.0264	+0.0144	18 / 37	100
C nll : 100 epochs, log-loss checkpoint	0/120	-0.0073	+0.0007	+0.0080	56 / 42	100

Appendix

Table 10: Instrumented scikit-learn MLP under the label null: six cells

\times

20 label draws at

O_{0}

; only the validation score changes. Arm A reproduces the stored estimator bit for bit; the C arms share one 100-epoch trajectory per fit and differ only in the checkpoint selected on the inner validation split. Gain is the mean fitted log-loss contrast; “vs

g

” is the mean held-out loss minus the frozen generator’s loss on the same label draw (a diagnostic, not a population excess risk); selected and run epochs are means over fits. Flags count the 120 draws of this fixed panel, which are not independent draws of models (Table 11 ).

	Flagged (of 20)				Output-only excess vs $g$ (nats)
Cell	A	B	C acc	C nll	C acc	C nll
MoE/DeiT/C-10	7/20	0/20	4/20	0/20	+0.0122	+0.0005
AR-block/Swin/C-10	1/20	0/20	2/20	0/20	+0.0141	+0.0005
AR-block/DeiT/C-100	6/20	0/20	4/20	0/20	+0.0144	+0.0009
Halting/DeiT/C-100	20/20	0/20	20/20	0/20	+0.0630	+0.0008
AR-block/ViT/C-100	3/20	0/20	1/20	0/20	+0.0160	+0.0006

Appendix

Table 11: Checkpoint-selection arms, per cell. Twenty label draws per cell at

O_{0}

with the MLP probe. Flagged counts are raw detections under arm A (original accuracy policy), B (log-loss policy) and the two fixed-trajectory arms C acc and C nll ; the excess columns are the fixed-trajectory arms’ mean held-out log loss of the output-only probe minus the frozen generator’s loss on the same labels.

	scikit-learn null		PyTorch null			PyTorch, null with Gaussian routing
Cell	acc	nll	acc	nll	last	acc	nll	last
AR-block/DeiT/C-100	4	0	20	0	0	19	0	0
AR-block/Swin/C-10	2	0	0	0	0	4	0	0
AR-full/DeiT/TIN	19	0	18	0	0	14	0	0
AR-block/ViT/C-100	1	0	20	0	0	19	0	0
MoE/DeiT/C-10	4	0	5	0	0	4	0	0

Appendix

Table 12: Checkpoint selection on fixed 100-epoch trajectories in two implementations, view

O_{0}

, MLP probe. Entries are raw-comparison flags (both bootstrap intervals above zero). scikit-learn: the instrumented stock estimator, arms C acc and C nll (Table 10 ). PyTorch: an independently implemented MLP with the same width, penalty definition, optimiser settings, folds, labels and scoring, read at the checkpoint with the best validation accuracy (acc), the lowest validation log loss (nll) or the final epoch (last). Null: labels from the frozen output-only generator with real routing (20 draws per cell) or with routing replaced by independent Gaussian features of the same width (Gaussian routing, same labels). Implanted signal: 10 draws per cell and strength.

Evidence / check	Setting	Result	Interpretation
Validation
Exact-label-null CPT	Common-generator benchmark	59/3,840 (1.54%)	All pooled gates pass; max cell 4%
Null comparator detections	RAW / GLOBAL / MATCHED	528 / 325 / 305 of 3,840	Contrast concentrated in MLP
Same, log-loss selection	MLP half re-run, paired	0 / 27 / 26 of 1,920; CPT 22	Selection rule, not function class
Power adequacy	Independent 5k/5k split	200/200 + 200/200	Linear endpoint only
Observed-label evidence

Appendix

Table 13: Validation checks and observed-label evidence establish different claims. The common-generator benchmark assesses null detections, under the original accuracy-based checkpoint selection and, for the MLP half, under log-loss selection; the independent-split benchmark assesses a linear power endpoint. R1 and R2 report distinct evidential criteria.

Comparator	View	$n$	$n_{00}$	$n_{01}$	$n_{10}$	$n_{11}$	acc (%)	nll (%)
RAW	$O_{0}$	600	292	0	308	0	51.33	0.00
	$O_{1}$	600	508	0	92	0	15.33	0.00
	$O_{2}$	360	303	0	57	0	15.83	0.00
	$O_{3}$	360	289	0	71	0	19.72	0.00
	all	1920	1392	0	528	0	27.50	0.00
GLOBAL	$O_{0}$	600	479	2	112	7	19.83	1.50

Appendix

Table 14: Table 1 resolved by output view. Views share label draws, so rows within a comparator are not independent.

	Accuracy policy	Log-loss policy
Seed-level CPT flags (recomputed runs)	19/272	21/272
both / accuracy only / log-loss only / neither	9 / 10 / 12 / 241
R1 cells (family–view)	2/88	2/88
Mean conditional contrast, recomputed runs (nats)	+0.000379	+0.001328

Appendix

Table 15: Observed-label MLP tests recomputed under log-loss checkpoint selection with the same conditional model, split and reference assignments, for all 68 runs (272 run–view tests). Seed-level counts are single tests. R1 is positive in 2 and 2 of 88 family–view cells under the two policies, but not in the same cells: AR-block/DeiT/C-100 at

O_{0}

becomes R1-positive; AR-full/DeiT/C-100 at

O_{2}

loses R1. Post-hoc re-analysis; the accuracy-policy results are those of the pre-specified procedure.

Cell	Ref. gain	Selection	RAW	Mean RAW	GLOBAL	MATCHED	CPT	CPT $\wedge$
	(nats)		flags	gain (nats)	flags	flags	flags	noise
Swin/C-10	0	accuracy	11/20	$+0.0267$	7/20	7/20	0/20	0/20
		log loss	0/20	$-0.0090$	0/20	0/20	0/20	0/20
Swin/C-10	0.0049	accuracy	16/20	$+0.0501$	13/20	11/20	3/20	3/20
		log loss	0/20	$-0.0040$	4/20	3/20	11/20	6/20
Swin/C-10	0.0098	accuracy	16/20	$+0.0654$	18/20	16/20	10/20	9/20

Appendix

Table 16: Matched synthetic panel at view

O_{1}

, MLP probe: the first 20 replicates of each setting of the independent-split benchmark (Table 4 ) analysed under the original accuracy-based stopping-and-selection policy and under the log-loss policy, sharing labels, routing, folds, controls, conditional model and reference assignments. Counts are per-replicate flags out of 20; the mean raw gain is the average fitted log-loss contrast over the 20 replicates. The reference gain is a property of the implanted synthetic signal, not an observed conditional-reference contrast. CPT

\wedge

noise is a per-replicate conjunction, not the cross-seed R2 rule. Twenty paired replicates give coarse resolution.

Reference gain (nats)	Arm	Flagged	$+$ R vs $g$ (nats)
0.0050	A: accuracy stopping and selection (stock)	29/60	+0.0163
0.0050	B: log-loss stopping and selection	0/60	+0.0069
0.0050	C acc : 100 epochs, accuracy checkpoint	20/60	+0.0141
0.0050	C nll : 100 epochs, log-loss checkpoint	0/60	+0.0069
0.0101	A: accuracy stopping and selection (stock)	37/60	+0.0107
0.0101	B: log-loss stopping and selection	2/60	+0.0020

Appendix

Table 17: Positive control with the same pipeline: labels and a synthetic routing matrix

R_{\lambda}=\lambda(T-\eta)e_{1}+\varepsilon

carrying a known conditional signal, six cells

\times

10 replicates per strength. Log-loss selection removes the null detections of Table 10 and also detects almost nothing here, while accuracy selection flags at close to its null rate either way. The last column shows that the augmented probe remains worse than the generator even where real signal exists.

Figure 5: Probe comparisons under the exact label null of Fig. 1 , and the role of checkpoint selection. A: percentages flagged by RAW, GLOBAL and MATCHED (both 95% bootstrap intervals above zero) and by CPT (both $p$ -values at most $0.025$ ). MLP results use accuracy-based checkpoint selection. 600 evaluations per probe at $O_{0}$ and $O_{1}$ and 360 at $O_{2}$ and $O_{3}$ , sharing 600 label vectors. B: the same six models at $O_{0}$ , 20 label draws each: every MLP fit runs for 100 epochs and its trajectory is read at the checkpoint with the best validation accuracy and at the one with the lowest validation log loss, in the stock scikit-learn estimator and in an independent PyTorch implementation. Right: mean held-out loss of the output-only probe minus the frozen generator’s loss on the same label draws, a diagnostic rather than a population excess risk (Tables 10 and 12 ).

Model	$\beta_{0}$	$\beta_{1}$	Mean p	SD p	Observed $T$	Log loss	Brier
MoE/DeiT/C10	0.807199	1.972213	0.9228	0.1354	0.9224	0.1840	0.0534
ARb/Swin/C10	0.989538	1.882043	0.9035	0.1605	0.9042	0.2156	0.0665
ARb/DeiT/C100	0.850323	1.240079	0.6767	0.2773	0.6922	0.4312	0.1416
ACT/DeiT/C100	0.909690	1.316111	0.7346	0.2551	0.7356	0.4072	0.1322
ARb/ViT/C100	0.752829	1.321237	0.7106	0.2729	0.7234	0.4102	0.1336
ARf/DeiT/TIN	0.568209	1.046911	0.5468	0.2906	0.5390	0.5082	0.1689

Appendix

Table 18: A single fitted generator per model defines the label benchmark. Coefficients use A; descriptive probability and score diagnostics use B.

View	Probe	Denominator	RAW	GLOBAL	MATCHED	CPT
O0	Linear	600	0 $\to$ 0	2 $\to$ 7	5 $\to$ 5	12 $\to$ 8
O0	MLP	600	286 $\to$ 308	109 $\to$ 119	92 $\to$ 91	13 $\to$ 10
O1	Linear	600	0 $\to$ 0	3 $\to$ 5	3 $\to$ 5	9 $\to$ 10
O1	MLP	600	102 $\to$ 92	93 $\to$ 85	85 $\to$ 94	14 $\to$ 10
O2	Linear	360	0 $\to$ 0	3 $\to$ 4	2 $\to$ 1	7 $\to$ 6
O2	MLP	360	59 $\to$ 57	63 $\to$ 52	67 $\to$ 53	6 $\to$ 8

Appendix

Table 19: The nonlinear contrast persists across label generators, while local comparator ordering changes. Entries compare historical fold-specific

\to

common-generator detections.

Model	Probe	O0	O1	O2	O3
MoE/DeiT/C10	Linear	2/100 (2.00%)	3/100 (3.00%)	1/60 (1.67%)	1/60 (1.67%)
MoE/DeiT/C10	MLP	1/100 (1.00%)	0/100 (0.00%)	2/60 (3.33%)	0/60 (0.00%)
ARb/Swin/C10	Linear	1/100 (1.00%)	1/100 (1.00%)	1/60 (1.67%)	0/60 (0.00%)
ARb/Swin/C10	MLP	1/100 (1.00%)	2/100 (2.00%)	1/60 (1.67%)	0/60 (0.00%)
ARb/DeiT/C100	Linear	1/100 (1.00%)	1/100 (1.00%)	1/60 (1.67%)	2/60 (3.33%)
ARb/DeiT/C100	MLP	2/100 (2.00%)	4/100 (4.00%)	1/60 (1.67%)	0/60 (0.00%)

Appendix

Table 20: Every common-generator CPT cell meets the pre-specified ceiling. All 48 model–view–probe cells are shown.

View	Probe	RAW	GLOBAL	MATCHED	CPT
O0	Linear	0/600 (0.00%)	2/600 (0.33%)	5/600 (0.83%)	12/600 (2.00%)
O0	MLP	286/600 (47.67%)	109/600 (18.17%)	92/600 (15.33%)	13/600 (2.17%)
O1	Linear	0/600 (0.00%)	3/600 (0.50%)	3/600 (0.50%)	9/600 (1.50%)
O1	MLP	102/600 (17.00%)	93/600 (15.50%)	85/600 (14.17%)	14/600 (2.33%)
O2	Linear	0/360 (0.00%)	3/360 (0.83%)	2/360 (0.56%)	7/360 (1.94%)
O2	MLP	59/360 (16.39%)	63/360 (17.50%)	67/360 (18.61%)	6/360 (1.67%)

Appendix

Table 21: Historical fold-specific controls differ in detection frequency. Counts compare identical test halves.

Mechanism/backbone/data	Probe	O0	O1	O2	O3
MoE/DeiT/C10	Linear	1/100 (1.00%)	1/100 (1.00%)	0/60 (0.00%)	0/60 (0.00%)
MoE/DeiT/C10	MLP	2/100 (2.00%)	1/100 (1.00%)	3/60 (5.00%)	1/60 (1.67%)
ARb/Swin/C10	Linear	1/100 (1.00%)	0/100 (0.00%)	0/60 (0.00%)	0/60 (0.00%)
ARb/Swin/C10	MLP	3/100 (3.00%)	2/100 (2.00%)	2/60 (3.33%)	0/60 (0.00%)
ARb/DeiT/C100	Linear	4/100 (4.00%)	2/100 (2.00%)	1/60 (1.67%)	1/60 (1.67%)
ARb/DeiT/C100	MLP	2/100 (2.00%)	2/100 (2.00%)	0/60 (0.00%)	2/60 (3.33%)

Appendix

Table 22: Historical CPT cell rates vary within the pre-specified ceiling. All six models, four views and two probes are shown (48 cells).

DeiT family	View	Probe	$\Delta_{\rm cond,log}$	$\Delta_{\rm cond,Brier}$	R1 seeds	R2 seeds
C100 block	O0	Linear	+0.003954	+0.001460	3/3	1/3
C100 block	O1	Linear	+0.002577	+0.000983	3/3	0/3
C100 block	O2	Linear	+0.002768	+0.001076	2/3	0/3
C100 block	O3	Linear	+0.002926	+0.001188	3/3	0/3
C100 full	O0	Linear	+0.001854	+0.000636	2/5	1/5
C100 full	O0	MLP	+0.005424	+0.001585	3/5	1/5

Appendix

Table 23: R1 evidence appears in five families, while the additional same-width robustness criterion R2 remains unmet. All 12 R1-positive family–probe–view cells are shown, under the primary working model Q1; MLP cells use the original accuracy-based checkpoint selection (log-loss sensitivity: Table 15 ; working-model sensitivity: Table 26 ).

DeiT family	View	$\Delta$ LL	Mahalanobis KS	PIT median KS	Acceptance	Hamming
C100 block	O0	0.013–0.016	0.146–0.171	0.014–0.016	0.477–0.484	0.99977–0.99981
C100 block	O1	0.079–0.093	0.157–0.183	0.014–0.015	0.426–0.436	0.99974–0.99975
C100 block	O2	0.067–0.075	0.154–0.182	0.013–0.015	0.404–0.408	0.99969–0.99980
C100 block	O3	0.101–0.133	0.177–0.188	0.013–0.015	0.321–0.326	0.99951–0.99971
C100 full	O0	0.018–0.020	0.175–0.221	0.012–0.014	0.480–0.486	0.99979–0.99983
C100 full	O2	0.048–0.077	0.178–0.226	0.013–0.015	0.403–0.421	0.99964–0.99980

Appendix

Table 24: Working-model diagnostics vary across R1-positive settings. Ranges span available seeds for each family and output view.

View	Probe	Score	$p$ , $S=50$	$p$ , $S=100$	Steps
$O_{0}$	Linear	log loss	0.1000	0.1000	0
$O_{0}$	Linear	Brier	0.1000	0.1125	1
$O_{0}$	MLP	log loss	0.5750	0.5875	1
$O_{0}$	MLP	Brier	0.4250	0.5125	7
$O_{1}$	Linear	log loss	0.0875	0.2375	12
$O_{1}$	Linear	Brier	0.0625	0.1250	5

Appendix

Table 25: Doubled-

S

check on c10-deit-ar_block-s0 with the corrected canonical view seeds:

p

-values recomputed from the 79 stored reference statistics at

S=50

(primary record) and

S=100

, and their difference in grid steps of

1/80

. ∗ : at most

0.025

. The pre-specified criterion is a change of at most one step for every

p

-value.

Family	Probe	View	Q1	Q2	Q3
AR-block/DeiT/C-100	Linear	$O_{0}$	3/3	3/3	3/3
AR-block/DeiT/C-100	Linear	$O_{1}$	3/3	3/3	3/3
AR-block/DeiT/C-100	Linear	$O_{2}$	2/3	2/3	2/3
AR-block/DeiT/C-100	Linear	$O_{3}$	3/3	3/3	3/3
AR-full/DeiT/C-100	Linear	$O_{0}$	2/5	2/5	2/5
AR-full/DeiT/C-100	MLP	$O_{0}$	3/5	3/5	3/5

Appendix

Table 26: Working-model sensitivity of the observed-label R1 cells. Q1 is the primary working model (ridge mean, shared residual correlation). Q2 replaces the mean by gradient boosting; Q3 keeps the selected Q1 mean and adds an output-dependent diagonal scale, with a shared residual correlation refitted on the scaled residuals. Q2 and Q3 are fixed sensitivity variants, fitted on half A with hyperparameters tuned there by three-fold held-out conditional log-likelihood; both fit less well than Q1 in all 103 fits. Each draws its own reference assignments; MLP cells use the original accuracy-based selection. Entries give the training seeds with both

p

-values at most

0.025

; × marks a cell that no longer meets the two-seed R1 rule. The last row counts R1 cells (seed-level flags in parentheses) in the comparison group: the Swin and ViT families with the same dataset and routing variant at the same probe and view, selected as R1-negative under Q1 and not a calibration set. Post-hoc; stability does not establish the true conditional law.

Cell	$L_{f}$	$\lambda$	Recorded gain	Positive	Power	Gate
ARb/DeiT/C100	22	0.3828815918	0.00993134	200/200	100%	pass
ARb/Swin/C10	8	0.5706916534	0.00993018	200/200	100%	pass

Appendix

Table 27: Both independent-split Linear power endpoints meet the adequacy gate. Each replicate draws 10k synthetic examples before disjoint 5k fitting and 5k testing.

DeiT family	Seed	View	Probe	Log loss $\times 10^{3}$	Brier $\times 10^{3}$	Joint
C10 block	1	O0	Linear	+2.466 [-1.011,+6.198]	+0.800 [-0.427,+2.054]	no
C10 block	1	O1	Linear	+0.754 [-2.048,+3.676]	+0.356 [-0.703,+1.493]	no
C10 block	1	O3	Linear	+1.044 [-2.091,+4.077]	+0.428 [-0.671,+1.533]	no
C10 block	2	O0	Linear	-0.401 [-3.385,+2.427]	-0.071 [-1.047,+0.866]	no
C10 block	2	O1	Linear	+1.256 [-1.517,+3.896]	+0.642 [-0.284,+1.587]	no
C10 block	2	O3	Linear	+1.321 [-1.720,+4.578]	+0.664 [-0.330,+1.701]	no

Appendix

Table 28: Only three seed comparisons meet both noise-score conditions, in different cells. All 28 same-half comparisons in R1-positive cells are shown.

DeiT family	View	Probe	Seed	p (log/Brier)	$\Delta_{\rm cond,log}$	$\Delta_{\rm cond,Brier}$	R1
C100 block	O0	Linear	0	0.0125/0.0125	+0.003883	+0.001371	yes
C100 block	O0	Linear	1	0.0125/0.0125	+0.003140	+0.001025	yes
C100 block	O0	Linear	2	0.0125/0.0125	+0.004838	+0.001984	yes
C100 block	O1	Linear	0	0.0250/0.0250	+0.002201	+0.000853	yes
C100 block	O1	Linear	1	0.0250/0.0125	+0.002309	+0.000865	yes
C100 block	O1	Linear	2	0.0125/0.0125	+0.003221	+0.001231	yes

Appendix

Table 29: Positive cell-level R1 findings include negative individual seed effects. All seed values for the 12 R1-positive cells are shown.

Family	Backbone	Dataset	$L_{f}$	Seeds
Halting / C10	DeiT-S	CIFAR-10	12	0,1,2
Halting / C100	DeiT-S	CIFAR-100	12	0,1,2
AR-block / C100	DeiT-S	CIFAR-100	22	0,1,2
AR-full / C100	DeiT-S	CIFAR-100	23	0,1,2,3,4
AR-block / C10	DeiT-S	CIFAR-10	22	0,1,2
AR-full / C10	DeiT-S	CIFAR-10	23	0,1,2

Appendix

Table 30: Routing width varies across families but is constant across seeds and output views. The manifest covers 22 families and 68 training runs.

Family	Backbone	Runs	Routing width	Accuracy range (%)
Halting / C10	DeiT-S	3	12	91.95–92.33
Halting / C100	DeiT-S	3	12	72.13–73.37
AR-block / C100	DeiT-S	3	22	68.42–70.13
AR-full / C100	DeiT-S	5	23	70.00–72.07
AR-block / C10	DeiT-S	3	22	69.47–90.55
AR-full / C10	DeiT-S	3	23	89.72–91.80

Appendix

Table 31: Training quality varies across retained model families. Routing widths and top-1 accuracy ranges cover all available runs.

Item	Resolution
AR-block/DeiT/C-100 classification	An interim reading labelled this family Type I on its shuffle contrast alone ( $+0.0021$ nats, 3/3 seeds, linear). Applying the pre-specified three-way criterion literally it does not qualify: the raw gain is $+0.0011$ with 0/3 seeds detected, and the noise contrast reaches 0/3 seeds on Brier. Recorded as a partial-contrast artifact; the family is Type 0.
Excluded training record	One nominal AR-full seed was found byte-identical in weights to the corresponding AR-block seed and was removed rather than repaired; an independently retrained replacement was used.
Repaired evaluation caches	Three Swin/C-100 routing caches had been overwritten; they were regenerated from the independent checkpoints and re-verified against their stored evaluation records.
Boundary cell	A historical $\alpha\!=\!0.05$ boundary observation in AR-full/DeiT/C-100 was adjudicated by two additional independent seeds ( $p=0.410$ , $p=0.410$ ); the family carries five seeds.
Later analysis amendments	Three, each recorded before the corresponding analysis was run and none changing the primary decision rule or taxonomy. (i) A contrast-specific power benchmark was added after the raw-gain benchmark (App. I ). (ii) The post-hoc matched-shuffle analysis initially averaged its 20 replicate predictions before scoring; since log loss and Brier are convex in the prediction this gave the control an ensemble advantage, so it was re-run scoring each replicate separately (App. J ). The corrected analysis is the one reported. (iii) A3: the three-way conjunction’s operating characteristics were bounded post hoc from the existing benchmark; the per-replicate indicators needed for an exact joint rate were not retained, so Fréchet bounds are reported (App. I ).

Appendix

Table 32: Documented corrections preserve the original decision rule and taxonomy. Entries identify excluded records, corrected inputs and analysis amendments.

Artifact / purpose	Record type	Identifier
Conditional results (599 artifacts)	Git commit	f4d4e9d
Conditional interpretation	Git commit	b25e882
R1 adjudication	Git commit	c4ce715
Correction specification	Git commit	dc837c8
Correction implementation	Git commit	a1a6288
Independent-split / same-half results	Git commit	a5ff97b

Appendix

Table 33: Immutable records separate results from interpretation. Identifiers locate the complete artifact history.

Artifact	SHA256
Conditional protocol	cca925094bf6f8094ad2bdb625ef84f2 58c67f3f7890873e42bc7166f1c2110e
Conditional result aggregate	56ffda52f85f92d937bf24d78238ff19 bf34eab5f731089d566b5e60b346cdb4
Correction result aggregate	9ae456f6e9deb9eaf0c37d01343ffe25 0bc1944ce0b72dda6fc5bbbc46de6654
Common-generator protocol	6bd38cbce3109e7ed64346adb336bf4d 37c90b395163f0690b0ba44be1b5f8cd
Common-generator result aggregate	3c23f20feb6ac518eccea0522d1c4d59 561262ce8845ca86d558c9ec93bf3442

Appendix

Table 34: Full SHA256 identifiers support byte-level verification. Each digest is split across two lines for readability.

Artifact key	Manuscript identity
adepth_c10	Halting / C10 / DeiT
adepth_c100	Halting / C100 / DeiT
deit_c100_block	AR-block / C100 / DeiT
deit_c100_full	AR-full / C100 / DeiT
deit_c10_block	AR-block / C10 / DeiT
deit_c10_full	AR-full / C10 / DeiT

Appendix

Table 35: Artifact family keys map to descriptive manuscript labels. Backbone and dataset retain the original manifest identity.

Explore similar work

Oct 1, 2026cs.AI

Auditing Routing Entropy as an Uncertainty Signal in Attention-Residual Transformers

Dynamic architectures leave a per-example routing trace beside each prediction, and diffuse routing is easy to read as a sign that the prediction is unreliable. We audit that reading for routing entropy in Attention-Residual (AR) variants of Swin-Tiny and DeiT-Small, trained from scratch on CIFAR-10/100 with a soft-binned calibration auxiliary loss, asking whether the trace carries information about correctness beyond what the model's own confidence already reveals. Three checks probe this increment: does a routing signal appear at fixed confidence, does it replicate across training seeds, and can a held-out predictor exploit it against output-only and shuffled-trace controls? A sensitivity audit then injects effects of known size and measures the fraction of each that the probes recover. No test in the fixed 30-test binned family survives multiplicity correction, and neither the nominal hit nor a borderline result recurs in its sibling seeds. Across 24 paired runs a scalar routing probe yields no pooled improvement in routing-stratified calibration, and an entropy-profile probe predicts correctness better than the same probe given shuffled profiles yet worse than a confidence-only predictor in both binary log-loss and Brier score: a gain over shuffled traces does not become a gain over the output. Conditioning on the complete logit vector leaves the corresponding comparison unresolved. The audit bounds how far these non-detections can be read: at an injected effect of 0.010 nats the profile probe recovers 24-59% of the oracle gain, and a reference-preserving correction probe recovers 8% and 23% in the two CIFAR-100 settings, below the threshold we fixed for applying it to real labels. The results establish control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.

Wenhao Liang, Lin Yue, Wei Emma Zhang +3

Adelaide University, Adelaide, Australia

Transformer Residual Streams Model Auditing Entropy Traces Imagenet

Ask PDF

May 11, 2026cs.CV

Probing Routing-Conditional Calibration in Attention-Residual Transformers

Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly accompany predictions with sample-specific internal routing traces and pair them with claims of calibration-relevant uncertainty. We ask a basic question: do these traces provide stable routing-specific evidence for post-hoc calibration beyond confidence? We study this in Attention-Residual transformers (Kimi Team, 2026) through a matched-confidence diagnostic suite that stratifies examples by routing-derived state, compares subgroup gaps against within-bin routing-permutation nulls, and evaluates matched post-hoc probes differing only in their auxiliary feature. Across our completed AR runs, scalar routing summaries do not provide stable evidence of routing-conditional miscalibration: weighted gaps remain small or seed-sensitive, and only

1

30

within-bin permutation tests rejects the conditional-null at

α=0.05

(only on one seed; not stable across seeds in that cell). AR-CondCal, a minimal

2

-D Nadaraya--Watson probe on confidence and routing-depth variance, lies within the seed-variance band of matched confidence-only and predictive-entropy controls and does not reliably improve worst-routing-tertile ECE; bandwidth-sensitivity checks (Scott multiples, CV-NLL, global-ECE oracle) do not change this. A full-vector MLP over

(c, H_1, \ldots, H_L)

can appear to improve over a linear confidence baseline, but the apparent gain disappears once a capacity-matched confidence-only MLP is included as a control, and shuffled routing profiles achieve comparable performance. Apparent routing-aware calibration gains in this AR setting should not be read as internal-state calibration until matched-confidence, bandwidth, capacity, and permutation controls rule out common confounds.

Wenhao Liang, Lin Yue, Wei Emma Zhang +4

Adelaide University · Australian Institute for Machine Learning (AIML), Adelaide University · The University of Queensland

Post-Hoc Transformer Architectures

Ask PDF

Jul 21, 2026cs.AI

The Knowing-Saying Gap: When Probes See Errors that Confidence Misses

Linear probes detect corrupted context in language models with near-perfect accuracy, yet this does not translate into reliable failure prediction. The result is a dissociation with direct implications for deployment monitoring. Across multi-hop arithmetic chains, probes that detect corruption turn out to be uninformative about final answer correctness; models forced into structured confidence formats collapse to two values with indistinguishable error rates; and probe persistence across hops fails to separate correct from incorrect outcomes, refuting our pre-registered "persistence beats peak" hypothesis. This pattern of knowing but not saying generalises across model families including reasoning models. As a real-time monitor, probe-based interventions are sharply model and error-type dependent: branch-and-pick is net-positive across models and uniquely non-breaking on Llama-3.1-8B (4 rescued, 0 broken), while reprompt and replace-prior break correct traces at roughly the rate they rescue wrong ones. Probe-based monitoring is a necessary complement to verbalised confidence, but no single intervention dominates, and the deployable answer is model-aware, error-type-aware routing.

Jyotin Goel, Ipshita Bandyopadhyay, Justin Shenk

Correctness Reasoning Gaps Corrupted Inputs Persistence

Ask PDF