Do ResNets Route? Sparse Interaction Experts in Residual Networks

Liang Yan; Siying Chen; Kaijie Chen; Bo Li; Jinghao Zhang; Mu Miao

cs.LGOct 2, 2026

Do ResNets Route? Sparse Interaction Experts in Residual Networks

Authors: Liang Yan, Siying Chen, Kaijie Chen, Bo Li, Jinghao Zhang, Mu Miao

Organizations: Fudan University · University of Washington · Tongji University · Shandong University ·

Abstract

Residual networks execute every block for every input, yet their functional contributions need not be input independent. We formulate a trained ResNet as a set function over binary residual-branch masks and apply Möbius inversion to decompose its output exactly into individual residual corrections and higher-order interactions. For smooth residual stacks, we show that each fixed $k$ -way interaction scales as $\mathcal{O}(λ^k)$ under residual scaling. Exhaustive analysis of ImageNet-pretrained ResNet-18 and ResNet-34 reveals that interaction mass peaks at orders five and ten, respectively, rather than at low orders. Reducing the residual scale shifts both spectra toward lower orders, but also changes model predictions. The interaction coefficients are concentrated in magnitude but not hard sparse, and prediction-preserving sparsity weakens with depth. Crucially, the dominant interactions vary across inputs and predicted classes around a shared global core, while their overall complexity changes little with sample difficulty. These results show that dense ResNets implement an implicit form of soft routing: every block is executed, but different inputs rely on different residual interaction experts. Routing can therefore emerge at the level of functional contribution without an explicit router or sparse execution.

Figures & tables

Figure 1: From dense residual execution to input-dependent interaction experts. (a) A standard ResNet executes every residual block for every input and contains no explicit router. For analysis, we apply binary gates to its

L

residual branches, evaluate all

2^{L}

masks, and use Möbius inversion to decompose the output exactly into interaction coefficients

\Delta_{S}

, one for each branch subset

S

. These gates are analytical interventions and do not alter ordinary dense inference. (b) Schematic of the resulting functional decomposition for different inputs. Each dot represents an interaction, positioned by its order

|S|

and sized by its predicted-class contribution

|\Delta_{S}v_{x}|

; the callouts identify the branches participating in the strongest interactions. The dominant terms combine a shared global core with input-dependent, class-structured variation. We refer to this concentration and variation of functional contributions as implicit soft routing , which is distinct from sparse execution or an explicit learned router.

	Method	Norm ratio	Rel. error ( $\downarrow$ )	Scaled error ( $\downarrow$ )	Cosine ( $\uparrow$ )	Top-1 agree ( $\uparrow$ )
18	$A_{x}^{\mathrm{raw}}$	$79.400$	$79.200$	$0.961$	$0.222$	$0.006$
	$A_{x}^{\mathrm{mean}}$	$0.310$	$0.977$	$0.961$	$0.222$	$0.006$
	$A_{x}^{\mathrm{ctr}}$	$77.400$	$77.200$	$0.959$	$0.229$	$0.006$
	Möbius 1 1 1 ResNet-18, before rounding: relative error $8.51\times 10^{-7}$ , optimally scaled error $8.55\times 10^{-7}$ .	1.000	0.000	0.000	1.000	1.000
34	$A_{x}^{\mathrm{raw}}$	$18{,}852.327$	$18{,}851.887$	$0.889$	$0.440$	$0.025$
	$A_{x}^{\mathrm{mean}}$	$0.288$	$0.910$	$0.889$	$0.440$	$0.025$

Table 1: Naive additive reconstructions versus exact Möbius reconstruction on ResNet-18 and ResNet-34. Entries are means, rounded to three decimal places. ResNet-18 uses

N=10{,}000

ImageNet validation images; ResNet-34 uses

N=1{,}000

Figure 2: Residual interaction-order spectra for scalar predicted-class and vector-logit readouts. The top and bottom rows show ResNet-18 and ResNet-34, respectively. From left to right, the columns report normalized interaction energy $\widetilde{E}_{k}$ , cumulative low-order energy $C_{\leq k}$ , effective interaction order $\kappa$ , and mean per-interaction energy $M_{k}$ on a logarithmic scale.

Figure 3: Inference-time residual-scaling sweep on pretrained ResNet-18 ( $N=10{,}000$ ) and ResNet-34 ( $N=1{,}000$ ), without retraining. Solid and dashed lines denote ResNet-18 and ResNet-34, respectively. (a) Effective interaction order $\kappa$ for the scalar and vector readouts. (b) Cumulative low-order energy $C_{\leq 3}$ . (c) Top-1 agreement of each scaled model with its prediction at $\lambda=1$ .

Figure 4: Top- $K$ interaction concentration and reconstruction. ResNet-18 results are shown on top and ResNet-34 results on the bottom. (a,e) Captured squared coefficient mass. (b,f) Relative logit error. (c,g) Agreement with the full-model predictions. (d,h) Order composition and enrichment of the vector Top- $K$ supports. Most coefficient mass is concentrated in a sparse interaction core, while a long tail progressively refines the final prediction.

Figure 5: Input dependence and difficulty sensitivity of residual interaction supports. Results are shown for ResNet-18 and ResNet-34 on $10{,}000$ ImageNet validation images. (a,b) Same-class minus different-class similarity for scalar, vector, and weighted interaction signatures, with label shuffling as a null control. (c,d) Input-wise Jaccard similarity with the global vector interaction set at approximately $13\%$ retention. (e,f) Spearman correlations between label-free difficulty measures and vector interaction complexity. (g,h) Negative log-likelihood versus the vector effective interaction count $N_{\mathrm{eff}}^{h}$ . Dominant interactions exhibit class-dependent variation within a shared global core, while their overall complexity is only weakly associated with input difficulty.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
Residual network, masks, and readouts
$x,\ N,\ C$	Input image; number of images; number of classes.
$L,\ [L];\ l,i,j\in[L]$	Number and index set of residual blocks; block indices.
$F_{l},\ s_{l},\ \phi_{l}$	Residual branch, shortcut, and activation of block $l$ .
$x_{l}(m)$	Hidden state at block $l$ under mask $m$ .
$m\in\{0,1\}^{L}$	Residual-branch mask; $m_{l}=0$ disables branch $l$ only.

Appendix

Table 2: Notation. Superscript

r\in\{v,h\}

marks the scalar (predicted-class) or vector (full-logit) readout; per-input statistics are averaged over images. Masks are analytical interventions, not learned routing decisions. Symbols used within a single appendix are defined there.

Model	Reconstruction	Norm ratio $(\rightarrow 1)$	Rel. error $(\downarrow)$	Scaled error $(\downarrow)$	Cosine $(\uparrow)$	Top-1 agree $(\uparrow)$
ResNet-18	Raw sum	$79.366$	$79.150$	$0.9606$	$0.2217$	$0.0064$
	Mean output	$0.3100$	$0.9771$	$0.9606$	$0.2217$	$0.0064$
	Centered sum	$77.412$	$77.190$	$0.9595$	$0.2288$	$0.0062$
	Möbius	$1.0000$	$8.506{\times}10^{-7}$	$8.550{\times}10^{-7}$	$1.0000$	$1.0000$
ResNet-34	Raw sum	$18{,}852.3$	$18{,}851.9$	$0.8890$	$0.4400$	$0.0250$
	Mean output	$0.2877$	$0.9104$	$0.8890$	$0.4400$	$0.0250$

Appendix

Table 3: Naive aggregation versus exact Möbius reconstruction. Entries are per-image means over fixed ImageNet-1K test subsets. ResNet-18 uses

N=10{,}000

images and

256

masks per image; ResNet-34 uses

N=1{,}000

images and

65{,}536

masks per image. Top-1 agreement is measured against the corresponding full-model prediction.

Figure 6: Naive masked-response aggregation fails, whereas Möbius inversion reconstructs the full logits. The top and bottom rows show ResNet-18 and ResNet-34, respectively. Columns report norm ratio, relative error, optimally scaled error, cosine similarity, and top-1 agreement with the full model. Bars and error bars denote per-image means and standard errors over the fixed ImageNet-1K test subsets.

Figure 7: Residual interaction-order spectra at the natural operating point. Results are shown for ResNet-18 (top; $N=10{,}000$ , $L=8$ ) and ResNet-34 (bottom; $N=1{,}000$ , $L=16$ ). From left to right, the columns show the normalized squared coefficient mass $\widetilde{E}_{k}$ , cumulative low-order mass $C_{\leq k}$ , effective interaction order $\kappa$ , and mean per-interaction mass $M_{k}$ . Scalar readouts use the class predicted by the full model, while vector readouts use all logits. Both models peak above $L/2$ , and the large high-order $M_{k}$ values show that this pattern is not explained solely by the number of subsets at each order.

Model	Readout	Peak $k$ ( $\widetilde{E}_{k}$ )	$C_{\leq 3}$	$C_{\leq L/2}$	$\kappa$	$M_{L}/M_{1}$
ResNet-18	Scalar	$5\;(0.2721)$	$0.1417$	$0.3545$	$5.001$	$32.21$
	Vector	$5\;(0.2951)$	$0.1356$	$0.3675$	$4.929$	$22.31$
ResNet-34	Scalar	$10\;(0.1951)$	$0.001153$	$0.2421$	$9.887$	$341.4$
	Vector	$10\;(0.1956)$	$0.001116$	$0.2419$	$9.884$	$296.4$

Appendix

Table 4: Summary of the interaction-order spectra.

C_{\leq L/2}

includes the midpoint order, while

M_{L}/M_{1}

compares the dataset-mean per-interaction masses at the highest and first orders. Values are rounded to approximately four significant digits.

$k$	Scalar	Vector	Scalar	Vector
	ResNet-18		ResNet-34
1	$0.008619$	$0.007611$	$0.000038$	$0.000036$
2	$0.032546$	$0.027096$	$0.000212$	$0.000196$
3	$0.100549$	$0.100901$	$0.000903$	$0.000884$
4	$0.212804$	$0.231893$	$0.003625$	$0.003595$
5	$\mathbf{0.272092}$	$\mathbf{0.295098}$	$0.012134$	$0.012054$

Appendix

Table 5: Complete normalized interaction-order spectra. Entries report the dataset-mean normalized squared coefficient mass

\widetilde{E}_{k}

. Six decimal places are retained so that the small low-order values of ResNet-34 are not rounded to zero.

Figure 8: Inference-time residual-scaling sweep. Results are shown for pretrained ResNet-18 ( $N=10{,}000$ ) and ResNet-34 ( $N=1{,}000$ ), without retraining. Solid and dashed lines denote ResNet-18 and ResNet-34, respectively. Pink and blue denote scalar and vector readouts. (a) Effective interaction order $\kappa$ . (b) Cumulative low-order squared coefficient mass $C_{\leq 3}$ . (c) Top-1 agreement between the scaled full-mask prediction and the corresponding prediction at $\lambda=1$ . Decreasing $\lambda$ shifts both models toward lower-order interactions, but large shifts coincide with substantial changes in prediction.

$\lambda$	$\kappa^{v}$	$\kappa^{h}$	$C_{\leq 3}^{v}$	$C_{\leq 3}^{h}$	Top-1 agreement
1.00	5.0009	4.9295	0.1417	0.1356	1.0000
0.95	4.9756	4.9002	0.1491	0.1447	0.9375
0.90	4.9448	4.8706	0.1573	0.1541	0.8759
0.85	4.9086	4.8389	0.1671	0.1640	0.8093
0.80	4.8635	4.8022	0.1786	0.1751	0.7314
0.75	4.8042	4.7572	0.1933	0.1882	0.6365

Appendix

Table 6: Residual-scaling sweep for ResNet-18. Entries are dataset means over

N=10{,}000

images and are rounded to four decimal places. Agreement compares the full-mask prediction at each

\lambda

with that at

\lambda=1

$\lambda$	$\kappa^{v}$	$\kappa^{h}$	$C_{\leq 3}^{v}$	$C_{\leq 3}^{h}$	Top-1 agreement
1.00	9.8872	9.8835	0.0012	0.0011	1.0000
0.95	9.8462	9.8390	0.0014	0.0013	0.9200
0.90	9.8027	9.7938	0.0017	0.0016	0.8780
0.85	9.7616	9.7474	0.0020	0.0020	0.8140
0.80	9.7013	9.6957	0.0025	0.0024	0.7430
0.75	9.6391	9.6373	0.0032	0.0030	0.6630

Appendix

Table 7: Residual-scaling sweep for ResNet-34. Entries are dataset means over

N=1{,}000

images and are rounded to four decimal places. Agreement compares the full-mask prediction at each

\lambda

with that at

\lambda=1

Figure 9: Per-input magnitude Top- $K$ reconstruction for ResNet-18. (a) Retained squared coefficient mass for scalar and vector rankings. (b–d) Vector relative error, cosine similarity, and agreement with the all-open model under magnitude, random, order-matched, and low-order selection. (e) Predicted-class margin ratio. (f) Order composition of the magnitude Top- $K$ sets; color denotes the selected fraction within each order and text denotes enrichment over combinatorial prevalence. (g,h) Mean per-image first-hit budgets and retained fractions for the stated error and mass thresholds.

Figure 10: Per-input magnitude Top- $K$ interaction reconstruction for ResNet-34. Results are averaged over $N=1{,}000$ unlabeled ImageNet-1K test images. The interaction dictionary contains $M=2^{16}-1=65{,}535$ non-empty terms; the shortcut-only baseline is always included and is not counted in $K$ . (a) Squared coefficient mass captured by independently ranked scalar predicted-class and vector-logit Top- $K$ sets. (b–d) Relative vector-logit error, cosine similarity, and agreement with the all-open model prediction under magnitude Top- $K$ , random- $K$ , order-matched random, and low-order selection. Agreement is measured against the full-model prediction rather than the ground-truth label. (e) Ratio of the reconstructed predicted-class margin to the full-model margin. (f) Order composition of the vector magnitude Top- $K$ sets across retained fractions; color denotes the selected fraction within each interaction order, and cell text gives its enrichment over combinatorial prevalence. (g,h) Effective interaction counts and their fractions of the full dictionary for vector-error and coefficient-mass thresholds; markers summarize the mean and median distributions across inputs. Coefficient mass is concentrated, but faithful reconstruction is much less sparse: retaining $50\%$ of the terms captures $97.90\%$ of the vector mass, yet yields relative error $13.87$ and only $0.60\%$ prediction agreement.

ResNet-18 ( $N=10{,}000$ , $M=255$ )
Retained	$K$	$\Gamma_{K}^{v}$	$\mathrm{Err}_{K}^{v}$	$\Gamma_{K}^{h}$	$\mathrm{Err}_{K}^{h}$	$\mathrm{Cos}_{K}$	Agree
1%	3	0.2445	0.9934	0.1450	4.0016	0.0019	0.0019
5%	13	0.6140	0.8625	0.4345	4.0782	0.1214	0.0063
10%	26	0.8161	0.6754	0.6548	3.6881	0.1671	0.0339
25%	64	0.9751	0.2644	0.9364	1.7686	0.4896	0.3048
50%	128	0.9988	0.0575	0.9971	0.2850	0.9554	0.8766

Appendix

Table 8: Magnitude Top-

K

concentration and reconstruction. Entries are dataset means; uncertainty terms are omitted for readability. Scalar and vector supports are ranked separately. Top-1 agreement is measured against the corresponding all-open model prediction.

Model	Retained	$K$	Relative error			Top-1 agreement
			Magnitude	Random	Order-matched	Magnitude	Random	Order-matched
ResNet-18	25%	64	1.7686	7.9048	7.3574	0.3048	0.0028	0.0044
	50%	128	0.2850	9.0794	7.7646	0.8766	0.0042	0.0068
ResNet-34	25%	16,384	35.5789	123.0253	117.9390	0.0020	0.0018	0.0006
	50%	32,768	13.8741	141.9837	111.8046	0.0060	0.0016	0.0024

Appendix

Table 9: Representative vector-readout baseline comparisons. Magnitude selection substantially improves numerical reconstruction over random and order-matched selection, but this does not necessarily translate into prediction agreement. Values are dataset means.

Model	Criterion	Mean $K$	$K/M$
ResNet-18	$\mathrm{Err}_{K}^{h}\leq 0.10$	150.60	59.059%
	$\mathrm{Err}_{K}^{h}\leq 0.05$	186.64	73.191%
	$\Gamma_{K}^{h}\geq 0.90$	55.77	21.870%
	$\Gamma_{K}^{h}\geq 0.95$	68.59	26.898%
	$\Gamma_{K}^{h}\geq 0.99$	99.85	39.155%
ResNet-34	$\mathrm{Err}_{K}^{h}\leq 0.10$	65,203.74	99.495%

Appendix

Table 10: Vector effective budgets. Mean per-image first-hit

K

and its fraction of the

M

non-empty interactions. Error thresholds use the first crossing of a potentially non-monotone curve; mass thresholds are monotone.

Figure 11: Input dependence and class structure of ResNet-18 interaction sets. Results use $N=10{,}000$ ImageNet-1K validation images and $M=255$ non-empty interactions. (a) Vector-set Jaccard similarity for same-class and different-class pairs, a uniform random- $K$ baseline, and per-image overlap with the global vector set. (b) Same-class minus different-class gaps for vector binary, vector weighted, and decision-conditioned scalar signatures, together with the vector label-shuffle control. (c) Same-class/different-class overlap lift for vector and scalar signatures. (d) Leave-one-out 1-nearest-neighbor class accuracy at $K=32$ , compared with the empirical $0.10\%$ chance level. (e) Similarity of random pairs drawn within the high- and low-confidence terciles. (f) Distribution of per-image overlap with the global vector set at $K=32$ ( $12.55\%$ retention). (g) Same-class and different-class pairwise Jaccard distributions at $K=32$ . The vector sets share a strong global core but retain a small class-dependent component, while the decision-conditioned scalar sets show substantially stronger class alignment.

Figure 12: Input dependence and class structure of ResNet-34 interaction sets. Results use the same $N=10{,}000$ class-balanced ImageNet-1K validation images and $M=65{,}535$ non-empty interactions. Panels (a), (c), and (e) include retained fractions from $0.05\%$ to $25\%$ ; the fraction-matched comparison in panel (b) begins at $1\%$ . Panel (f) reports per-image overlap with the global vector set at $K=8{,}520$ ( $13.0\%$ retention), matching the retained fraction used for ResNet-18 rather than its absolute $K$ . Panel (g) shows the same-class and different-class Jaccard distributions at the fixed $K=32$ used by the nearest-neighbor diagnostic. The positive class gaps replicate the ResNet-18 trend, but high global overlap again indicates class-structured variation around a shared interaction core rather than disjoint routes.

ResNet-18 ( $M=255$ )
Retained	$K$	$J_{\mathrm{same}}^{h}$	$J_{\mathrm{diff}}^{h}$	$\Delta_{\mathrm{class}}^{h}$	Lift h	$J_{\mathrm{rand}}$	$J_{\mathrm{global}}$
1.18%	3	0.6334	0.5972	0.0362	1.061	0.0069	0.6900
3.14%	8	0.6623	0.6318	0.0305	1.048	0.0167	0.7312
5.10%	13	0.6885	0.6560	0.0325	1.050	0.0268	0.7443
10.20%	26	0.7778	0.7557	0.0222	1.029	0.0537	0.8072
12.55%	32	0.8329	0.8150	0.0179	1.022	0.0683	0.8705

Appendix

Table 11: Vector interaction-set overlap. Entries are dataset means.

J_{\mathrm{same}}

and

J_{\mathrm{diff}}

use ground-truth ImageNet classes;

J_{\mathrm{rand}}

is the uniform random-

K

baseline; and

J_{\mathrm{global}}

is the mean per-image overlap with

\mathcal{G}_{K}

. Uncertainty columns are omitted for readability.

ResNet-18
Retained	$K$	$\Delta_{h}$	$\Delta_{w}$	$J_{\mathrm{same}}^{v}$	$J_{\mathrm{diff}}^{v}$	$\Delta_{v}$	$\Delta_{\mathrm{shuf}}$
1.18%	3	0.0362	0.0361	0.1911	0.0710	0.1200	0.0014
3.14%	8	0.0305	0.0327	0.2535	0.1289	0.1246	0.0019
5.10%	13	0.0325	0.0347	0.2994	0.1765	0.1229	0.0005
10.20%	26	0.0222	0.0279	0.3844	0.2804	0.1040	-0.0009
12.55%	32	0.0179	0.0256	0.4195	0.3248	0.0947	0.0003

Appendix

Table 12: Class-alignment gaps across signatures and controls.

\Delta_{h}

is binary vector Jaccard,

\Delta_{w}

is weighted vector Jaccard,

\Delta_{v}

is binary predicted-class scalar Jaccard, and

\Delta_{\mathrm{shuf}}

is the vector gap after label permutation. Scalar results are decision-conditioned because the selected output coordinate can differ across images. Values are means.

Model	Vector 1-NN	Scalar 1-NN	Chance	Max conf. gap
ResNet-18	0.0039	0.2423	0.0010	0.0045
ResNet-34	0.0033	0.0991	0.0010	0.0031

Appendix

Table 13: Signature diagnostics. Leave-one-out 1-NN uses binary Jaccard at the fixed absolute budget

K=32

; this is not a fraction-matched comparison across architectures, and deterministic tie handling makes the values diagnostic. The final column is the largest absolute confidence-tercile gap over the evaluated

K

grid.

Figure 13: Sample difficulty and residual interaction complexity for ResNet-18. Results use $N=10{,}000$ ImageNet-1K validation images and $M=255$ non-empty interactions. (a) Tie-correct Spearman correlations between label-aware difficulty measures and vector-readout complexity. (b) The same analysis for negative confidence and negative predicted-class margin, both oriented so that larger values denote harder inputs. (c,d) Corresponding results for the decision-conditioned scalar readout. (e) Relative hard–easy change in the vector metrics, $100(\mu_{\mathrm{hard}}-\mu_{\mathrm{easy}})/|\mu_{\mathrm{easy}}|$ , between the top and bottom NLL quartiles. (f) Relative change in the vector metrics between misclassified and correctly classified images. Panels (e,f) display $T_{>3}^{h}$ , whereas the tables report its equivalent complement $C_{\leq 3}^{h}=1-T_{>3}^{h}$ to avoid redundant columns. (g,h) NLL versus the vector effective interaction count $N_{\mathrm{eff}}^{h}$ and low-order mass $C_{\leq 3}^{h}$ (shown as a percentage), respectively. Associations are uniformly weak; the clearest change is a small broadening of the interaction-magnitude distribution, not a shift toward higher interaction orders.

Figure 14: Sample difficulty and residual interaction complexity for ResNet-34. Results are computed over a class-balanced subset of $N=10{,}000$ ImageNet-1K validation images and all $M=65{,}535$ non-empty residual interactions. All difficulty scores are oriented so that larger values indicate harder inputs. (a) Tie-correct Spearman correlations between label-aware difficulty measures, including NLL, prediction error, and negative true-class margin, and vector-readout measures of interaction breadth, order, and Top- $K$ compressibility. (b) Corresponding correlations for the label-free measures of negative confidence and negative predicted-class margin. (c,d) Scalar-readout counterparts of (a,b), where the scalar coordinate is the class predicted by the all-open model. (e) Relative changes in vector-readout complexity between the highest- and lowest-NLL quartiles. (f) Relative changes between misclassified and correctly classified images. These comparisons cover effective interaction counts, interaction order, high-order coefficient mass, and reconstruction budgets. (g,h) Binned distributions of NLL against the vector effective interaction count $N_{\mathrm{eff}}^{h}$ and low-order coefficient mass $C_{\leq 3}^{h}$ , respectively; color intensity denotes the number of images in each bin. Across all analyses, the correlations and group differences remain small: harder inputs neither recruit substantially broader interaction supports nor shift coefficient mass toward higher orders, despite the large interaction dictionary.

Difficulty	$N_{\rm eff}^{h}$	$N_{\rm ent}^{h}$	$\bar{k}^{h}$	$\kappa^{h}$	$C_{\leq 3}^{h}$	$K_{.10}^{\rm err}$	$K_{.05}^{\rm err}$	$K_{.90}^{\rm mass}$	$K_{.95}^{\rm mass}$	$K_{.99}^{\rm mass}$
ResNet-18
NLL	0.0313	0.0352	0.0157	0.0182	-0.0080	0.0711	0.0489	0.0267	0.0232	0.0238
Wrong	0.0186	0.0237	0.0063	0.0083	0.0002	0.0264	-0.0009	0.0170	0.0116	0.0103
$-$ True margin	0.0268	0.0306	0.0137	0.0162	-0.0073	0.0575	0.0351	0.0237	0.0187	0.0187
$-$ Confidence	0.0383	0.0407	0.0154	0.0191	-0.0080	0.0883	0.0796	0.0334	0.0298	0.0335
$-$ Predicted margin	0.0339	0.0355	0.0102	0.0145	-0.0052	0.0665	0.0598	0.0315	0.0254	0.0290

Appendix

Table 14: Tie-correct Spearman correlations for vector-readout complexity. Every difficulty score is oriented so that larger values mean harder inputs.

C_{\leq 3}^{h}

is shown instead of the redundant

T_{>3}^{h}=1-C_{\leq 3}^{h}

Difficulty	$N_{\rm eff}^{v}$	$N_{\rm ent}^{v}$	$\bar{k}^{v}$	$\kappa^{v}$	$C_{\leq 3}^{v}$
ResNet-18
NLL	0.0533	0.0637	-0.0355	-0.0456	0.0275
Wrong	0.0234	0.0333	-0.0186	-0.0320	0.0211
$-$ True margin	0.0478	0.0571	-0.0369	-0.0435	0.0290
$-$ Confidence	0.0654	0.0757	-0.0326	-0.0447	0.0250
$-$ Predicted margin	0.0550	0.0639	-0.0365	-0.0443	0.0286

Appendix

Table 15: Tie-correct Spearman correlations for scalar-readout complexity. The scalar coordinate is the class predicted by the all-open model and is fixed across masks for each image.

ResNet-18
Quantity	Easy	Hard	Hard $-$ Easy	Relative (%)
$N_{\rm eff}^{h}$	89.1949	89.4823	0.2874	0.322
$N_{\rm ent}^{h}$	62.7012	63.0648	0.3635	0.580
$\kappa^{h}$	4.9283	4.9319	0.0036	0.073
$C_{\leq 3}^{h}$	0.135677	0.135351	-0.000326	-0.241
$K_{.10}^{\rm err}$	148.7320	151.5508	2.8188	1.895

Appendix

Table 16: Loss-quartile comparison. Easy and hard denote the bottom and top NLL quartiles. Relative change is

100(\mathrm{Hard}-\mathrm{Easy})/|\mathrm{Easy}|

. Values are dataset means; no uncertainty columns are included. For

C_{\leq 3}

, the relative percentage can appear comparatively large because the ResNet-34 baseline is close to zero; the absolute gap should be used to assess its practical size.

ResNet-18
Quantity	Correct	Wrong	Wrong $-$ Correct	Relative (%)
$N_{\rm eff}^{h}$	89.2743	89.4619	0.1876	0.210
$N_{\rm ent}^{h}$	62.7944	63.0482	0.2538	0.404
$\kappa^{h}$	4.9300	4.9320	0.0020	0.041
$C_{\leq 3}^{h}$	0.135411	0.135413	$2.22\!\times\!10^{-6}$	0.002
$K_{.10}^{\rm err}$	150.5648	151.3084	0.7436	0.494

Appendix

Table 17: Correctness comparison. The ResNet-18 subset contains

6{,}994

correct and

3{,}006

incorrect predictions; the ResNet-34 subset contains

7{,}355

correct and

2{,}645

incorrect predictions. Relative change is

100(\mathrm{Wrong}-\mathrm{Correct})/|\mathrm{Correct}|

. Absolute gaps are the more informative scale for the small

C_{\leq 3}

baseline.

Model	$N_{\rm eff}^{h}$	$\kappa^{h}$	$C_{\leq 3}^{h}$	$K_{.10}^{\rm err}$	$K_{.95}^{\rm mass}$	$N_{\rm eff}^{v}$	$\kappa^{v}$
ResNet-18	-0.0306	-0.001003	$9.652\!\times\!10^{-5}$	0.1311	-0.0086	0.0913	-0.007939
ResNet-34	-5.7765	-0.000242	$2.762\!\times\!10^{-7}$	0.8280	-5.1910	0.9821	0.000174

Appendix

Table 18: Class-controlled NLL slopes. Each entry is the raw

\beta_{1}

from a within-class fixed-effect regression. Because the quantities have different units and scales, their signed magnitudes should be interpreted within each column rather than compared directly across metrics or architectures.

Residual Learning

Residual learning studies methods that learn incremental updates or error corrections, helping deep networks and hybrid models refine predictions.

See all 158 papers

Jun 11, 2026

When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals

Block Attention Residuals (Block AttnRes) by replace fixed additive residuals with a learned softmax over earlier depth-source representations, surfacing cross-layer routing as an inspectable tensor in the forward pass. This is a tempting interpretability target: information flow normally inferred indirectly is now directly observable. We ask whether such exposure suffices for mechanistic interpretation. We probe two same-scale (

0.6

B) Block AttnRes checkpoints under identical routing-ablation interventions: a vanilla Qwen3 inference-wrapped through a deterministic recency-bias schedule that the codebase admits as a routing-equivalent loading path, and a Block AttnRes Qwen3 trained from scratch with routing as part of optimisation. The wrapped baseline's routing weights are content-independent and reproduce the schedule's analytic prediction. The trained AttnRes checkpoint instead exhibits three localised routing motifs: an embedding-source pathway through early-layer MLP, a current-state pathway through early-layer attention and MLP, and an older-history pathway through late-layer attention. Beyond this stratification, we find a sharp dissociation between average routing mass and causal importance: in both sublayers, the largest mass slice is not the largest causal contribution, and one source family carries appreciable mass with no detectable causal role under intervention. Architectural exposure of routing is therefore necessary but not sufficient for mechanistic interpretation: structured depth routing emerges only when routing has been part of training, and even then, descriptive routing summaries should be treated as candidate hypotheses to be tested by causal interventions, not as evidence of mechanism in their own right.

Aydin Javadov

ETH Zurich

Transformer Interpretability Attention Residual Connections Mechanistic Interpretability LLM Routing Causal Interpretability Attention Routing

Ask PDF

Jun 9, 2026

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points. No existing work has defined a measurable RG order parameter, tested it under controlled variation of the input distribution, or made quantitative predictions that are empirically verified. We study the simplest architecture for which the analogy is tractable: a pure MLP residual stack trained on masked token prediction over synthetic Markov chain sequences with known spectral properties. We report three findings. (i) The effective rank of the residual stream decreases monotonically with depth after training, consistent with progressive integration of irrelevant degrees of freedom. (ii) This rank collapse is selective: it occurs for chains with short correlation length approximately 1 but is absent for chains with long correlation length approximately 7, measured at the position level to control for mean-pooling artifacts. The network preserves exactly the degrees of freedom relevant to the prediction task, the content of the RG relevance criterion. (iii) Inter-layer kernel drift is concentrated at one or two specific transitions, with the remainder of the network near a fixed point, consistent with a discrete fixed-point plateau. Together these findings constitute the first quantitative, position-level evidence that MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

Parviz Haggi-Mani, Irina Rish

Universit´e de Montr´eal · Mila - Quebec AI Institute

Multilayer Perceptrons Neural Network Training Dynamics

Ask PDF

Oct 22, 2026

Early learning in large residual networks

We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We analyze networks in a sequential limit where first width and depth diverge, then the number of training data points diverges, and finally the input dimension diverges. This is a more refined setup compared to the typical mean-field theory analysis in which depth and width diverge but the number of training datapoints and input dimension are fixed. Our main technical result is a diagrammatic computation of both the network output and the hidden representations after the few steps of gradient descent in terms of the statistics of inner products of inputs from the training set. One of our principle conceptual findings is a negative result: in the limit of large input dimension the network output can learn only a linear function of its inputs early in training since higher learned monomials are more and more suppressed in the input dimension and vanish in the large dimension limit. This suggests that mean-field limits of ResNets are surprisingly rigid in the regime of high input dimension and may not be sufficient toy models for understanding feature learning. Finally, we investigate how to ensure hyperparameter transfer for the learning rate as a function of dataset size and input dimension. We test our predictions numerically and find excellent agreement with our theory.

Vladimir Narovlansky, Boris Hanin

ORFE Department, Princeton University, Princeton, NJ 08544, USA · Racah Institute of Physics, The Hebrew University of Jerusalem, Jerusalem 91904, Israel

Neural Network Training Dynamics Residual Learning Representation Learning

Ask PDF

Abstract

Residual Learning

Explore similar work

When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals

Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

Early learning in large residual networks