The Residual Stream's Effective Depth

Barak Gahtan; Ido Galil; Alex M. Bronstein

cs.LGSep 25, 2026

The Residual Stream's Effective Depth

Authors: Barak Gahtan, Ido Galil, Alex M. Bronstein

Organizations: Technion Israel Institute of Technology · Nvidia · Technion Israel Institute of Technology & ISTA Institute of Science and Technology Austria

Abstract

We introduce effective depth ( $\Deff$ ), a scalar diagnostic that treats the layer-wise residual stream of a transformer as a discrete-time process, measures how representation similarity decays with layer distance, and aggregates that profile into one number. Across sixteen decoder-only language models, $\Deff$ separates a structural consequence of residual accumulation from an empirical one: even maximally diverse orthogonal updates have the closed-form reference $F_L = 2L/(L+1)<2$ , yet fifteen of sixteen default measurements lie below $F_L$ (Qwen3.5: 32--44%, OLMo-2: 40--41%, Pythia: 23--28%). Matched references show that the gap is not caused by the persistent initial state or update-size imbalance, but is largely a calibrated signature of correlated residual updates rather than evidence that depth is unused. Symmetric position-0, token-normalisation, and top-PC controls show the regime is not reducible to BOS or top-PC artefacts: the lone above-reference default outlier joins the same regime, and all sixteen models are sub-reference after token-normalisation or top-1-PC removal. Intermediate checkpoints show that the regime is established early in OLMo-2 and stable through 5T tokens, while Pythia-1.4B follows a distinct decreasing trajectory. A controlled residual-carry intervention supports the mechanism, and $\Deff$ is best read as a global accumulated-state diagnostic, not as a capability score or pruning method.

Figures & tables

Figure 1: Gap to the orthogonal-update reference

F_{L}=2L/(L+1)

for the sixteen-model sweep (

N{=}10{,}000

, FineWeb-Edu,

K{=}L{-}1

). Positive values indicate

D_{\mathrm{eff}}<F_{L}

, i.e. accumulated states show additional redundancy beyond the orthogonal-update reference; negative values indicate

D_{\mathrm{eff}}>F_{L}

. Fifteen of sixteen default measurements are sub-reference, with gaps clustering by model family. Gemma-3-12b is the lone above-reference default outlier; position-0 and norm controls in Figure 2 move it into the sub-reference regime. Full numeric values are reported in Table 2 .

Figure 2: Position-0 and norm control variants for a six-model representative set. Points show gap to

F_{L}

in percent; the vertical line marks the reference. “First non-BOS” uses position 1 only, “token-normed” normalises each token before pooling, and top-PC controls remove the leading cross-sample components before CKA. The controls can move the measured gap substantially, especially for Gemma-3-12B and BOS-sensitive Llama/Qwen cases, but all controlled variants in this panel are sub-reference. Exact values and the full sixteen-model table are in Appendix S4.

Figure 3: Effective-depth training dynamics. Left: gap to

F_{L}=2L/(L+1)

as a percentage, vs. training tokens. Right:

D_{\mathrm{eff}}/L

vs. training tokens. Three OLMo-2 sizes (1B, 7B, 13B) converge to a stable

40

–

41\%

gap by the earliest landed post-initialisation checkpoints and persist through 5T tokens; Pythia-1.4B shows a distinct monotonic decrease. The figure plots all landed checkpoints (25 of 35 planned); missing OLMo-2 points correspond to unavailable or failed revisions and are not interpolated.

Model	$L$	Trained	Random	$\|\Delta\|$
OLMo-2-1B	16	0.070	0.074	0.004
Qwen3.5-0.8B	24	0.045	0.060	0.016
Qwen3.5-2B	24	0.045	0.063	0.018
Qwen3.5-4B	32	0.034	0.047	0.013
Llama-3.1-8B	32	0.046	0.035	0.011
OLMo-2-7B	32	0.037	0.036	0.001

Table 1: Multi-family null baseline (

N{=}10{,}000

, FineWeb-Edu). Trained and random-weight

D_{\mathrm{eff}}/L

remain in the same low range across model families for completed pairs (maximum absolute difference 0.018). For OLMo-2-1B/7B/13B, Qwen3.5-4B/9B/27B, and Llama-3.1-8B, the random column reports the mean of five independent seeds; for the other rows it is a single architecture-default initialisation.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$L$	$d$	$D_{\mathrm{eff}}$	$D_{\mathrm{eff}}/L$	gap (%)	$\hat{\varrho}(1)$
Pythia-70M	6	512	1.23	0.205	$+28.2$	0.862
Pythia-160M	12	768	1.34	0.112	$+27.2$	0.856
Pythia-410M	24	1024	1.46	0.061	$+23.8$	0.940
Pythia-1B	16	2048	1.46	0.091	$+22.7$	0.914
Pythia-1.4B	24	2048	1.42	0.059	$+26.2$	0.954
OLMo-2-1B	16	2048	1.12	0.070	$+40.5$	0.972

Appendix

Table 2: Effective depth (

N{=}10{,}000

, FineWeb-Edu,

K{=}L{-}1

). Absolute

D_{\mathrm{eff}}

remains

O(1)

across scales. All models except Gemma-3-12b sit below the orthogonal-update reference

F_{L}=2L/(L+1)

(positive gap), with the gap clustering by family; Gemma-3-12b is the lone above-reference default outlier.

Construction	$D_{\mathrm{eff}}(h)/L$	$D_{\mathrm{eff}}(f)/L$	$\hat{\varrho}_{h}(1)$
Independent states	0.258	–	0.093
Orthogonal states	0.996	–	0.0001
Orthogonal updates (scale sweep)	0.031–0.056	0.258	0.911–0.9999

Appendix

Table S1: Controlled synthetic constructions. Independent and orthogonal states show that the metric rises when states are genuinely decorrelated; orthogonal-update sweeps show that

D_{\mathrm{eff}}(h)

can remain low even when the updates themselves are diverse.

Figure S1:

D_{\mathrm{eff}}/L

for synthetic constructions (green, blue) and real models (red). Orthogonal states recover

D_{\mathrm{eff}}/L=0.996

; independent states give 0.258; real models fall well below both, confirming that the low values reflect genuine redundancy in the accumulated residual stream.

Figure S2: Depth sweep for synthetic residual constructions. Orthogonal states recover

D_{\mathrm{eff}}=L

, whereas accumulated orthogonal updates obey

D_{\mathrm{eff}}=2L/(L+1)<2

and therefore have

D_{\mathrm{eff}}/L=O(1/L)

. The finite-sample iid reference matches the positive CKA bias observed in the sampled iid control; it is shown only as an estimator calibration.

$\\|f\\|/\\|h\\|$ scale	$D_{\mathrm{eff}}(h)/L$	$D_{\mathrm{eff}}(f)/L$	$\hat{\varrho}_{h}(1)$
0.01	0.031	0.258	0.9999
0.05	0.032	0.258	0.998
0.10	0.034	0.258	0.992
0.30	0.044	0.258	0.964
0.50	0.049	0.258	0.945
1.00	0.054	0.258	0.924

Appendix

Table S2: Orthogonal-update construction (

N{=}10{,}000

L{=}32

d{=}1024

). Diverse updates can coexist with very low accumulated-state depth.

Figure S3: Orthogonal-update sweep (

N{=}10{,}000

L{=}32

d{=}1024

D_{\mathrm{eff}}(f)/L

(updates, red) remains constant at 0.258 regardless of scale, while

D_{\mathrm{eff}}(h)/L

(accumulated, blue) stays in the real-model range (gray band). Diverse updates do not imply diverse accumulated states.

Model	$L$	$D_{\mathrm{eff}}$	gap to $F_{L}$	gap to $F_{\mathrm{norm}}$	gap to $F_{h_{0}+\mathrm{norm}}$	gap to $F_{h_{0}+\mathrm{corr}}$
Pythia-70M	6	1.23	+28.3	+59.0	+59.0	-14.7
Pythia-160M	12	1.34	+27.4	+54.5	+54.5	-24.6
Pythia-410M	24	1.46	+23.7	+46.6	+46.6	-39.0
Pythia-1B	16	1.45	+23.0	+34.9	+34.9	-38.6
Pythia-1.4B	24	1.42	+26.0	+42.1	+42.1	-36.9
OLMo-2-1B	16	1.12	+40.4	+64.1	+61.7	-5.8

Appendix

Table S3: Matched reference baselines for

D_{\mathrm{eff}}

F_{L}

is the universal equal-update reference;

F_{\mathrm{norm}}

matches measured update covariance norms;

F_{h_{0}+\mathrm{norm}}

additionally retains the persistent initial state;

F_{h_{0}+\mathrm{corr}}

also retains the measured update-similarity lag profile. Gaps are relative to each reference, in percent.

Model	$L$	$D_{\mathrm{eff}}/L$	gap%	first non-BOS	token-normed	top-1 PC	top-3 PC	max pos-0 norm ratio
		Default		Control variants (gap to $F_{L}$ , %)
Pythia-70M	6	0.2047	+28.3	-6.5	+30.1	+20.0	+11.4	10.0 $\times$ at $L_{3}$
Pythia-160M	12	0.1117	+27.4	+22.1	+33.7	+27.7	+23.3	18.4 $\times$ at $L_{4}$
Pythia-410M	24	0.0610	+23.7	+8.7	+38.4	+34.3	+27.9	42.1 $\times$ at $L_{9}$
Pythia-1B	16	0.0906	+23.0	+13.7	+40.0	+35.3	+29.7	35.3 $\times$ at $L_{6}$
Pythia-1.4B	24	0.0592	+26.0	+14.1	+41.8	+37.2	+30.9	24.6 $\times$ at $L_{5}$

Appendix

Table S4: Position-0 and norm control variants of

D_{\mathrm{eff}}

for the full model set (

N{=}10{,}000

, FineWeb-Edu). “Default” columns reproduce the canonical pooling used in Table 2 of the main paper. “First non-BOS” replaces the pooled mean with position 1 only; “token-normed” divides each token’s hidden state by its

\ell_{2}

norm before mean pooling; “top-1/3 PC” projects out dominant cross-sample directions from each layer’s pooled matrix. The only above-reference model in the canonical setup, Gemma-3-12b, becomes strongly sub-reference under every control variant.

Model	Default	first non-BOS	token-normed	top-1 PC	top-3 PC
Gemma-3-4B	$+2.2$	$+35.0$	$+30.8$	$+23.4$	$+25.4$
Gemma-3-27B	$-8.3$	$+25.5$	$+7.4$	$+3.0$	$+28.0$

Appendix

Table S5: Additional Gemma-3 control variants. Values are gaps to

F_{L}

in percent. The default-pooling anomaly persists across larger Gemma-3 sizes, but all controls move both extra models into the sub-reference regime.

Model	Intervention	Default	first non-BOS	token-normed	top-1 PC	top-3 PC
Gemma-3-4B	none	$+2.2$	$+35.0$	$+30.8$	$+23.4$	$+25.4$
	no post	$+48.1$	$+48.5$	$+48.3$	$+40.8$	$+47.5$
	no QK	$-60.2$	$+31.3$	$+6.3$	$-8.4$	$+8.6$
	no post/no QK	$+48.2$	$+48.5$	$+48.4$	$+47.7$	$+47.6$
Gemma-3-12B	none	$-17.0$	$+35.3$	$+34.4$	$+31.4$	$+26.6$
	no post	$+47.6$	$+48.9$	$+48.3$	$+47.2$	$+46.0$

Appendix

Table S6: Gemma-scale forward-pass interventions. Values are gaps to

F_{L}

in percent under the same

N{=}10{,}000

FineWeb-Edu extraction as Table S5 . “No post” replaces the post-attention and post-MLP RMSNorms by identities; “no QK” replaces the query/key RMSNorms by identities.

Variant	Default	first pos-1	token-normed	top-1 PC	top-3 PC	max pos-0 ratio
none	$+43.7{\pm}0.2$	$+43.9{\pm}0.1$	$+43.7{\pm}0.2$	$+42.8{\pm}0.1$	$+42.2{\pm}0.1$	$1.00{\pm}0.00$
no post	$+47.1{\pm}0.1$	$+43.6{\pm}0.1$	$+47.1{\pm}0.1$	$+46.0{\pm}0.1$	$+44.7{\pm}0.2$	$1.20{\pm}0.00$
no QK	$+43.7{\pm}0.2$	$+43.9{\pm}0.0$	$+43.7{\pm}0.2$	$+42.8{\pm}0.1$	$+42.2{\pm}0.1$	$1.00{\pm}0.00$
no post/no QK	$+47.1{\pm}0.1$	$+43.6{\pm}0.1$	$+47.1{\pm}0.1$	$+46.0{\pm}0.1$	$+44.7{\pm}0.2$	$1.19{\pm}0.00$

Appendix

Table S7: Gemma-3-4B initialisation-only norm probes. Values are gaps to

F_{L}

in percent, averaged over three random seeds with standard deviations. All jobs use max_steps=0 ; no training is performed. The pretrained Gemma-style default anomaly is absent at random initialisation.

Variant	Default	first pos-1	token-normed	top-1 PC	top-3 PC	max pos-0 ratio	val loss
none	$+41.3{\pm}0.6$	$+33.0{\pm}0.5$	$+41.0{\pm}0.6$	$+35.7{\pm}2.7$	$+32.6{\pm}2.1$	$1.48{\pm}0.04$	$5.06{\pm}0.01$
no post	$+40.7{\pm}0.3$	$+22.8{\pm}1.7$	$+40.6{\pm}0.4$	$+36.8{\pm}1.7$	$+33.9{\pm}3.0$	$5.50{\pm}1.17$	$5.08{\pm}0.01$
no QK	$+41.7{\pm}0.3$	$+31.3{\pm}2.3$	$+41.6{\pm}0.3$	$+36.2{\pm}2.9$	$+32.9{\pm}2.7$	$1.79{\pm}0.18$	$5.07{\pm}0.01$
no post/no QK	$+40.7{\pm}0.3$	$+26.7{\pm}0.9$	$+40.8{\pm}0.3$	$+36.6{\pm}2.0$	$+34.3{\pm}2.8$	$5.06{\pm}1.24$	$5.09{\pm}0.01$

Appendix

Table S8: Gemma-3-270M short-pretraining norm probes. Values are final checkpoint gaps to

F_{L}

in percent, averaged over three seeds with standard deviations. These runs use a short FineWeb-Edu pretraining budget and are not intended to match the full Gemma training pipeline. None of the variants reproduces the pretrained Gemma anomaly.

Model	$L$	$K{=}5$	$K{=}10$	$K{=}15$	$K{=}25$	$K{=}L{-}1$
Pythia-70M	6	0.239	–	–	–	–
Pythia-160M	12	0.145	0.112	–	–	0.110
Pythia-1.4B	24	0.116	0.076	0.064	–	0.057
Qwen3.5-4B	32	0.102	0.060	0.045	0.035	0.034
OLMo-2-7B	32	0.103	0.061	0.047	0.037	0.035
Llama-3.1-8B	32	0.106	0.064	0.052	0.046	0.045

Appendix

Table S9: Sensitivity of

D_{\mathrm{eff}}/L

to maximum lag

K

. The paper uses full

K=L{-}1

throughout. For most models values stabilize by

K=25

, while for Qwen3.5-27B the gap between

K=25

and

K=L{-}1

is 0.009.

Figure S4: Four independent aggregation methods applied to the same CKA similarity matrices. Despite different mathematical formulations, all methods agree on the model ordering and confirm that deep models (

L\geq 32

) have effective depth below 10% of

L

Model	Trained	Random mean	Random range	mean $\pm 2\sigma$
OLMo-2-1B	0.0702	0.0741	[0.0739, 0.0743]	[0.0737, 0.0745]
OLMo-2-7B	0.0369	0.0361	[0.0360, 0.0362]	[0.0359, 0.0362]
OLMo-2-13B	0.0294	0.0286	[0.0286, 0.0287]	[0.0285, 0.0287]
Qwen3.5-4B	0.0350	0.0469	[0.0468, 0.0470]	[0.0468, 0.0471]
Qwen3.5-9B	0.0369	0.0442	[0.0441, 0.0442]	[0.0441, 0.0443]
Qwen3.5-27B	0.0211	0.0217	[0.0216, 0.0218]	[0.0216, 0.0218]

Appendix

Table S10: Five-seed random-weight stability for

D_{\mathrm{eff}}/L

(

N{=}10{,}000

, FineWeb-Edu,

K{=}L{-}1

). The narrow random-weight bands show that the null baseline is nearly deterministic for a fixed architecture and initialisation scheme; trained values fall outside the random range in all seven cases.

Figure S5: Trained vs. random-weight

D_{\mathrm{eff}}/L

across model families for completed pairs (sorted by depth

L

). Bar heights remain in the same low range for trained and random, indicating a strong residual-geometric component.

Figure S6:

D_{\mathrm{eff}}/L

vs. geodesic efficiency across all model–dataset pairs. Pearson

r=0.65

. The moderate correlation confirms that the CKA-based diagnostic and the geometric measure capture related but distinct aspects of layer utilisation.

Model	CKA	Unbiased HSIC-CKA	Cosine
Pythia-70M	0.239	0.240	0.277
OLMo-2-1B	0.071	0.071	0.137
Llama-3.1-8B	0.035	0.035	0.051
Mistral-7B	0.036	0.036	0.047
OLMo-2-7B	0.037	0.038	0.054
OLMo-2-13B	0.033	0.033	0.046

Appendix

Table S11: Auxiliary metric-robustness check:

D_{\mathrm{eff}}/L

under three similarity metrics on WikiText-103. CKA and unbiased HSIC-CKA agree within 0.002; cosine similarity is slightly higher but follows the same pattern.

Model	$L$	Instruct	Base	$\|\Delta\|$
Qwen3.5-0.8B	24	0.045	0.045	$<$ 0.001
Qwen3.5-4B	32	0.035	0.035	$<$ 0.001
Llama-3.1-8B	32	0.047	0.047	0.002

Appendix

Table S12: Instruction-tuned vs. base model

D_{\mathrm{eff}}/L

(

N{=}10{,}000

). Differences are negligible (

\leq 0.002

), indicating that post-training alignment leaves the residual-stream geometry largely unchanged.

Model	$L$	$D_{\mathrm{eff}}/L$	$D_{\mathrm{eff}}^{\mathrm{upd}}/L$	$\hat{\varrho}^{\mathrm{upd}}(1)$	$\\|f\\|/\\|h\\|$
Pythia-160M	12	0.112	0.154	0.648	0.54
OLMo-2-1B	16	0.070	0.080	0.846	0.50
Qwen3.5-4B	32	0.034	0.045	0.787	0.31
Llama-3.1-8B	32	0.046	0.045	0.806	0.37
OLMo-2-13B	40	0.029	0.041	0.796	0.29
Qwen3.5-27B	64	0.021	0.033	0.601	0.18

Appendix

Table S13: Accumulated

D_{\mathrm{eff}}/L

vs. update-based

D_{\mathrm{eff}}^{\mathrm{update}}/L

(

N{=}10{,}000

, FineWeb-Edu). Synthetic orthogonal-update controls show that accumulation alone can keep

D_{\mathrm{eff}}(h)

low; this table asks whether real-model updates are also correlated. They are.

$\gamma$	val. loss	$D_{\mathrm{eff}}(h)/L$	$\hat{\varrho}_{h}(1)$	$\\|f\\|/\\|h\\|$
1.00	3.46 [3.30, 3.58]	0.094 [0.093, 0.095]	0.96	0.32
0.75	1.76 [1.71, 1.86]	0.092 [0.087, 0.098]	0.95	0.82
0.50	3.35 [3.35, 3.35]	0.124 [0.112, 0.137]	0.84	0.82
0.25	3.35 [3.35, 3.35]	0.268 [0.211, 0.325]	0.52	1.25

Appendix

Table S14: Residual-carry intervention on a 12-layer nanoGPT ( shakespeare_char , iter 5000). Values are mean across seeds with [min, max] in brackets (

n{=}3

seeds for

\gamma\in\{1.0,0.75\}

n{=}2

for

\gamma\in\{0.5,0.25\}

). At

\gamma{=}1.0

, training has already begun to overfit by iter 5000 (best validation loss 1.56 at iter 1500).

Figure S7: Controlled residual-carry intervention on a 12-layer nanoGPT. Bold lines are the across-seed mean at each

\gamma

; faint lines are individual seeds. Top:

D_{\mathrm{eff}}(h)/L

and

D_{\mathrm{eff}}(\mathrm{update})/L

trajectories across training, color-coded by

\gamma

. Bottom: validation loss and the update-to-state norm ratio. Substantially reducing

\gamma

(

\gamma{\leq}0.5

) raises

D_{\mathrm{eff}}(h)/L

from initialisation onward in every seed, but training fails;

\gamma{=}0.75

leaves

D_{\mathrm{eff}}(h)/L

statistically indistinguishable from

\gamma{=}1.0

while training slightly better.

Model	$L$	Point $D_{\mathrm{eff}}/L$	Bootstrap mean	95% CI	CI width
Llama-3.1-8B	32	0.0461	0.0461	[0.0459, 0.0462]	0.0003
OLMo-2-7B	32	0.0357	0.0356	[0.0356, 0.0357]	0.0001
Qwen3.5-4B	32	0.0337	0.0337	[0.0336, 0.0337]	0.0001
Qwen3.5-9B	32	0.0358	0.0358	[0.0357, 0.0359]	0.0002
Qwen3.5-27B	64	0.0211	0.0211	[0.0210, 0.0212]	0.0002

Appendix

Table S15: Passage-bootstrap

D_{\mathrm{eff}}/L

on five representative architectures (

N{=}10{,}000

B{=}100

). The bootstrap mean matches the point estimate to four decimal places, and the 95% CI width is at most

3\times 10^{-4}

in every case. At the three-decimal precision used in Table 2 of the main paper, the interval collapses to the reported point estimate, so we omit it there.

Model	Method	$k{=}0$	$k{=}1$	$k{=}2$	$k{=}4$	$k{=}6$	$k{=}8$
OLMo-2-1B	BI	6.5	9.1	17.9	85.0	2,021	16,091
OLMo-2-1B	Local $D_{\mathrm{eff}}$	6.5	31.0	1,322	87,606	190,276	2,768,762
Llama-3.1-8B	BI	4.6	5.0	5.8	10.0	18.5	56.1
Llama-3.1-8B	Local $D_{\mathrm{eff}}$	4.6	431.1	662.0	34,625	89,183	487,295

Appendix

Table S16: Boundary check: perplexity after

k

skipped layers, ranked by BI vs. by local

\Delta D_{\mathrm{eff}}

. BI is substantially better at predicting safe removals on both models, consistent with

D_{\mathrm{eff}}

being a global accumulated-state diagnostic rather than a per-layer importance signal. Spearman correlations between the two rankings are

\hat{\varrho}_{s}=-0.27

(

p=0.31

) for OLMo-2-1B and

\hat{\varrho}_{s}=-0.19

(

p=0.30

) for Llama-3.1-8B.

Model	$L$	$D_{\mathrm{eff}}/L$	gap (%)	PPL 0	$k_{1.10}$	$f_{1.10}$
OLMo-2-1B	16	0.070	$+40.5$	6.54	0	0.000
Pythia-1.4B	24	0.059	$+26.2$	8.11	0	0.000
Mistral-7B	32	0.048	$+20.8$	3.94	1	0.031
Llama-3.1-8B	32	0.046	$+24.1$	4.60	1	0.031
OLMo-2-7B	32	0.036	$+40.6$	5.08	1	0.031
Qwen3.5-9B	32	0.036	$+40.6$	6.37	1	0.031

Appendix

Table S17: Per-model BI pruning tolerance (

N{=}10{,}000

held-out FineWeb-Edu). PPL 0 is the baseline;

k_{1.10}

is the largest

k\in\{0,1,2,4,6,8\}

at which

\mathrm{PPL}_{k}\leq 1.10\cdot\mathrm{PPL}_{0}

;

f_{1.10}=k_{1.10}/L

		$\mathrm{frac}_{1.10}$ vs $D_{\mathrm{eff}}/L$		$k_{1.10}$ vs $D_{\mathrm{eff}}/L$
Subset	$n$	Pearson $r$ ( $p$ )	Spearman $\varrho$ ( $p$ )	Pearson $r$ ( $p$ )	Spearman $\varrho$ ( $p$ )
Full panel	8	$-0.74$ ( $0.04$ )	$-0.79$ ( $0.02$ )	$-0.90$ ( $0.002$ )	$-0.93$ ( $0.001$ )
Drop OLMo-2-1B	7	$-0.59$ ( $0.16$ )	$-0.67$ ( $0.10$ )	$-0.87$ ( $0.01$ )	$-0.91$ ( $0.005$ )
Drop Qwen3.5-4B	7	$-0.82$ ( $0.03$ )	$-0.80$ ( $0.03$ )	$-0.94$ ( $0.002$ )	$-0.91$ ( $0.005$ )
Drop both	6	$-0.68$ ( $0.14$ )	$-0.66$ ( $0.15$ )	$-0.92$ ( $0.009$ )	$-0.86$ ( $0.03$ )

Appendix

Table S18: Correlations of BI pruning tolerance with

D_{\mathrm{eff}}/L

at the full

n{=}8

panel and three leave-one-out subsets that drop the largest- and smallest-

D_{\mathrm{eff}}/L

endpoints. Pre-registered primary metric:

\mathrm{frac}_{1.10}

. Secondary: raw

\mathrm{tolerance\_k}_{1.10}

. All

p

-values are two-sided.

Family	Models	MMLU	ARC	HellaSwag	GSM8K-CoT
Pythia	70M, 160M, 410M, 1B, 1.4B	5/5	5/5	5/5	0/5
OLMo-2	1B, 7B, 13B	3/3	3/3	3/3	2/3
Qwen3.5	0.8B, 2B, 4B, 9B, 27B	5/5	5/5	5/5	3/5
Llama	3.1-8B	1/1	1/1	1/1	1/1
Mistral	7B	1/1	1/1	1/1	1/1
Gemma	3-12b	1/1	1/1	1/1	1/1

Appendix

Table S19: Capability-evaluation coverage. MMLU, ARC-Challenge, and HellaSwag are available for all 16 models. GSM8K-CoT is evaluated only for models with at least 4B parameters; the Qwen3.5-27B and Gemma-3-12b GSM8K values exhibit suspicious answer-extraction failures and are not used for claims.

Panel	Task	Pearson $r$	$p$	Spearman $\varrho$	$p$
All 16	MMLU	$-0.640$	$0.008$	$-0.931$	$<0.001$
All 16	ARC	$-0.664$	$0.005$	$-0.842$	$<0.001$
All 16	HellaSwag	$-0.718$	$0.002$	$-0.794$	$<0.001$
All 16	GSM8K-CoT	$-0.291$	$0.484$	$-0.349$	$0.396$
Modern only	MMLU	$-0.749$	$0.008$	–	–
Modern only	ARC	$-0.600$	$0.051$	–	–

Appendix

Table S20: Exploratory correlation between benchmark score and

D_{\mathrm{eff}}/L

. Values report Pearson

r

and Spearman

\varrho

with two-sided

p

-values. All-model correlations are informative but confounded by family and era; the within-Qwen3.5 panel is the cleanest within-family scaling check, but remains collinear with scale and depth and is not a predictive-validity claim.

Dropped family	MMLU	ARC	HellaSwag	GSM8K-CoT
Gemma	$-0.696$ ( $0.004$ )	$-0.730$ ( $0.002$ )	$-0.785$ ( $0.001$ )	$-0.021$ ( $0.964$ )
Llama	$-0.636$ ( $0.011$ )	$-0.663$ ( $0.007$ )	$-0.719$ ( $0.003$ )	$-0.305$ ( $0.505$ )
Mistral	$-0.637$ ( $0.011$ )	$-0.663$ ( $0.007$ )	$-0.724$ ( $0.002$ )	$-0.190$ ( $0.683$ )
OLMo-2	$-0.617$ ( $0.025$ )	$-0.638$ ( $0.019$ )	$-0.708$ ( $0.007$ )	$-0.197$ ( $0.709$ )
Pythia	$-0.749$ ( $0.008$ )	$-0.600$ ( $0.051$ )	$-0.426$ ( $0.192$ )	–
Qwen3.5	$-0.564$ ( $0.071$ )	$-0.606$ ( $0.048$ )	$-0.699$ ( $0.017$ )	$-0.848$ ( $0.070$ )

Appendix

Table S21: Leave-one-family-out robustness for all-16 correlations between score and

D_{\mathrm{eff}}/L

. Entries are Pearson

r

with two-sided

p

-values in parentheses. MMLU and ARC remain negative under every family deletion; HellaSwag weakens after dropping Pythia, indicating an era/family confound.

Model	$\varrho_{s}(\mathrm{GW},\mathrm{CKA})$	$\varrho_{s}(\|i{-}j\|,\mathrm{GW})$	$\varrho_{s}(\|i{-}j\|,\mathrm{CKA})$
Pythia-410M	$-0.86$	$+0.45$	$-0.48$
Llama-3.1-8B	$-0.68$	$+0.54$	$-0.89$
OLMo-2-7B	$-0.47$	$+0.44$	$-0.82$

Appendix

Table S22: Gromov-Wasserstein pilot. GW distance is strongly anti-correlated with

\mathrm{CKA}

over layer pairs, and increases with layer distance, so the regime described in the paper is visible under an alignment-free optimal transport lens as well.

Transformer Interpretability

Transformer interpretability studies how attention, feed-forward layers, and hidden representations support model predictions and behavior.

See all 338 papers

Jul 20, 2026

An Analysis of Residual-Stream Geometry Across Transformer Depth

We propose a transition-centred geometric analysis of transformer residual streams. Relative displacement measures how far representations move between consecutive layers, and orthogonal Procrustes analysis separates each transition into a rigid rotation and a non-rigid residual. Across six instruction-tuned models, on code generation and cross-lingual translation, these measurements reveal reproducible depth regularities. Relative displacement is strongly layer-dependent; typically larger early and late, with a quieter middle third; and nearly invariant across conditions within each model. Rotation magnitude is nearly constant across depth, while Procrustes residual and angle concentration remain depth-modulated, with residual peaking at the final transition. During generation, non-English targets show larger final-layer displacement and residual than English targets. We present these as descriptive geometric regularities, not as measures of computational effort or causal explanations. The contribution is a measurement framework for residual-stream transitions and evidence that, in the settings studied here, depth curves are model-dependent and largely condition-stable.

Sunit Bhattacharya, Ravi Shankar Kolli

ProRata AI

Transformer Interpretability Representation Geometry in Language Models

Ask PDF

Oct 6, 2026

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

As data propagates through a Transformer, the norm of its hidden states grows by orders of magnitude with depth, a phenomenon framed as 'curse of depth' and nearly universally treated as a pathology to be suppressed. We take the opposite view. Across 16 pre-trained LLMs from 9 families, spanning dense, mixture-of-experts and hybrid architectures and Pre-, Peri- and Post-Norm designs, we find that this growth reflects an emergent depth-positional encoding, carried by the only learned per-layer gain on the residual stream, the normalization weight

γ

: with depth,

γ

grows in magnitude and rotates in direction, jointly encoding the layer index. We make this depth-conditioned encoding explicit with LayerRoPE, an implicit analog of RoPE along the depth axis, which replaces all layerwise

γ

vectors with a single shared vector and depth-conditioned scalars, at a net reduction in parameters and

<0.02\%

change in FLOPs. Across a model ladder scaled up to

100

B+ tokens, LayerRoPE consistently outperforms Pre-, Post- and Peri-Norm and Layer-Norm Scaling, reaching Pre-Norm's 1.3B loss with

3.4\times

less compute; LayerRoPE is the only approach that shows strong convergence and improves near monotonically as depth scales to 512 layers. It improves learning-rate sensitivity by

3

10\times

, and transfers naively to and consistently improves looped latent models and Vision Transformers. Inspecting its learned schedule inverts the prevailing premise: LayerRoPE does not shrink the residual stream but widens it, damping what each block reads while amplifying what it writes. Depth stability, our results suggest, calls not for suppressing the residual stream, but for depth-conditioned regulation of the computational blocks it feeds.

Shikhar Srivastava, Christopher Kanan

University of Rochester

Transformer Positional Encoding Layer Normalization

Ask PDF

Mar 17, 2026

Residual Stream Duality in Modern Transformer Architectures

Recent work has made clear that the residual pathway is not mere optimization plumbing; it is part of the model's representational machinery. We agree, but argue that the cleanest way to organize this design space is through a two-axis view of the Transformer. A decoder evolves information along two ordered dimensions: sequence position and layer depth. Self-attention already provides adaptive mixing along the sequence axis, whereas the residual stream usually performs fixed addition along the depth axis. If we fix a token position and treat layer index as the ordered variable, then a causal depth-wise residual attention read is exactly the same local operator as causal short sliding-window attention (ShortSWA), except written over depth rather than over sequence. This is the core residual stream duality behind Transformer

^2

. This perspective also clarifies the recent literature. ELC-BERT and DenseFormer already show that learned aggregation over depth can outperform uniform residual accumulation, while Vertical Attention, DeepCrossAttention (DCA), MUDDFormer, and Attention Residuals move further toward explicit attention-based routing over earlier layers. The key point, however, is that operator-level duality does not imply systems-level symmetry. For large-scale autoregressive models, sequence-axis ShortSWA is usually the more hardware-friendly placement because it reuses token-side sliding-window kernels, KV-cache layouts, and chunked execution. If the goal is instead to change the shortcut itself, Deep Delta Learning (DDL) is the cleaner intervention because it modifies the residual operator directly rather than adding a separate cross-layer retrieval path. Our recommendation is therefore simple: use DDL when the shortcut is the object of interest, and use sequence-axis ShortSWA when the goal is local adaptive mixing.

Yifan Zhang

Transformer Attention Mechanisms Attention Residual Connections

Ask PDF

Abstract

Transformer Interpretability

Explore similar work

An Analysis of Residual-Stream Geometry Across Transformer Depth

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition

Residual Stream Duality in Modern Transformer Architectures