DepthBench: Measuring How Residual Connections Enable More Computational Depth

Keyu Wang; Yangyi Huang; Jiale Kang; David González-Martínez; Weiyang Liu; Shiwei Liu

cs.CVSep 26, 2026

DepthBench: Measuring How Residual Connections Enable More Computational Depth

Authors: Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu

Organizations: ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center · The Chinese University of Hong Kong

Abstract

Depth is a natural way to increase the computational capacity in Transformers, yet the contribution of deeper layers can diminish as depth grows larger. Recent approaches enhance normalization (\text{e.g.}, LayerNorm Scaling) or residual connections (\text{e.g.}, mHC, AttnRes) to enable better information flow and depth utilization. However, it remains unclear whether they truly translate increased architectural depth into effective computational depth, and whether their reported gains stem from better access to information across depth, or unaccounted-for confounding factors. In this paper, we introduce DepthBench, a controlled benchmark for studying computational depth across various architectures. We systematically vary the width--depth aspect ratio ( $d_{\text{model}}/n_{\text{layer}}$ ) from shallow--wide to deep--narrow shapes, while keeping the model size and pre-training recipe fixed. Across 10 representative architectures, we find that the benefit of allocating more capacity to depth is strongly architecture-dependent. Standard Pre-LN and most of its norm- and scaling-based variants provide little benefit and can even degrade performance as models become deeper and narrower, whereas HC and Full AttnRes improve consistently even at extreme deep shapes. These gains extend beyond pre-training loss and consistently translate into improved domain-specific performance and effective computation. Controlled layer-level analyses further show that the gains of HC and Full AttnRes are associated with more effective utilization of additional layers, revealing distinct mechanisms of computational depth across architectures. Overall, our results identify residual connection design as a key determinant of whether depth can serve as a meaningful scaling axis by enabling additional architectural depth to translate into effective computation.

Figures & tables

Figure 1: Left: Validation loss across aspect ratios under an approximately fixed 400M parameter budget and shared pre-training recipe. Right: Model sizes and aspect ratios of representative industry-released models with Pre-LN architecture, overlaid with shaded regions illustrating the contrasting shape preferences observed in DepthBench .

Method	Update rule	Weight	Source
Baseline
Pre-LN ( Xiong et al., 2020 )	$h_{\ell}=h_{\ell-1}+F_{\ell}(\mathrm{LN}(h_{\ell-1}))$	Fixed	$h_{\ell-1}$
LayerNorm variants
Sandwich-LN ( Ding et al., 2021 )	$h_{\ell}=h_{\ell-1}+\mathrm{LN}_{\mathrm{out}}\!\left(F_{\ell}(\mathrm{LN}_{\mathrm{in}}(h_{\ell-1}))\right)$	Fixed	$h_{\ell-1}$
LNS ( Sun et al., 2026 )	$h_{\ell}=h_{\ell-1}+F_{\ell}\!\left(\frac{1}{\sqrt{\ell}}\mathrm{LN}(h_{\ell-1})\right)$	Fixed	$h_{\ell-1}$
DeepNorm ( Wang et al., 2024 )	$h_{\ell}=\mathrm{LN}\!\left(\alpha h_{\ell-1}+F_{\ell}(h_{\ell-1})\right)$	Fixed	$h_{\ell-1}$

Table 1: Overview of residual update mechanisms for the architectures ( Team et al., 2026b ) . Architectures differ primarily in how layer

\ell

receives and aggregates information from preceding layers. Weight : whether the mixing coefficients are fixed or input-dependent (dynamic). Source : which earlier representations layer

\ell

can access.

Setting	Matched Budget	# Shapes	Aspect Ratios	Purpose
Main benchmark	$\approx$ 400M total	7	$9.1$ – $76.0$	Comprehensive architecture comparison
Iso-backbone control	$\approx$ 300M backbone	6	$6.5$ – $78.0$	Control for backbone-size variation
Multi-scale suite	200M–500M total	3	$\approx$ 28, $\approx$ 43, $\approx$ 76	Test consistency across model scales
Validation at scale	$\approx$ 1.6B total	3	$27.9,\ 43.2,\ 73.1$	Validate trends at larger scale

Table 2: Overview of the experimental settings in DepthBench .

Layers	Hidden	Intermediate	Heads	Head Dim.	Aspect Ratio	Backbone Size	Total Size	Total Diff.
16	1216	3248	16	76	76.00	284M	407M	$+0.28\%$
20	1120	2992	16	70	56.00	301M	414M	$+2.14\%$
24	1024	2736	16	64	42.67	302M	405M	$0.00\%$
28	960	2560	16	60	34.29	310M	406M	$+0.21\%$
32	896	2400	16	56	28.00	309M	399M	$-1.49\%$
42	800	2144	16	50	19.05	324M	404M	$-0.31\%$

Table 3: 400M model configurations across different width–depth aspect ratios.

Pre-LN	Sandwich-LN	LNS	DeepNorm	KEEL	MoDA	Full AttnRes	Block AttnRes	HC	mHC
$2\times 10^{-3}$	$2\times 10^{-3}$	$1\times 10^{-2}$	$1\times 10^{-3}$	$2\times 10^{-3}$	$2\times 10^{-3}$	$2\times 10^{-3}$	$2\times 10^{-3}$	$2\times 10^{-3}$	$2\times 10^{-3}$

Table 4: Optimal learning rate across architectures.

Figure 2: Left: 400M main results of validation loss across 10 architectures. Right: Validation loss across a wider range of shapes, including extreme small aspect ratios, under fixed total size (400M, top) and fixed backbone size (300M, bottom) for Pre-LN, HC and Full AttnRes.

Figure 3: Validation loss for Pre-LN, Full AttnRes, and HC under three aspect ratios. Left: {200M, 300M, 400M, 500M} models. Right: 1.6B models.

Figure 4: NLL results of 400M models on Coding, STEM and Math categories. We report negative log-likelihood across model aspect ratios and architectures. Each panel shows the average NLL over two representative tasks: MBPP ( Austin et al., 2021 ) and HumanEval ( Chen et al., 2021 ) for coding (left) , SciQ ( Welbl et al., 2017 ) and GPQA ( Rein et al., 2023 ) for STEM (middle) , and GSM8K ( Cobbe et al., 2021 ) and MATH-500 ( Lightman et al., 2024 ) for math (right) .

Figure 5: Angular distance from the initial layer $\ell$ (x-axis) and its subsequent $n^{th}$ layer (y-axis) . Angular distance ranges in [0, 1] and yellow indicates smaller distances and dark blue indicates larger distances. Here, we use a unified learning rate of $2\times 10^{-3}$ for all models (only LNS does not use the optimal learning rate), because the learning rate significantly affects the learned representations, as shown in Appendix C . We exclude DeepNorm because it explodes at a learning rate of $2\times 10^{-3}$ .

Figure 6: (a,c) Fraction of layer pairs in the last

3/4

of the network exceeding the causal-score (

>0.45

) and permutation-score (

>0.15

) thresholds, respectively. (b,d) Pairwise causal and permutation score maps for

L=32

models. Higher causal scores indicate stronger cross-layer dependence, while higher permutation scores indicate lower layer interchangeability.

Figure 7: Explicit residual-path weights $w_{i\to\ell}$ . Each cell shows the mixing weight of sublayer output $f_{i}$ to the pre-norm input of a later sublayer $\ell$ , averaged over validation calibration tokens. (a) Pre-LN: $w_{i\to\ell}\equiv 1$ . (b) Full AttnRes: $w_{i\to\ell}$ is the per-token softmax weight on $f_{i}$ . (c) HC: $w_{i\to\ell}=\beta_{i}^{\top}A_{i+1}\cdots A_{\ell-1}\alpha_{\ell}$ , combining write mappings, residual mappings, and read mappings.

Figure 8: Left: Performance drop after removing a single layer. Right: The KL divergence between layerwise and final prediction distributions.

Figure 9: Block AttnRes depth-mixing mechanism at

L=16

24

32

. At each layer, the share of the depth-mixing weight in Eq. 9 placed on the last layer output (

\alpha_{\ell}

, green) versus on previous block outputs (

\sum_{\mathbf{v}}\beta_{\ell,\mathbf{v}}

, grey); dashed lines mark block boundaries, where the set of saved block outputs is updated.

Figure 10: Residual transport in HC and mHC. (a) Effective rank across aspect ratios, with shapes

(L,d)

below each tick. Effective rank is the exponential of the entropy of the normalized singular values. (b) Mean absolute cosine similarity between propagated stream directions in

L=32

models. (c) Singular values of their full residual products, normalized by the largest. Statistics are averaged over sampled held-out FineWeb-Edu data; bands and error bars show 95% document-bootstrap intervals.

Figure 11: Measured prefill cost including FLOPs, KV cache memory, Peak GPU memory and GPU hours at sequence length

=4096

, model size at about 400M, BF16, A100-SXM4-40GB. The dashed line in (a) corresponds to validation loss.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Peak LR	$5\times 10^{-3}$	$2\times 10^{-3}$	$1\times 10^{-3}$	$5\times 10^{-4}$	$2\times 10^{-4}$
Pre-LN MoDA	2.775	2.764	2.773	2.864	–
Post-LN MoDA	4.893	3.842	2.912	2.933	3.443

Appendix

Table 5: Validation loss for MoDA normalization variants (

L=24

model).

Architecture	Parameters	$\Delta$ vs. Pre-LN
Pre-LN / LNS / DeepNorm	405.41M	—
Sandwich-LN	405.46M	$+0.049$ M
KEEL	405.45M	$+0.048$ M
AttnRes (Full / Block)	405.51M	$+0.098$ M
HC ( $n{=}4$ )	405.70M	$+0.296$ M
mHC ( $n{=}4$ )	410.13M	$+4.72$ M

Appendix

Table 6: Total parameters and overhead relative to Pre-LN at

L=24

(

d_{\text{model}}{=}1024

Layers	Hidden	Intermediate	Heads	Head Dim.	Aspect Ratio	Backbone Size	Backbone Diff.	Total Size
16	1248	3328	16	78	78.00	299M	$-1.11\%$	425M
24	1024	2736	16	64	42.67	302M	$\phantom{+}0.00\%$	405M
32	896	2400	16	56	28.00	309M	$+2.26\%$	399M
42	768	2048	16	48	18.29	297M	$-1.69\%$	375M
70	608	1632	16	38	8.69	312M	$+3.15\%$	373M
84	544	1456	16	34	6.48	299M	$-0.99\%$	354M

Appendix

Table 7: Iso-backbone 300M model configurations

Layers	Hidden	Intermediate	Heads	Head Dim.	Aspect Ratio	Backbone Size	Total Size	Total Diff.
200M – 4B training tokens
24	672	1792	16	42	28.00	130M	198M	$-3.42\%$
18	768	2048	16	48	42.67	127M	205M	$0.00\%$
12	896	2400	16	56	74.67	116M	206M	$+0.69\%$
300M – 6B training tokens
28	800	2144	16	50	28.57	216M	296M	$+1.09\%$

Appendix

Table 8: Model configurations for the depth scaling ladder across 200M–500M scales.

Layers	Hidden	Intermediate	Q/KV Heads	Head Dim.	Aspect Ratio	Backbone Size	Total Size	Total Diff.
28	2048	6144	16 / 8	128	73.14	1.409B	1.615B	$0.00\%$
40	1728	5184	16 / 8	108	43.20	1.433B	1.607B	$-0.51$ %
54	1504	4512	16 / 8	94	27.85	1.466B	1.617B	$+0.11\%$

Appendix

Table 9: 1.6B model configurations across different width–depth aspect ratios.

Figure 12: Learn rate sweep over the architectures. We choose the same optimal learning rate for Block AttnRes as for Full AttnRes. For mHC, we used Legacy near-uniform init for the learning rate sweep, which differs very slightly from the results reported in the main text using 0/-8 gap-8 init .

Figure 13: The analyses in Section 4.1 can be affected by the choice of learning rate.

Panel of Figure 14 — Figure 14: Causal score for all architectures at $L=16$ , $24$ , and $32$ (top to bottom).

Panel of Figure 15 — Figure 15: Permutation score for all architectures at $L=16$ , $24$ , and $32$ (top to bottom).

Panel of Figure 16 — Figure 16: Angular distance for all architectures at $L=16$ , $24$ , and $32$ (top to bottom).

Attention Residual Connections

Attention residual connections use learned, input-dependent routing to combine earlier layer outputs instead of relying only on fixed skips.

See all 23 papers

Scaling Adaptive Depth with Norm-Agnostic Residual Networks
Jun 15, 2026Tomás Figliolia, Beren MillidgeEfficient Neural Network Inference Dynamic Neural Networks
Zyphra San Francisco, CA
ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling
Feb 9, 2026Yilang Zhang, Bingcong Li, Niao He +1Residual Learning Neural Network Optimization
ML Research, Morgan Stanley, New York, NY 10019 · Department of CS, ETH Zürich, 8092 Zürich, Switzerland · Department of ECE, University of Minnesota, Minneapolis, MN 55455
LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
Oct 6, 2026Shikhar Srivastava, Christopher KananTransformer Positional Encoding
University of Rochester

Jun 15, 2026J/K move · Enter open · S save

Scaling Adaptive Depth with Norm-Agnostic Residual Networks

Tomás Figliolia, Beren Millidge

Zyphra San Francisco, CA

Efficient Neural Network Inference Dynamic Neural Networks Transformer Residual Learning

Open paper AskPDF arXiv

Residual architectures are ubiquitous in deep learning, but they suffer from a subtle structural limitation: the norm of the residual stream can grow rapidly with depth. As a result, updates from later layers become small relative to the accumulated residual state. This reduces their impact on the representation and limits the benefits of scaling models in depth. To address this, we introduce NAG, a norm-agnostic residual architecture that separates magnitude from directional information in the residual stream, preserving meaningful layer contributions throughout depth and preventing later updates from being systematically suppressed by residual-norm growth. Importantly, NAG introduces only a negligible number of additional parameters and relies on simple operations that are easily kernel-fusible, preserving training efficiency in practice. We show that this architecture outperforms baseline Transformers, with gains that increase substantially as depth grows, enabling effective training of much deeper models. The norm-agnostic formulation also leads to an interpretable Mixture-of-Depths (MoD) mechanism that adaptively skips both attention and MLP layers. Beyond serving as a post-training accuracy-compute tradeoff, this mechanism can be used as a pretraining-time scaling strategy: under iso-FLOP training, compute saved by reducing per-token forward-pass cost can be reinvested into training on more tokens while keeping the total parameter count and KV-cache budget fixed. In our experiments, moderate Mixture-of-Depths rates of approximately 20%-25% match full-depth baseline performance under equal training compute while substantially reducing the number of executed layer parameters and forward-pass FLOPs. These results identify sparsity in depth as a new scaling axis for fixed-compute training, enabling very deep yet FLOP-efficient models.

Abstract

Attention Residual Connections

Explore similar work

Scaling Adaptive Depth with Norm-Agnostic Residual Networks

ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling

LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition