Residual Stream Duality in Modern Transformer Architectures

Yifan Zhang

cs.LGMar 17, 2026

Residual Stream Duality in Modern Transformer Architectures

Authors: Yifan Zhang

Abstract

Recent work has made clear that the residual pathway is not mere optimization plumbing; it is part of the model's representational machinery. We agree, but argue that the cleanest way to organize this design space is through a two-axis view of the Transformer. A decoder evolves information along two ordered dimensions: sequence position and layer depth. Self-attention already provides adaptive mixing along the sequence axis, whereas the residual stream usually performs fixed addition along the depth axis. If we fix a token position and treat layer index as the ordered variable, then a causal depth-wise residual attention read is exactly the same local operator as causal short sliding-window attention (ShortSWA), except written over depth rather than over sequence. This is the core residual stream duality behind Transformer $^2$ . This perspective also clarifies the recent literature. ELC-BERT and DenseFormer already show that learned aggregation over depth can outperform uniform residual accumulation, while Vertical Attention, DeepCrossAttention (DCA), MUDDFormer, and Attention Residuals move further toward explicit attention-based routing over earlier layers. The key point, however, is that operator-level duality does not imply systems-level symmetry. For large-scale autoregressive models, sequence-axis ShortSWA is usually the more hardware-friendly placement because it reuses token-side sliding-window kernels, KV-cache layouts, and chunked execution. If the goal is instead to change the shortcut itself, Deep Delta Learning (DDL) is the cleaner intervention because it modifies the residual operator directly rather than adding a separate cross-layer retrieval path. Our recommendation is therefore simple: use DDL when the shortcut is the object of interest, and use sequence-axis ShortSWA when the goal is local adaptive mixing.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of Residual Stream Duality. A modern Transformer evolves information along two ordered dimensions: sequence position and layer depth. Computing the target state $h_{t}^{(\ell)}$ via explicit depth-wise residual attention operates identically to causal short sliding-window attention (ShortSWA) on the sequence axis, effectively bridging the representation mechanisms of the two pathways.

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 535 papers

Jul 21, 2026

Dual Attention Residuals

Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain multiple residual trajectories. These capabilities have largely been studied in isolation, and assigning an independent retriever to each stream still prevents one trajectory from influencing depth selection in another. We propose Dual Attention Residuals (DAR), which brings multi-stream interaction into historical retrieval through reciprocal cross-stream addressing. For each target stream, DAR computes depth weights from normalized states in the opposite stream and applies them to values from the target stream's own history. The retrieved states are combined for an unchanged Transformer branch and updated through constrained gated writes; a block-form variant operates on block-level histories to control overhead. Across dense models from 0.1B to 1B parameters and a 7B sparse-MoE model, DAR consistently improves validation loss over standard residual Transformers and Attention Residuals. Routing ablations show that the gain cannot be explained by an additional stream or value projection alone. Representation and intervention analyses further show that reciprocal cross-stream selection preserves depth-wise diversity and avoids the redundancy or functional imbalance observed in alternative two-stream designs.

Xingda Yu, Yining Li, Xinzhang Liu +5

TeleAI

Transformer Cross-Attention Attention Residual Connections Residual Learning

Ask PDF

May 13, 2026

Delta Attention Residuals

Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, standard Attention Residuals still attend over cumulative hidden states in previous layers, which are highly redundant. We show that this redundancy leads to routing collapse in deeper layers: attention weights become low-contrast and closer to uniform (max weight

{\approx}

0.2), limiting the model's ability to select informative states in previous layers. This raises a key but underexplored design question: what layer-wise representations should be routed in Attention Residuals? To answer this question, we propose Delta Attention Residuals, which attend over deltas -- the change introduced by each sublayer (

\mathbf{v}_i = \mathbf{h}_{i+1} - \mathbf{h}_i

) -- instead of cumulative states. Delta representations are structurally diverse and yield higher-contrast attention distributions (max weight

{\approx}

0.6), enabling more selective and effective routing across layers. This principle applies at both per-sublayer and block granularity. Across all tested scales (220M--7.6B), Delta Attention Residuals consistently outperform both standard residuals and Attention Residuals, with 1.7--8.2% validation perplexity gains. Delta Attention Residuals also enables converting pretrained checkpoints into Delta Attention Residuals via standard fine-tuning. Code is available at https://github.com/wdlctc/delta-attention-residuals-code.

Cheng Luo, Zefan Cai, Junjie Hu

Independent Researcher · University of Wisconsin–Madison

Cross-Layer Attention Attention Residual Connections Residual Learning Transformer Attention Attention Mechanisms Attention Routing

Ask PDF

Sep 25, 2026

The Residual Stream's Effective Depth

We introduce effective depth (

\Deff

), a scalar diagnostic that treats the layer-wise residual stream of a transformer as a discrete-time process, measures how representation similarity decays with layer distance, and aggregates that profile into one number. Across sixteen decoder-only language models,

\Deff

separates a structural consequence of residual accumulation from an empirical one: even maximally diverse orthogonal updates have the closed-form reference

F_L = 2L/(L+1)<2

, yet fifteen of sixteen default measurements lie below

F_L

(Qwen3.5: 32--44%, OLMo-2: 40--41%, Pythia: 23--28%). Matched references show that the gap is not caused by the persistent initial state or update-size imbalance, but is largely a calibrated signature of correlated residual updates rather than evidence that depth is unused. Symmetric position-0, token-normalisation, and top-PC controls show the regime is not reducible to BOS or top-PC artefacts: the lone above-reference default outlier joins the same regime, and all sixteen models are sub-reference after token-normalisation or top-1-PC removal. Intermediate checkpoints show that the regime is established early in OLMo-2 and stable through 5T tokens, while Pythia-1.4B follows a distinct decreasing trajectory. A controlled residual-carry intervention supports the mechanism, and

\Deff

is best read as a global accumulated-state diagnostic, not as a capability score or pruning method.

Barak Gahtan, Ido Galil, Alex M. Bronstein

Technion Israel Institute of Technology · Nvidia · Technion Israel Institute of Technology & ISTA Institute of Science and Technology Austria

Transformer Interpretability Decoder-Only Language Models Representational Similarity Analysis

Ask PDF

Abstract

Transformer

Explore similar work

Dual Attention Residuals

Delta Attention Residuals

The Residual Stream's Effective Depth