Attention Residual Connections

Follow this topic Ask about this topic

Attention residual connections study how a network can retrieve and mix information from earlier layers, rather than passing it forward only through fixed residual additions. Common approaches use attention over layer outputs or blocks, with variants routing layer updates, multiple streams, or compressed representations to control computation and memory. Research evaluates these designs in language-model scaling and downstream benchmarks, as well as in vision, diffusion, and recurrent models. Results are promising, but added overhead, the conditions under which routing improves performance, and whether routing weights reflect causal contributions remain open questions.

Latest papers

23 papers3 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

Depth Memory
Oct 13, 2026David González-Martínez, Philipp Nazari, T. Konstantin Rusch +1Memory-Augmented Language Models Associative Memory
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center +2
Multi-Stream Attention Residuals
Oct 9, 2026Xiaohan Qin, Xiangdong Zhang, Tuo Dai +7Attention Residual Connections Residual Learning
School of AI, Shanghai Jiao Tong University · Shanghai Innovation Institute · Dots Studio, Xiaohongshu Inc. +2
Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
Oct 8, 2026Pengxiang Li, Dilxat Muhtar, Di He +3Recurrent Transformers Attention Residual Connections
The Hong Kong Polytechnic University · Independent Researcher · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences +6
Structured Residual Connectivity Matters for Diffusion Transformers
Sep 27, 2026Yuhe Liu, Xinyin Ma, Gongfan Fang +2Diffusion Transformer Diffusion-Based Image Generation
National University of Singapore · Shanghai Jiao Tong University
The Extender: A Log-Structured Transformer
Sep 26, 2026Jakob ErikssonTransformer Transformer Inference
UIC · University of Illinois Chicago
DepthBench: Measuring How Residual Connections Enable More Computational Depth
Sep 26, 2026Keyu Wang, Yangyi Huang, Jiale Kang +3Attention Residual Connections Residual Learning
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +1
Dataset-Dependent Effects of Cross-Depth Aggregation and Soft-Routed Experts in EEG Foundation Model Fine-Tuning
Sep 15, 2026Mingyang Jiang, Yamin Li, Daniel Moyer +3Attention Residual Connections EEG Decoding
Department of Computer Science, Vanderbilt University · Department of Biomedical Informatics and Data Science, Yale University
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Aug 2, 2026Kehan WangAttention Residual Connections Autoregressive Language Modeling
Kimi Team
SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation
Jul 23, 2026Junsong Chen, Jincheng Yu, Yitong Li +11Video Diffusion Models Attention Residual Connections
NVIDIA
Multi-Head Attention Residuals
Jul 22, 2026Cheng Luo, Zefan Cai, Junjie HuTransformer Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
Dual Attention Residuals
Jul 21, 2026Xingda Yu, Yining Li, Xinzhang Liu +5Transformer Cross-Attention
TeleAI
Low-Rank Attention Residuals
Jun 19, 2026Jonathan SuLow-Rank Attention Decoder-Only Language Models
Independent Researcher
ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation
Jun 12, 2026Yanzhao Guo, Wenkai Chen, Jianwei ZhangLanguage-Conditioned Robot Manipulation Efficient VLM Inference
Shanghai Jiao Tong University · Technical Aspects of Multimodal Systems (TAMS), Department of Informatics, Universität Hamburg
When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals
Jun 11, 2026Aydin JavadovTransformer Interpretability Attention Residual Connections
ETH Zurich
Attention Sinks and Outliers in Attention Residuals
May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM Quantization Attention Residual Connections
Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5
Delta Attention Residuals
May 13, 2026Cheng Luo, Zefan Cai, Junjie HuCross-Layer Attention Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V
Apr 12, 2026Chirag ShindeTransformer Attention Attention Residual Connections
Independent Researcher
XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation
Mar 28, 2026Xinyu Liu, Qing Xu, Zhen ChenAttention Residual Connections Medical Image Segmentation
Imperial College London · University of Nottingham · The Hong Kong Polytechnic University
GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways
Mar 23, 2026Soudeep Ghoshal, Himanshu BuckchashConvolutional Neural Networks Attention Residual Connections
Kalinga Institute of Industrial Technology, Bhubaneswar, India · IMC University of Applied Sciences Krems, Krems, Austria
Residual Stream Duality in Modern Transformer Architectures
Mar 17, 2026Yifan ZhangTransformer Attention Mechanisms
Attention Residuals
Mar 16, 2026Kimi Team, Guangyu Chen, Yu Zhang +34Attention Residual Connections Residual Learning
Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias
Mar 10, 2026Borun D ChowdhuryLost-in-the-Middle Effect Transformer
Meta, London, W1T 1FB, UK
A Structural Theory of Position Bias in Transformers
Feb 18, 2026Hanna Herasimchyk, Robin Labryga, Tomislav Prusina +1Lost-in-the-Middle Effect Position Bias
University of Hamburg Hamburg, Germany

Oct 13, 2026J/K move · Enter open · S save

Depth Memory

David González-Martínez, Philipp Nazari, T. Konstantin Rusch, Shiwei Liu

ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center+2

Memory-Augmented Language Models Associative Memory Attention Residual Connections

Open paper AskPDF arXiv

Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connections (mHC) and Attention Residuals (AttnRes), instead use more expressive depth aggregation mechanisms that improve language modeling performance at scale. In this work, we consider the mixing of representations over depth as a problem of storing and retrieving information across layers. Formalizing this perspective, we introduce Depth Memory (DM), a unified framework that describes the standard pre-layer-norm (PreLN) transformer, mHC, and AttnRes in terms of streams that carry representations through the network. One such stream, for example, could be an explicit memory stream that is queried to enrich the input to every layer, and whose output is subsequently written back into the memory for use by later layers. For AttnRes, this memory stream is a growing cache of explicit layer representations. For mHC, it is an expanded residual stream. Building on this view, we propose Linear Depth Memory (LDM), which formulates information retrieval across depth as a regression problem solved using recursive least squares. The resulting architecture propagates three streams across depth: the standard residual stream, a linear associative memory stream, and an inverse covariance stream. For high-throughput training and inference, we propose a hardware-aware implementation of LDM that reduces peak activation memory and memory traffic. LDM consistently achieves the lowest perplexities among PreLN, mHC and AttnRes baselines at the 400M and 1.5B parameter scales, alongside competitive zero-shot accuracy.