Attention Residual Connections

Overview

Attention residual connections study how a network can retrieve and mix information from earlier layers, rather than passing it forward only through fixed residual additions. Common approaches use attention over layer outputs or blocks, with variants routing layer updates, multiple streams, or compressed representations to control computation and memory. Research evaluates these designs in language-model scaling and downstream benchmarks, as well as in vision, diffusion, and recurrent models. Results are promising, but added overhead, the conditions under which routing improves performance, and whether routing weights reflect causal contributions remain open questions.

Latest papers

CardsList

26 papers2 papers in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

  1. Loopback Transformer: Towards a Unified View of Residual Looped Transformers

    Nov 4, 2026Xuan Ouyang, Zefan Cai, Chenlong Zhang +3TransformerRecurrent Transformers

  2. ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals

    Nov 3, 2026Yuekun Yao, Yupei Du, Dawei Zhu +1Recurrent TransformersAttention Residual Connections

  3. Depth Memory

    Oct 13, 2026David González-Martínez, Philipp Nazari, T. Konstantin Rusch +1Memory-Augmented Language ModelsAssociative Memory

  4. Multi-Stream Attention Residuals

    Oct 9, 2026Xiaohan Qin, Xiangdong Zhang, Tuo Dai +7Attention Residual ConnectionsResidual Learning

  5. Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

    Oct 8, 2026Pengxiang Li, Dilxat Muhtar, Di He +3Recurrent TransformersAttention Residual Connections

  6. Structured Residual Connectivity Matters for Diffusion Transformers

    Sep 27, 2026Yuhe Liu, Xinyin Ma, Gongfan Fang +2Diffusion TransformerDiffusion-Based Image Generation

  7. The Extender: A Log-Structured Transformer

    Sep 26, 2026Jakob ErikssonTransformerTransformer Inference

  8. DepthBench: Measuring How Residual Connections Enable More Computational Depth

    Sep 26, 2026Keyu Wang, Yangyi Huang, Jiale Kang +3Attention Residual ConnectionsResidual Learning

  9. Dataset-Dependent Effects of Cross-Depth Aggregation and Soft-Routed Experts in EEG Foundation Model Fine-Tuning

    Sep 15, 2026Mingyang Jiang, Yamin Li, Daniel Moyer +3Attention Residual ConnectionsEEG Decoding

  10. Multi-Head Attention Residuals

    Jul 22, 2026Cheng Luo, Zefan Cai, Junjie HuTransformerAttention Residual Connections

  11. Dual Attention Residuals

    Jul 21, 2026Xingda Yu, Yining Li, Xinzhang Liu +5TransformerCross-Attention

  12. ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation

    Jun 12, 2026Yanzhao Guo, Wenkai Chen, Jianwei ZhangLanguage-Conditioned Robot ManipulationEfficient VLM Inference

  13. Attention Sinks and Outliers in Attention Residuals

    May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM QuantizationAttention Residual Connections

  14. Delta Attention Residuals

    May 13, 2026Cheng Luo, Zefan Cai, Junjie HuCross-Layer AttentionAttention Residual Connections

  15. XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation

    Mar 28, 2026Xinyu Liu, Qing Xu, Zhen ChenAttention Residual ConnectionsMedical Image Segmentation

  16. GradAttn: Replacing Fixed Residual Connections with Task-Modulated Attention Pathways

    Mar 23, 2026Soudeep Ghoshal, Himanshu BuckchashConvolutional Neural NetworksAttention Residual Connections

  17. Attention Residuals

    Mar 16, 2026Kimi Team, Guangyu Chen, Yu Zhang +34Attention Residual ConnectionsResidual Learning

  18. A Structural Theory of Position Bias in Transformers

    Feb 18, 2026Hanna Herasimchyk, Robin Labryga, Tomislav Prusina +1Lost-in-the-Middle EffectPosition Bias