Attention Residual Connections

Follow this topic Ask about this topic

Momentum

1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 10

All topics

Dataset-Dependent Effects of Cross-Depth Aggregation and Soft-Routed Experts in EEG Foundation Model Fine-Tuning
Sep 15, 2026Mingyang Jiang, Yamin Li, Daniel Moyer +3Attention Residual Connections EEG Decoding
Department of Computer Science, Vanderbilt University · Department of Biomedical Informatics and Data Science, Yale University
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Aug 2, 2026Kehan WangAttention Residual Connections Autoregressive Language Modeling
Kimi Team
SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation
Jul 23, 2026Junsong Chen, Jincheng Yu, Yitong Li +11Video Diffusion Models Attention Residual Connections
NVIDIA
Multi-Head Attention Residuals
Jul 22, 2026Cheng Luo, Zefan Cai, Junjie HuTransformer Attention Residual Connections
Independent Researcher · University of Wisconsin–Madison
Dual Attention Residuals
Jul 21, 2026Xingda Yu, Yining Li, Xinzhang Liu +5Transformer Cross-Attention
TeleAI
Low-Rank Attention Residuals
Jun 19, 2026Jonathan SuLow-Rank Attention Decoder-Only Language Models
Independent Researcher
ReactVLA: Fast and Lightweight Reactive Robot Manipulation via Improved Mean Flow Action Generation
Jun 12, 2026Yanzhao Guo, Wenkai Chen, Jianwei ZhangLanguage-Conditioned Robot Manipulation Efficient VLM Inference
Shanghai Jiao Tong University · Technical Aspects of Multimodal Systems (TAMS), Department of Informatics, Universität Hamburg
When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals
Jun 11, 2026Aydin JavadovTransformer Interpretability Attention Residual Connections
ETH Zurich
Attention Sinks and Outliers in Attention Residuals
May 18, 2026Haozheng Luo, Haoran Dai, Shaoyang Zhang +10LLM Quantization Attention Residual Connections
Department of Computer Science, Northwestern University · Department of Computer Science, Illinois Institute of Technolog · Department of Computer Science and Engineering, University of Michigan +5
Delta Attention Residuals
May 13, 2026Cheng Luo, Zefan Cai, Junjie HuAttention Residual Connections Residual Learning
Independent Researcher · University of Wisconsin–Madison

Sep 15, 2026, cs.LGJ/K move · Enter open · S save

Dataset-Dependent Effects of Cross-Depth Aggregation and Soft-Routed Experts in EEG Foundation Model Fine-Tuning

Mingyang Jiang, Yamin Li, Daniel Moyer, Fan Ma+2

Department of Computer Science, Vanderbilt University · Department of Biomedical Informatics and Data Science, Yale University

Attention Residual Connections EEG Decoding Mixture-of-Experts Models EEG Foundation Models

Open paper AskPDF arXiv

EEG decoding tasks can rely on different temporal dynamics and cross-channel relationships. We test whether specialized modules improve a fully fine-tuned EEG foundation model by augmenting CBraMod with cross-depth Attention Residuals (AttnRes) and two soft-routed expert banks. Across matched three-seed experiments on FACED, ISRUC, SEED-V, and PhysioNet-MI, the complete model changes mean balanced accuracy relative to full fine-tuning by -0.12, +1.27, +0.77, and -1.27 points, respectively. AttnRes alone improves mean balanced accuracy on three datasets, whereas adding experts on top of AttnRes helps only FACED and SEED-V. These gains come with substantial overhead: AttnRes requires 2.11 to 2.88x runtime and 1.78 to 2.67x memory, while the complete model requires 2.41 to 3.04x runtime and 1.86 to 2.85x memory. Overall, the added modules produce dataset-dependent, sometimes opposing effects rather than consistent gains over full fine-tuning.

Momentum

Related topics

Latest papers 10