Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

On the Necessity of Attention-FFN Split in Vision Transformers
Oct 7, 2026Junhyeok Kim, Jinyeong Kim, Jae Wan Park +1Vision Transformer Attention Mechanisms
Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.
The Identifiability and Observability of Deep Normalized Attention
Oct 7, 2026Pranav Venkata KondaAttention Mechanisms Representation Identifiability
Columbia University
Hardware-aware Calibrated Clustered Attention for Efficient Visual Geometric Transformers
Oct 7, 2026Weitian Wang, Shubham Rai, Cecilia De La Parra +1Efficient ViTs Attention Mechanisms
Robert Bosch GmbH, Germany · Ruhr University Bochum, Germany
Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Oct 6, 2026Ryan Solgi, Parsa Madinei, Zheng ZhangAttention Mechanisms Low-Rank Attention
University of California-Santa Barbara Santa Barbara, CA, USA
Random Feature Gaussian Process Attention: Linear-Time Probabilistic Attention with Calibrated Uncertainty
Oct 6, 2026Amir Mohammad Mahfoozi, Zi Yang, Ying Li +1Transformer Attention Attention Mechanisms
Department of Computer Engineering, Sharif University of Technology · School of Computing and Data Science, The University of Hong Kong
Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models
Oct 6, 2026Dayan Pan, Jingyuan Wang, Xie YuAttention Mechanisms Parameter-Efficient Fine-Tuning
School of Computer Science and Engineering, MOE Engineering Research Center of Advanced Computer Application Technology, Beihang University Beijing, China · School of Computer Science and Engineering, School of Economics and Management, Beihang University Beijing, China
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University
Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay
Oct 4, 2026Yiyuan Luo, Vaggos ChatziafratisText Embeddings Attention Mechanisms
University of California, Santa Cruz
More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
Oct 3, 2026Noam Elata, Itay Lamprecht, Mikey Shechter +3LLM Inference Acceleration Sparse Attention
Technion – Haifa, Israel · Intel, Israel · Corma +1
Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention
Oct 1, 2026Michael Fore, James Mason Inder, Mrishika Nair +2Multivariate Time Series Forecasting In-Context Learning
Amazon
Permutation-Robust Decision Modeling with Candidate-Independent Block-Causal Attention
Oct 1, 2026Guy AmitPermutation-Invariant Learning Attention Mechanisms
Independent Researcher
Attention Kernels for Learning Maps Between Heavy-Tailed Measures
Sep 30, 2026Kailen Hargenrader, Edoardo Calvello, Bohan ChenSoftmax Attention Attention Mechanisms
Computing and Mathematical Sciences California Institute of Technology · Lawrence Berkeley National Laboratory University of California, Berkeley ICSI
Cluster Attention Neural Operators for Solving Parametric Partial Differential Equations
Sep 30, 2026Ming Zhong, Antonio Colanera, Gianluigi Rozza +1PDE Solving Attention Mechanisms
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 100049, China. · State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China. · International School for Advanced Studies (SISSA), Trieste 34136, Italy. +1
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
Sep 30, 2026Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5AI Accelerator Inference LLM Inference Acceleration
Faculty of Mathematics University of Vienna, Austria · Huawei Heisenberg Research Center, Munich, Germany · Huawei Technologies Co. Ltd
Attention Function as an Intrinsic Inductive Bias: How Models' Behavior Diverges in Novel Contexts
Sep 30, 2026Dong Gyun Kang, Megha Thukral, Kwangsoo KimAttention Head Analysis OOD Generalization
College of Computing, Georgia Institute of Technology · Department of Transdisciplinary Medicine, Seoul National University Hospital
When Context Changes: Understanding Update Failures in LLMs
Sep 30, 2026Junyu Guo, Yuchen Fang, Shangding Gu +3Belief Updating in LLMs Attention Mechanisms
University of California, Berkeley
Concept-Grounded Attention: A Controlled Evaluation of Graph-Injected Attention, Temporal Versioning, and Epistemic Status
Sep 30, 2026Sachin Dev Duggal, Pradyumna Swarnalatha Ramanna, Alexandros VassiliadesLLM Grounding Temporal Reasoning in Language Models
SeKondBrain AI Labs, London, United Kingdom.
No Scale Left Behind: Multi-Scale Autoencoder with Bi-directional Attention for Time Series Anomaly Detection
Sep 29, 2026Jiaheng Guo, Haochen Zhang, Yu-Chao Huang +3Reconstruction-Based Anomaly Detection Attention Mechanisms
UNITES Lab, University of North Carolina at Chapel Hill
Retrieval Capacity of Self-Attention Under Competition
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-Attention Long-Context Language Modeling
University of Luxembourg, Luxembourg
Pixel-Level Transformers in Remote Sensing: A Canopy Height Case Study
Sep 29, 2026Sven Ligensa, Jan Pauls, Karsten Schrödter +2Remote Sensing Image Understanding Remote Sensing
University of Münster Münster, Germany · Laboratoire des Sciences du Climat et de l’Environnement Paris, France
SMat-Attention: Structured Long-Context Sequence Modeling
Sep 28, 2026Emile Anand, Abdullah Ateyeh, Archer Wang +1Long-Context Modeling Linear Attention
Massachusetts Institute of Technology
Quasi Linear Kernel Attention with Infinite Capacity
Sep 28, 2026Nicolaj Rux, Johannes Hertrich, Sebastian NeumayerTransformer Expressivity Attention Mechanisms
Faculty of Mathematics, Chemnitz University of Technology, 09126 Chemnitz, Germany · Institute of Computer Science, University of Göttingen, 37073 Göttingen, Germany
Dual-Stream Simultaneous Translation via 2D Grid Attention
Sep 28, 2026Yu Pu, Wei-Qiang ZhangTransformer Attention Attention Mechanisms
Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China
TSGate: Timestep-Aware Gated Attention for Diffusion Transformers
Sep 28, 2026Boyu Zhang, Yifan Liu, Shuxia Lin +3Diffusion Transformer Gated Attention
Alibaba Token Hub, Alibaba Group · Southeast University
Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
Sep 28, 2026Junlin Chen, Daize Dong, Huanwei Di +9Softmax Attention Efficient Attention
Rutgers University · Carnegie Mellon University · Oracle +2
Eyes on the Road: A Naturalistic Comparison of MTW Rider Gaze in Urban Indian Traffic
Sep 27, 2026Prerak Srivastava, Bhaiya Vaibhaw Kumar, Kavita VemuriVisual Attention Eye Tracking
IIIT Hyderabad Hyderabad, India
SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval
Sep 27, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungMemory-Augmented Neural Networks Persistent Memory for Language Models
College of Pharmacy, Chungnam National University Daejeon, Republic of Korea · Department of Computer Science & Engineering, Chungnam National University Daejeon, Republic of Korea
FluidRain: Incompressible Rain Flow as an Attention Bias for Loop-in-Loop Video Deraining
Sep 24, 2026Pu Wang, Yongcong Wang, Wenhao Li +6Attention Mechanisms
Shandong University · Central South University · Nanjing University of Science and Technology +2
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Sep 23, 2026Qucheng Gao, Zuyi Yang, Xiao ChenSelf-Attention Chaotic Dynamical Systems
Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA
Memory Attention
Sep 23, 2026Jiale KangMemory-Augmented Language Models LLM Inference Acceleration

Oct 7, 2026, cs.CVJ/K move · Enter open · S save

On the Necessity of Attention-FFN Split in Vision Transformers

Junhyeok Kim, Jinyeong Kim, Jae Wan Park, Seong Jae Hwang

Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.

Vision Transformer Attention Mechanisms Transformer FFNs

Open paper AskPDF arXiv

The standard Transformer architecture relies on a rigid pattern that alternates Attention and Feed-Forward Network (FFN) layers. Despite its widespread adoption, the inductive bias imposed by this strict separation has not been systematically examined. In this work, we investigate the necessity of the Attention-FFN dichotomy in Vision Transformers (ViTs). To facilitate this analysis, we introduce the AttenFeed module, a unified component that integrates the functional properties of both Attention and FFN. Based on this module, we devise the unified Vision Transformer (uViT), which replaces the conventional alternating Attention-FFN structure with a sequence of AttenFeed modules. We then use uViT as a control group that relaxes the Attention-FFN dichotomy of the standard ViT and systematically compare the two models across multiple datasets and model scales. Our experiments reveal that the Attention-FFN dichotomy can hinder performance at smaller model scales due to the rigid parameter allocation of ViTs. The AttenFeed module and uViT serve as new analytical tools for understanding the Attention-FFN structure and offer theoretical insights into the heuristically designed architecture of conventional ViTs.

Momentum

Related topics

Latest papers 378