Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
Jul 25, 2026Dhruvil S, Fenil Sojitra, Ravirajsinh ChauhanTransformer Interpretability Low-Rank Attention
1Indian Institute of Technology Madras · 2P P Savani University
The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers
Jul 25, 2026Byeong Hoon YoonSoftmax Attention Transformer Expressivity
Independent Researcher
What Softmax Throws Away: Mass-Aware Attention for Evidence Accumulation
Jul 24, 2026Minwoo Yu, Young-guk HaTemporal GNNs Temporal Link Prediction
Smart Computing Laboratory, Department of Computer Science & Engineering, Konkuk University, Seoul 05029, Republic of Korea.
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Jul 22, 2026Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun MoonLow-Rank Attention Transformer Attention
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea · Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, Republic of Korea
MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsEfficient Transformer Inference Grouped-Query Attention
University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil
Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making
Jul 21, 2026Yuyang Shen, Shan Dai, Daimin ChenNon-Stationary RL Sequential Decision Making
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data · University of International Business and Economics
L1 Augmented Attention as an Improved Vector Similarity Metric
Jul 20, 2026Kurt GoddenTransformer Attention Language Modeling
Walsh College, Troy Michigan, USA
Attention from Above: A Multimodal Model for Drone-Based Object Localization
Jul 20, 2026Hyun-Ki JungOpen-Vocabulary Object Detection Small Object Detection
University of Seoul, Seoul, South Korea
Spectral-Morphological Attention U-Net: An Efficient Network for Active Wildfire Detection
Jul 17, 2026Yugong Zeng, Jonathan WuRemote Sensing Image Segmentation Wildfire Detection
Department of Electrical and Computer Engineering University of Windsor 401 Sunset Avenue Ontario, Canada N9B 3P4
Long-Context Fine-Tuning with Limited VRAM
Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-Tuning KV-Cache Offloading
BMW Group
Screening Is Effective for Visual Recognition
Jul 15, 2026Shunya Shimomura, Kazuhiro HottaVision Transformer Attention Mechanisms
Meijo University
DAGR: State-Conditioned Goal Representations via Difference-Aware Goal Cross-Attention
Jul 15, 2026Xing Lei, Wenyan Yang, Xuetao Zhang +1Reinforcement Learning Attention Mechanisms
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Department of Electrical Engineering and Automation, Aalto University · School of Engineering, Westlake University
Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning
Jul 13, 2026Vishwajith RameshMemory-Augmented Neural Networks Attention Mechanisms
Vy Labs, Inc.
Causal Supervision of Attention for Affective Behaviour Analysis
Jul 13, 2026Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis +1Causal Representation Learning Facial Action Unit Detection
Institute of Digital Games, University of Malta, Malta · AI Department, University of Malta, Malta
CUST: Clustered Unit-level Similarity Transformer for Lightweight Image Super-Resolution
Jul 13, 2026Jeongsoo KimVision Transformer Efficient ViTs
Independent Researcher
Controlling Motion Transfer in Diffusion Transformers via Attention Heads
Jul 13, 2026Sunyoung Jung, Jiwoo Park, Yoonseok Choi +3Diffusion Model Guidance Diffusion Transformer
Yonsei University · LG Electronics · University of California, Merced
Learning To Focus: Anatomy-Guided Attention Regularization for Medical Image Classification
Jul 12, 2026Tonmoy Hossain, Atiqur Rahman, Farhana Hossain Swarnali +1Medical Image Classification Attention Mechanisms
Department of Computer Science, University of Virginia, Virginia, USA · Ahsanullah University of Science and Technology, Dhaka, Bangladesh · Kahlert School of Computing, University of Utah, Utah, USA +1
Geometry-aware Gaussian Prior and Axial Attention for Cervical Cytology Image Classification
Jul 11, 2026Yating Li, Cheng Ye, Nenan Lyu +2Computational Pathology Medical Image Classification
School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China · School of Information Science and Technology, University of Science and Technology of China, Hefei 230026, China · Department of Gynecologic Oncology, National Cancer Center, National Clinical Research Center for Cancer, Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100730, China
On Locality and Length Generalization in Visual Reasoning
Jul 10, 2026Pulkit Madan, Sanjay Haresh, Reza Ebrahimi +3Visual Reasoning Length Generalization
Qualcomm AI Research*
LEEVLA: Seeing What Matters in Latent Environment Evolution for Vision-Language-Action
Jul 9, 2026Qi Lyu, Baicheng Liu, Xudong Wang +3Latent Dynamics Modeling Vision-Language-Action Models
State Key Laboratory of Robotics and Intelligent Systems · Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences +1
STAGformer: A Spatio-temporal Agent Graph Transformer for Micro Mobility Demand Forecasting
Jul 7, 2026Ye ZihaoAttention Mechanisms Spatiotemporal Forecasting
Department of System Engineering City University of Hong Kong
LBR: Towards Mitigating Length Bias in Large Language Models for Recommendation
Jul 5, 2026Hongchen Li, Bohao Wang, Jingbang Chen +5Large Language Model-Based Recommendation Attention Mechanisms
State Key Laboratory of Blockchain and Data Security, Zhejiang University. · College of Computer Science and Technology, Zhejiang University. · Zhejiang University Hangzhou, Zhejiang, China +3
Beyond Scene Priors: Fine-Grained Traffic Scene Reasoning with Benchmarking and Query-Guided Small-Object Focus
Jul 5, 2026Waikit Xiu, Qiang Lu, Zian Wang +4Visual Question Answering VLMs for Autonomous Driving
1The University of Hong Kong · 2Sun Yat-Sen University · 3The Hong Kong University of Science and Technology (Guangzhou)
MACRO: Training-free Multi-plane Attention for Closeup Render Optimization
Jul 4, 2026Nitzan Hodos, Roy Amoyal, Lior Fritz +33D Gaussian Splatting Image Enhancement
Amazon Prime Video · Technion, Israel Institute of Technology · Ben Gurion University +1
Sequential Correlations Change In-Context Learning: Effective Context Length and Architectural Mismatch
Jul 4, 2026Mary Letey, Yue M. Lu, Cengiz Pehlevan +1In-Context Learning Attention Mechanisms
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Society of Fellows and Center for Brain Science, Harvard University
Learning to Generate Multiple Objects from Dense and Occluded Layouts
Jul 3, 2026Bach-Hoang Ngo, Si-Tri Ngo, Hieu Le +1Image Generation Compositional Image Generation
University of Science, Ho Chi Minh City, Vietnam · 2Vietnam National University, Ho Chi Minh City, Vietnam · 3UNC Charlotte, North Carolina, United States
LBTCap: A Lightweight Bilateral Transformer for Real-Time Remote Sensing Image Change Captioning
Jul 3, 2026Licheng Zhang, Siew-Kei Lam, Naveed AkhtarRemote Sensing Image Understanding Image Captioning
School of Computing and Information Systems, The University of Melbourne, Melbourne, 3010, Victoria, Australia · Nanyang Technological University, Singapore · The University of Melbourne, Melbourne, 3010, Victoria, Australia
Self-Gating Attention for Efficient Time Series Forecasting
Jul 2, 2026Dezheng Wang, Tong Chen, Wei Yuan +3Time Series Forecasting Linear Attention
School of Automation, Southeast University, 210096, China · School of Electrical Engineering and Computer Science, The University of Queensland, QLD 4072, Australia
DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction
Jul 2, 2026Susmit Agrawal, Matthias Bethge, Matthias KümmererVisual Attention Attention Mechanisms
University of Tübingen · Tübingen AI Center · IMPRS-IS
HistoSeg++: Delving deeper with attention and multiscale feature fusion for biomarker segmentation
Jul 2, 2026Saad Wazir, Rao Faizan, Daeyoung KimU-Net Multi-Scale Feature Fusion
School of Computing, KAIST Daejeon, Republic of Korea · Kyung Hee University (Global Campus) Yongin-si, Republic of Korea

Jul 25, 2026, cs.LGJ/K move · Enter open · S save

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan

1Indian Institute of Technology Madras · 2P P Savani University

Transformer Interpretability Low-Rank Attention Attention Head Analysis Attention Mechanisms Representation Geometry in Language Models Multi-Head Attention+1

Open paper AskPDF arXiv

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference. Despite its adoption in massive production models, no prior work has studied what information this bottleneck preserves or discards, nor how it reshapes internal transformer circuits. We present the first comprehensive mechanistic interpretability study of MLA, training a 114M-parameter transformer (pretrained on a web/code/math mixture, fine-tuned on TinyStories) and analyzing its representations through SVD, attention head taxonomy, linear probing, and a disruption-attribution analysis. Our key findings are: (1) the cKV bottleneck learns a pure content representation, preserving entity identity (98% retention) while discarding positional information, validating MLA's separation of content from position via RoPE; (2) induction heads co-locate at a single layer (Layer 12), unlike their distributed formation in standard MHA; (3) a single "semantic hub" layer (Layer 15) simultaneously exhibits the highest SVD effective rank and strongest disruption-attribution score; and (4) the bottleneck is globally over-provisioned, using only 46% of its capacity on average. These findings suggest MLA does not merely compress attention passively, but reshapes how the model organizes content, position, and circuit structure. We view this as an initial data point and detail scope limitations in Section 5.

Momentum

Related topics

Latest papers 378