Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

ClusterAttention: A training-free speedup of bidirectional attention
Aug 27, 2026Kasper Nordenram, Amelie DittmannEfficient Attention Attention Mechanisms
Independent researcher
Ask Self, Ask Others: Relation Is All You Need
Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieSelf-Attention Decoder-Only Language Models
Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore
Attention from Action, for Action: Emergent Visual Bottlenecks for Policy Learning
Aug 13, 2026Zheyu Zhuang, Ruiyu Wang, Nick Heppert +4Visual Attention Visuomotor Policy Learning
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden · Department of Computer Science, University of Freiburg, Germany · Department of Informatics, Universität Hamburg, Germany
Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage
Aug 12, 2026Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu +3Spiking Transformers Efficient Attention
School of Computing, National University of Singapore · 2Sea AI Lab · 4Shanghai Advanced Research Institute, Chinese Academy of Sciences +2
PolarSym: Polar Geometry-aware Attention for CAD Floorplan Parsing
Aug 12, 2026Kerui Chen, Yiqing Wang, Kangzhou Xin +2Panoptic Segmentation Transformer Attention
School of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou, China · Third Dimension (Henan) Software Technology, Zhengzhou, China
LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
Aug 12, 2026Qiuwu Chen, Zimo Liu, Yuchen Li +8Language Model Pretraining Memory-Augmented Language Models
AIGCode · South China University of Technology · Pazhou Laboratory
FLARE++: Low-rank attention with dynamic attention routing
Aug 12, 2026Vedant Puri, Yongjie Jessica Zhang, Levent Burak KaraLow-Rank Attention PDE Surrogate Modeling
Department of Mechanical Engineering Carnegie Mellon University Pittsburgh, Pennsylvania, USA
Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention
Aug 11, 2026Vicente OpazoLow-Rank Attention Attention Mechanisms
CENIA
Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry
Aug 11, 2026Liangchen GeAttention Mechanisms
Linearized 2-Simplicial Attention
Aug 10, 2026Aritra Das, Dhruman Gupta, Debayan GuptaLinear Attention Attention Mechanisms
1Truth Audit Labs
CDGC-Net: 3D Medical Image Segmentation with Cooperative Dual-Scale Self-Attention and Grouped Channel Modeling
Aug 9, 2026Zheyang Jing, Qin Lu, Jianwang Li +3Self-Attention Multi-Scale Feature Fusion
Nanchang Hangkong University, Nanchang, China
Rethinking Attention Locality in Spiking Transformers
Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Spiking Transformers Spiking Neural Networks
1Zhejiang University · 2Westlake University · 3Peking University
Gated Spatial Redundancy Projection for Pathology Transformer Attentions
Aug 8, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Cancer Survival Prediction Self-Attention
Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montréal, Canada · Mila - Quebec AI Institute, Montreal, Canada
DoGMA: A Central-Dogma-Guided Foundation Model for Multi-Omics Alignment and Multi-Task Learning in Oncology
Aug 8, 2026Junfei Ling, Bangzheng Pu, Bingsen Xue +3Cancer Survival Prediction Clinical Outcome Prediction
Institute of Medical Robotics, Shanghai Jiao Tong University · Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China · Institute of Data Science, The University of Hong Kong +1
Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection
Aug 5, 2026Mihailo Ilić, Miloš Savić, Vladimir Kurbalija +3Memory-Augmented Neural Networks Federated Learning Aggregation
Department of Mathematics and Informatics, Faculty of Sciences, University of Novi Sad, Serbia · Department of Informatics, Modeling, Electronics, and Systems, University of Calabria, Italy
Attention-based representations for multi-task computation
Aug 4, 2026Daniel Hsu, Mingyue XuRepresentation Learning Multi-Task Learning
Columbia University · Purdue University
When Should Graph Attention Be Sparse? Learning a Per-Edge Tsallis Index
Aug 3, 2026Kleyton da Costa, Bernardo ModenesiGraph Attention Networks Attention Mechanisms
University College London & Holistic AI · University of Utah
TextNCA: Neural Cellular Automata for Language Modeling via Hierarchical Local Attention
Aug 3, 2026Avni Mittal, Avinash Anand, Ashutosh Kumar +7Neural Cellular Automata Autoregressive Language Modeling
Independent Researcher · Singapore Institute of Technology, Singapore · IIIT Delhi, India +2
DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling
Aug 3, 2026Yixiao Qian, Song Chen, Pengkai Wang +3Memory-Augmented Language Models KV Caching
College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · Hong Kong Polytechnic University +1
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
Aug 2, 2026Sen SongTransformer Self-Attention
Independent Researcher, Guangzhou, China
Role-Decoupled Attention Residuals: Separating Matching and Content Retrieval Across Depth
Aug 2, 2026Kehan WangAttention Residual Connections Autoregressive Language Modeling
Kimi Team
Opt.Gear Technical Report
Aug 2, 2026Juneyoung Park, Youngwook KwonOn-Device Language Model Inference Long-Context Language Model Inference
Opt.Gear Team
AttnLink: Turning Attention into Schema Links for Text-to-SQL
Aug 1, 2026Jinwang Song, Tao Liu, Haowen Zheng +3Schema Linking Text-to-SQL
ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Jul 31, 2026Yuhang Zhan, Lisi Chen, Shuo ShangSoftmax Attention KV Caching
University of Electronic Science and Technology of China
A Compositional Theory of Causally Masked Transformers
Jul 29, 2026Franz Nowak, Ryan Cotterell, Reda BoumasmoudTransformer Expressivity Transformer
1ETH Zürich
Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding
Jul 29, 2026Weiye Shi, Fanxu Meng, Muhan ZhangSpeculative Decoding LLM Inference Acceleration
Institute for Artificial Intelligence, Peking University
Intrinsic and Triangulation-Agnostic Attention: A Simple and Powerful Approach for Learning on Meshes
Jul 27, 2026Ashwath Shetty, Zihan Zhu, Soeren Pirk +13D ViTs Geometric Representation Learning
Université de Montréal & Mila, Canada · Kiel University, Germany
Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
Jul 27, 2026Hao YeSoftmax Attention Rotary Positional Embeddings
Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, University of Chinese Academy of Sciences · Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences · University of Chinese Academy of Sciences
SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
Jul 27, 2026Yu Cui, Yi Xu, Jiahao Wang +6Representation Collapse Transformer
Zhejiang University Hangzhou, China · Alibaba Group Beijing, China
Variational-Ising-Attention:Tailored Attention Matters for Science
Jul 26, 2026Rui WangAttention Mechanisms
Tsinghua University Department of Chemical Engineering

Aug 27, 2026, cs.LGJ/K move · Enter open · S save

ClusterAttention: A training-free speedup of bidirectional attention

Kasper Nordenram, Amelie Dittmann

Independent researcher

Efficient Attention Attention Mechanisms

Open paper AskPDF arXiv

We introduce ClusterAttention, a general training-free speedup of bidirectional attention at large token counts. We point out two common assumptions in contemporary training-free methods; attention sparsity, and context that can be leveraged, such as structure in the input or multiple similar forward passes, and show when they fail. Our proposed method utilizes a fast attention-aware recursive clustering method, and compensation of excluded clusters through their mean. The clustering method gives power-of-two cluster sizes, allowing block-sparse attention to match dense attention in GPU throughput. On TabPFN-3 arXiv:2605.13986, a model where none of the assumptions hold, ClusterAttention is to our knowledge the first method to provide a substantial speedup over the default attention, while consistently keeping over 99% of its accuracy. On the largest dataset from the TALENT benchmark suite, it makes processing of the training dataset close to 8x faster at nearly 11x attention speedup. ClusterAttention is also competitive with domain-specific methods, while avoiding any of the domain-specific engineering. On video-generation with Wan 2.1-T2V-14B arXiv:2503.20314 it produces output closer to dense attention at a larger speedup (1.8x vs 1.4x) than SVOO arXiv:2603.18636, a leading method in this domain, with both evaluated without offline calibration.

Momentum

Related topics

Latest papers 378