Self-Attention

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 195

All topics

Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling
Oct 6, 2026Ryan Solgi, Parsa Madinei, Zheng ZhangAttention Mechanisms Low-Rank Attention
University of California-Santa Barbara Santa Barbara, CA, USA
An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
Oct 4, 2026Przemysław RolaTransformer Interpretability Self-Attention
Department of Mathematics, Kraków University of Economics
Universal interpolation for deep residual self-attention networks
Oct 1, 2026Sibylle Marcotte, Joan BrunaSoftmax Attention Transformer
Department of Computer Science New York University
Contrastive Attention Mitigates Spectral Bias in Spiking Transformers
Oct 1, 2026Xiaoli Liu, Malu Zhang, Yang YangSpiking Transformers Self-Attention
School of Computer Science and Engineering, University of Electronic Science and Technology of China
Retrieval Capacity of Self-Attention Under Competition
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateSelf-Attention Long-Context Language Modeling
University of Luxembourg, Luxembourg
PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention
Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-Attention LLM Inference Acceleration
The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
Sep 23, 2026Qucheng Gao, Zuyi Yang, Xiao ChenSelf-Attention Chaotic Dynamical Systems
Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA
Prescriptive SVD-Inspired Attention via Spectral Energy Retention
Sep 21, 2026Vasileios Arampatzakis, Vasileios Sevetlidis, George PavlidisTransformer Interpretability Low-Rank Attention
Athena Research Center, Greece
Increasing Skill Level Recruits Deeper Attention Layers in a Frozen Chess Transformer
Sep 20, 2026David LitmanTransformer Interpretability Attention Head Analysis
Computational Neurobiology Laboratory, Salk Institute
It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
Sep 8, 2026Raito Kiya, Satoki Ohashi, Kosuke Sato +6Self-Attention Attention Sinks
1Tohoku University · 2SOKENDAI · 3NINJAL +3
The Head Complexity of Boolean Functions in Single-Layer Attention
Sep 3, 2026Rajmohan Rajaraman, Ravi Sundaram, Amanuel TesfayeSelf-Attention Attention Mechanisms
Northeastern University.
Multi-Head Self Attention is a Parameter Identification Mechanism
Sep 1, 2026W. Ross MorrowSoftmax Attention Parameter Identifiability
TANGO: Treating Tokens as Operators
Aug 22, 2026Joshua NunleyTransformer FFNs Transformer
Department of Informatics Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
Ask Self, Ask Others: Relation Is All You Need
Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieSelf-Attention Decoder-Only Language Models
Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore
Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference
Aug 10, 2026Burc GokdenGraph Attention Networks LLM Inference
Fromthesky Research Labs LLC, Oregon, USA
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1Transformer Self-Attention
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data
Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention
Aug 9, 2026Qucheng Gao, Zuyi Yang, Xiao ChenDynamical Systems Self-Attention
Department of Physics, Boston College, Chestnut Hill, Massachusetts 02467, USA · Department of Physics, Tsinghua University, Beijing 100084, China
CDGC-Net: 3D Medical Image Segmentation with Cooperative Dual-Scale Self-Attention and Grouped Channel Modeling
Aug 9, 2026Zheyang Jing, Qin Lu, Jianwang Li +3Self-Attention Multi-Scale Feature Fusion
Nanchang Hangkong University, Nanchang, China
Gated Spatial Redundancy Projection for Pathology Transformer Attentions
Aug 8, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Cancer Survival Prediction Self-Attention
Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montréal, Canada · Mila - Quebec AI Institute, Montreal, Canada
Faster Query-Key Learning Sharpens Attention in Self-Attention Models
Aug 7, 2026Rahul Vashisht, Harish G. RamaswamySelf-Attention
Department of CSE, IIT Madras, India · Department of DSAI, IIT Madras, India.
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
Aug 2, 2026Sen SongTransformer Self-Attention
Independent Researcher, Guangzhou, China
Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex
Jul 25, 2026Chanwoo Park, Asuman OzdaglarSelf-Attention Regret Minimization
1MIT EECS
Inhibited Self-Attention: Sharpening Focus in Vision Transformers
Jul 14, 2026Peter R. D. van der Wal, Nicola Strisciuglio, George AzzopardiSelf-Attention Vision Transformer
University of Groningen · University of Twente · 3Stellenbosch University +1
LoSA-Net: A Localized and Scale-Adaptive Network for Boundary-Sensitive Prediction of Perineural Invasion in 3D MRI
Jul 13, 2026Youngung Han, Hyunsu Go, Kyeonghun Kim +9Self-Attention Multi-Scale Feature Fusion
Seoul National University, Seoul, Republic of Korea · OUTTA, Seoul, Republic of Korea · Chung-Ang University, Seoul, Republic of Korea +3
From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Self-Attention Transformer Attention
School of Software Technology, Zhejiang University, China · College of Computer Science and Technology, Zhejiang University, China · Alibaba Cloud, China
Towards Robust EEG Decoding Based on Riemannian Self-Attention
Jun 24, 2026Shaocheng Jin, Tao Zhou, Rui Wang +4Brain-Computer Interfaces Self-Attention
School of Artificial Intelligence and Computer Science Jiangnan University Wuxi, China · Department of Information Engineering and Computer Science University of Trento Trento, Italy · Centre for Vision, Speech and Signal Processing (CVSSP) University of Surrey Guildford, UK
Communicability-Inspired Positional Encoding (CIPE)
Jun 24, 2026Yipeng Zhang, Zhongtian Sun, Pietro Liò +1Graph Representation Learning Self-Attention
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore 637371, Singapore · School of Computing, University of Kent, Canterbury, Kent CT2 7NZ, UK · Department of Computer Science and Technology, University of Cambridge, 15 JJ Thomson Avenue, Cambridge CB3 0FD, UK
Attention mechanism for scalable mesh-based neural surrogates of free-surface fluids
Jun 22, 2026Federico Lanteri, Massimiliano CremonesiNeural Surrogate Modeling Self-Attention
Department of Civil and Environmental Engineering, Politecnico di Milano Piazza Leonardo da Vinci, 32, 20133, Milano, Italy
Kuramoto Attention: Synchronizing Self-Attention on the Torus
Jun 10, 2026Joshua NunleyDynamical Systems Self-Attention
Department of Informatics, Luddy School of Informatics, Computing, and Engineering Cognitive Science Program Indiana University Bloomington
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
Jun 9, 2026Alejandro García-Castellanos, Maurice Weiler, Erik J BekkersRotary Positional Embeddings Self-Attention
AMLab, University of Amsterdam · MIT CSAIL
Interpolation between Convolution and Attention via K-Nearest Neighbors
May 31, 2026Mingi KangSelf-Attention Convolutional Neural Networks
Bowdoin College
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
May 29, 2026Harry Jake Cunningham, Nicola Muca CironeTransformer Interpretability Feature Attribution
Department of Computer Science, University College London, London, UK. · Cartesia.AI, San Francisco, US.
VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
May 28, 2026Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral +4Video Diffusion Models Self-Attention
1Virginia Tech · 2fal
Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
May 28, 2026Masaaki Imaizumi, Masanori Koyama, Noboru Isobe +1Transformer Self-Attention
The University of Tokyo, Tokyo, Japan · RIKEN Advanced Intelligene Project, Tokyo, Japan · Kyoto University, Kyoto, Japan
Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics
May 28, 2026Matthew Smart, Soumya Ganguly, Nilava Metya +2Self-Attention Empirical Bayes
Lewis-Sigler Institute for Integrative Genomics, Princeton University, Princeton, NJ, USA · Department of Mathematics, Rutgers University, Piscataway, NJ, USA · Department of Physics and Astronomy, Rutgers University, Piscataway, NJ, USA +1
Parallax: Parameterized Local Linear Attention for Language Modeling
May 27, 2026Yifei Zuo, Dhruv Pai, Zhichen Zeng +3Language Model Pretraining Self-Attention
Northwestern University · Tilde Research · University of Washington
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
May 27, 2026Xiuying Wei, Caglar GulcehreMemory-Augmented Language Models Self-Attention
EPFL, CLAIRE Lausanne, Switzerland
PLS in the Mirror of Self-Attention
May 27, 2026Jiangsheng, YouRecursive Least Squares Self-Attention
Jason
Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference
May 27, 2026Alan FerrariEfficient Transformer Inference Self-Attention
Knowledge Lab AG Zürich, Switzerland
UNIQUE: Universal Top-k Sparse Attention for Training-free Inference and Sparsity-aware Training
May 26, 2026Keqi Deng, Shaoshi Ling, Ruchao Fan +1Self-Attention Long-Context Language Model Inference
Microsoft, USA
Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective
May 26, 2026Hyunmin Cho, Woo Kyoung Han, Kyong Hwan JinSelf-Attention Transformer Attention
Department of Electrical Engineering, Korea University, Seoul, South Korea.
AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers
May 26, 2026Semi Lee, Hyejin Go, Hyesong ChoiToken Merging Self-Attention
Electronic Engineering · Soongsil University · Seoul, South Korea
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion
May 25, 2026Tuna Tuncer, Felix Becker, Thomas PfeilVideo Diffusion Models Self-Attention
1Technical University of Munich · 2Tensordyne
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
May 25, 2026Xintong Yang, Hao Gu, Binxing Xu +6Memory-Augmented Language Models Self-Attention
The Hong Kong University of Science and Technology · Zhejiang University
Quaternion Self-Attention with Shared Scores
May 24, 2026Shogo Yamauchi, Tohru Nitta, Hideaki TamoriSelf-Attention Attention Mechanisms
The Asahi Shimbun Company, Tokyo, Japan · Tokyo Woman’s Christian University, Tokyo, Japan.
Interdomain Attention: Beyond Token-Level Key-Value Memory
May 23, 2026Naoki Kiyohara, Harrison Bo Hua Zhu, Riccardo El Hassanin +4Self-Attention Long-Context Language Modeling
Approaching I/O-optimality for Approximate Attention
May 22, 2026Pál András Papp, Aleksandros Sobczyk, Anastasios ZouziasSelf-Attention Efficient Attention
Computing Systems Lab, Huawei Technologies, Zurich, Switzerland
Improved Belief-Attention in Vision Task
May 22, 2026Guoqiang ZhangVisual Attention Self-Attention
University of Exeter
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Interpretability Circuit Discovery
ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention
May 21, 2026Joe SharrattMixed-Precision Quantization Self-Attention
Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
May 21, 2026Hangyue Zhao, Paul Caillon, Erwan Fagnou +1Self-Attention Structured Sparsity
ESPCI PSL, Paris, France · LAMSADE, Université Paris Dauphine - PSL, Paris, France
ASAP: Attention Sink Anchored Pruning
May 21, 2026Jaehyuk Lee, Hanyoung Kim, Yanggee Kim +1Self-Attention Efficient ViTs
Department of Mathematics Korea University Seoul, Republic of Korea · Korea University Seoul, Republic of Korea
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
May 21, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural Networks Self-Attention
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Manifold-Guided Attention Steering
May 20, 2026Ian Li, Kapilesh Guruprasad, Raunak Sengupta +3Language Model Steering Self-Attention
University of California, San Diego
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoSelf-Attention KV Caching
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
Towards Understanding Self-Pretraining for Sequence Classification
May 20, 2026Omar Coser, Loredana Zollo, Paolo Soda +1Self-Attention Masked Language Modeling
PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
May 20, 2026Yanyi Lyu, Letian Chen, Futing Sun +3Self-Attention Diffusion Language Model Inference
Harbin Institute of Technology (Shenzhen)
Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-Attention Long-Context Language Modeling
The Hong Kong University of Science and Technology · The University of Hong Kong · Zhejiang University +1

Oct 6, 2026, cs.LGJ/K move · Enter open · S save

Tucker Bottleneck Attention for Multi-Dimensional Sequence Modeling

Ryan Solgi, Parsa Madinei, Zheng Zhang

University of California-Santa Barbara Santa Barbara, CA, USA

Attention Mechanisms Low-Rank Attention Self-Attention Weather Forecasting

Open paper AskPDF arXiv

The quadratic cost of self-attention limits scalability to long sequences from multidimensional data. We introduce Tucker bottleneck attention (TuBA), which exploits low-rank tensor structure for efficient global token mixing. TuBA projects hidden tensors into compact Tucker cores, performs multi-head self-attention and linear projections on the cores, and writes updates back to the ambient space, enabling subquadratic computation. Its autoregressive extension combines bidirectional interactions within cores with causal attention across cores. On video prediction and global weather forecasting, TuBA achieves favorable accuracy-efficiency trade-offs over standard and efficient attention and task-specific models. Compared to standard self-attention, TuBA reduces error and computation by up to 24.7% and 66.6% for video prediction and 37.1% and 85.1% for autoregressive weather forecasting, with speedups up to 4.27 times. Low-rank Tucker cores and multi-frame generation also outperform full-rank attention and frame-by-frame generation, respectively.

Momentum

Related topics

Latest papers 195