Self-Attention

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 195

All topics

Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoSelf-Attention Long-Context Language Modeling
Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling
Apr 20, 2026Yujie Chen, Tailai Chen, Yifeng Gao +4Self-Attention Long-Context Language Model Inference
Shanghai Jiao Tong University · University of California, San Diego · Carnegie Mellon University
HeadRank: Decoding-Free Passage Reranking via Preference-Aligned Attention Heads
Apr 19, 2026Juyuan Wang, Chenxing Wang, Yuchen Fang +8Attention Head Analysis Self-Attention
Weixin Group, Tencent, China · 2South China University of Technology, Guangzhou, China
Open-TQ-Metal: Fused Compressed-Domain Attention for Long-Context LLM Inference on Apple Silicon
Apr 18, 2026Sai VegasenaAI Accelerator Inference Self-Attention
Ensue
SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models
Apr 18, 2026Junnan Liu, Xinyan Liu, Peifeng Gao +4GPU Acceleration Self-Attention
Harbin Institute of Technology (Weihai) · University of Hong Kong, Hong Kong, China · City University of Hong Kong, Hong Kong, China
AdaSplash-2: Faster Differentiable Sparse Attention
Apr 16, 2026Nuno Gonçalves, Hugo Pitorro, Vlad Niculae +4GPU Kernel Optimization Self-Attention
Instituto Superior Técnico, Universidade de Lisboa · Instituto de Telecomunicações · Carnegie Mellon University +4
Dispatch-Aware Ragged Attention for Pruned Vision Transformers
Apr 16, 2026Seifeldin Abdellatif, Ahmad AlmasriEfficient Transformer Inference Visual Attention
College of Engineering Al Ain University Abu Dhabi, United Arab Emirates
Expressivity of Transformers: A Tropical Geometry Perspective
Apr 16, 2026Ye Su, Yong LiuTransformer Expressivity Transformer
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · University of Chinese Academy of Sciences, Beijing 101407, China · Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China
A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation
Apr 16, 2026Yuval Ran-Milo, Hila Ofek, Shahar MendelSelf-Attention Decoder-Only Language Models
Tel Aviv University
Gating Enables Curvature: A Geometric Expressivity Gap in Attention
Apr 16, 2026Satwik Bathula, Anand A. JoshiInformation Geometry Neural Representation Geometry
Department of Electrical and Computer Engineering University of Southern California
Sinkhorn doubly stochastic attention rank decay analysis
Apr 9, 2026Michela Lapenna, Rita Fioresi, Bahman GharesifardSoftmax Attention Self-Attention
Department of Physics and Astronomy, University of Bologna, Bologna, Italy · Department of Pharmacy and Biotechnologies, University of Bologna Bologna, Italy · Department of Mathematics and Statistics, Queen’s University, Kingston, ON, Canada
Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
Invertible Query-Key Coupling Composes with Attention Mechanisms
Apr 2, 2026Barak Gahtan, Alex M. BronsteinInvertible Neural Networks Self-Attention
Technion – Israel Institute of Technology · ISTA – Institute of Science and Technology Austria
Screening Is Enough
Apr 1, 2026Ken M. NakanishiSelf-Attention Long-Context Language Modeling
Center for Emergent Matter Science (CEMS), RIKEN · Graduate School of Science, The University of Tokyo
HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
Mar 30, 2026Yufei Xu, Fanxu Meng, Fan Jiang +11Self-Attention LLM Inference Acceleration
Xiaohongshu
Learning When to Attend: Conditional Memory Access for Long-Context LLMs
Mar 18, 2026Sakshi Choudhary, Aditya Chattopadhyay, Luca Zancato +4Self-Attention Long-Context Language Modeling
Department of Electrical and Computer Engineering, Purdue University, USA · AWS Agentic AI, USA
WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Mar 17, 2026Hanna Lee, Tan Dat Nguyen, Jaehoon Kang +1TTS Synthesis Self-Attention
Korea Advanced Institute of Science and Technology, Republic of Korea · Sungkyunkwan University, Republic of Korea
Beyond Pairwise Attention: Higher-Order Modular Attention for Efficient Sequence Learning
Mar 11, 2026Shirin Amiraslani, Xin GaoSelf-Attention Transformer Attention
Department of Mathematics and Statistics York University Toronto, ON M3J 1P3, Canada
StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
Mar 7, 2026Duy M. H. Nguyen, Tuan A. Tran, Duong Nguyen +17Token Merging Image Segmentation
University of Stuttgart · DFKI · Max Planck Research School +10
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences
Feb 6, 2026Alessio Pecilli, Matteo RosatiSelf-Attention Transformer Attention
Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy
FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangVideo Diffusion Models Self-Attention
1Monash University · 2Zhejiang University
Poly-attention: a general scheme for higher-order self-attention
Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanSelf-Attention Compositional Reasoning
Computer Science, Columbia University, New York, NY 10027.
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling
Jan 17, 2026Xingyue Huang, Xueying Ding, Mingxuan Ju +3Self-Attention Long-Context Language Modeling
University of Oxford · Carnegie Mellon University · Snap Inc.
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
Dec 18, 2025Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra +1Self-Attention Long-Context Language Model Inference
Microsoft Research India.
Block Sparse Flash Attention
Dec 7, 2025Daniel Ohayon, Itay Lamprecht, Itay Hubara +3GPU Acceleration Self-Attention
Technion – Israel Institute of Technology, Haifa, Israel · Intel - Habana Labs · Intel – Habana Labs, Tel Aviv, Israel

Apr 20, 2026, cs.LGJ/K move · Enter open · S save

Sessa: Selective State Space Attention

Liubomyr Horbatko

Self-Attention Long-Context Language Modeling Persistent Memory for Language Models Recurrent Transformers Attention Mechanisms State Space Models

Open paper AskPDF arXiv

Modern sequence modeling is dominated by two families: Transformers, whose self-attention can access arbitrary elements of the visible sequence, and structured state-space models, which propagate information through an explicit recurrent state. These mechanisms face different limitations on long contexts: when attention is diffuse, the influence of individual tokens is diluted across the effective support, while recurrent state propagation can lose long-range sensitivity unless information is actively preserved. As a result, both mechanisms face challenges in preserving and selectively retrieving information over long contexts. We propose Sessa, a decoder that places attention inside a recurrent feedback path. This creates many attention-based paths through which past tokens can influence future states, rather than relying on a single attention read or a single recurrent chain. We prove that, under explicit assumptions and matched regimes, Sessa admits power-law memory tails

O(\ell^{-β})

for

0 < β< 1

, with slower decay than in the corresponding Transformer and Mamba-style baselines. We further give an explicit construction that achieves this power-law rate. Under the same assumptions, Sessa is the only model class among those considered that realizes flexible selective retrieval, including profiles whose influence does not decay with distance. Consistent with this theoretical advantage, across matched experiments, Sessa achieves the strongest performance on long-context benchmarks while remaining competitive with Transformer and Mamba-style baselines on short-context language modeling.

Momentum

Related topics

Latest papers 195