Self-Attention

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 195

All topics

Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks
May 5, 2026Yaobo ZhangRotary Positional Embeddings Self-Attention
School of Physics, Ningxia University Yinchuan, 750021, China
HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
May 5, 2026Jorge L. Ruiz WilliamsSelf-Attention KV Caching
Independent Researcher
Cascade Token Selection for Transformer Attention Acceleration
May 4, 2026Stephen J. ThomasSelf-Attention Transformer Attention
Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015
Linearizing Vision Transformer with Test-Time Training
May 4, 2026Yining Li, Dongchen Han, Zeyu Liu +3Efficient Transformer Inference Self-Attention
Tsinghua University, Beijing, China
StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
May 4, 2026Jaber Jaber, Osama JaberGPU Kernel Optimization Self-Attention
RightNow AI
When Attention Collapses: Residual Evidence Modeling for Compositional Inference
May 4, 2026Niklas HoubaGravitational-Wave Astronomy Self-Attention
Institute for Particle Physics and Astrophysics ETH Zurich
SAGA: A Robust Self-Attention and Goal-Aware Anchor-based Planner for Safe UAV Autonomous Navigation
May 4, 2026Junhao Wei, Yanxiao Li, Dexing Yao +9Self-Attention Roadmap-Based Motion Planning
Faculty of Applied Sciences, Macao Polytechnic University, Macao 999078, China · Pazhou Lab (Huangpu), Guangzhou 510555, China · College of Animal Science and Technology, Zhongkai University of Agriculture and Engineering, Guangzhou 510225, China +3
Projection-Free Transformers via Gaussian Kernel Attention
May 4, 2026Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1Transformer Self-Attention
The Pennsylvania State University, University Park, USA
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Focus and Dilution: The Multi-stage Learning Process of Attention
May 2, 2026Zheng-An Chen, Pengxiao Lin, Zhi-Qin John Xu +1Self-Attention Transformer Attention
School of Mathematical Sciences, Shanghai Jiao Tong University. · Institute of Natural Sciences, Shanghai Jiao Tong University. · MOE-LSC, Shanghai Jiao Tong University. +2
DARE: Diffusion Language Model Activation Reuse for Efficient Inference
May 1, 2026Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang +3Self-Attention Diffusion Language Model Inference
Chandra Family Department of Electrical and Computer Engineering The University of Texas at Austin · Cornell University
Borrowed Geometry: Cross-Distribution Head-Importance Fingerprints of Frozen Pretrained Gemma 4 31B
May 1, 2026Abay BektursunCross-Modal Learning Attention Head Analysis
Independent research
Attention Is Where You Attack
Apr 30, 2026Aviral Srivastava, Sourav PandaSelf-Attention Adversarial Attacks
Amazon · The Pennsylvania State University
Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models
Apr 29, 2026Vijay Sadashivaiah, Georgios Dasoulas, Judith Mueller +1Softmax Attention Self-Attention
Merck & Co., Inc., Cambridge, MA, USA
Emergent Self-Attention from Astrocyte-Gated Associative Memory Dynamics
Apr 28, 2026Arnau Vivet, Alex ArenasSoftmax Attention Dynamical Systems
1Departament d’Enginyeria Inform`atica i Matem`atiques, Universitat Rovira i Virgili, 43007 Tarragona, Spain · 2Complexity Science Hub Vienna, Metternichgasse 8, 1030 Vienna, Austria
Learning to Rotate: Temporal and Semantic Rotary Encoding for Sequential Modeling
Apr 27, 2026Hailing Cheng, Daqi Sun, Xinyu LuRotary Positional Embeddings Self-Attention
LinkedIn Inc. Mountain View, CA 94043
Salca: A Sparsity-Aware Hardware Accelerator for Efficient Long-Context Attention Decoding
Apr 27, 2026Wang Fan, Wei Cao, Xi Zha +5Self-Attention Long-Context Language Model Inference
Fudan University
Kwai Summary Attention Technical Report
Apr 27, 2026Chenglong Chu, Guorui Zhou, Guowang Zhang +35Self-Attention KV Caching
Kuaishou
Nautile-370M: Spectral Memory Meets Attention in a Small Reasoning Model
Apr 27, 2026Maixent ChenebauxSelf-Attention Small Language Models
Trickstr
Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
Apr 27, 2026Amogh Sheth, Biruk Assefa, Yi Wen Huang +2LLM Pruning Self-Attention
Edison Academy Magnet School · State University of New York College at Plattsburgh · The University of Texas at Austin +2
ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers
Apr 26, 2026Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1Efficient Transformer Inference Softmax Attention
Advanced Computer Vision Laboratory, National Yang Ming Chiao Tung University
Transformer as an Euler Discretization of Score-based Variational Flow
Apr 26, 2026Huadong LiaoRepresentation Learning Transformer
HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token Compression Self-Attention
1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2
Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
Apr 26, 2026Giansalvo CirrincioneTransformer Interpretability Representation Collapse
aLaboratoire LTI, Université de Picardie Jules Verne, Chemin du Thil, 80025 Amiens, France
Dissociating Decodability and Causal Use in Bracket-Sequence Transformers
Apr 24, 2026Aryan Sharma, Cutter Dawes, Shivam RavalTransformer Interpretability Self-Attention
1Yale University · 2Independent · 3Harvard University
Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
Apr 23, 2026Boxun Xu, Yuming Du, Zichang Liu +7Autoregressive Diffusion Self-Attention
Meta Superintelligence Labs · University of California, Santa Barbara · Work done at Meta
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Efficient Transformer Inference Self-Attention
Harvard University
Stream-CQSA: Exact Out-of-Memory Recovery for Attention
Apr 22, 2026Yiming Bian, Joshua M. AkeySelf-Attention Memory-Efficient Inference
Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540
Simplified Sparse Attention via Gist Tokens
Apr 22, 2026Yuzhen Mao, Michael Y. Li, Emily B. FoxSelf-Attention Long-Context Language Model Inference
Stanford University
The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content
Apr 21, 2026Yuqi Zhang, Di ZhangRetrieval-Augmented Generation Self-Attention
Xi’an Jiaotong-Liverpool University

May 5, 2026, cs.LGJ/K move · Enter open · S save

Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks

Yaobo Zhang

School of Physics, Ningxia University Yinchuan, 750021, China

Rotary Positional Embeddings Self-Attention Transformer Attention Positional Encoding

Open paper AskPDF arXiv

Relative positional encodings determine which functions of query-key lag can enter the primitive attention logit. RoPE supplies a rotary phase, while ALiBi supplies an additive distance bias. Motivated by group-theoretic views of linear translation-invariant positional encodings, we study a non-semisimple case in which a complex rotary eigenvalue and a nilpotent response live in the same defective Jordan block. The resulting relative operator generates oscillatory-polynomial features such as

e^{-γd}\cos(ωd)

e^{-γd}\sin(ωd)

d e^{-γd}\cos(ωd)

, and

d e^{-γd}\sin(ωd)

, for causal lag

d=i-j\geq 0

. Thus the construction realizes a distance-modulated phase basis

d e^{iωd}

, rather than merely adding a separate distance channel to RoPE. We formulate Exact Jordan-RoPE as a non-semisimple one-parameter representation, give its real block form, and specify the contragredient query action required by non-orthogonal positional maps. We also distinguish this exact representation from stabilized variants whose bounded shear improves numerical behavior but breaks the exact group law. Kernel-level diagnostics and a Jordan-friendly synthetic language-model task show that the coupled Jordan basis is useful when the target contains distance-modulated phase interactions. On a small WikiText-103 byte language model, a scaled-exact variant improves over RoPE and direct-sum baselines within the Jordan family, while RoPE+ALiBi remains strongest overall. The evidence is structural rather than a broad performance claim.

Momentum

Related topics

Latest papers 195