Linear Attention

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, up 43% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction
May 22, 2026Chong Cheng, Peilin Tao, Nanjie Yao +93D ViTs Streaming 3D Reconstruction
HKUST(GZ) · Horizon Robotics · CASIA +2
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
Fast and Stable Triangular Inversion for Delta-Rule Linear Transformers
May 20, 2026Aleksandros Sobczyk, Gioele Gottardo, Christos K. Matzoros +4Efficient Transformer Inference AI Accelerator Inference
Computing Systems Lab Huawei Technologies2026 Switzerland
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1
KVBuffer: IO-aware Serving for Linear Attention
May 18, 2026Longwei Zou, Lin ZhongLLM Inference Serving Linear Attention
Department of Computer Science Yale University New Haven, CT 06511
Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion
May 15, 2026Kunyang Li, Mubarak Shah, Yuzhang ShangMemory-Augmented Neural Networks Video Diffusion Models
Institute of Artificial Intelligence, University of Central Florida
Representative Attention For Vision Transformers
May 14, 2026Yuntong Li, Hainuo Wang, Hengxing Liu +2Vision Transformer Efficient ViTs
College of Intelligence and Computing, Tianjin University, Tianjin 300350, China
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
May 13, 2026Chenyu Zhou, Hongpei Li, Yuerou Liu +3Transformer Attention Linear Attention
Shanghai Jiao Tong University · Northwestern University · Huazhong University of Science and Technology +1
Exact Linear Attention
May 13, 2026Weinuo OuEfficient Transformer Inference Self-Attention
Wuyi University
The Routing and Filtering Structure of Attention
May 12, 2026Shafayeth Jamil, Rehan KapadiaAttention Head Analysis Self-Attention
Ming Hsieh Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA 90089
Variational Linear Attention: Stable Associative Memory for Long-Context Transformers
May 11, 2026Vishal Pandey, Gopal SinghEfficient Attention Linear Attention
Independent Researcher London, UK · Metriqual Athens, GR
PhysEDA: Physics-Aware Learning Framework for Efficient EDA With Manhattan Distance Decay
May 11, 2026Zetao YangEfficient Neural Network Inference Reward Shaping
School of Mathematics and Statistics Beijing Institute of Technology Beijing, China
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Adaptive Memory Decay for Log-Linear Attention
May 7, 2026Yaxita Amin, Helen Zichen Li, Mengfan Zhang +1Long-Context Modeling Linear Attention
University of Maryland
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Momentum Methods Linear Attention
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Linearizing Vision Transformer with Test-Time Training
May 4, 2026Yining Li, Dongchen Han, Zeyu Liu +3Efficient Transformer Inference Self-Attention
Tsinghua University, Beijing, China
ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers
Apr 26, 2026Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1Efficient Transformer Inference Softmax Attention
Advanced Computer Vision Laboratory, National Yang Ming Chiao Tung University
LayerBoost: Layer-Aware Attention Reduction for Efficient LLMs
Apr 23, 2026Mohamed Ali Souibgui, Jan Fostier, Rodrigo Abadía-Heredia +3Transformer Attention LLM Inference Acceleration
Openchip & Software Technologies, S.L.
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI
LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel
Apr 22, 2026Zhe Feng, Sen Lian, Changwei Wang +5Efficient ViTs Efficient Attention
1MAIS,Institute of Automation, Chinese Academy of Sciences · usr 3China Electronics Data Corporation · 5The Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology +4
FG $^2$ -GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Apr 21, 2026Pingwei Sun, Yuxuan Hu, Jianchao Tan +6Long-Context Modeling Linear Attention
Meituan · Beijing, China
StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation
Mar 24, 2026Zhiyuan Chen, Yuxuan Zhong, Fan Wang +4Linear Attention In-Context Learning
Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, China
Low-latency Event-based Object Detection with Spatially-Sparse Linear Attention
Mar 6, 2026Haiqing Hao, Zhipeng Sui, Rong Zou +4Efficient Object Detection Linear Attention
State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing, China · Robotics and Perception Group, University of Zurich, Zurich, Switzerland · ShanghaiTech University, Shanghai, China
RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
Feb 12, 2026Kaicheng Xiao, Haotian Li, Liran Dong +1Memory-Augmented Neural Networks Long-Context Language Modeling
The Chinese University of Hong Kong
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference Acceleration Linear Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
Exact Flow Linear Attention: Exact Solution from Continuous-Time Dynamics
Dec 14, 2025Jingdi Lei, Di Zhang, Soujanya PoriaDynamical Systems Linear Attention
Nanyang Technological University · Fudan University
Selective Rotary Position Embedding
Nov 21, 2025Sajad Movahedi, Timur Carstensen, Arshia Afzal +3Softmax Attention Rotary Positional Embeddings
ELLIS Institute Tübingen1, LIONS, EPFL2, University of Freiburg3, Max-Planck-Institute for Intelligent Systems4, Prior Labs5
LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling
Oct 21, 2025Yang Liu, Yunjiang Jiang, Ayush Agarwal +3Recommender Systems Efficient Inference
Meta
MatchAttention: Embedding Explicit Matching Constraints into Attention for Efficient Stereo Matching
Oct 16, 2025Tingman Yan, Tao Liu, Chenghao Li +4Image Matching Stereo Matching
School of Control Science and Engineering, Dalian University of Technology, Dalian, 116024, China · Pika Labs · Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China +1
Neural Architecture Discovery via Autonomous Evolution
Jul 24, 2025Weixian Xu, Yixiu Liu, Yang Nan +4Autonomous Research Agents AI Agents for Scientific Discovery
Shanghai Jiao Tong University · SII · GAIR +1

May 22, 2026, cs.CVJ/K move · Enter open · S save

HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction

Chong Cheng, Peilin Tao, Nanjie Yao, Guanzhi Ding+8

HKUST(GZ) · Horizon Robotics · CASIA+2

3D ViTs Streaming 3D Reconstruction 3D Reconstruction Camera Pose Estimation Linear Attention

Open paper AskPDF arXiv

Online 3D reconstruction requires estimating camera pose and scene geometry under strict causal and bounded-memory constraints. Existing methods often suffer from drift, jitter, or collapse on long sequences. We trace these failures to a fundamental mismatch. Streaming geometry is inherently temporally heterogeneous, with evidence ranging from short-lived correspondences to persistent global scale. However, current architectures impose uniform and pathological influence patterns. For example, sliding windows enforce hard cutoffs, while ungated recurrence and causal attention cause cache saturation and spike-like attention sinks. To resolve this, we formalize geometric propagation as an \emph{evidence influence kernel} and propose HorizonStream, a long-horizon Transformer that explicitly factorizes this kernel. For the long-range temporal factor, Geometric Linear Attention learns channel-wise decay rates to enable bounded, multi-timescale propagation of geometric evidence. For the short-range spatial factor, Geometric Local Attention with Spatiotemporal RoPE performs reliable 3D matching while suppressing attention sinks. Finally, Metric Readout Tokens recover stable scale and rigid pose directly from the persistent geometric state. Extensive experiments show that HorizonStream, trained on only 48-frame clips, generalizes stably to sequences exceeding 10,000\ frames with constant memory and linear time, achieving state-of-the-art streaming 3D reconstruction performance. Project Page: https://3dagentworld.github.io/horizonstream/

Momentum

Related topics

Latest papers 95