Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption
Jul 1, 2026Xiaomeng Fu, Jia Li, Yiming Hu +5Long-Horizon Video Generation Streaming Video Generation
Institute of Information Engineering, Chinese Academy of Sciences · AMAP, Alibaba Group · The University of Hong Kong
RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation
Jul 1, 2026Yaofu Liu, Wanli Lan, Jinxi Li +2Mixed-Precision Quantization Rotary Positional Embeddings
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Independent Researcher, China
Enhancing Oracle Bone Inscription Recognition via Multi-Scale Layer Attention
Jun 30, 2026Chaowen Yan, Kaishen Wang, Yong Wang +2Multi-Scale Feature Fusion Attention Mechanisms
aAcademy of Chinese Traditional Culture, Sichuan Normal University, Chengdu, China · bCollege of Computer Science, Sichuan University, Chengdu, China · cSchool of Artificial Intelligence, Sichuan University, Chengdu, China
MAGE: View-guided Point Cloud Completion with Efficient Modality Alignment and Adaptive Geometry Enhancement
Jun 30, 2026Weize Quan, Zhengwei Wu, Kai Wang +1Point Cloud Completion Cross-Modal Alignment
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · School of Artificial Intelligence, University of Chinese Academy Sciences, Beijing, China. · GIPSA-lab, Univ. Grenoble Alpes, CNRS, Grenoble INP, 38000 Grenoble, France.
Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding
Jun 29, 2026Seongro Yoon, Donghyeon Cho, Jinsun Park +1Vision Transformer Facial Expression Recognition
Inria, Université Côte d’Azur, France · Hanyang University, South Korea · Pusan National University, South Korea
Learning High-Level Decision Making with an Interaction-Aware Attention-Based Network in Autonomous Driving
Jun 29, 2026Marcelo Contreras, Willi Poh, Christoph Stiller +1Autonomous Driving Attention Mechanisms
NODE lab, University of Alberta, Edmonton, AB, T6G 1H9 Canada · MRT Institute, Karlsruhe Institute of Technology, Karlsruhe, Germany
Adaptive Financial Transformer with Regime-Gated Attention for Stock Return Prediction
Jun 28, 2026Dishan SarkarMultivariate Time Series Forecasting Time Series Forecasting
Department of Artificial Intelligence and Machine Learning Birla Institute of Technology, Mesra Ranchi, India
NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation
Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Long-Context Language Model Inference Sliding-Window Attention
FARS
DnA: Denoising Attention for Visual Tasks
Jun 25, 2026Ron Campos, Subhajit Maity, Xin Li +2Visual Attention Vision Transformer
University of Central Florida, USA · University of North Carolina at Charlotte, USA
CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
Jun 25, 2026Sayak DuttaLinear Attention Delta-Rule Linear Attention
Privacy Vulnerabilities of Attention Layers in Tabular Foundation Models and Protection of High-Risk Queries
Jun 24, 2026Tânia Carvalho, Maxime CordyTabular Foundation Models Membership Inference Attacks
SnT, University of Luxembourg · Luxembourg
Enhancing Layer Interaction Using Key-Correlated Layer Attention
Jun 24, 2026Jianlong Xiong, ChuanBo Xie, Le Yu +2Efficient Attention Linear Attention
College of Computer Science, Sichuan University, Chengdu, 610065, China · Department of Ultrasound, Zigong Maternal and Child Health Care Hospital, China
Lightweight PCGAE-Net: Parallel CrossGate Attention and Bottleneck AutoEncoder for Efficient 5G Channel Prediction
Jun 24, 2026Uma Kishore Godavarti, K. Giridhar, Vanani Prince Dharmendrabhai +2Autoencoders Attention Mechanisms
Network Modem Team, Samsung R&D India Bangalore · Department of Electrical Engineering, Indian Institute of Technology Madras, Chennai, India
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
Jun 24, 2026Luke McDermott, Robert W. Heath, Rahul ParhiLifelong Learning Agents Long-Context Modeling
UC San Diego
ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory
Jun 23, 2026Habibullah AkbarMemory-Augmented Language Models Long-Context Language Modeling
Kreasof AI Jakarta, Indonesia
All Routes Lead to Collapse
Jun 21, 2026K. R. BalasubramanianRepresentation Collapse Attention Sinks
Independent Research
Gated MLPs as Symmetry-Broken Rank-1 Bilinear Attention
Jun 20, 2026Nathan BreslowMultilayer Perceptrons Symmetry Breaking
CoSA: Correlation-Guided Change Attention with Learnable Residual Gating for Remote Sensing Change Detection
Jun 20, 2026Abdirashid Omar, Jonghyuk ParkRemote Sensing Change Detection Attention Mechanisms
Department of Data Science, Graduate School of Kookmin University Seoul 02707, Republic of Korea
Fidelity- and Perception-Aware Local Implicit Attention for Arbitrary-Scale Image Super-Resolution
Jun 20, 2026Yu-Syuan Xu, Hao-Lun Sun, Hao-Wei Chen +2Diffusion Models for Image Restoration Attention Mechanisms
1National Taiwan University · 2MediaTek Inc.
Low-Rank Attention Residuals
Jun 19, 2026Jonathan SuLow-Rank Attention Decoder-Only Language Models
Independent Researcher
The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups
Jun 18, 2026Przemyslaw MusialskiLie Group Methods Attention Mechanisms
New Jersey Institute of Technology
Attention as Frustrated Synchronization
Jun 17, 2026Joshua NunleyOscillatory Neural Networks Autoregressive Language Modeling
Cognitive Science Program Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington
Hierarchical Attention via Domain Decomposition
Jun 16, 2026Stephan Köhler, Oliver RheinbachLow-Rank Attention PDE Operator Learning
Faculty of Mathematics and Computer Science Technische Universität Bergakademie Freiberg, 09596 Freiberg, Germany
Attention Alignment Between Humans and Vision-Language Models
Jun 16, 2026Isaac R. Christian, Udith Haputhanthrige, Hanna Hornfeld +4VLM Evaluation Visual Attention
Princeton Neuroscience Institute, Princeton University · Department of Psychology, Princeton University · Department of Computer Science, Princeton University +2
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Jun 15, 2026Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4Neural Network Initialization Linear Attention
1The University of Sydney · 2Together AI · 5Microsoft +2
QK-Normed MLA: QK normalization without full key caching
Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV Caching Transformer Attention
School of Data Science, The Chinese University of Hong Kong, Shenzhen.
Rethinking the Role of Efficient Attention in Hybrid Architectures
Jun 13, 2026Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6Attention Head Analysis Long-Context Modeling
1Tsinghua University · 2OpenBMB
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels
Jun 12, 2026Tiantian ZhangEfficient Attention Linear Attention
Columbia University
Zero-shot generalization of transformer neural operators to larger domains
Jun 12, 2026Armand de Villeroché, Sibo Cheng, Vincent Le Guen +5Local Attention PDE Operator Learning
CEREA, ENPC, EDF R&D, Institut Polytechnique de Paris, Île-de-France, France · SINCLAIR AI Laboratory, Saclay, Île-de-France, France · EDF R&D, Île-de-France, France +1
WAM4D: Fast 4D World Action Model via Spatial Register Tokens
Jun 12, 2026Ying Li, Xiaobao Wei, Jiajun Cao +10World Models for Robotics Robotic Manipulation
1Peking University · 2The Hong Kong University of Science and Technology · 3Beijing Innovation Center of Humanoid Robotics

Jul 1, 2026, cs.CVJ/K move · Enter open · S save

Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption

Xiaomeng Fu, Jia Li, Yiming Hu, Yong Wang+4

Institute of Information Engineering, Chinese Academy of Sciences · AMAP, Alibaba Group · The University of Hong Kong

Long-Horizon Video Generation Streaming Video Generation Local Attention KV-Cache Compression Attention Mechanisms Autoregressive Video Generation

Open paper AskPDF arXiv

Autoregressive (AR) streaming models have emerged as a powerful paradigm for long video generation. However, the linearly growing Key-Value (KV) cache poses a significant bottleneck, leading to memory overload and degraded inference throughput. A common compression method is to drop redundant KV tokens, which often breaks long-range dependencies, resulting in temporal flickering and identity loss. In this paper, we propose Instance-Specific Parametric Absorption (ISPA), a novel framework that shifts the KV cache compression from discarding to distilling. The core idea is to transit a subset of layers from Full-Attention (F-Layers) to memory-efficient Local-Attention (L-Layers) by "absorbing" historical context into the model's weights. Specifically, during a brief warmup phase, ISPA monitors the output discrepancy between global and local attention. At the transition point, we solve a closed-form least-squares problem to compute an instance-specific weight modulation that compensates for the missing history. Experiments across architectures (1.3B to 14B) demonstrate that ISPA can remove up to 50% of the KV cache with near-lossless visual quality. We hope this perspective encourages future work to explore parametric memory consolidation beyond external token-level cache management for streaming generative models.

Momentum

Related topics

Latest papers 378