Attention Mechanisms

Follow this topic Ask about this topic

Momentum

42 papers in the last four weeks, up 100% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 378

All topics

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
May 31, 2026Dong Le, Thong Nguyen, Cong-Duy Nguyen +1In-Context Retrieval Long-Context Language Modeling
Nanyang Technological University · National University of Singapore · VinUniversity
Efficient Temporal Modeling for Mobile Sleep Staging via Lightweight Random Attention
May 31, 2026Guisong Liu, Pengfei Wei, Jainsong Zhang +1Sleep Stage Classification Attention Mechanisms
School of Biological Science and Medical Engineering Southeast University Nanjing, China · School of Computer Science & Software Engineering Shenzhen University Shenzhen, China · Donders Institute for Brain, Cognition and Behaviour Radboud University Medical Center Nijmegen, The Netherlands
Global-Local Attention Decomposition for Terrain Encoding in Humanoid Perceptive Locomotion
May 30, 2026Shengcheng Fu, Yang Zhang, Zhanxiang Cao +4Legged Robotics Humanoid Robot Locomotion
Tongji University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China +1
Functional Attention: From Pairwise Affinities to Functional Correspondences
May 29, 2026Jiefang Xiao, Maolin Gao, Simon Weber +2PDE Operator Learning Attention Mechanisms
Technical University of Munich, Germany · Munich Center for Machine Learning (MCML), Germany · PIXL, Department of Computer Science, University of Oxford, United Kingdom +1
MultiAct: Text-to-Motion Generation from Composite Text via Tailored Attention Guidance
May 29, 2026Nathan Sala, Ofir Abramovich, Ariel Shamir +3Text-to-Motion Generation Cross-Modal Attention
Tel Aviv University, Israel · Reichman University, Israel · CYENS Centre of Excellence, Cyprus +1
Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
May 29, 2026Ruina Hu, Chen Wang, Lai Wei +5Reinforcement Learning with Verifiable Rewards Visually Grounded Reasoning
1Harbin Institute of Technology · 2Zhongguancun Academy · 4Nankai University +4
A Novel Global Context-aware Deep Neural Network for Enhanced Brain Tumor Segmentation using Magnetic Resonance Images
May 28, 2026Sourjya Mukherjee, Ananya Bhattacharjee, R. MuruganU-Net Medical Image Segmentation
Biomedical Imaging Laboratory, Department of Electronics and Communication Engineering, National Institute of Technology Silchar, Silchar, Assam, India, 788010
VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion
May 28, 2026Hidir Yesiltepe, Jiazhen Hu, Tuna Han Salih Meral +4Video Diffusion Models Self-Attention
1Virginia Tech · 2fal
Improving Adversarial Robustness of Attribution via Implicit Regularization
May 28, 2026Amir Mehrpanah, Matteo Gamba, Hossein AzizpourTransformer Interpretability Neural Network Robustness
Department of Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden · Department of Computer Science, Brown University, USA · Science for Life Laboratory, Stockholm, Sweden
Gated Graph Attention Networks with Learnable Temperature
May 28, 2026Zhongtian Ma, Hao Wu, Yexin Zhang +2Graph Attention Networks Graph Neural Networks
School of Cybersecurity, Northwestern Polytechnical University, Xi’an 710072, China · Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China
High-Fidelity Industrial Crash Dynamics Prediction via Geometry-Aware Operator Learning with Memory-Efficient Low-Rank Attention
May 26, 2026Deepak Akhare, Mohammad Amin Nabian, Corey Adams +2Low-Rank Attention Surrogate Modeling
Department of Aerospace and Mechanical Engineering, University of Notre Dame, Notre Dame, IN · NVIDIA, San Jose, CA · General Motors, Detroit, MI
LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation
May 26, 2026Gabriele Cesa, Thomas Hehn, Aleix Torres-Camps +4Rotary Positional Embeddings Parallel Decoding
Qualcomm AI Research∗
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
May 25, 2026Waleed Razzaq, Yun-Bo ZhaoRepresentation Learning Uncertainty Quantification
Department of Automation, University of Science & Technology of China, Hefei, China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center
Triplet-Block Diffusion RWKV
May 25, 2026Ke Lin, Yiyang Luo, Zhaolong Su +2Diffusion Language Model Inference Recurrent Transformers
William & Mary · HKUST · Cornell
ADMFormer: An Adaptive-Decomposition Transformer with Time-Varying Masked Spatial Attention for Traffic Forecasting
May 25, 2026Ruiwen Gu, Qitai Tan, Yahao Liu +1Attention Mechanisms Spatiotemporal Forecasting
Shenzhen Ubiquitous Data Enabling Key Lab Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · School of Computer Science and Engineering University of Electronic Science and Technology of China, Chengdu, China
Cross-Stage Attention Multi-Expert Network for Radiologist-Inspired Breast Ultrasound Diagnosis
May 25, 2026Xinyang Zhai, Chong Yang, Ruizhi ZhangMixture of Experts Medical Image Classification
Quaternion Self-Attention with Shared Scores
May 24, 2026Shogo Yamauchi, Tohru Nitta, Hideaki TamoriSelf-Attention Attention Mechanisms
The Asahi Shimbun Company, Tokyo, Japan · Tokyo Woman’s Christian University, Tokyo, Japan.
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
May 23, 2026Spandan PratyushTransformer Interpretability Sparse Attention
Independent Researcher, New Delhi 110070, India
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisSelf-Attention Gated Attention
Independent Researcher, Athens, Greece.
Patch Hierarchical Attention Transformer for Efficient Particle Jet Tagging
May 20, 2026Aaron Wang, Zihan Zhao, Alan Xia +5Jet Tagging Efficient Attention
University of Illinois Chicago Chicago, IL 60607, USA · University of California San Diego La Jolla, CA 92093, USA · California Institute of Technology Pasadena, CA 91125, USA +1
DASH: Fast Differentiable Architecture Search for Hybrid Attention in Minutes on a Single GPU
May 20, 2026Weizhe Chen, Miao Zhang, Junpeng Jiang +3LLM Inference Efficiency Attention Mechanisms
Harbin Institute of Technology (Shenzhen)
RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers
May 20, 2026Yuxi Liu, Zekun Zhang, Yixiang Cai +3Rotary Positional Embeddings Low-Rank Attention
Peking University · University of Electronic Science and Technology of China · Beijing University of Posts and Telecommunications
DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
May 18, 2026Yuxiang Huang, Nuno M. T. Gonçalves, Federico Alvetreti +5Self-Attention Long-Context Modeling
1Tsinghua University · 2Instituto Superior Técnico, Universidade de Lisboa. · 3Instituto de Telecomunicações. +6
RAVE: Re-Allocating Visual Attention in Large Multimodal Models
May 18, 2026Xi Leng, Xinhong Ma, Ziqiang Dong +4Visual Attention Self-Attention
Qwen Business Unit of Alibaba · The Chinese University of Hong Kong, Shenzhen · Beijing Institute of Technology
Optimising CSRNet with parameter-free attention mechanisms for crowd counting in public transport
May 18, 2026Aida Rostamza, Enrico Del Re, Joshua Cherian Varughese +1Intelligent Transportation Systems Crowd Counting
Johannes Kepler University Linz, Austria, Department Intelligent Transport Systems, Altenberger Straße 69, 4040 Linz, Austria.
Babel: Jailbreaking Safety Attention via Obfuscation Distribution Optimized Sampling
May 18, 2026Ziwei Wang, Jing Chen, Ruichao Liang +6LLM Safety LLM Jailbreak Attacks
Wuhan University · Nanyang Technological University · Southeast University +1
CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models
May 18, 2026Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid +3VLM Evaluation Vision-Language Models
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh · KAUST
LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models
May 17, 2026Xinting Jiang, Junyi Luo, Ruichen Qi +4On-Device Language Model Inference Hardware-Aware NAS
Brown University · University of Michigan · Google Research
Hamiltonian-Inspired Attention Mechanism for Scalable RF Transmitter Fingerprinting
May 17, 2026Chitraksh Singh, Monisha Dhanraj, Akram SheriffTransformer Attention Physics-Informed ML
Frondeur Labs · Cisco Systems

May 31, 2026, cs.CLJ/K move · Enter open · S save

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention

Dong Le, Thong Nguyen, Cong-Duy Nguyen, Anh Tuan Luu

Nanyang Technological University · National University of Singapore · VinUniversity

In-Context Retrieval Long-Context Language Modeling Linear Attention Attention Mechanisms

Open paper AskPDF arXiv

Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state, but it consistently lags on in-context retrieval and long-context tasks. Existing remedies act on the write side of memory through gating, delta updates, or kernel feature maps, but the read step is left unchanged: every past key contributes additively to the output, so useful targets are diluted by the bulk of stored vectors. We borrow one specific piece of softmax's geometry to construct a cheap read-time contraction of the query. A second-order Taylor expansion of the softmax log-partition at the isotropic-attention point gives a local quadratic model whose curvature coincides with the running key covariance, a quantity that can be maintained with the same recurrent/chunkwise mechanism as the linear-attention state. The associated linear operator contracts the query along the high-variance directions of memory before it reads the state. We call this mechanism Curvature-Conditioned Query (CCQ). CCQ modifies only the read step and is composable with any linear-attention backbone. Attached to GLA and Gated DeltaNet, it improves perplexity, zero-shot downstream accuracy, S-NIAH retrieval at and beyond the training context, length-extrapolation perplexity from 4K to 20K, and LongBench accuracy.

Momentum

Related topics

Latest papers 378