Attention Mechanisms

Follow this topic Ask about this topic

Attention mechanisms let a model select and combine information from other tokens, image regions, or data points, supporting tasks such as language understanding, visual recognition, and retrieval. The standard approach uses query-key-value comparisons with softmax, while sparse, linear or kernel-based, local, and hybrid methods aim to reduce the cost of processing long sequences; positional encodings and cross-attention adapt interactions to order, geometry, or another input. Researchers also study attention heads and circuits to understand model behavior, alongside mathematical analyses of expressivity, training dynamics, and representation collapse. A central challenge is balancing accuracy and generalization against memory and computation, while determining when attention weights provide reliable explanations rather than merely useful internal signals.

Latest papers

517 papers61 papers in the last four weeks, up 30% on the four weeks before. 0.4% of all new papers.

From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs
Mar 3, 2026Pengyu Lai, Yixiao Chen, Dewu Yang +3PDE Surrogate Modeling Neural Operators
School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China
Action-Guided Attention for Video Action Anticipation
Mar 2, 2026Tsung-Ming Tai, Sofia Casarin, Andrea Pilzer +2Temporal Video Understanding Attention Mechanisms
Free University of Bozen-Bolzano · NVIDIA
Chunk-wise Attention Transducers for Fast and Accurate Streaming Speech-to-Text
Feb 27, 2026Hainan Xu, Vladimir Bataev, Travis M. Bartley +1Streaming ASR Attention Mechanisms
NVIDIA Corporation.
FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking
Feb 27, 2026Sifan Zhou, Jiahao Nie, Ziyu Zhao +2Visual Object Tracking Point Cloud Learning
School of Automation Southeast University, Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education Nanjing, China · School of Information Technology and Artificial Intelligence, Zhejiang University of Finance and Economics Hangzhou, China · Big Data Institute Central South University Changsha, China
SHINE: Sequential Hierarchical Integration Network for EEG and MEG
Feb 27, 2026Xiran Xu, Yujie Yan, Songyi Li +4Electroencephalography Neural Processes
Speech and Hearing Research Center, School of Intelligence Science and Technology, Peking University, China · National Key Laboratory of General Artificial Intelligence, China · Center for BioMed-X Research, Academy for Advanced Interdisciplinary Studies, Peking University, China
Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition
Feb 27, 2026Mohammadreza Heidarianbaei, Mareike Dorozynski, Hubert Kanyamahanga +2Open-Vocabulary Semantic Segmentation Remote Sensing Image Segmentation
Institute of Photogrammetry and GeoInformation Leibniz University Hannover Germany
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Feb 26, 2026Jeongin Bae, Baeseong Park, Gunho Park +7Transformer Attention Attention Mechanisms
NAVER Cloud
Identifying Crucial Attention Heads for Multilingual Language Models: Retrieval and Retrieval-Transition Heads
Feb 25, 2026Shaswat Patel, Vishvesh Trivedi, Yue Han +2Multilingual Language Models Cross-Lingual Reasoning in Language Models
New York University
Beyond Attention: True Adaptive World Models via Spherical Kernel Operator
Feb 25, 2026Vladimer KhasiaWorld Model Learning Attention Mechanisms
Independent Researcher
Enhancing Renal Tumor Malignancy Prediction: Deep Learning with Automatic 3D CT Organ Focused Attention
Feb 25, 2026Zhengkang Fan, Chengkun Sun, Russell Terry +2Medical Image Classification 3D Medical Imaging
Department of Health Outcomes & Biomedical Informatics, University of Florida, Gainesville, FL, USA · Department of Computer & Information Sciences, Temple University, Philadelphia, PA, USA · Department of Urology, University of Florida, Gainesville, FL, USA
Position-Aware Sequential Attention for Accurate Next Item Recommendations
Feb 24, 2026Timur Nabiev, Evgeny FrolovAttention Mechanisms Sequential Recommendation
Skolkovo Institute of Science and Technology · AIRI
Nonparametric Teaching of Attention Learners
Feb 24, 2026Chen Zhang, Jianghui Wang, Bingyang Cheng +7Training Data Selection Data-Efficient Learning
The University of Hong Kong · King Abdullah University of Science and Technology · Independent Researcher
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
Feb 23, 2026Yuxing Tian, Fengran Mo, Weixu Zhang +2LLM Reranking Learning to Rank
University of Montreal · McGill University & MILA · International Digital Economy Academy
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Training Dynamics
TML Lab, EPFL
U-FedTomAtt: Ultra-lightweight Federated Learning with Attention for Tomato Disease Recognition
Feb 18, 2026Romiyal George, Sathiyamohan Nishankar, Selvarajah Thuseethan +4Lightweight Neural Networks Federated Learning
Department of Computer Engineering, University of Peradeniya, Sri Lanka. · Faculty of Science and Technology, Charles Darwin University, Australia. · Canadian Institute for Cybersecurity, University of New Brunswick, Canada. +1
CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
Feb 17, 2026Bradley McDanel, Steven Li, Harshit KhaitanCross-Layer Attention LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
RPT-SR: Regional Prior attention Transformer for infrared image Super-Resolution
Feb 17, 2026Youngwan Jin, Incheol Park, Yagiz Nalcakan +3Remote Sensing Infrared Image Super-Resolution Image Super-Resolution
Yonsei University · BK21 Graduate Program in Intelligent Semiconductor Technology
Feature Recalibration Based Olfactory-Visual Multimodal Model for Enhanced Rice Deterioration Detection
Feb 16, 2026Rongqiang Zhao, Hengrui Hu, Yijing Wang +2Multimodal Classification Multimodal Fusion
Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China. · National Key Laboratory of Smart Farm Technologies and Systems, Harbin 150001, China.
Competition for attention predicts good-to-bad tipping in AI
Feb 16, 2026Neil F. Johnson, Frank Y. HuoLLM Safety Language Modeling
Physics Department and Dynamic Online Networks Laboratory, George Washington University, Washington D.C. 20052, U.S.A.
Compressed Sensing for Capability Localization in Large Language Models
Feb 11, 2026Anna Bair, Yixuan Even Xu, Mingjie Sun +1Attention Head Analysis LLM Interpretability
Carnegie Mellon University
Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
Feb 11, 2026Keston Aquino-MichaelsTransformer Self-Attention
No Way Labs
FGAA-FPN: Foreground-Guided Angle-Aware Feature Pyramid Network for Oriented Object Detection
Feb 11, 2026Jialin MaMulti-Scale Feature Fusion Object Detection
International school of BUPT, Beijing University Of Posts And Telecommunications, Beijing,China.
Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inference
Feb 9, 2026Yifei Gao, Lei Wang, Rong-Cheng Tu +3Self-Attention KV Caching
University College London, London, UK · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences (CAS), Shenzhen 518055, China · The Chinese University of Hong Kong +1
ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference
Feb 8, 2026Jitai Hao, Qiang Huang, Yaowei Wang +2Self-Attention KV Caching
Harbin Institute of Technology (Shenzhen), China
Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences
Feb 6, 2026Alessio Pecilli, Matteo RosatiSelf-Attention Transformer Attention
Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy
FlashBlock: Attention Caching for Efficient Long-Context Block Diffusion
Feb 5, 2026Zhuokun Chen, Jianfei Cai, Bohan ZhuangVideo Diffusion Models Self-Attention
1Monash University · 2Zhejiang University
Poly-attention: a general scheme for higher-order self-attention
Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanSelf-Attention Compositional Reasoning
Computer Science, Columbia University, New York, NY 10027.
STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs
Feb 2, 2026Weikang Meng, Liangyu Huo, Yadan Luo +4LLM Inference Acceleration Linear Attention
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing
Feb 2, 2026Egor Serov, Ilya Kuleshov, Alexey ZaytsevEfficient Neural Network Inference Neural Controlled Differential Equations
Applied AI Institute · Risk AI Research Lab
Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
Jan 28, 2026Fengrui Zuo, Zhiwei Ke, Yiming Liu +3Diffusion Language Model Inference Token Pruning
School of Computer Science and Technology, University of Science and Technology of China · Suzhou Institute of Advanced Research, University of Science and Technology of China

Mar 3, 2026J/K move · Enter open · S save

From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs

Pengyu Lai, Yixiao Chen, Dewu Yang, Rui Wang+2

School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China

PDE Surrogate Modeling Neural Operators Physics-Informed Neural Operators Transformer Attention Attention Mechanisms

Open paper AskPDF arXiv

Partial differential equations (PDEs) are fundamental for modeling complex physical systems, yet classical numerical solvers face prohibitive computational costs in high-dimensional and multi-scale regimes. While Transformer-based neural operators have emerged as powerful data-driven alternatives, they conventionally treat all discretized spatial points as uniform, independent tokens. This monolithic approach ignores the intrinsic scale separation of physical fields, applying computationally prohibitive global attention that redundantly mixes smooth large-scale dynamics with high-frequency fluctuations. Rethinking Transformers through the lens of complex dynamics, we propose DynFormer, a novel dynamics-informed neural operator. Rather than applying a uniform attention mechanism across all scales, DynFormer explicitly assigns specialized network modules to distinct physical scales. It leverages a Spectral Embedding to isolate low-frequency modes, enabling a Kronecker-structured attention mechanism to efficiently capture large-scale global interactions with reduced complexity. Concurrently, we introduce a Local-Global-Mixing transformation. This module utilizes nonlinear multiplicative frequency mixing to implicitly reconstruct the small-scale, fast-varying turbulent cascades that are slaved to the macroscopic state, without incurring the cost of global attention. Integrating these modules into a hybrid evolutionary architecture ensures robust long-term temporal stability. Extensive memory-aligned evaluations across four PDE benchmarks demonstrate that DynFormer achieves up to a 95% reduction in relative error compared to state-of-the-art baselines, while significantly reducing GPU memory consumption. Our results establish that embedding first-principles physical dynamics into Transformer architectures yields a highly scalable, theoretically grounded blueprint for PDE surrogate modeling.