Transformer Attention

Follow this topic Ask about this topic

Transformer attention uses queries, keys, and values to decide which tokens should contribute information to one another, with positional encodings and masks shaping those interactions. Work seeks both to improve this basic mechanism and to reduce its costs, using approaches such as sparse, linear, kernel, and local attention, as well as efficient GPU kernels and compressed key-value caches. Researchers also add structural or domain-specific biases and study attention’s role in learning, retrieval, stability, and interpretability across language, vision, graphs, and time series. A central tension remains: methods that reduce computation or constrain interactions can improve speed and memory use, but may lose accuracy or long-range retrieval quality, and no single design is best across settings.

Latest papers 308

41 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

All topics

From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs
Mar 3, 2026Pengyu Lai, Yixiao Chen, Dewu Yang +3PDE Surrogate Modeling Neural Operators
School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China
On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functions
Mar 3, 2026Linyan Gu, Lihua Yang, Feng ZhouTransformer Expressivity Neural Network Approximation Theory
School of Mathematics, Sun Yat-sen University · School of Information Science, Guangdong University of Finance and Economics
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
Feb 28, 2026April FuVLM Hallucination Mitigation Vision-Language Models
Independent Researcher
Transformers are Stateless Differentiable Neural Computers
Feb 27, 2026Bo Tang, Weiwei XieTransformer Memory-Augmented Neural Networks
Department of Electrical and Computer Engineering, Worcester Polytechnic Institute, Worcester, MA 01609 · Department of Data Science, Worcester Polytechnic Institute, Worcester, MA 01609
Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention
Feb 26, 2026Jeongin Bae, Baeseong Park, Gunho Park +7Transformer Attention Attention Mechanisms
NAVER Cloud
Incremental Learning of Sparse Attention Patterns in Transformers
Feb 22, 2026Oğuz Kaan Yüksel, Rodrigo Alvarez Lucendo, Nicolas FlammarionSelf-Attention Transformer Attention
TML Lab, EPFL
Toward Manifest Relationality in Transformers via Symmetry Reduction
Feb 21, 2026J. François, L. RaveraTransformer Transformer Attention
Mons University (UMONS), 20 Place du Parc, 7000 Mons, Belgium. · Grupo de Investigación en Física Teórica, Universidad Católica De La Santísima Concepción, Alonso de Ribera 2850, Concepción, Chile.
Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training
Feb 21, 2026Seyed Morteza EmadiTransformer Attention Attention Quantization
UNC-Chapel Hill, Kenan-Flagler Business School, Chapel Hill, NC, USA
Exact Attention Sensitivity and the Geometry of Transformer Stability
Feb 21, 2026Seyed Morteza EmadiTransformer Attention Softmax Attention
UNC-Chapel Hill, Kenan-Flagler Business School Chapel Hill, NC, USA
LORA-CRAFT: Cross-layer Rank Adaptation via Frozen Tucker Decomposition of Pre-trained Attention Weights
Feb 19, 2026Kasun Dewage, Marianna Pensky, Shankadeep Mondal +1Tensor Decomposition Low-Rank Adaptation
University of Central Florida Orlando, FL 32816, USA
In-Context Learning in Linear vs. Quadratic Attention Models: An Empirical Study on Regression Tasks
Feb 19, 2026Ayush Goel, Arjun Kohli, Sarvagya SomvanshiIn-Context Learning Transformer Attention
University of California, Berkeley
Selective Synchronization Attention
Feb 16, 2026Hasi HaysSelf-Attention Transformer Attention
Department of Chemical Engineering, University of Arkansas, Fayetteville, AR 72701, USA
Central Dogma Transformer II: An AI Microscope for Understanding Cellular Regulatory Mechanisms
Feb 9, 2026Nobuyuki OtaTransformer Interpretability Single-Cell Perturbation Modeling
Independent Researcher, Burlingame, CA, USA
Quantum Attention by Overlap Interference: Predicting Classical and Many-Body Quantum Sequences
Feb 6, 2026Alessio Pecilli, Matteo RosatiSelf-Attention Transformer Attention
Dipartimento di Ingegneria Civile, Informatica e delle Tecnologie Aeronautiche, Universit`a degli Studi Roma Tre, Via della Vasca Navale 79, 00146 Rome, Italy
Poly-attention: a general scheme for higher-order self-attention
Feb 2, 2026Sayak Chakrabarti, Toniann Pitassi, Josh AlmanSelf-Attention Compositional Reasoning
Computer Science, Columbia University, New York, NY 10027.
Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
Jan 11, 2026Jaewon Sok, Jewon Yeom, Seonghyeon Park +2LLM Pruning Attention Head Analysis
Department of Rural Systems Engineering, Seoul National University · Graduate School of Data Science, Seoul National University · Department of Aerospace Engineering, Seoul National University
Architecture Before the Formula: Training-Dependent Computational Semantics in Transformers
Jan 10, 2026Luis F. Rosario FreytesTransformer Attention Neural Network Interpretability
GeoTransolver: Learning Physics on Irregular Domains Using Multi-scale Geometry Aware Physics Attention Transformer
Dec 23, 2025Corey Adams, Rishikesh Ranade, Ram Cherukuri +1Neural Surrogate Modeling Structural Mechanics
NVIDIA Corporation, Santa Clara, US
Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer
Nov 24, 2025Haoyu Wu, Jingyi Xu, Qiaomu Miao +2Rotary Positional Embeddings Video Diffusion Models
Stony Brook University, NY, USA · UNC-Charlotte, NC, USA
Controllably Efficient Language Models
Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathEfficient Transformer Inference Long-Context Language Modeling
New York University
A Unified Geometric Space for Topological Alignment Between Transformer-Based Models and Human Brain Networks
Oct 28, 2025Silin Chen, Yuzhong Chen, Caiwei Wang +9Neural Representation Geometry Transformer Attention
The Clinical Hospital of Chengdu Brain Science Institute, MOE-K Lab for NeuroInformation, Brain-Apparatus Communication Institute, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu. 611731, China. · School of Automation, Northwestern Polytechnical University, Xi’an 710072, China. · Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA. +1
Higher Embedding Dimension Creates a Stronger World Model for a Simple Sorting Task
Oct 21, 2025Brady Bhalla, Honglu Fan, Nancy Chen +1Transformer Interpretability Representation Learning
California Institute of Technology · Pasadena, CA 91125 · Google DeepMind +3
Qubit-centric Transformer for Surface Code Decoding
Oct 13, 2025Seong-Joon Park, Hee-Youl Kwak, Yongjune KimNeural Decoding Quantum Error Correction
Samsung Electronics Company, Ltd., Suwon 16677, South Korea · Institute of Artificial Intelligence, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea · Department of Electrical, Electronic and Computer Engineering, University of Ulsan, Ulsan 44610, South Korea +1
Accelerating Attention with Basis Decomposition
Oct 2, 2025Jialin ZhaoSelf-Attention Transformer Attention
Department of Computer Science Tsinghua University
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
Sep 12, 2025Rupert Mitchell, Kristian KerstingLanguage Model Pretraining Softmax Attention
Department of Computer Science, TU Darmstadt, Darmstadt, Germany · Hessian Center for AI (hessian.AI), Darmstadt, Germany · German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany +1
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Sep 4, 2025Peter RacioppoSelf-Attention Long-Context Language Modeling
Independent Researcher, Los Angeles, CA, USA.
Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
Jul 21, 2025Andrew GracykRiemannian Optimization Transformer Attention
NABLA: Neighborhood Adaptive Block-Level Attention
Jul 17, 2025Dmitrii Mikhailov, Aleksey Letunovskiy, Maria Kovaleva +6Video Diffusion Models Diffusion Transformer
Kandinsky Lab, Moscow, Russia
A foundation model with multi-variate parallel attention to generate neuronal activity
Jun 25, 2025Francesco Carzaniga, Michael Hersche, Abu Sebastian +2Electroencephalography EEG Decoding
IBM Research – Zurich, Rüschlikon, Switzerland · Department of Neurology, Inselspital, Sleep-Wake-Epilepsy-Center, Bern University Hospital, Bern University, Bern, Switzerland
Ordinary Least Squares as an Attention Mechanism
Apr 13, 2025Philippe Goulet CoulombeTransformer Attention Linear Regression
Département des Sciences Économiques, Université du Québec à Montréal

Mar 3, 2026J/K move · Enter open · S save

From Complex Dynamics to DynFormer: Rethinking Transformers for PDEs

Pengyu Lai, Yixiao Chen, Dewu Yang, Rui Wang+2

School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai, 200240, China

PDE Surrogate Modeling Neural Operators Physics-Informed Neural Operators Transformer Attention Attention Mechanisms

Open paper AskPDF arXiv

Partial differential equations (PDEs) are fundamental for modeling complex physical systems, yet classical numerical solvers face prohibitive computational costs in high-dimensional and multi-scale regimes. While Transformer-based neural operators have emerged as powerful data-driven alternatives, they conventionally treat all discretized spatial points as uniform, independent tokens. This monolithic approach ignores the intrinsic scale separation of physical fields, applying computationally prohibitive global attention that redundantly mixes smooth large-scale dynamics with high-frequency fluctuations. Rethinking Transformers through the lens of complex dynamics, we propose DynFormer, a novel dynamics-informed neural operator. Rather than applying a uniform attention mechanism across all scales, DynFormer explicitly assigns specialized network modules to distinct physical scales. It leverages a Spectral Embedding to isolate low-frequency modes, enabling a Kronecker-structured attention mechanism to efficiently capture large-scale global interactions with reduced complexity. Concurrently, we introduce a Local-Global-Mixing transformation. This module utilizes nonlinear multiplicative frequency mixing to implicitly reconstruct the small-scale, fast-varying turbulent cascades that are slaved to the macroscopic state, without incurring the cost of global attention. Integrating these modules into a hybrid evolutionary architecture ensures robust long-term temporal stability. Extensive memory-aligned evaluations across four PDE benchmarks demonstrate that DynFormer achieves up to a 95% reduction in relative error compared to state-of-the-art baselines, while significantly reducing GPU memory consumption. Our results establish that embedding first-principles physical dynamics into Transformer architectures yields a highly scalable, theoretically grounded blueprint for PDE surrogate modeling.