Transformer Attention

Follow this topic Ask about this topic

Recent momentum

-78%

2 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-14

1 new paper

A weekly snapshot of new work published in Transformer Attention.

Period ending 2026-09-07

1 new paper

A weekly snapshot of new work published in Transformer Attention.

88 papers

Latest in Transformer Attention

Browse all topics →

NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers
Sep 23, 2026Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang +1Transformer Attention Batch Normalization
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Sep 2, 2026Runlin Shi, Bojian Yin, Guoqi LiTransformer Attention Rotary Position Embedding
Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras
Sep 1, 2026Jiming Feng, Junliang LiTransformer Attention Transformer Architectures
Disentangling the Expressivity of RoPE
Aug 12, 2026Selim Jerad, Anej Svete, Jiaoda Li +1Rotary Position Embedding Expressivity
Rethinking Attention Locality in Spiking Transformers
Aug 9, 2026Zeqi Zheng, Zizheng Zhu, Yuping Yan +3Transformer Attention Binary Spikes
Gated Spatial Redundancy Projection for Pathology Transformer Attentions
Aug 8, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Whole-Slide Images Computational Pathology
Spectral Outliers Reveal Dominant Learned Structure in Transformer Attention
Aug 8, 2026Kasun Dewage, Marianna Pensky, Suranadi De Silva +1Transformer Attention Outliers
Beyond Attention: Signed Integrated Gradients Attribution in a BiomeGPT-Style Microbiome Transformer
Aug 6, 2026Oren NelsonTraining Data Attribution Transformer Attention
Attention-Only White-Box Transformer via LeJEPA-Based Self-Supervised Pretraining
Aug 4, 2026Yang Bai, Linyuan Wang, Haoyang Jiang +3Transformer Attention Self-Supervised
Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Design
Aug 2, 2026Sen SongTransformer Attention Softmax Attention
Phase Structure in Rotary Attention: A Spectral Framework for Semantic Continuity and Execution-Boundary Governance
Jul 28, 2026Abraham ChachamovitsRotary Position Embedding Transformer Attention
SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation
Jul 27, 2026Yu Cui, Yi Xu, Jiahao Wang +6Transformer Attention Recommendation
A Coulomb Particle Model for Learning Kernel Attention in Transformers
Jul 26, 2026Masoud Badiei Khuzani, Sharath Honnaiah, Atiq Islam +2Transformer Attention Feature Learning
Interior interpretability with attention rollout: contraction and propagation profiles in Transformers
Jul 24, 2026Umberto Biccari, Qian Huang, Enrique ZuazuaTransformer Attention Model Interpretability
RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
Jul 23, 2026Zahra Yousefijamarani, Alaa AlameldeenTransformer Attention Transformer Architectures
MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsQuadratic Attention Transformer Attention
A Controlled Study of Attention-Only Transformers
Jul 20, 2026Henry Ndubuaku, Karen Mosoyan, Jakub Mroz +5Transformer Attention Transformer Encoder
Toward Localizing and Repairing Bias in Transformer Attention Heads
Jul 14, 2026Sigma JahanTransformer Attention Transformer Architectures
MMA-Former: Multi-Window Mixture-of-Head Attention Transformer for Adaptive PNI Prediction in 3D MRI
Jul 13, 2026Youngung Han, Induk Um, Kyeonghun Kim +9Magnetic Resonance Imaging Progressive Attention Fusion
From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Jul 12, 2026Binbin Lin, Wei Chen, Yalun Li +3Transformer Attention Transformer Architectures
FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention
Jul 8, 2026Athanasios ZerisTransformer Attention Fourier
X-LogSMask: Expand Transformer for Graph-Structured Data
Jul 2, 2026Leyan Li, Rennong Yang, Zhenxing Zhang +1Heterogeneous Graph Transformers Transformer Attention
Morphing into Hybrid Attention Models
Jun 29, 2026Disen Lan, Jianbin Zheng, Yuxi Ren +5Hybrid Transformer Transformer Attention
Adaptive Financial Transformer with Regime-Gated Attention for Stock Return Prediction
Jun 28, 2026Dishan SarkarTransformer Attention Transformer Architectures
Communicability-Inspired Positional Encoding (CIPE)
Jun 24, 2026Yipeng Zhang, Zhongtian Sun, Pietro Liò +1Heterogeneous Graph Transformers Positional Encoding
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Jun 23, 2026Vatsal Baherwani, Zixi Chen, Shikai Qiu +2Transformer Attention Dynamic Sparse Attention
Predictive Repair Management Using a Multi-Head Attention Transformer and Online Learning
Jun 19, 2026Xinyao Zhang, Willie Cade, Karl R. Haapala +2Predictive Maintenance Deep Learning Framework
Scalable Hierarchical Attention Transformers for Multi-Turn Jailbreak Detection in Long Conversations
Jun 19, 2026Chenhui Hu, Muhammed Salih, Sudipto Guha +1Jailbreak Evaluations Transformer Attention
Complementary Attention Head Pruning for Efficient Transformers
Jun 17, 2026Yaniv Livertovsky, Shahar Somin, Gonen SingerTransformer Attention Heterogeneous Pruning Scores
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Jun 16, 2026Viet-Hoang Tran, Vinh Khanh Bui, Van-Hoan Trinh +2Positional Encoding Transformer Attention
Adaptive inference and function vectors in deep transformers
Jun 15, 2026Ravin Raj, Gautam ReddyTransformer Architectures Transformer Attention
RATS! Patches Talk Through Registers: Emergent Parts in Register Attention Transformers
Jun 12, 2026Timing Yang, Predrag Neskovic, Jansen Seheult +4Self-Supervised Vision Transformers Transformer Attention
Transformer Based Model for Spatiotemporal Feature Learning in EEG Emotion Recognition
Jun 9, 2026Xinglong Cui, Dian GuTransformer Attention Spatiotemporal
Integrating gene regulatory priors into Transformer attention with scTransformer for interpretable scRNA-seq analysis
Jun 8, 2026Mikele Milia, Louis Fabrice Tshimanga, Henning Mueller +2Single-Cell Rna Sequencing Transformer Attention
Hasse Diagrams for Attention: A Partial Order Framework for Designing Transformer Masks
Jun 8, 2026Chentao Li, Han GuoTransformer Attention Causal Attention
Chiaroscuro Attention: Spending Compute in the Dark
Jun 6, 2026Prateek Kumar SikdarTransformer Attention Representational Capacity
Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers
Jun 6, 2026Philip QuirkeTransformer Attention Transformer Architectures
Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram
Jun 4, 2026Athanasios ZerisTransformer Attention Multimodal Transformers
Signed Dual Attention: Capturing Signed Dependencies in Time Series Forecasting
Jun 3, 2026Balthazar Courvoisier, Tristan CazenaveTransformer Attention Time Series
Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
May 29, 2026Harry Jake Cunningham, Nicola Muca CironeTransformer Attention Importance
ConTrans: Learning Text-enhanced Local-global Temporal Representations for Zero-shot Temporal Action Localization
May 29, 2026Kanchan Keisham, Thenukan Pathmanathan, Thangarajah AkilanTemporal Action Localization Multiscale
Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
May 28, 2026Anej Svete, William Merrill, Ryan Cotterell +1Transformer Attention Transformer Architectures
Accelerating HEVC Intra Partitioning via a CNN-Hierarchical Attention Transformer Hybrid
May 27, 2026Krishna Kumar Sharma, Somdyuti PaulSelf-Supervised Vision Transformers Neural Video Codecs
Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
May 25, 2026Athanasios ZerisTransformer Attention Positional Encoding
Generic Interpretation Approach for Transformer Models Incorporating Heterogenous Attention Structures
May 25, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenTransformer Attention Multimodal Transformers
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
May 23, 2026Spandan PratyushTransformer Attention Dynamic Sparse Attention
Every Component is a Lookup: Token Attribution and Composition from a Single Decomposition
May 22, 2026Po-Kai Chen, Niki van Stein, Aske PlaatTransformer Attention Transformer Architectures
Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers
May 22, 2026Yongzhong XuTransformer Attention Transformer Architectures
Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
May 21, 2026Athanasios ZerisTransformer Attention Inductive Bias
Patch Hierarchical Attention Transformer for Efficient Particle Jet Tagging
May 20, 2026Aaron Wang, Zihan Zhao, Alan Xia +5High Energy Physics Transformer Attention
Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics
May 19, 2026Lucky VermaModel Weights Transformer Attention
Hamiltonian-Inspired Attention Mechanism for Scalable RF Transmitter Fingerprinting
May 17, 2026Chitraksh Singh, Monisha Dhanraj, Akram SheriffRadio Frequency Transformer Attention
Attention Dispersion in Dynamic Graph Transformers: Diagnosis and a Transferable Fix
May 15, 2026Jinhao Zhang, Kangfei Zhao, Qiuhao Zeng +1Transformer Attention Transformer Architectures
Exact Linear Attention
May 13, 2026Weinuo OuQuadratic Attention Transformer Attention
Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
May 12, 2026Lingdong Li, Hangming Zhang, Qiang YuSpiking Neural Networks Transformer Attention
Transformer Interpretability from Perspective of Attention and Gradient
May 12, 2026Yongjin Cui, Xiaohui Fan, Huajun ChenTransformer Attention Vision Transformer
Beyond Similarity: Temporal Operator Attention for Time Series Analysis
May 11, 2026Jevon Twitty, Vinh Pham, Nitiwith Rotchanarak +4Spatial Attention Transformer Attention
RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
May 11, 2026Byeongchan Kim, Arijit Sehanobish, Avinava Dubey +2Positional Encoding Transformer Attention
Complex-Valued Phase-Coherent Transformer
May 11, 2026Leona HiokiTransformer Attention Unified Transformer
CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring
May 10, 2026Li LixingInstruction-Tuned Models Cross-Attention

Sep 23, 2026 · cs.LGJ/K move · Enter open · S save

NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers

Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang, Ronghui Mu

University of Exeter

Newton-Schulz (NS) iteration has recently been used in the Muon optimizer to transform update matrices during the training of large language models. Motivated by its spectral effect, we investigate applying NS directly to Transformer attention representations. We introduce Newton-Schulz Attention (NS-Attn.), a parameter-free transformation applied to the output of each attention head. Each head output is arranged as a feature-by-token matrix and normalized by its Frobenius norm. We then apply a finite NS polynomial step and restore the original norm. The objective is to reduce spectral concentration and increase effective rank before standard head merging and output projection. Across ViT and Swin on CIFAR-10 and CIFAR-100, NS-Attn. improves final-epoch accuracy in all 12 matched-seed comparisons, with mean gains of 0.25--0.83 percentage points. ViT ablations show higher mean accuracy with one iteration than with two. Spectral analysis further shows reduced leading-eigenvalue concentration and increased effective rank. These gains incur additional inference latency.

Transformer Attention Batch Normalization

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Recent digests

1 new paper

1 new paper

Latest in Transformer Attention