Transformers

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

42 new papers

A weekly snapshot of new work published in Transformers.

Period ending 2026-09-14

3 new papers

A weekly snapshot of new work published in Transformers.

Inside this field

Focused directions

Focused topic

Transformer Architectures

Within Transformers

684 papersExplore

Focused topic

Quadratic Attention

Within Transformers

144 papersExplore

Focused topic

Dynamic Sparse Attention

Within Transformers

109 papersExplore

Focused topic

Transformer Attention

Within Transformers

89 papersExplore

Focused topic

Transformer Encoder

Within Transformers

80 papersExplore

Focused topic

Transformer Layers

Within Transformers

80 papersExplore

Focused topic

Softmax Attention

Within Transformers

74 papersExplore

Focused topic

Unified Transformer

Within Transformers

40 papersExplore

Focused topic

Multi-Head Self-Attention

Within Transformers

35 papersExplore

Focused topic

Autoregressive Transformers

Within Transformers

33 papersExplore

Focused topic

Global Attention Mechanism

Within Transformers

20 papersExplore

1,208 papers

Latest in Transformers

Browse all topics →

Simply Stabilizing the Loop via Fully Looped Transformer
May 11, 2026Rao Fu, Zixuan Yang, Jiankun Zhang +4Transformer Architectures Batch Normalization
Robust Basis Spline Decoupling for the Compression of Transformer Models
May 11, 2026Joppe De Jonghe, Van Tien Pham, Mariya IshtevaTransformer Architectures Context Compression
Rethinking Random Transformers as Adaptive Sequence Smoothers for Sleep Staging
May 11, 2026Guisong Liu, Xin Gao, Martin Dresler +2Sleep Stage Classification Sleep
Dissecting Jet-Tagger Through Mechanistic Interpretability
May 11, 2026Saurabh Rai, Sanmay GangulyMechanistic Interpretability High Energy Physics
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahQuadratic Attention Efficient Long-Context Inference
Continuous Latent Contexts Enable Efficient Online Learning in Transformers
May 11, 2026Emile Anand, Abdullah Ateyeh, Xinyuan Cao +1In-Context Learning Transformer Architectures
Probing Routing-Conditional Calibration in Attention-Residual Transformers
May 11, 2026Wenhao Liang, Lin Yue, Wei Emma Zhang +4Post-Hoc Transformer Architectures
The Association of Transformer-based Sentiment Analysis with Symptom Distress and Deterioration in Routine Psychotherapy Care
May 11, 2026Douglas K. Faust, Peter Awad, Alexandre Vaz +1Sentiment Analysis Cognitive Behavioral Therapy
Pretraining large language models with MXFP4 on Native FP4 Hardware
May 11, 2026Musa Cim, Poovaiah Palangappa, Miro Hodak +3W4A4 Quantization Fp8
TokaMind for Power Grid: Cross-Domain Transfer from Fusion Plasma
May 10, 2026JC Wu, Norton Lee, Kai Siang ChenPower Systems Unified Transformer
Nectar: Neural Estimation of Cached-Token Attention via Regression
May 10, 2026João Monteiro, Michal Klein, Pierre Ablin +1Softmax Attention Efficient Long-Context Inference
CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring
May 10, 2026Li LixingInstruction-Tuned Models Cross-Attention
One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning
May 10, 2026Bowen He, Juncheng Dong, Lin Lin +1Deep Reinforcement Learning In-Context Learning
Benchmarking Transformer and xLSTM for Time-Series Forecasting of Heat Consumption
May 10, 2026Marja Wahl, Daniel R. Bayer, Sven Rausch +1Electricity Price Forecasting Bidirectional Long Short-Term Memory
Spatial-Frequency Gated Swin Transformer for Remote Sensing Single-Image Super-Resolution
May 10, 2026Md Aminur Hossain, Parekh Valkesh, Ayush V. Patel +3Swin Frequency-Spatial Fusion Module
ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
May 10, 2026Qian Chen, Junqiao Zhao, Hongtu Zhou +4Soft Actor-Critic Sparse Rewards
The Transformer as a Polar State Estimator
May 10, 2026Peter RacioppoUnified Transformer Transformer Architectures
Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
May 10, 2026Gabriel Smithline, Chris MascioliTransformer Attention Feed-Forward Network
Uncertainty-Aware Token Importance Estimation in Spiking Transformers
May 10, 2026Wenxuan Liu, Zecheng Hao, Tong Bu +2Transformer Architectures Importance
Kinetic theory for Transformers and the lost-in-the-middle phenomenon
May 9, 2026Mitia Duerinckx, Borjan Geshkovski, Stefano RossiMean-Field Limit Causal Attention
RigidFormer: Learning Rigid Dynamics using Transformers
May 9, 2026Zhiyang Dou, Minghao Guo, Haixu Wu +3Rigid-Body Dynamics Transformer Architectures
Emergent Semantic Role Understanding in Language Models
May 9, 2026Carla Griffiths, Mirco MusolesiLarge Language Model Pretraining Sentence-Level Classification
SHIELD: Scalable Optimal Control with Certification using Duality and Convexity
May 9, 2026Hansung Kim, Siddharth H. Nair, Francesco BorrelliModel Predictive Control Optimal Control
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Transformer Layers Large Models
Character-Level Transformer for Tajik-Persian Transliteration with a Parallel Lexical Corpus
May 9, 2026Mullosharaf K. ArabovPersian Text Corpora
CATO: Charted Attention for Neural PDE Operators
May 9, 2026Chun-Wun Cheng, Sifan Wang, Carola-Bibiane Schönlieb +1Neural Operators Discretization
When and Why Grouping Attention Heads Accelerates Muon Optimization
May 9, 2026Hongtao Zhang, Wenjie Zhou, Wei Chen +1Multi-Head Self-Attention Muon
Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity
May 9, 2026Zhongjie Shi, Wenjing LiaoTransformer Architectures Approximation
Causal Dimensionality of Transformer Representations: Measurement, Scaling, and Layer Structure
May 9, 2026Nilesh Sarkar, Dawar Jyoti DekaSparse Autoencoder Features Transformer Architectures
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuQuadratic Attention Sequence Modeling
Scaling Limits of Long-Context Transformers
May 8, 2026Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2Softmax Attention Context Length
Transformers Can Implement Preconditioned Richardson Iteration for In-Context Gaussian Kernel Regression
May 8, 2026Mingsong Yan, Dongyang Li, Charles Kulick +1Kernel Ridge Regression Transformer Attention
Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization
May 8, 2026Navid Rezazadeh, Arash Gholami DavoodiGumbel-Softmax Relaxation Transformer Attention
Context-Gated Associative Retrieval: From Theory to Transformers
May 8, 2026Moulik Choraria, Argyrios Gerogiannis, Vidhata Jayaraman +2Dense Associative Memory In-Context Learning
When Attention Beats Fourier: Multi-Scale Transformers for PDE Solving on Irregular Domains
May 8, 2026Brandon Yee, Pairie Koh, Jack Rodriguez +1Partial Differential Equation Benchmarks Fourier
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Singular Value Decomposition Large Language Model Compression
Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
May 8, 2026Noboru Isobe, Daisuke Inoue, Masaaki ImaizumiTransformer Architectures Mean-Field Limit
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3Recurrent Model Transformer Layers
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3Dynamic Sparse Attention Efficient Long-Context Inference
Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization
May 8, 2026Veeti Ahvonen, Damian Heiman, Antti Kuusisto +2Transformer Encoder Transformer Architectures
Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
May 8, 2026Jin Xu, Camille Couturier, Victor Rühle +2Transformer Layers Transformer Architectures
Dynamic Mode Decomposition along Depth in Vision Transformers
May 8, 2026Nishant Suresh Aswani, Saif Eddin JabariSelf-Supervised Vision Transformers Vision Transformer
Approximation Error Upper and Lower Bounds for Hölder Class with Transformers
May 8, 2026Xin He, Yuling Jiao, Xiliang Lu +1Transformer Architectures Generalization Bounds
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Dynamic Sparse Attention Deepseek
Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
May 8, 2026Zixuan Xie, Xinyu Liu, Claire Chen +3In-Context Learning Transformer Architectures
Amortized-Precision Quantization for Early-Exit Vision Transformers
May 8, 2026Rui Fang, Hsi-Wen Chen, Ming-Syan ChenMixed-Precision Quantization Self-Supervised Vision Transformers
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
May 8, 2026Mana Sakai, Masaaki ImaizumiGeneralization Bounds Transformer Architectures
TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts
May 8, 2026Jeimin Jeon, Hyunju Lee, Bumsub HamSelf-Supervised Vision Transformers Neural Architecture Search
Attention Transfer Is Not Universally Effective for Vision Transformers
May 8, 2026Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4Self-Supervised Vision Transformers Vision Transformer
SAFformer:Improving Spiking Transformer via Active Predictive Filtering
May 8, 2026Zequan Xie, Weiming Zeng, Yunhua Chen +3Spiking Neural Networks Hybrid Transformer
LoopQ: Quantization for Recursive Transformers
May 8, 2026Rui Fang, Hsi-Wen Chen, Ming-Syan ChenWeight-Only Quantization Quantization
Adaptive Memory Decay for Log-Linear Attention
May 7, 2026Yaxita Amin, Helen Zichen Li, Mengfan Zhang +1Quadratic Attention Multilayer Perceptrons
Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
May 7, 2026Ernie Chu, Vishal M. PatelPre-Trained Video Diffusion Models Few-Step Generation
ProtSent: Protein Sentence Transformers
May 7, 2026Dan Ofer, Oriel Perets, Michal Linial +1Protein Data Bank Global Embeddings
UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
May 7, 2026Minbin Huang, Han Shi, Chuanyang Zheng +5Mixture-of-Experts Models Experts
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehDynamic Sparse Attention Key-Value Cache
Transformers Efficiently Perform In-Context Logistic Regression via Normalized Gradient Descent
May 7, 2026Chenyang Zhang, Yuan CaoIn-Context Learning Transformer Architectures
Long Context Pre-Training with Lighthouse Attention
May 7, 2026Bowen Peng, Subho Ghosh, Jeffrey QuesnelleSoftmax Attention Causal Reasoning
Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models
May 7, 2026Amir Rezaei Balef, Mykhailo Koshil, Katharina EggenspergerTabular Foundation Models Transformer Layers
Cubit: Token Mixer with Kernel Ridge Regression
May 7, 2026Chuanyang Zheng, Jiankai Sun, Yihang Gao +6Transformer Attention Sequence Modeling

May 11, 2026 · cs.LGJ/K move · Enter open · S save

Simply Stabilizing the Loop via Fully Looped Transformer

Rao Fu, Zixuan Yang, Jiankun Zhang, Jing Ma+3

Hong Kong Baptist University · Jilin University

Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading additional computation for improved performance without increasing parameter count or context length. Because the number of loop iterations can be adjusted at inference, it also provides a natural mechanism for balancing performance and test-time compute. However, Looped Transformer still suffers from training instability when the number of loop iterations increases. Our analysis reveals that this instability stems from two sources: gradient oscillation and residual explosion. To address these two problems, we propose the Fully Looped Transformer, which introduces two parameter-free modifications: (1) Fully Looped Architecture, which distributes inter-loop signals across all layers to mitigate residual explosion; (2) Attention Injection, which reuses the existing attention block to suppress gradient oscillation. These modifications stabilize training dynamics, enabling the Fully Looped Transformer to be trained stably up to 12 loop iterations, whereas other baseline looped models collapse in this regime. In milder settings where Looped Transformer does not collapse, Fully Looped Transformer still improves average downstream-task performance by up to 13.2%. Overall, our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets by varying loop iterations at inference.

Transformer Architectures Batch Normalization Loop Modifications

Open paper Ask about it PDF