Recurrent Transformers

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, against 2 the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 115

All topics

Simply Stabilizing the Loop via Fully Looped Transformer
May 11, 2026Rao Fu, Zixuan Yang, Jiankun Zhang +4Test-Time Scaling Recurrent Transformers
Hong Kong Baptist University · Jilin University
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahTransformer Inference Self-Attention
Featherless AI · Eleuther AI
LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models
May 10, 2026Taekhyun Park, Yongjae Lee, Dohee Kim +1Recurrent Transformers LLM Post-Training
Department of Data Science Pusan National University Busan, Republic of Korea · Department of Industrial Engineering Pusan National University Busan, Republic of Korea · Department of Artificial Intelligence Engineering Changwon National University Changwon, Republic of Korea
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
Lattice Deduction Transformers
May 9, 2026Liam Davis, Leopold Haller, Alberto Alfarano +1Recurrent Transformers Neuro-Symbolic Reasoning
Amherst College · Axiom · Barnard College, Columbia University
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3KV-Cache Management Memory-Efficient Inference
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
LoopQ: Quantization for Recursive Transformers
May 8, 2026Rui Fang, Hsi-Wen Chen, Ming-Syan ChenRecurrent Transformers Language Modeling
National Taiwan University
Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning
May 6, 2026William T. Redman, Erik C. Johnson, Brian RobinsonCompositional Reasoning Continual Learning
Johns Hopkins Applied Physics Lab
RD-ViT: Recurrent-Depth Vision Transformer for Semantic Segmentation with Reduced Data Dependence Extending the Recurrent-Depth Transformer Architecture to Dense Prediction
May 5, 2026Renjie HeMedical Image Analysis Recurrent Transformers
Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
Apr 30, 2026Thea AvissRecurrent Neural Networks Recurrent Transformers
Fifth Dimension
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Apr 24, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóLow-Rank Adaptation Recurrent Transformers
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
Apr 23, 2026Grigory SapunovMemory-Augmented Neural Networks Recurrent Transformers
Intento
Hyperloop Transformers
Apr 23, 2026Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon KimLLM Compression Hyper-Connections
Massachusetts Institute of Technology
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Efficient Transformer Inference Self-Attention
Harvard University
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Apr 22, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisLanguage Model Scaling Laws Recurrent Neural Networks
Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI
One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models
Apr 20, 2026Chris Cameron, Wangzheng Wang, Nikita Ivanov +3Recurrent Transformers Diffusion Models
Huawei Technologies · University of Waterloo · University of Toronto
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoSelf-Attention Long-Context Language Modeling
The Topological Trouble With Transformers
Apr 18, 2026Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne LiuTransformer FFNs Recurrent Neural Networks
Google DeepMind
Stability and Generalization in Looped Transformers
Apr 16, 2026Asher LabovichNeural Network Generalization Dynamical Systems
Relational Preference Encoding in Looped Transformer Internal States
Apr 10, 2026Jan KirinPairwise Preference Learning Reward Modeling
Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
Mar 23, 2026Hung-Hsuan ChenEfficient Transformer Inference Recurrent Transformers
Computer Science and Information Engineering National Central University Taoyuan, Taiwan
On the Residual Scaling of Looped Transformers: Stability and Transferability
Date pendingShaowen Wang, Bingrui Li, Ge Zhang +3Residual Learning Hyperparameter Transfer
Tsinghua University · ByteDance Seed · M-A-P
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Date pendingShaowen Wang, Ge Zhang, Kairong Luo +6Language Model Scaling Laws Transformer Attention
Tsinghua University · ByteDance Seed · M-A-P

May 11, 2026, cs.LGJ/K move · Enter open · S save

Simply Stabilizing the Loop via Fully Looped Transformer

Rao Fu, Zixuan Yang, Jiankun Zhang, Jing Ma+3

Hong Kong Baptist University · Jilin University

Test-Time Scaling Recurrent Transformers Neural Network Training Dynamics

Open paper AskPDF arXiv

Scaling model performance typically requires increasing model size. Looped Transformer offers a compelling alternative by iteratively reusing the same Transformer blocks, trading additional computation for improved performance without increasing parameter count or context length. Because the number of loop iterations can be adjusted at inference, it also provides a natural mechanism for balancing performance and test-time compute. However, Looped Transformer still suffers from training instability when the number of loop iterations increases. Our analysis reveals that this instability stems from two sources: gradient oscillation and residual explosion. To address these two problems, we propose the Fully Looped Transformer, which introduces two parameter-free modifications: (1) Fully Looped Architecture, which distributes inter-loop signals across all layers to mitigate residual explosion; (2) Attention Injection, which reuses the existing attention block to suppress gradient oscillation. These modifications stabilize training dynamics, enabling the Fully Looped Transformer to be trained stably up to 12 loop iterations, whereas other baseline looped models collapse in this regime. In milder settings where Looped Transformer does not collapse, Fully Looped Transformer still improves average downstream-task performance by up to 13.2%. Overall, our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets by varying loop iterations at inference.

Momentum

Related topics

Latest papers 115