Recurrent Transformers

Follow this topic Ask about this topic

Recurrent Transformers study models that repeatedly apply shared Transformer blocks, pass hidden states or memory between tokens, or combine recurrence with attention. This can add test-time computation and support long-context and state-tracking tasks while trading parameters or cache memory for iterative work. Common approaches include depth-looped Transformers, recurrent and associative memory, hybrid attention–state-space layers, and adaptive halting, with stability controls and distillation used to make recurrence practical. Results span language modeling and retrieval, algorithmic tests such as Sudoku, ARC, and parity, and vision and audio tasks; a central tension is that extra loops can improve reasoning or length generalization but can also destabilize states, lose useful information, or add latency and memory costs.

Latest papers

183 papers53 papers in the last four weeks, up 112% on the four weeks before. 0.3% of all new papers.

Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning
May 6, 2026William T. Redman, Erik C. Johnson, Brian RobinsonCompositional Reasoning Continual Learning
Johns Hopkins Applied Physics Lab
RD-ViT: Recurrent-Depth Vision Transformer for Semantic Segmentation with Reduced Data Dependence Extending the Recurrent-Depth Transformer Architecture to Dense Prediction
May 5, 2026Renjie HeMedical Image Analysis Recurrent Transformers
Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX, USA
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
Apr 30, 2026Thea AvissRecurrent Neural Networks Recurrent Transformers
Fifth Dimension
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
Apr 24, 2026Hector Borobia, Elies Seguí-Mas, Guillermina Tormo-CarbóLow-Rank Adaptation Recurrent Transformers
VRAIN – Valencian Research Institute for Artificial Intelligence, Universitat Politècnica de València, Valencia, Spain · Department of Economics and Social Sciences, Universitat Politècnica de València, Valencia, Spain · Department of Business Organisation, Universitat Politècnica de València, Valencia, Spain
Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
Apr 23, 2026Grigory SapunovMemory-Augmented Neural Networks Sudoku Solving
Intento
Hyperloop Transformers
Apr 23, 2026Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon KimLLM Compression Hyper-Connections
Massachusetts Institute of Technology
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language Modeling Self-Attention
Harvard University
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Apr 22, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisTruncated Backpropagation Through Time Language Model Scaling Laws
Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences
Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language Modeling Recurrent Neural Networks
1MIT · 2Liquid AI
One Step Forward and K Steps Back: Better Reasoning with Denoising Recursion Models
Apr 20, 2026Chris Cameron, Wangzheng Wang, Nikita Ivanov +3Abstract Reasoning Recurrent Transformers
Huawei Technologies · University of Waterloo · University of Toronto
Sessa: Selective State Space Attention
Apr 20, 2026Liubomyr HorbatkoAttention Mechanisms Self-Attention
The Topological Trouble With Transformers
Apr 18, 2026Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne LiuTransformer FFNs Recurrent Neural Networks
Google DeepMind
Stability and Generalization in Looped Transformers
Apr 16, 2026Asher LabovichNeural Network Generalization Dynamical Systems
Hierarchical vs. Flat Iteration in Shared-Weight Transformers
Apr 15, 2026Sang-Il HanTransformer Recurrent Transformers
Korea University of Technology and Education · Biomonster Co.
Parcae: Scaling Laws For Stable Looped Language Models
Apr 14, 2026Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick +1Language Model Scaling Laws Recursive Language Models
University of California, San Diego · Together AI
A Mechanistic Analysis of Looped Reasoning Language Models
Apr 13, 2026Hugh Blayney, Álvaro Arroyo, Johan Obando-Ceron +4Recurrent Transformers LLM Interpretability
University of Oxford · Mila – Quebec AI Institute · Universit´e de Montréal +1
TAPNext++: What's Next for Tracking Any Point (TAP)?
Apr 12, 2026Sebastian Jung, Artem Zholus, Martin Sundermeyer +6Point Tracking Recurrent Transformers
German Aerospace Center (DLR) · Karlsruhe Institute of Technology (KIT) · Mila - Quebec AI Institute +7
Relational Preference Encoding in Looped Transformer Internal States
Apr 10, 2026Jan KirinPairwise Preference Learning Reward Modeling
ELT: Elastic Looped Transformers for Visual Generation
Apr 10, 2026Sahil Goyal, Swayam Agrawal, Gautham Govind Anil +3Image Generation Transformer
Loom: A Scalable Analytical Neural Computer Architecture
Apr 9, 2026Mehmet Kerem TurkcanNeural Algorithmic Reasoning Recurrent Transformers
Columbia University
Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Apr 9, 2026Harsh Kohli, Srinivasan Parthasarathy, Huan Sun +1Position Extrapolation Implicit Reasoning in Language Models
The Ohio State University
SE-Enhanced ViT and BiLSTM-Based Intrusion Detection for Secure IIoT and IoMT Environments
Apr 6, 2026Afrah Gueriani, Hamza Kheddar, Ahmed Cherif Mazari +2Hybrid Transformer Architectures IoT Intrusion Detection
LSEA Lab., Faculty of Technology, University of MEDEA, Medea 26000, Algeria · LSEA Lab, Faculty of Science, University of MEDEA, Medea 26000, Algeria · Department of Computer Engineering, Gazi University, Ankara, Turkey +1
BiTA: Bidirectional Gated Recurrent Unit-Transformer Aggregator in a Temporal Graph Network Framework for Alert Prediction in Computer Networks
Apr 3, 2026Zahra Makki Nayeri, Mohsen RezvaniHybrid Transformer Architectures Temporal GNNs
Faculty of Computer Engineering, Shahrood University of Technology, Shahrood, Iran
Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation
Apr 2, 2026Jaber Jaber, Osama JaberRecurrent Transformers Decoder-Only Language Models
RightNow AI
Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence
Apr 1, 2026Liu XiaoRecurrent Transformers Autoregressive Language Modeling
Liu Xiao
S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
Apr 1, 2026Jack YoungParameter-Efficient Fine-Tuning Recurrent Transformers
Indiana University
LA-Sign: Looped Transformers with Geometry-aware Alignment for Skeleton-based Sign Language Recognition
Mar 30, 2026Muxin Pu, Mei Kuan Lim, Chun Yong Chong +1Sign Language Recognition Recurrent Transformers
School of Information Technology, Monash University · S-Lab, Nanyang Technological University
Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping
Mar 25, 2026Yao Chen, Yilong Chen, Yinqi Yang +9Adaptive Computation Recurrent Transformers
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
Thinking Deeper, Not Longer: Memory-Efficient Test-Time Reasoning with Depth-Recurrent Transformers for Compositional Generalization
Mar 23, 2026Hung-Hsuan ChenEfficient Transformer Inference Recurrent Transformers
Computer Science and Information Engineering National Central University Taoyuan, Taiwan
Hybrid Associative Memories
Mar 20, 2026Leon Lufkin, Tomás Figliolia, Beren Millidge +1Associative Memory Recurrent Transformers
1Zyphra

May 6, 2026J/K move · Enter open · S save

Shortcut Solutions Learned by Transformers Impair Continual Compositional Reasoning

William T. Redman, Erik C. Johnson, Brian Robinson

Johns Hopkins Applied Physics Lab

Compositional Reasoning Continual Learning Recurrent Transformers Shortcut Learning LLM Mathematical Reasoning

Open paper AskPDF arXiv

Identifying and exploiting common features across domains is at the heart of the human ability to make analogies, and is believed to be crucial for the ability to continually learn. To do this successfully, general and flexible computational strategies must be developed. While the extent to which Transformer neural network models can perform compositional reasoning has been the subject of intensive recent investigation, little work has been done to systematically understand how well these models can leverage their representations to learn new, related experiences. To address this gap, we expand the previously developed Learning Equality and Group Operations (LEGO) framework to a continual learning (CL) setting ("continual LEGO"). Using this continual LEGO experimental paradigm, we study the capability of feedforward and recurrent Transformer models to perform CL. We find that BERT, a canonical feedforward Transformer model, learns shortcut solutions that limits its ability to generalize and prevents strong forward transfer to new experiences. In contrast, we find evidence supporting the hypothesis that ALBERT, a recurrent version of BERT, learns a For loop-esque solution, which leads to better CL performance. When applying BERT and ALBERT models to a CL setting that requires composition across experiences, we find that both model families fail. Our investigation suggests that ALBERT models can have their performance drop rescued by use of training strategies that combine data across experiences, but this is not true for BERT models, where a detrimental shortcut solution becomes entrenched with initial training. Our results demonstrate that the recurrent ALBERT model may have an inductive bias better suited for CL and motivate future investigation of the interplay between Transformer architecture and computational solutions that emerge in modern models and tasks.