Recurrent Transformers

Follow this topic Ask about this topic

Recurrent Transformers study models that repeatedly apply shared Transformer blocks, pass hidden states or memory between tokens, or combine recurrence with attention. This can add test-time computation and support long-context and state-tracking tasks while trading parameters or cache memory for iterative work. Common approaches include depth-looped Transformers, recurrent and associative memory, hybrid attention–state-space layers, and adaptive halting, with stability controls and distillation used to make recurrence practical. Results span language modeling and retrieval, algorithmic tests such as Sudoku, ARC, and parity, and vision and audio tasks; a central tension is that extra loops can improve reasoning or length generalization but can also destabilize states, lose useful information, or add latency and memory costs.

Latest papers

183 papers53 papers in the last four weeks, up 112% on the four weeks before. 0.3% of all new papers.

Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning
Jun 11, 2026Jiayu Yang, Chao Chen, Shengen Wu +6RL for Language Model Reasoning Mechanistic Interpretability
HKUST(GZ) · University of Cambridge · NTU +2
$μ$ VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models
Jun 10, 2026Egor Cherepanov, Nikita Kachaev, Daniil Zelezetsky +6Memory-Augmented VLMs Recurrent Transformers
CogAI Lab, Moscow, Russia · MIRAI, Moscow, Russia
Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers
Jun 3, 2026Yacouba Kaloga, Shashi Kumar, Shakeel A. Sheikh +3Inference-Time Scaling Test-Time Scaling
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic
CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability
May 31, 2026Chad A. CappsRecurrent Neural Networks Autoregressive Language Modeling
Independent Researcher
Looped Transformers with Layer Normalization Provably Learn the Power Method
May 30, 2026Lyumin Wu, Chenyang Zhang, Yuan CaoRecurrent Transformers Layer Normalization
School of Computing & Data Science, The University of Hong Kong
Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation
May 29, 2026Haozhou ZhangCoT Reasoning Persistent Memory for Language Models
Department of Mathematics and Statistics University of Idaho
Déjà View: Looping Transformers for Multi-View 3D Reconstruction
May 28, 2026Alessandro Burzio, Tobias Fischer, Sven Elflein +9Multi-View 3D Reconstruction Recurrent Transformers
NVIDIA · University of Modena and Reggio Emilia, AImageLab · ETH Zürich +1
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
May 28, 2026Markus Frey, Behzad Shomali, Joachim Koehler +1Recurrent Transformers Language Model Scaling
Lamarr Institute · Fraunhofer IAIS · University of Bonn
Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers
May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural Networks Memory-Augmented Language Models
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model Pretraining Transformer Inference
Microsoft · University of Washington · University of Wisconsin-Madison
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
May 26, 2026Xiao-Wen Yang, Ziyu Han, Xi-Hua Zhang +4Latent Dynamics Modeling Test-Time Scaling
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Nanjing, China
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
May 25, 2026Sangyun Lee, Sean McLeish, Tom Goldstein +1Memory-Augmented Language Models Long-Context Language Model Inference
Carnegie Mellon University · University of Maryland
Triplet-Block Diffusion RWKV
May 25, 2026Ke Lin, Yiyang Luo, Zhaolong Su +2Diffusion Language Model Inference Recurrent Transformers
William & Mary · HKUST · Cornell
Training-Free Looped Transformers
May 22, 2026Lizhang Chen, Jonathan Li, Chen Liang +2Transformer Inference Recurrent Transformers
University of Texas at Austin
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
May 18, 2026Jucheng Shen, Barbara Su, Anastasios KyrillidisTransformer Interpretability Recurrent Transformers
Rice University
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferGated Attention Recurrent Transformers
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
Solve the Loop: Attractor Models for Language and Reasoning
May 12, 2026Jacob Fein-Ashley, Paria RashidinejadLanguage Model Pretraining Recurrent Transformers
University of Southern California
Recurrent Transformer-Based Near- and Far-Field THz Wideband Channel Estimation for UM-MIMO
May 12, 2026Dmitry Artemasov, Alexander Shmatok, Kirill Andreev +3Recurrent Transformers Wireless Communications
Center for Next Generation Wireless and IoT, Skolkovo Institute of Science and Technology, Moscow, Russia · Department of IEOR, Indian Institute of Technology Bombay, India · Faculty of Computer and Engineering Sciences, Innopolis University, Innopolis, Russia
Latent Chain-of-Thought Improves Structured-Data Transformers
May 11, 2026Carson Dudley, Samet OymakCoT Reasoning Time Series Forecasting
Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
May 11, 2026Michal Byra, Pawel Olszowiec, Grzegorz Stefanski +2Vision Transformer Recurrent Transformers
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
Simply Stabilizing the Loop via Fully Looped Transformer
May 11, 2026Rao Fu, Zixuan Yang, Jiankun Zhang +4Test-Time Scaling Recurrent Transformers
Hong Kong Baptist University · Jilin University
Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
May 11, 2026Daniel Goldstein, Eugene CheahTransformer Inference Self-Attention
Featherless AI · Eleuther AI
LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models
May 10, 2026Taekhyun Park, Yongjae Lee, Dohee Kim +1Recurrent Transformers LLM Post-Training
Department of Data Science Pusan National University Busan, Republic of Korea · Department of Industrial Engineering Pusan National University Busan, Republic of Korea · Department of Artificial Intelligence Engineering Changwon National University Changwon, Republic of Korea
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
Lattice Deduction Transformers
May 9, 2026Liam Davis, Leopold Haller, Alberto Alfarano +1Sudoku Solving Abstract Interpretation
Amherst College · Axiom · Barnard College, Columbia University
Kaczmarz Linear Attention
May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language Modeling Recurrent Transformers
School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3KV-Cache Management Memory-Efficient Inference
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
LoopQ: Quantization for Looped Language Models
May 8, 2026Rui Fang, Hsi-Wen Chen, Chia-Ming Cheng +1Recurrent Transformers Language Modeling
National Taiwan University

Jun 11, 2026J/K move · Enter open · S save

Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning

Jiayu Yang, Chao Chen, Shengen Wu, Yinhong Liu+5

HKUST(GZ) · University of Cambridge · NTU+2

RL for Language Model Reasoning Mechanistic Interpretability Recurrent Transformers RL Post-Training Latent CoT Reasoning

Open paper AskPDF arXiv

Latent chain-of-thought compresses reasoning through continuous hidden-state recurrence, but existing approaches are difficult to optimize with standard on-policy reinforcement learning (RL) and hard to analyze mechanistically. We introduce SWITCH, a switchable latent reasoning framework that addresses both challenges with two discrete boundary tokens: <swi> enters latent reasoning and </swi> exits it. These discrete anchors make policy ratios well-defined for standard on-policy RL while providing natural intervention points for mechanistic analysis. We train SWITCH with a visible-to-latent curriculum and Switch-GRPO, which propagates gradients through recurrent latent computation. SWITCH consistently outperforms prior hidden-state-recurrence approaches at similar model scales. Mechanistic analysis reveals that (i) <swi> implements a localized, learned switching policy rather than a stylistic artifact; (ii) the latent step performs problem-specific, causally important computation; and (iii) this computation is concentrated in a single hidden-state transition upon entering latent mode. Together, these results show that hidden-state-recurrence latent reasoning can be both effectively optimized with on-policy RL and directly analyzed mechanistically, enabling us to study how RL reshapes internal reasoning computation. Code is available at https://github.com/LARK-AI-Lab/SWITCH.