Recurrent Transformers

Ask about this topic

Overview

Recurrent Transformers study models that repeatedly apply shared Transformer blocks, pass hidden states or memory between tokens, or combine recurrence with attention. This can add test-time computation and support long-context and state-tracking tasks while trading parameters or cache memory for iterative work. Common approaches include depth-looped Transformers, recurrent and associative memory, hybrid attention–state-space layers, and adaptive halting, with stability controls and distillation used to make recurrence practical. Results span language modeling and retrieval, algorithmic tests such as Sudoku, ARC, and parity, and vision and audio tasks; a central tension is that extra loops can improve reasoning or length generalization but can also destabilize states, lose useful information, or add latency and memory costs.

Latest papers

CardsList

182 papers53 papers in the last four weeks, up 112% on the four weeks before. 0.3% of all new papers.

  1. CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening

    Oct 28, 2026Wenyu Du, Stephen ChungEfficient Language Model InferenceRecurrent Transformers

    DualverseAI

  2. Depth through Recurrence: Towards Ultra-Efficient On-Device ASR

    Oct 27, 2026Chen Feng, Tianyi Xu, Ramchalam Kinattinkara Ramakrishnan +5Automatic Speech RecognitionModel Compression

    Qualcomm AI Research † · McGill University

  3. LoopFold: Parallelizing Recurrent Depth in Looped Transformers

    Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Recurrent TransformersEfficient Transformer Inference

    Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

  4. StateLoop: Persistent State for Looped Transformers

    Oct 26, 2026Feichen Song, Boyi Zeng, He Li +7Persistent Memory for Language ModelsLanguage Modeling

    Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

  5. Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks

    Oct 25, 2026Szymon Mazurek, Alberto Presta, Michał ByraVision-Language ModelsRecurrent Transformers

    Samsung AI Center Warsaw, Poland · AGH University of Krakow, Poland · ACC Cyfronet AGH, Poland +1

  6. LSTM-Style Looped Transformers with Shared KV Caches

    Oct 23, 2026Andreas KirschRecurrent TransformersKV-Cache Management

  7. Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping

    Oct 23, 2026Yudong Zhang, Wei Ding, Kaili Zheng +2Efficient Language Model InferenceLLM Inference Acceleration

    Tsinghua University · Baidu Inc.

  8. How Much Gradient Does Truncated Backprop Lose in Looped Transformers?

    Oct 20, 2026Aur Shalev MerinTruncated Backpropagation Through TimeBackpropagation

    Independent researcher

  9. Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

    Oct 20, 2026Cheng Liang, Tengfei Zhang, Wenjie DU +6Recurrent Language ModelsRecurrent Transformers

    Shanghai Jiaotong University · University of Science and Technology of China · Nanyang Technological University +1

  10. HistoLoop: Iterative Contextual Inference with a Looped Transformer for Histology-Based Spatial Gene Expression Prediction

    Oct 20, 2026Ziang Liu, Jie Liu, Shuai Li +2Spatial Gene Expression PredictionGene Expression Prediction from Histopathology Images

    Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology · College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China

  11. Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

    Oct 19, 2026Xuyuan Xiong, Siyu Chen, Zihan Shao +5Truncated Backpropagation Through TimeEfficient Language Model Training

    MMLab, The University of Hong Kong · Yale University

  12. Internalizing Chain-of-thought into Looped Transformers

    Oct 19, 2026Kehang Zhou, Pinyan LuRecurrent TransformersLatent Reasoning

    Institute for Theoretical Computer Science Shanghai University of Finance and Economics

  13. P2N: Reusing Deep Representations for Greater Effective Depth

    Oct 18, 2026Yiqin Hao, Boyi Zeng, Zitong Wang +7Efficient Language Model InferenceRecurrent Transformers

    LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai AI Laboratory

  14. UpLoop: Upcycling Large Language Models with Gated Loop

    Oct 18, 2026Tieyuan Chen, Huabin Liu, Yingtao Zhang +4Recurrent Language ModelsRecurrent Transformers

    Inclusion AI, Ant Group · Shanghai Jiao Tong University · Tsinghua University

  15. Block Recursive Transformers for Structured Parameter Sharing

    Oct 18, 2026Josep Marques, Hyunyoung A. Cho, Youngeun Kim +4TransformerRecurrent Transformers

    Sungkyunkwan University · Korea University

  16. Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks

    Oct 16, 2026Magnus Sesodia, Juil Sock, Philip Torr +1Recurrent TransformersAlgorithmic Reasoning

    University of Oxford · BBC R&D

  17. Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design

    Oct 16, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Recurrent TransformersRecursive Language Models

    Institute of Foundation Models · CMU · USC

  18. Causal Latent Workspaces for Looped Transformers

    Oct 16, 2026Ilia ZaznovAutoregressive Language ModelingMemory-Augmented Language Models

  19. LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models

    Oct 15, 2026Zixuan Liu, Nan Xue, Rui Bao +4Speculative DecodingLLM Inference Acceleration

    Shanghai Jiao Tong University · Tongji University

  20. Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers

    Oct 14, 2026Peilong Zhou, Hongtao Zhang, Yulong Huang +3Recurrent TransformersNeural Algorithmic Reasoning

    SKLP, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences +1

  21. Test-Time Forecasting: Recurrent Depth Enables In-Context Dynamics Prediction

    Oct 14, 2026Jeffrey B. Lai, John Quinn, Anthony Bao +1Dynamical SystemsIn-Context Learning

    Oden Institute University of Texas at Austin · Department of Physics University of Texas at Austin

  22. Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

    Oct 13, 2026Sarah Pan, Ali BehrouzLong-Context Language Model InferenceMemory-Augmented Language Models

    Massachusetts Institute of Technology · Cornell University

  23. Recurrent In-Context Learning for Tabular Foundation Models

    Oct 10, 2026Tejas Vaidhya, Gaurav Sahu, Irina RishTabular Foundation ModelsTabular ICL

    Mila – Quebec AI Institute · Nolano AI · Unified Sciences

  24. Learning to Extrapolate in Depth with Looped Transformers

    Oct 10, 2026Huitao Yang, Jianhao Huang, Zixuan Wang +1Position ExtrapolationRecurrent Transformers

    University of California, Los Angeles · Princeton University

  25. Adaptive-Depth Is a Finishing Stage

    Oct 9, 2026Andrei Piterbarg, Sean McLeish, Dhruv Pai +1Efficient Language Model TrainingAdaptive Learning

    University of Pennsylvania · University of Maryland

  26. One Model to Loop Them All: Elastic Latent Recurrence for Language Models

    Oct 8, 2026Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano +3Language ModelingRecurrent Language Models

    NVIDIA

  27. TraceRelay: Attention-Aligned Recurrence over Rolling Traces

    Oct 8, 2026Sungwoo Goo, Hwi-yeol Yun, Sangkeun JungSequence ModelingPersistent Memory for Language Models

    College of Pharmacy, Chungnam National University · Department of Computer Science & Engineering, Chungnam National University

  28. Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

    Oct 8, 2026Pengxiang Li, Dilxat Muhtar, Di He +3Recurrent TransformersAttention Residual Connections

    The Hong Kong Polytechnic University · Independent Researcher · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences +6

  29. From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers

    Oct 8, 2026Gerard Grau García, Arnau Padrés Masdemont, Niccolò Grillo +3Efficient Language Model ReasoningContinuous Latent Reasoning

    Qualcomm AI Research · MIT · University of Cambridge +1

  30. ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

    Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache QuantizationEfficient Transformer Inference

    KAIST · Yonsei University · Seoul National University