Recurrent Transformers

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, against 2 the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 115

All topics

Compressing History into Memory: Distilling Transformers into Recurrent Transformers
Jun 19, 2026Philippe Weinzaepfel, Christian Wolf, Mert Bülent Sariyildiz +2Robotics Recurrent Transformers
Naver Labs Europe, Meylan, France
Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memory
Jun 17, 2026Yanan NiuMemory-Augmented Neural Networks Knowledge Editing
Kuaishou Technology
A Hybrid LSTM--Vision Transformer Architecture for Predicting HRRR Forecast Errors
Jun 17, 2026David Aaron Evans, Jay C. Rothenberger, Kara J. Sulia +2Multivariate Time Series Forecasting Recurrent Transformers
Atmospheric Sciences Research Center, University at Albany, SUNY, Albany, NY, USA · University of Oklahoma, Norman, OK, USA · State Weather Risk Communication Center, University at Albany, SUNY, Albany, NY, USA
Looped World Models
Jun 16, 2026Hongyuan Adam Lu, Z. L. Victor Wei, Qun Zhang +28World Model Learning Recurrent Transformers
FaceMind Research Asia
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Jun 16, 2026Sajad Movahedi, Vera Milovanović, Shlomo Libo Feigin +5Recurrent Transformers Algorithmic Reasoning
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center · ETH Zurich · Swiss Institute of Bioinformatics +2
LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling
Jun 16, 2026Jian Yang, Shawn Guo, Wei Zhang +16Efficient Transformer Inference Test-Time Scaling
1Beihang University · 2IQuest Research · 4Renming University of China +1
The Reservoir Attention Network: Cross-Pass State in Pretrained Transformers via Content-Addressable Reservoir Injection
Jun 14, 2026Emma LeonhartMemory-Augmented Neural Networks Persistent Memory for Language Models
Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models
Jun 12, 2026Rituraj Sharma, Tu VuRecurrent Neural Networks Recurrent Transformers
Virginia Tech
Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning
Jun 11, 2026Jiayu Yang, Chao Chen, Shengen Wu +6RL for Language Model Reasoning Mechanistic Interpretability
HKUST(GZ) · University of Cambridge · NTU +2
$μ$ VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models
Jun 10, 2026Egor Cherepanov, Nikita Kachaev, Daniil Zelezetsky +6Memory-Augmented VLMs Recurrent Transformers
CogAI Lab, Moscow, Russia · MIRAI, Moscow, Russia
Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers
Jun 3, 2026Yacouba Kaloga, Shashi Kumar, Shakeel A. Sheikh +3Inference-Time Scaling Test-Time Scaling
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic
CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability
May 31, 2026Chad A. CappsRecurrent Neural Networks Autoregressive Language Modeling
Independent Researcher
Looped Transformers with Layer Normalization Provably Learn the Power Method
May 30, 2026Lyumin Wu, Chenyang Zhang, Yuan CaoRecurrent Transformers Layer Normalization
School of Computing & Data Science, The University of Hong Kong
Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation
May 29, 2026Haozhou ZhangCoT Reasoning Persistent Memory for Language Models
Department of Mathematics and Statistics University of Idaho
Déjà View: Looping Transformers for Multi-View 3D Reconstruction
May 28, 2026Alessandro Burzio, Tobias Fischer, Sven Elflein +9Multi-View 3D Reconstruction Recurrent Transformers
NVIDIA · University of Modena and Reggio Emilia, AImageLab · ETH Zürich +1
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
May 28, 2026Markus Frey, Behzad Shomali, Joachim Koehler +1Recurrent Transformers Language Model Scaling
Lamarr Institute · Fraunhofer IAIS · University of Bonn
Tensor Memory: Fixed-Size Recurrent State for Long-Horizon Transformers
May 26, 2026Kabir Swain, Sijie Han, Daniel Karl I. Weidele +2Memory-Augmented Neural Networks Memory-Augmented Language Models
Massachusetts Institute of Technology, Cambridge, MA, USA · University of Toronto, Toronto, Canada · IBM Research, Cambridge, MA, USA
Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
May 26, 2026Zeyi Huang, Xuehai He, LiLiang Ren +8Language Model Pretraining Transformer Inference
Microsoft · University of Washington · University of Wisconsin-Madison
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models
May 26, 2026Xiao-Wen Yang, Ziyu Han, Xi-Hua Zhang +4Latent Dynamics Modeling Test-Time Scaling
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Nanjing, China
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
May 25, 2026Sangyun Lee, Sean McLeish, Tom Goldstein +1Memory-Augmented Language Models Long-Context Language Model Inference
Carnegie Mellon University · University of Maryland
Triplet-Block Diffusion RWKV
May 25, 2026Ke Lin, Yiyang Luo, Zhaolong Su +2Diffusion Language Model Inference Recurrent Transformers
William & Mary · HKUST · Cornell
Training-Free Looped Transformers
May 22, 2026Lizhang Chen, Jonathan Li, Chen Liang +2Transformer Inference Recurrent Transformers
University of Texas at Austin
Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
May 21, 2026Ali Hatamizadeh, Yejin Choi, Jan KautzLong-Context Language Modeling Recurrent Transformers
NVIDIA
LT2: Linear-Time Looped Transformers
May 20, 2026Chunyuan Deng, Yizhe Zhang, Rui-Jie Zhu +4Efficient Transformer Inference Recurrent Transformers
1Rice University · 2Apple · 3UC Santa Cruz +1
One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
May 18, 2026Jucheng Shen, Barbara Su, Anastasios KyrillidisTransformer Interpretability Recurrent Transformers
Rice University
Provably Shorter Scratchpads in Hybrid DeltaNet-Attention Decoders
May 15, 2026Tomasz SteiferGated Attention Recurrent Transformers
Centre for Credible AI Warsaw University of Technology Rektorska 4, Warszawa 00-614, Poland
Solve the Loop: Attractor Models for Language and Reasoning
May 12, 2026Jacob Fein-Ashley, Paria RashidinejadLanguage Model Pretraining Recurrent Transformers
University of Southern California
Recurrent Transformer-Based Near- and Far-Field THz Wideband Channel Estimation for UM-MIMO
May 12, 2026Dmitry Artemasov, Alexander Shmatok, Kirill Andreev +3Recurrent Transformers Wireless Communications
Center for Next Generation Wireless and IoT, Skolkovo Institute of Science and Technology, Moscow, Russia · Department of IEOR, Indian Institute of Technology Bombay, India · Faculty of Computer and Engineering Sciences, Innopolis University, Innopolis, Russia
Latent Chain-of-Thought Improves Structured-Data Transformers
May 11, 2026Carson Dudley, Samet OymakCoT Reasoning Time Series Forecasting
Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
May 11, 2026Michal Byra, Pawel Olszowiec, Grzegorz Stefanski +2Vision Transformer Recurrent Transformers
Samsung AI Center, Warsaw, Poland · Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland

Jun 19, 2026, cs.CVJ/K move · Enter open · S save

Compressing History into Memory: Distilling Transformers into Recurrent Transformers

Philippe Weinzaepfel, Christian Wolf, Mert Bülent Sariyildiz, Guillaume Bono+1

Naver Labs Europe, Meylan, France

Robotics Recurrent Transformers Knowledge Distillation

Open paper AskPDF arXiv

Transformers are AI's workhorse but their computational cost becomes prohibitive when processing long sequences. We target long-horizon streaming vision and robotics applications, where it is particularly impractical to store and maintain a history of observations. Recurrent Transformers address this limitation by maintaining fixed-size memory but their performance lags behind that of transformers operating over the full observation history. We argue that this gap does not stem from architectural limitations, but from differences in how these models learn to compress past information. Without access to an observation history, recurrent models must explicitly decide what to retain in memory at each step, a significantly harder learning problem. In this work, we propose a distillation approach that transfers the compression strategy of a classical full-history transformer to a recurrent variant. We enable this by designing a teacher model that explicitly compresses its observation history into a fixed-size bottleneck representation and directly supervise the student's memory with this bottleneck representation, effectively aligning the two compression mechanisms. We show that this approach allows to train a recurrent latent robotic memory with linear-time complexity on the Mem-RPE task while substantially narrowing the performance gap to full-history transformers. We additionally validate the same principle on streaming visual question answering (VQA) and observe improved recurrent predictions thanks to memory distillation

Momentum

Related topics

Latest papers 115