Recurrent Transformers

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, against 2 the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 115

All topics

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals
Oct 7, 2026Heejun Kim, Junyoung Lee, SangLyul Cho +3KV-Cache Quantization Efficient Transformer Inference
KAIST · Yonsei University · Seoul National University
Recurrent Looped Transformer
Oct 6, 2026Yifan Zhang, Jichen Feng, Shihan QinLength Generalization Recurrent Transformers
Princeton University · University of Pennsylvania
TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity
Oct 6, 2026Zijian Li, Xiangchen Song, Gongxu Luo +7Synthetic Data Pretraining Tabular Foundation Models
Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence · Guangdong University of Technology +1
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model Training Recurrent Transformers
Institute of Foundation Models · CMU · USC
Balancing Memory Pathways: Analyzing and Improving Memory Utilization in Hybrid LMs
Oct 5, 2026Hyunji Lee, Joykirat Singh, Zaid Khan +5Long-Context Language Modeling Recurrent Transformers
UNC Chapel Hill · The University of Texas at Austin · Mila +1
LiFT: Loop Flow Transformers
Oct 4, 2026Mohammad Mahdi Derakhshani, Pedro M. P. Curvo, Gertjan J. Burghouts +2Diffusion Transformer Recurrent Transformers
VISLab · University of Amsterdam · TNO, Intelligent Imaging +1
Universal Test-Time Training
Oct 4, 2026Zefan Cai, Qinzhe Hu, Ziqiao Ma +2Memory-Augmented Language Models Recurrent Transformers
University of Wisconsin–Madison · University of Michigan–Ann Arbor · Adobe
Decoding Looped Transformers Better for (Almost) Free
Oct 1, 2026Weihao Liu, Huangjie Zheng, Tianrong Chen +5Efficient Transformer Inference Language Model Decoding
Apple
LAST: Looped Audio Spectrogram Transformer
Oct 1, 2026Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina +1Audio Understanding Recurrent Transformers
Georgia Institute of Technology · University of California San Diego · Google DeepMind
A foundation for systematic analysis of transformers and RNNs for tractography
Oct 1, 2026Emmanuelle Renauld, Philippe Poulin, Hugo Larochelle +2Recurrent Neural Networks Recurrent Transformers
Computer Sciences Department, Université de Sherbrooke · Mila, Université de Montréal · Imeka Solutions Inc, Sherbrooke, QC, Canada
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
Oct 1, 2026Di He, Pengxiang Li, Da Chang +3Mixture-of-Experts Language Models Recurrent Transformers
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Scaling Laws for Looped Mixture of Experts
Sep 30, 2026Yanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiMixture-of-Experts Language Models Language Model Scaling Laws
Meta AI
Looped Diffusion Transformer
Sep 30, 2026Yong Xien Chng, Tianyi Chen, Wenwen Tong +7Diffusion Transformer Recurrent Transformers
SenseTime Research · LeapLab, Tsinghua University · Nanyang Tehnological Univesity
What Limits Recursive Reasoning Models: Optimization, Architecture and Test-Time Scaling
Sep 30, 2026Yuliana Shakhvalieva, Dmitrii Kharchev, Viacheslav Bezrukov +4Neural Algorithmic Reasoning Recurrent Transformers
RND NLP, DAIMLD, Russian Federation
Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Sep 30, 2026Jiaju Wu, Yi Hu, Muhan ZhangTransformer Interpretability Transformer Attention
Institute for Artificial Intelligence, Peking University. · School of Mathematical Sciences, Peking University.
LoopVL: Recurrent Visual Intelligence
Sep 29, 2026Zhe Qian, Ziyang Gong, Zhongxing Xu +9Vision-Language Models Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China · Baidu · Shanghai Jiao Tong University +5
Pretraining Latent Information Feedback Transformers with Teacher Supervision
Sep 29, 2026Dor Tirosh, Ido Amos, Mor GevaLanguage Model Pretraining Transformer
Blavatnik School of Computer Science and AI, Tel Aviv University · The Hebrew University of Jerusalem
LoopICL: Looping a single transformer block to solve tabular tasks
Sep 28, 2026Amir Rezaei Balef, Katharina EggenspergerTabular Foundation Models Cost-Aware Inference
TU Dortmund University, Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Dortmund, Germany · University of Tübingen, Tübingen, Germany
How to Loop MoE: Flatten the Experts, Untie the Attention
Sep 28, 2026Shouren Wang, Chuang Ma, Mohsen Hariri +6Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Case Western Reserve University · Kyoto University · NII LLMC
Improving Test-Time Scaling with Adaptive Looped Transformers
Sep 28, 2026Yichen You, Tianyu Fu, Aosong Feng +4Test-Time Scaling Recurrent Transformers
Tsinghua University · Yale University
Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization
Sep 28, 2026Huzi Cheng, Zhewei ZhangLLM Reasoning with Graphs LLM Interpretability
University of Minnesota · Independent Researcher
Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers
Sep 28, 2026Guanghao Li, Zihan Su, Hao Yu +6Speculative Decoding Recurrent Transformers
Tsinghua University · Southern University of Science and Technology · The University of Hong Kong +2
DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
Sep 28, 2026Julian Boesch, Andrew Wee, Alexander StranzlIn-Context Retrieval Recurrent Transformers
Purdue University · Obit Research · State University of New York at Stony Brook.
LoopTrack: A Simple Baseline for Parameter-Efficient Transformer Tracking
Sep 27, 2026Liang Peng, Chenxiao Li, Libo Zhang +2Visual Object Tracking Recurrent Transformers
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University · University of North Texas · Institute of Software, Chinese Academy of Sciences
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer Inference Recurrent Transformers
ELLIS Institute Tübingen Max Planck Institute for Intelligent Systems Tübingen AI Center
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA
GTR: Gated Token Recurrence for Efficient Dense Prediction
Sep 22, 2026Zhe Feng, Longfei Liu, Wei Liu +7Efficient Neural Network Inference Gated Attention
Didi International Business Group · Intellindust AI Lab · Institute of Automation, Chinese Academy of Sciences +2
Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models
Sep 17, 2026Ha Van Dau, Thanh Tung Khuat, Nguyen Thanh DungLLM Evaluation Recurrent Transformers
NuverxAI - AI & Creative Innovation Company Limited · Ho Chi Minh City University of Technology (HCMUT)
Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers
Sep 17, 2026Aras Kavuncu, Muhammad Burhan HafezMemory-Augmented Neural Networks Recurrent Transformers
University of Southampton, Electronics and Computer Science
How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
Sep 16, 2026Zixi Chen, Akshay Vegesna, Samip Dahal +1Language Model Scaling Laws Recurrent Transformers
New York University · Q Labs

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

Heejun Kim, Junyoung Lee, SangLyul Cho, Dongsu Han+2

KAIST · Yonsei University · Seoul National University

KV-Cache Quantization Efficient Transformer Inference Low-Bit Quantization Recurrent Transformers

Open paper AskPDF arXiv

Looped Transformers improve parameter efficiency by repeatedly applying shared Transformer blocks over multiple recurrent loops, increasing computational depth without increasing the parameter count. However, KV cache memory still scales with the number of loops, becoming a key memory bottleneck that limits batch size and inference throughput. KV cache quantization can alleviate this bottleneck, but existing methods often suffer substantial accuracy degradation at aggressive low-precision regimes. We observe that looped Transformers offer a unique opportunity: KV states across loops are highly similar. Based on this observation, we propose ResidualQuant, which uses the final-loop KV states as a reference and represents the remaining loops with low-precision residuals. Our method further combines least-square scaling and rotations applied to the residuals, as well as loop-wise mixed precision, to enable accurate quantization down to INT2 while retaining efficient reconstruction. Across multiple looped Transformer models and mathematical reasoning and code generation benchmarks, ResidualQuant consistently improves the accuracy-memory tradeoff over state-of-the-art rotation-based KV quantization. In particular, our method retains accuracy close to BF16 under mixed-precision settings while reducing theoretical KV storage by 80.7%, achieving up to 13.0% higher accuracy than the rotation-based baseline at the same memory budget. On an RTX 5090, the reduced KV memory traffic improves fixed-batch decode throughput by up to 2.73x, while the smaller memory footprint enables up to 2x larger batches, improving peak throughput by up to 4.15x.

Momentum

Related topics

Latest papers 115