Recurrent Language Models

Follow this topic Ask about this topic

Latest papers

40 papers20 papers in the last four weeks, up 300% on the four weeks before. 0.1% of all new papers.

MIRA: Mitigating Pole Geometry Bias in Recurrent Language Models
Nov 6, 2026Róisín Luo, Ihsan Ullah, Christian Gagné +2Recurrent Language Models Recurrent Neural Networks
University of Galway · Research Ireland – D-REAL · Université Laval (IID) +3
Dory1.0-4B: A Looped Normalized-GPT
Nov 2, 2026Maxim Fishman, Simeng Sun, Shantanu Acharya +6LLM Inference Recurrent Language Models
A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models
Nov 2, 2026Jackson Petty, William Merrill, Ashish SabharwalState Tracking in Language Models LLM Evaluation
Allen Institute for AI · New York University · Toyota Technological Institute at Chicago
Quantizing Recurrent Language Models Is Easier Than You Think
Nov 1, 2026Philipp Nazari, Neehal Tumma, Daniela Rus +1LLM Quantization Recurrent-State Quantization
Max Planck Institute for Intelligent Systems · ETH Zürich · Tübingen AI Center +3
Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
Oct 29, 2026Dmytro Babarytskyi, Oleksandr BabarytskyiRecurrent Language Models Hybrid Attention
Kharkiv National University of Radio Electronics Kharkiv, Ukraine · Military Law Institute Yaroslav Mudryi National Law University Kharkiv, Ukraine
LinQuant: Functional Invariances for Low-Precision Linear Recurrences
Oct 29, 2026Neehal Tumma, Philipp Nazari, Rohan Cherukuri +2LLM Quantization Recurrent-State Quantization
MIT CSAIL · Max Planck Institute for Intelligent Systems · ETH Zürich +2
ParaTTT: Parallelizing Test-Time Training Layers with Deep Memories
Oct 28, 2026Noel Loo, Neehal Tumma, T. Konstantin Rusch +3Test-Time Training Recurrent Language Models
MIT CSAIL · Liquid AI · Max Planck Institute for Intelligent Systems
Think Twice: The Metis-1.6 Technical Report
Oct 27, 2026Giulianno VollmerLLM Training Efficient Language Model Training
Lernex Research
ScaleRLT: Exact Log-Probabilities and Fast Deferred Rollouts for Reinforcement Learning on Looped Transformers
Oct 25, 2026Dezhen Lu, Zixiang Ni, Runguo LiRL for Language Models Efficient Language Model Inference
University of Tübingen, Tübingen, Germany. · Xi’an Jiaotong University, Xi’an, China. · University of Illinois Urbana-Champaign, Urbana, IL, USA.
Agreement rather than normalisation keeps depth reusable in looped language models
Oct 22, 2026Ivan Viakhirev, Kirill Borodin, Maxim Abramov +1Recurrent Language Models Recursive Language Models
lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia
How Depth Growth and Looping Shape In-Context Learning in Language Models
Oct 22, 2026Ferdinand Kapl, Kaitlin Maile, Stefan Bauer +1In-Context Learning Parameter Sharing
Technical University of Munich · Helmholtz AI, Munich · Munich Center for Machine Learning (MCML) +1
Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction
Oct 20, 2026Yuzhou HongAutoregressive Language Modeling Latent Dynamics Modeling
Zhejiang Sci-Tech University
Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder
Oct 20, 2026Cheng Liang, Tengfei Zhang, Wenjie DU +6Recurrent Language Models Recurrent Transformers
Shanghai Jiaotong University · University of Science and Technology of China · Nanyang Technological University +1
PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking
Oct 20, 2026Pan Wang, Yihao Hu, Guangyuan Dong +8Length Generalization Recurrent Language Models
Ant International · National University of Singapore
Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing
Oct 20, 2026Xi Yang, Guodong Liu, Cheng Peng +7Recurrent Language Models Clinical NLP
Advanced Intelligence and Research Eli Lilly and Company Indianapolis, IN, USA · Dept. of Health Outcomes and Biomedical Informatics University of Florida Gainesville, FL, USA · Higher Education and Research NVIDIA Santa Clara, CA, USA +2
Dual-Slot: From Serial Loops to Parallel Slots in Language Models
Oct 18, 2026Yiqin Hao, Zitong Wang, Boyi Zeng +7Autoregressive Language Modeling Recurrent Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai AI Laboratory
UpLoop: Upcycling Large Language Models with Gated Loop
Oct 18, 2026Tieyuan Chen, Huabin Liu, Yingtao Zhang +4Recurrent Language Models Recurrent Transformers
Inclusion AI, Ant Group · Shanghai Jiao Tong University · Tsinghua University
ReLaDrift: Recurrent Language Generation via Drifting for Test-Time Scaling
Oct 16, 2026Yuki Otsuka, Daisuke Oba, Naoaki OkazakiTest-Time Scaling Recurrent Language Models
Institute of Science Tokyo · AIST · NII LLMC
DeepThink: A Conceptual Model for Knowledge-Addressable Recurrent Language Modeling
Oct 12, 2026Weiguo ChenRecurrent Language Models Language Modeling
NUDT
Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models
Oct 10, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Arya Fayyazi +2Recurrent Language Models Refusal Behavior in Language Models
University of Southern California (USC), USA
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
Oct 9, 2026Ke Wan, Chen ChenRecurrent Language Models LLM Inference Acceleration
Department of Computer Science, University of Virginia
One Model to Loop Them All: Elastic Latent Recurrence for Language Models
Oct 8, 2026Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano +3Language Modeling Recurrent Language Models
NVIDIA
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model Training Recurrent Transformers
Institute of Foundation Models · CMU · USC
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model Training LLM Training
RND NLP, DAIMLD, Russian Federation
KV-streams for Efficient Compaction in Agentic Reinforcement Learning
Sep 28, 2026Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda +15Recurrent Language Models Agentic RL
Mila · Microsoft · Université de Montréal +9
Log-Depth Recurrent Language Modeling
Sep 23, 2026Yiqin Wang, Nuri Cingillioglu, Charles PertRecurrent Language Models Long-Context Language Modeling
Department of Computing Imperial College London London, United Kingdom
Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
Sep 23, 2026Ke Wan, Chen ChenLong-Context Language Model Inference Recurrent Transformers
Department of Computer Science University of Virginia Charlottesville, VA, USA
Temporal Recurrence Favors Fewer Layers
Sep 11, 2026Ivan Anokhin, Johan Obando-Ceron, Irina Rish +1Recurrent Neural Networks Recurrent Language Models
Mila – Qu´ebec AI Institute · Universit´e de Montr´eal · Sakana AI
Fast Weight Attention for Continual Learning
Aug 27, 2026Yifan Zhang, Steve Ta, Jasper Zhang +8Recurrent Language Models Memory-Augmented Language Models
Bytedance Seed · Princeton University · Hyperbolic Labs +1
Allocating Recurrent Compute in Looped Language Models
Aug 18, 2026Ruhai Lin, Yiyang Guo, Rui-Jie Zhu +2Recurrent Language Models Recurrent Transformers
University of California, Santa Cruz

Nov 6, 2026J/K move · Enter open · S save

MIRA: Mitigating Pole Geometry Bias in Recurrent Language Models

Róisín Luo, Ihsan Ullah, Christian Gagné, Dong Li+1

University of Galway · Research Ireland – D-REAL · Université Laval (IID)+3

Recurrent Language Models Recurrent Neural Networks Parallel Training of Recurrent Networks

Open paper AskPDF arXiv

We show that the Transformer's recurrent successors --- such as selective state-space models (SSMs), Receptance Weighted Key Value (RWKV), and Retentive Network (RetNet) --- can be characterized as infinite-impulse-response (IIR) systems. The zero--pole geometries of these IIR systems characterize the temporal modes they can represent. However, to trade parallelism, the algebraic structures of the parameterizations of these recurrent systems often give rise to poles constrained to the real axis, thereby inducing an architectural bias in their recurrent dynamics. To unify these recurrent architectures and eliminate their pole geometry bias, inspired by the lattice--ladder realization of general IIR systems, we present Mira (Modal IIR Recurrent Architecture), a unified family of linear-time recurrent architectures for building large language models (LLMs) with a highly parallel algebraic structure. Each Mira block realizes a higher-order IIR system that represents temporal modes from token streams through a lattice--ladder structure, followed by a feed-forward network (FFN). This realization enables the model to represent the temporal modes associated with both real and complex-conjugate poles. We pretrain Mira-S, a 12-layer small-scale LLM with 0.14B parameters composed entirely of Mira blocks. Throughout the pretraining, Mira-S achieves lower perplexity than strong recurrent baselines, exhibits stronger context-window generalization beyond the training length, and attains higher throughput with a dedicated CUDA kernel.