Linear RNNs

RNN: Recurrent Neural Network

Ask about this topic

Overview

Linear RNNs study how a sequence can be represented by a compact state that is updated linearly as each new input arrives. Common approaches include diagonal or rotation-based recurrences, gated and delta-rule updates, and structured or sparse memory; scan algorithms make many of these models efficient to train in parallel. They are evaluated on long-context language modeling, retrieval and associative recall, state tracking, and memory-dependent control, where they can offer lower decoding costs than attention. The central tension is how to extend useful memory and recall without sacrificing stability, expressivity, or efficiency, especially under partial observability and finite precision.

Latest papers

CardsList

24 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. Persistence, Exposure, and Credit Routing in Linear Temporal Predictive Coding

    Oct 26, 2026Akira Tanase, Julian Gould, Jeffrey SeelyPredictive CodingTemporal Credit Assignment

    Sakana AI · Imperial College London

  2. A theory of noisy linear recurrent neural network learning dynamics

    Oct 26, 2026John J. Vastola, Kanaka RajanRecurrent Neural NetworksNeural Network Training Dynamics

    Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University

  3. Sector-Ring Linear Reservoirs: Sector-Wise Decay-Timescale Allocation With State Reuse Across Candidate Settings

    Oct 19, 2026Yudai Ebato, Sou Nobukawa, Yusuke Sakemi +3Reservoir ComputingLinear RNNs

    Graduate School of Information and Computer Science, Chiba Institute of Technology, Narashino, Chiba 275-0016, Japan · Department of Computer Science, Chiba Institute of Technology, Narashino, Chiba 275-0016, Japan · Research Center for Mathematical Engineering, Chiba Institute of Technology, Narashino, Chiba 275-0016, Japan +4

  4. Finite-Horizon Fisher Memory in Two-Sided Power-Bounded Recurrent Systems

    Sep 30, 2026Jeonghoon LeeRecurrent Neural NetworksLinear RNNs

    Attractor Dynamics Inc.

  5. Linear Recurrent Memory Suffices to Distil a World-Model Policy for Robot Air Hockey

    Sep 30, 2026F. Olivia Fan, Oliver ObstRecurrent Neural NetworksRobotic RL

    UNSW Sydney, NSW 2052, Australia

  6. CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memory

    Sep 28, 2026Yixiao Chen, Shuojin Yang, Shi-Min HuLinear RNNsLinear Attention

    Tsinghua University

  7. Length-Independent State Tracking Under a Parallel Scan

    Sep 26, 2026Julien Brandoit, Arthur Fyon, Thomas Braipson +5Finite-State MachineRecurrent Neural Networks

    Montefiore Institute, University of Liège Liège, Belgium

  8. How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

    Aug 31, 2026Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi DawsonRecurrent Neural NetworksLinear RNNs

    Independent Researcher, United Kingdom · Department of Electrical and Electronic Engineering, University of Buea, Cameroon · minoHealth AI Labs, Ghana

  9. Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences

    Aug 23, 2026Ziyan Chen, Zhongzhu Zhou, Linhe Zheng +2Neural Scaling LawsLanguage Model Scaling Laws

    The University of Sydney · Together AI · The Pennsylvania State University

  10. Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

    Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context RetrievalMemory-Augmented Language Models

    Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen

  11. Dynamics and Representation Structure of Local Approximations to Gradient-Based Learning in Linear Recurrent Neural Networks

    May 29, 2026Ezekiel Williams, Alexandre Payeur, Guillaume LajoieTruncated Backpropagation Through TimeRecurrent Neural Networks

    Department of Mathematics and Statistics, Universit´e de Montr´eal, Canada · Mila Qu´ebec, Canada · Imperial College London, United Kingdom

  12. Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learning

    May 29, 2026Yike Zhao, Onno Eberhard, Malek Khammassi +2Partially Observable RLMarkov Models

    EPFL, Lausanne, Switzerland · Max Planck Institute for Intelligent Systems, Tübingen, Germany

  13. Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations

    May 27, 2026Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1Parameter SharingTransformer Attention

    Carnegie Mellon University · Google Research

  14. Kaczmarz Linear Attention

    May 9, 2026Jiaxuan Zou, Ruifeng Ren, Yong LiuLong-Context Language ModelingRecurrent Transformers

    School of Mathematics and Statistics Xi’an Jiaotong University Xi’an, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing, China

  15. MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

    May 7, 2026Yulong Huang, Xiang Liu, Hongxiang Huang +5Momentum MethodsLinear Attention

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

  16. How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

    May 6, 2026Mariia SeleznovaNeural Network InitializationRecurrent Neural Networks

    Ludwig-Maximilians-Universität München · Max Planck Institute of Quantum Optics

  17. Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences

    Apr 22, 2026Neehal Tumma, Noel Loo, Daniela RusLong-Context Language ModelingRecurrent Neural Networks

    1MIT · 2Liquid AI

  18. Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall

    Apr 9, 2026Yasong FanLLM Inference AccelerationAssociative Memory

    Independent Researcher

  19. Optimal Decay Spectra for Linear Recurrences

    Apr 8, 2026Yang CaoLinear RNNsLong-Range Dependency Modeling

  20. Why Are Linear RNNs More Parallelizable?

    Mar 4, 2026William Merrill, Hongjian Jiang, Yanhong Li +2Circuit ComplexityRecurrent Neural Networks

    Allen Institute for AI · Rheinland-Pfälzische Technische Universität · Max-Planck Institute for Software Systems

  21. When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training

    Feb 25, 2026Alexander Morgan, Ummay Sumaya Khan, Lingjia Liu +1Recurrent Neural NetworksNeural Network Training Dynamics

    dept. of ECE, Virginia Tech, Alexandria, VA, USA

  22. Learning State-Tracking from Code Using Linear RNNs

    Feb 16, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3Recurrent Neural NetworksState Tracking

    University of Freiburg · Prior Labs · Microsoft Research +2

  23. Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks

    Oct 15, 2024Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha +3In-Context LearningLinear RNNs

    Center for Cognitive Interaction Technology CITEC, Universität Bielefeld, Germany · Department of Electrical and Electronics Engineering, Birla Institute of Technology and Science Pilani, India · Department of Computer Science, Royal Holloway, University of London, United Kingdom

  24. RotRNN: Modelling Long Sequences with Rotations

    Jul 9, 2024Kai Biegun, Rares Dolga, Jake Cunningham +1Recurrent Neural NetworksLong-Context Modeling

    UCL AI Centre London, UK · UCL AI Centre & UiPath London, UK