Recurrent Neural Networks

Also known as RNN

Recurrent neural networks study how a model can process ordered inputs by repeatedly updating an internal state, which is useful for streaming data, long-term memory, and decisions under partial observability. Common approaches include gated LSTMs and GRUs, linear and state-space recurrences, continuous-time and spiking networks, and weight-shared loops for iterative reasoning; training ranges from backpropagation through time to online learning and parallel scans. Researchers test these models on state-tracking and recall tasks, language modeling, time-series forecasting, control, and physical-system prediction. A central tension is how to preserve useful information and stable learning over long sequences while keeping computation efficient and matching the accuracy and flexibility of Transformers.

Latest papers

302 papers47 papers in the last four weeks, up 15% on the four weeks before. 0.3% of all new papers.

Memory Caching: RNNs with Growing Memory
Feb 27, 2026Ali Behrouz, Zeman Li, Yuan Deng +3Recurrent Neural Networks Memory-Augmented Language Models
ReDON: Recurrent Diffractive Optical Neural Processor with Reconfigurable Self-Modulated Nonlinearity
Feb 27, 2026Ziang Yin, Qi Jing, Raktim Sarma +3Diffractive Neural Networks Photonic Computing
Arizona State University, Tempe, AZ, USA · Center for Integrated Nanotechnologies, Sandia National Laboratories, Albuquerque, NM, USA · Rensselaer Polytechnic Institute, Troy, NY, USA
Stream Neural Networks: Epoch-Free Learning with Persistent Temporal State
Feb 25, 2026Amama PathanDynamic Neural Networks Streaming Algorithms
BLACKLOOP Research (Independent Research)
Pulse-Driven Neural Architecture: Learnable Oscillatory Dynamics for Robust Continuous-Time Sequence Processing
Feb 25, 2026Paras SharmaRecurrent Neural Networks Oscillatory Neural Networks
When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training
Feb 25, 2026Alexander Morgan, Ummay Sumaya Khan, Lingjia Liu +1Recurrent Neural Networks Neural Network Training Dynamics
dept. of ECE, Virginia Tech, Alexandria, VA, USA
Benchmarking State Space Models, Transformers, and Recurrent Networks for US Grid Forecasting
Feb 24, 2026Sunki Hong, Jisoo LeeForecasting Benchmarks Electricity Load Forecasting
Gramm AI
Subgroups of $U(d)$ Induce Natural RNN and Transformer Architectures
Feb 20, 2026Joshua NunleyRecurrent Neural Networks Transformer
Cognitive Science Program & Department of Informatics, Indiana University
Leakage and Second-Order Dynamics Improve Hippocampal RNN Replay
Feb 20, 2026Josue Casco-Rodriguez, Nanda H. Krishna, Richard G. BaraniukRecurrent Neural Networks Langevin Dynamics
Department of Electrical and Computer Engineering, Rice University, USA · Mila - Quebec AI Institute & Université de Montréal, Canada
On the "Induction Bias" in Sequence Models
Feb 20, 2026M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal +1Transformer State Tracking
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Learning Long-Range Dependencies with Temporal Predictive Coding
Feb 20, 2026Tom Potter, Oliver RhodesPredictive Coding Recurrent Neural Networks
International Centre for Neuromorphic Systems, The University of Manchester, Manchester, United Kingdom
Deepmechanics
Feb 20, 2026Abhay Shinde, Aryan Amit Barsainyan, Jose Siguenza +3Symplectic RNNs Lagrangian Neural Networks
Deep Forest Sciences, Mumbai, Maharashtra, India · Deep Forest Sciences, Bengaluru, Karnataka, India · Deep Forest Sciences, Palo alto, California, USA +1
Parallelizable Neural Turing Machines
Feb 18, 2026Gabriel Faria, Arnaldo Candido JuniorNeural Algorithmic Reasoning Recurrent Neural Networks
University of São Paulo∗ · ∗Instituto de Ciências Matemáticas e de Computação (ICMC), University of São Paulo. · São Paulo State University
Wideband Power Amplifier Behavioral Modeling Using an Amplitude Conditioned LSTM
Feb 17, 2026Abdelrahman Abdelsalam, You FeiNonlinear System Identification Long Short-Term Memory Networks
School of Electronic Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China
Continuous-Time Piecewise-Linear Recurrent Neural Networks
Feb 17, 2026Alena Brändle, Lukas Eisenmann, Florian Götz +1Recurrent Neural Networks Dynamical Systems
Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Mannheim, Germany · Faculty of Physics and Astronomy, Heidelberg University · Hector Institute for Artificial Intelligence in Psychiatry, CIMH +2
A unified theory of feature learning in RNNs and DNNs
Feb 17, 2026Jan P. Bauer, Kirsten Fischer, Moritz Helias +1Recurrent Neural Networks Representation Learning
Gatsby Computational Neuroscience Unit, UCL, London, UK · IAS-6, Forschungszentrum Juelich, Juelich, Germany
A comparative study of transformer models and recurrent neural networks for path-dependent composite materials
Feb 17, 2026Petter Uvdal, Mohsen MirkhalafNeural Surrogate Modeling Recurrent Neural Networks
Department of Physics, University of Gothenburg, Origovägen 6B, 41296 Gothenburg, Sweden
Drift-Diffusion Matching: Embedding dynamics in latent manifolds of asymmetric neural networks
Feb 16, 2026Ramón Nartallo-Kaluarachchi, Renaud Lambiotte, Alain GorielyRecurrent Neural Networks Dynamical Systems
Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, United Kingdom · Centre for Eudaimonia and Human Flourishing, University of Oxford, 7 Stoke Pl, Oxford, OX3 9BX, United Kingdom · Complexity Science Hub, Metternichgasse 8, Vienna, 1030, Austria
Learning State-Tracking from Code Using Linear RNNs
Feb 16, 2026Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3Recurrent Neural Networks State Tracking
University of Freiburg · Prior Labs · Microsoft Research +2
Random matrix theory of sparse neuronal networks with heterogeneous timescales
Dec 14, 2025Thiparat Chotibut, Oleg Evnin, Weerawit HorinouchiRandom Matrix Theory Recurrent Neural Networks
Chula Intelligent and Complex Systems, Department of Physics, Faculty of Science, Chulalongkorn University, Bangkok, Thailand · High Energy Physics Research Unit, Faculty of Science, Chulalongkorn University, Bangkok, Thailand · Theoretische Natuurkunde, Vrije Universiteit Brussel (VUB) and International Solvay Institutes, Brussels, Belgium +2
Learnability Window in Gated Recurrent Neural Networks
Dec 5, 2025Lorenzo LiviGated Recurrent Unit Recurrent Neural Networks
OPIT – Open Institute of Technology
Variability Aware Recursive Neural Network (VARNN): A Residual-Memory Model for Capturing Temporal Deviation in Sequence Regression Modeling
Oct 10, 2025Haroon Gharwi, Yue Dai, Kai ShuMemory-Augmented Neural Networks Residual Learning
Department of Computer Science, Illinois Institute of Technology, Chicago, IL, USA · Department of Computer Science, Emory University, Atlanta, GA, USA
Deep Regularized RNNs for Virtual Analog Modeling
Sep 19, 2025V. Valtteri Kallinen, Lauri Juvela, Thom ShersonRecurrent Neural Networks
Dept. of Information and Communications Engineering (DICE) Aalto University Espoo, Finland · Neural DSP Technologies Oy Helsinki, Finland
Recurrent State Encoders for Efficient Neural Combinatorial Optimization
Sep 5, 2025Tim Dernedde, Daniela Thyssens, Lars Schmidt-ThiemeRecurrent Neural Networks Combinatorial Optimization
Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim
Dynamical stability for dense patterns in attractor neural networks
Jul 14, 2025Uri Cohen, Máté LengyelNeural Network Memorization Associative Memory
Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, UK · Center for Cognitive Computation, Department of Cognitive Science, Central European University, Budapest, Hungary
Neural Langevin Machine: a local asymmetric learning rule can be creative
Jun 30, 2025Zhendong Yu, Weizhong Huang, Haiping HuangNeural Network Generalization Image Generation
PMI Lab, School of Physics, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China · Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices, Sun Yat-sen University, Guangzhou 510275, People’s Republic of China
Hierarchical Reasoning Model
Jun 26, 2025Guan Wang, Jin Li, Yuhao Sun +6Recurrent Neural Networks Algorithmic Reasoning
Sapient Intelligence, Singapore · Tsinghua University
R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks
Apr 1, 2025Nicholas H. Barbara, Ruigang Wang, Ian R. ManchesterNeural Network Robustness Recurrent Neural Networks
Australian Centre for Robotics (ACFR) and the School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney, Australia
Koopman-informed recurrent neural networks
Oct 30, 2024Erik Lien Bolager, Ana Čukarska, Iryna Burak +2Neural Network Optimization Time Series Forecasting
Munich Data Science Institute · School of Computation, Information and Technology, Technical University of Munich, Germany · Interdisciplinary Center for Scientific Computing (IWR) +1
Enhancing Cognitive Workload Classification Using Integrated LSTM Layers and CNNs for fNIRS Data Analysis
Jul 22, 2024Mehshan Ahmed Khan, Houshyar Asadi, Mohammad Reza Chalak Qazani +6Recurrent Neural Networks Convolutional Neural Networks
Institute for Intelligent Systems Research and Innovation, Deakin University, 75 Pigdons Rd, Waurn Ponds, VIC, 3216, Geelong, Australia
RotRNN: Modelling Long Sequences with Rotations
Jul 9, 2024Kai Biegun, Rares Dolga, Jake Cunningham +1Recurrent Neural Networks Long-Context Modeling
UCL AI Centre London, UK · UCL AI Centre & UiPath London, UK

Feb 27, 2026J/K move · Enter open · S save

Memory Caching: RNNs with Growing Memory

Ali Behrouz, Zeman Li, Yuan Deng, Peilin Zhong+2

Recurrent Neural Networks Memory-Augmented Language Models Long-Context Language Modeling

Open paper AskPDF arXiv

Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context length. While plausible for retrieval tasks, it causes quadratic complexity and so has motivated recent studies to explore viable subquadratic recurrent alternatives. Despite showing promising preliminary results in diverse domains, such recurrent architectures underperform Transformers in recall-intensive tasks, often attributed to their fixed-size memory. In this paper, we introduce Memory Caching (MC), a simple yet effective technique that enhances recurrent models by caching checkpoints of their memory states (a.k.a. hidden states). Memory Caching allows the effective memory capacity of RNNs to grow with sequence length, offering a flexible trade-off that interpolates between the fixed memory (i.e.,

O(L)

complexity) of RNNs and the growing memory (i.e.,

O(L^2)

complexity) of Transformers. We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules. Our experimental results on language modeling, and long-context understanding tasks show that MC enhances the performance of recurrent models, supporting its effectiveness. The results of in-context recall tasks indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.