Recurrent Neural Networks

Also known as RNN

Ask about this topic

Overview

Recurrent neural networks study how a model can process ordered inputs by repeatedly updating an internal state, which is useful for streaming data, long-term memory, and decisions under partial observability. Common approaches include gated LSTMs and GRUs, linear and state-space recurrences, continuous-time and spiking networks, and weight-shared loops for iterative reasoning; training ranges from backpropagation through time to online learning and parallel scans. Researchers test these models on state-tracking and recall tasks, language modeling, time-series forecasting, control, and physical-system prediction. A central tension is how to preserve useful information and stable learning over long sequences while keeping computation efficient and matching the accuracy and flexibility of Transformers.

Latest papers

CardsList

303 papers47 papers in the last four weeks, up 15% on the four weeks before. 0.3% of all new papers.

May 17, 2026

Von Economo neurons enable reliable social skill acquisition in recurrent spiking neural networks: a computational account with clinical predictions

Von Economo neurons (VENs) are selectively lost in behavioural-variant frontotemporal dementia (bvFTD) and reduced in autism spectrum conditions (ASC), yet their computational role in social learning remains unexplained. We train a spiking neural network (the VENCircuit) embedding VEN-like projection neurons (K=40, 2% of total) in a recurrent pyramidal circuit across 50 matched random initialisations with and without VENs. The network is trained on a controlled binary classification task; we make no claim to model social cognition directly. VEN-intact networks converged in 49/50 cases (98%) versus 35/50 (70%) for VEN-ablated networks (Fisher's exact OR=21.0, 95% CI 2.7-167, p=8.7e-5). Failed ablated networks showed complete absence of learning, inconsistent with a speed-of-learning account. Phase-ablation experiments show VEN removal is most disruptive during mid-training (epochs 5-25), when a co-adaptive dependency forms in the pyramidal circuit. We derive a formal account showing VENs provide a direct gradient pathway immune to Jacobian instabilities affecting the recurrent circuit. Inference-time VEN ablation caused a significant performance drop (Wilcoxon p=0.022), ranging from no change (16/20 networks) to catastrophic collapse (0.989 to 0.620). VENs function as acquisition scaffolds whose developmental absence produces stochastic learning failure - a computational analogue of variable social skill acquisition in ASC - with falsifiable predictions for organoid and electrophysiology studies.

Esila Keskin

School of Computing and Creative Technologies, University of the West of England, Bristol, UK

Neural ProcessesRecurrent Neural NetworksSpiking Neural NetworksNeural Network Training Dynamics
AskPDF
May 16, 2026

Parallel Recursive LSTM

Transformers have become the dominant architecture for sequence modeling by using self-attention to enable expressive and highly parallel processing. However, the resulting quadratic time and memory costs limit efficiency in long-context settings. Recurrent models such as LSTMs provide explicit nonlinear state updates and strong state-tracking capabilities, yet their strictly sequential computation limits parallelism. We introduce the Parallel Recursive LSTM (PR-LSTM), a hierarchical recurrent architecture that replaces left-to-right recurrence with recursive nonlinear state composition over a balanced computation tree. Tokens are first mapped independently to latent states, which are then recursively merged by a learned gated composition block. This structure uses the reduction pattern underlying parallel scans as a fixed execution schedule, rather than assuming an associative recurrence. As a result, PR-LSTM retains nonlinear gated state representations while reducing recurrent parallel depth from linear to logarithmic. Empirically, PR-LSTM achieves strong sequence-length generalization on formal-language benchmarks, solving more tasks than standard RNN, LSTM, and Transformer baselines, while avoiding the quadratic scaling of attention. These results suggest that recurrent computation can be reorganized hierarchically to expose parallelism without restricting the transition dynamics to linear or associative forms.

Tristan Gaudreault, Yongyi Mao

School of Electrical Engineering and Computer Science University of Ottawa Ottawa, ON

Recurrent Neural NetworksLanguage ModelingLong Short-Term Memory Networks
AskPDF
May 16, 2026

Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction

Streaming 3D reconstruction under a strict constant-memory budget hinges on how the recurrent state is updated as the stream evolves. We profile TTT3R-style per-token gates across five benchmarks and discover a structural bottleneck: the gate is intrinsically bounded in magnitude (median 0.310.31; never exceeding 0.60.6) and nearly frame-invariant, yielding an effective memory horizon of only ∼\sim3 frames per state token, which serves as the structural origin of long-sequence drift. We trace this to a missing axis: existing inference-time methods modulate updates only at the per-token, intra-frame level, while the orthogonal frame-level question of how strongly each frame should contribute to the state has been treated as content-independent. We close this gap with a scalar frame-level gate αt∈(0,1]α_t \in (0, 1] derived in closed form from frame-to-frame changes of internal features---a graded write weight, inspired by classical Simultaneous Localization and Mapping (SLAM) keyframe selection, that never discards a frame and requires no parameters, no training, and no extra forward pass. Across six benchmarks spanning camera pose, video depth, and 3D reconstruction at sequence lengths up to 4,6614,661 frames, our gate cuts ATE by 51%51\% on long TUM-RGBD pose sequences, reduces AbsRel by 13.0%13.0\% on Bonn video depth, and on KITTI long-sequence pose estimation surpasses both LongStream and Keyframe-VO in average ATE, while retaining strictly constant memory at zero training cost.

Kejun Ren, Lei Jin, Tianxin Huang +2

Beijing University of Posts and Telecommunications, Beijing, China · School of Computing and Data Science, The University of Hong Kong

Streaming 3D Reconstruction3D ReconstructionDepth EstimationRecurrent Neural NetworksVisual OdometryVideo Depth Estimation
AskPDF
May 14, 2026

PDRNN: Modular Data-driven Pedestrian Dead Reckoning on Loosely Coupled Radio- and Inertial-Signalstreams

Modern pedestrian dead reckoning (PDR) systems rely on fusing noisy and biased estimates of position, velocity, and calibrated orientation derived from loosely coupled sensors to determine the current pose of a localized object. However, discrepancies in the sampling rates of sensor-specific estimation methods and unreliable transmission pose significant challenges. And traditional methods often fail to effectively fuse multimodal sensor data during dynamic movements characterized by high accelerations, velocities, and rapidly varying orientations. To address these limitations, we propose a simple recurrent neural network (RNN) architecture capable of implicitly forecasting asynchronous sensor data streams from diverse estimation methods along reference trajectories. The proposed approach introduces PDRNN, a modular hybrid AI-assisted PDR system that handles each component as an independent ensemble of machine learning (ML) models to estimate both key parameter means and variances. Separate ML-based models are employed to estimate orientation, (un)directed velocity or distance from acceleration and gyroscope data, with optional absolute positioning from synchronized radio systems such as 5G for stabilization. A final fusion model combines these outputs, position, velocity, and orientation, while using uncertainty estimates to enhance system robustness. The modular design allows individual components to be updated, fine-tuned, or replaced without affecting the entire system. Experiments on dynamic sports movement data show that PDRNN achieves superior accuracy and precision compared to classic and ML-based methods, effectively avoiding error accumulation common in black-box approaches. And PDRNN offers forecast capabilities and better component control despite increased system complexity.

Peter Bauer, Andreas Porada, Felix Ott +2

Fraunhofer Institute for Integrated Circuits IIS, 90411 Nürnberg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany

Recurrent Neural NetworksRobot LocalizationMultimodal Sensor FusionModular Neural Networks
AskPDF
May 14, 2026

Exploitation of Hidden Context in Dynamic Movement Forecasting: A Neural Network Journey from Recurrent to Graph Neural Networks and General Purpose Transformers

Forecasting within signal processing pipelines is crucial for mitigating delays, particularly in predicting the dynamic movements of objects such as NBA players. This task poses significant challenges due to the inherently interactive and unpredictable nature of sports, where abrupt changes in velocity and direction are prevalent. Traditional approaches, including (S)ARIMA(X), Kalman filters (KF), and Particle filters (PF), often struggle to model the non-linear dynamics present in such scenarios. Machine learning (ML) methods, such as long short-term memory (LSTM) networks, graph neural networks (GNNs), and Transformers, offer greater flexibility and accuracy but frequently fail to explicitly capture the interplay between temporal dependencies and contextual interactions, which are critical in chaotic sports environments. In this paper, we evaluate these models and assess their strengths and weaknesses. Experimental results reveal key performance trade-offs across input history length, generalizability, and the ability to incorporate contextual information. ML-based methods demonstrated substantial improvements over linear models across forecast horizons of up to 2s. Among the tested architectures, our hybrid LSTM augmented with contextual information achieved the lowest final displacement error (FDE) of 1.51m, outperforming temporal convolutional neural network (TCNN), graph attention network (GAT), and Transformers, while also requiring less data and training time compared to GAT and Transformers. Our findings indicate that no single architecture excels across all metrics, emphasizing the need for task-specific considerations in trajectory prediction for fast-paced, dynamic environments such as NBA gameplay.

Lukas Schelenz, Shobha Rajanna, Denis Gosalci +6

Fraunhofer Institute for Integrated Circuits IIS, 90411 Nürnberg, Germany · Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany

Graph Neural NetworksMulti-Agent Trajectory PredictionRecurrent Neural NetworksTrajectory Prediction
AskPDF
May 14, 2026

Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learning

History-dependent policies induced by recurrent neural networks (RNNs) rely on latent hidden state dynamics, making verification in partially observable reinforcement learning (RL) challenging. Existing RNN verification tools typically rely on restrictive modeling assumptions or coarse over-approximations of the hidden state space, which can lead to overly conservative or inconclusive results. We propose RNN\textbf{RNN} Pro\textbf{Pro}babilistic Ve\textbf{Ve}rification (RNN-ProVe\texttt{RNN-ProVe}), a probabilistic framework that estimates the likelihood\textit{estimates the likelihood} of undesired behaviors in RNN-based policies. RNN-ProVe\texttt{RNN-ProVe} uses policy-driven sampling to approximate the set of hidden states that are feasible under a trained policy, and derives statistical error bounds to produce bounded-error, high-confidence estimates of behavioral violations. Experiments on partially observable single-agent and cooperative multi-agent tasks show that RNN-ProVe\texttt{RNN-ProVe} yields more quantitative, feasibility-aware probabilistic guarantees than existing tools, while scaling to recurrent and multi-agent settings.

Luca Marzari, Enrico Marchesini

1TU Wien, Vienna, Austria · 2Massachusetts Institute of Technology, Cambridge (MA), USA

Neural Network VerificationPartially Observable RLRecurrent Neural Networks
AskPDF
May 14, 2026

Spontaneous symmetry breaking and Goldstone modes for deep information propagation

In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes, which allow coherent information propagation over long distances and times. In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry and may therefore support analogous Goldstone-like degrees of freedom. We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations, providing a mechanism for stable information flow without relying on architectural stabilizers such as residual connections or normalization. In feedforward networks, this results in improved trainability and representational diversity across layers. In recurrent settings, we demonstrate the same mechanism is valuable for long-term memory by propagating information over recurrent iterations, thereby improving performance of RNNs and GRUs on long-sequence modeling tasks.

Nabil Iqbal, T. Anderson Keller, Yue Song +2

Dept. of Mathematical Sciences, Durham University · Kempner Institute, Harvard University · College of AI, Tsinghua University +2

Symmetry BreakingEquivariant Neural NetworksRecurrent Neural Networks
AskPDF
May 13, 2026

Parallel Scan Recurrent Neural Quantum States for Scalable Variational Monte Carlo

Neural-network quantum states have emerged as a powerful variational framework for quantum many-body systems, with recent progress often driven by massively parallel architectures such as transformers. Recurrent neural network quantum states, however, are frequently regarded as intrinsically sequential and therefore less scalable. Here we revisit this view by showing that modern recurrent architectures can support fast, accurate, and computationally accessible neural quantum state simulations. Using autoregressive recurrent wave functions together with recent advances in parallelizable recurrence, we develop variational ansätze, called parallel scan recurrent neural quantum states (PSR-NQS), which can be trained efficiently within variational Monte Carlo in one and two spatial dimensions. We demonstrate accurate benchmark results and show that, with iterative retraining, our approach reaches two-dimensional spin lattices as large as 52×5252\times52 while remaining in agreement with available quantum Monte Carlo data. Our results establish recurrent architectures as a practical and promising route toward scalable neural quantum state simulations with modest computational resources.

Ejaaz Merali, Mohamed Hibat-Allah, Mohammad Kohandel +2

Department of Physics and Astronomy, University of California, Davis, California 95616, USA · Department of Applied Mathematics, University of Waterloo, Waterloo, ON N2L 3G1, Canada · Vector Institute, Toronto, Ontario, M5G 0C6, Canada +1

Neural Quantum StatesVariational Monte CarloRecurrent Neural NetworksQuantum Many-Body Physics
AskPDF
May 13, 2026

Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catastrophic gradient instability during Backpropagation Through Time. In this work, we break this stalemate with Phasor Memory Network (PMNet), a novel architecture that structurally resolves memory volatility through Unitary Phasor Dynamics and Hierarchical Learnable Anchors. Rather than relying on brute-force scaling, we present a mechanistic proof-of-concept in a controlled byte-level setting. By constraining recurrent state updates to phase rotations on a complex unit circle, PMNet preserves gradient norms and inherently prevents divergence without the need for specialized initialization. We empirically demonstrate the active actuation of the memory module through a synthetic Copy-Paste task, where PMNet utilizes an expansive 85-slot hierarchical memory tree (=∑h=144h−1=\sum^{4}_{h=1}4^{h-1}) to achieve near 100% exact retrieval across temporal distances that completely exceed the local sliding window attention's receptive field. Furthermore, despite being a compact 119M parameter model trained on 18.8B tokens, PMNet matches the zero-shot long-context robustness of a Mamba model that is three times larger. Our ablation studies and gradient analyses confirm that the historical failure of explicit memory was a structural alignment problem, which PMNet effectively overcomes, providing a theoretically grounded foundation for scalable sequence modeling.

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

College of Pharmacy Chungnam National University Daejeon, South Korea · Department of Computer Science & Engineering Chungnam National University Daejeon, South Korea

Memory-Augmented Neural NetworksMemory-Augmented Language ModelsLong-Context Language ModelingPersistent Memory for Language ModelsRecurrent Neural NetworksMemorization in Generative Models
AskPDF
May 12, 2026

State-Space NTK Collapse Near Bifurcations

Rich feature learning in tasks that unfold over time often requires the model to pass through bifurcations, constituting qualitative changes in the underlying model dynamics. We develop a local theory of gradient descent near these transitions through the empirical state-space neural tangent kernel (sNTK). Our central finding is that bifurcations both dominate and simplify learning dynamics: near bifurcations, we can reduce sNTK to a rank-one operator corresponding to learning in a classical normal form system, providing an analytically tractable description of the local learning geometry, even for high-dimensional recurrent systems. Concretely, we give a procedure for decomposing sNTK into bifurcation-relevant and residual channels, showing that near commonly codimension-1 bifurcations the relevant channel is a rank-one operator that is highly amplified. This amplification causes the bifurcation channel to dominate the full sNTK. Thus, bifurcations locally warp the learning landscape, funneling gradient descent into a few critical dynamical directions and making the nearby kernel and loss geometry predictable from classical normal forms. We illustrate this in a student-teacher recurrent neural network: the first learned bifurcation coincides with a sharp collapse in sNTK effective rank and the emergence of a dominant parameter direction whose restricted sNTK closely matches the landscape predicted by the scalar pitchfork normal form. Finally, we show that low-rank natural gradient methods resolve the resulting learning instability near bifurcations with very little overhead over SGD.

James Hazelden, Eric Shea-Brown

University of Washington

Recurrent Neural NetworksNeural Network Training DynamicsNeural Tangent Kernel
AskPDF
May 12, 2026

Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction

Reconstructing nonlinear dynamical systems (DS) from data (DSR) is a fundamental challenge in science and engineering, but it inherently relies on sequential models. Recent breakthroughs for sequential models have produced algorithms that parallelize computation along sequence length TT, achieving logarithmic time complexity, O(log⁡T)\mathcal{O}(\log T). Since sequence lengths have been practically limited due to the linear runtime complexity O(T)\mathcal{O}(T) of classical backpropagation through time, this opens new avenues for DSR. This paper studies two prominent classes of parallel-in-time algorithms for this task, both of which leverage parallel associative scans as their core computational primitive. The first class comprises models with linear yet non-autonomous dynamics and a nonlinear readout, such as modern State Space Models (SSMs), while the second consists of general nonlinear models which can be parallelized using the DEER framework. We find that the linear training-time recurrence of the first class of models imposes limitations that often hinder learning of accurate nonlinear dynamics. To address this, we augment DEER with Generalized Teacher Forcing (GTF), a novel variant within the more general nonlinear framework that ensures stable and effective learning of nonlinear dynamics across arbitrary sequence lengths. Using GTF-DEER, we investigate the benefits of training on extremely long sequences (T>104T>10^4) for DSR. Our results show that access to such long trajectories significantly improves DSR if the data features long time scales. This work establishes GTF-DEER as a robust tool for data-driven discovery and underscores the largely untapped potential of long-sequence learning in modeling complex DS.

Florian Hess, Florian Götz, Daniel Durstewitz

1Dept. of Theoretical Neuroscience, Central Institute of Mental Health,2026 Mannheim, Germany · Faculty of Mathematics and Computer Science, Heidelberg University, Germany · Faculty of Physics and Astronomy, Heidelberg University, Germany +1

Nonlinear System IdentificationRecurrent Neural NetworksState Space Models
AskPDF
May 12, 2026

Train Short, Act Long: Multistability and the Horizon Independence of Recurrent Policies

In partially observable reinforcement learning (RL), a decision often depends on a cue seen many steps earlier, and training at such horizons is costly. Ideally, a policy trained on short horizons would remain optimal at arbitrarily long ones, making the training horizon a matter of convenience. We formalize this property as delay-horizon independence (DHI): optimality at every horizon, at a memory, computation and precision cost that is bounded in the horizon. We study agents that carry the cue in a recurrent memory of fixed size, whose cost does not grow with the delay. This paper characterizes DHI through the dynamics of that memory, in theory and on benchmarks. Although quantified over all horizons, DHI reduces to a property of a single map: a recurrent policy is DHI if and only if it admits a continuous readout of the memory, invariant under the idle dynamics and separating the cues. Monostable memory then precludes DHI at any parameter values, whereas multistable memory enables it. Monostability covers all contracting affine recurrences, hence state space models and gated linear recurrent neural networks (RNNs). Under arbitrarily small perturbations of the idle dynamics, multistability is the only mechanism supporting DHI at bounded memory. On the T-maze, the effective horizon of monostable models resists hyperparameter search: selected horizons fall short by up to 745×745\times when retrained. Multistable models trained on short horizons stay optimal up to 10610^6, on the T-maze and on MiniGrid. Current parallelizable RNNs thus trade horizon independence for scalability: parallelizable multistable architectures are the direction this points to.

Asad Bakija, Florent De Geeter, Julien Brandoit +2

Dept. of Electrical Engineering and Computer Science University of Liège Liège, Belgium · University of Liège

Dynamical SystemsReinforcement LearningPartially Observable RLRecurrent Neural NetworksGeneralization in RLLong-Horizon RL
AskPDF
May 12, 2026

Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons

Cortical neurons are complex, multi-timescale processors wired into recurrent circuits, shaped by long evolutionary pressure under stringent biological constraints. Mainstream machine learning, by contrast, predominantly builds models from extremely simple units, a default inherited from early neural-network theory. We treat this as a normative architectural question. How should one split a fixed parameter budget PP between the number of units NN, per-unit effective complexity kek_e, and per-unit connectivity kck_c? What controls the optimal allocation? This calls for a model in which per-unit complexity can be tuned independently of width and connectivity. Accordingly, we introduce the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons, chosen to mirror functional components of cortical neurons. The architecture allows for individually adjusting NN, kek_e, and kck_c and trains stably across orders of magnitude in scale. We evaluate the model on two qualitatively different sequence benchmarks: the neuromorphic SHD-Adding task and Enwik8 character-level language modeling. Performance improves monotonically along each of the three axes individually. Under a fixed budget, a clear non-trivial optimum emerges in their tradeoff, and larger budgets favor both more numerous and more complex neurons. A closed-form information-theoretic proxy captures these tradeoffs and attributes the diminishing returns at two ends to: per-neuron signal-to-noise saturation and across-neuron redundancy. A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the proxy. This suggests that the simple-unit default in ML is not obviously optimal once this tradeoff surface is probed, and offers a normative lens on cortex's reliance on complex spatio-temporal integrators.

Aaron Spieler, Georg Martius, Anna Levina

University of Tübingen, Germany · Max Planck Institute for Biological Cybernetics, Tübingen, Germany · Max Planck Institute for Intelligent Systems, Tübingen, Germany

Recurrent Neural NetworksNeural Scaling LawsNeural Network Expressivity
AskPDF
May 12, 2026

Hardware-Software Co-Design of Scalable, Energy-Efficient Analog Recurrent Computations

Always-on AI applications, from environmental sensors to biomedical implants, require ultra-low power consumption. Analog circuits offer a path to sub-microwatt inference, yet existing analog implementations are limited to feedforward architectures: extending them to recurrent dynamics has been considered impractical due to noise accumulation through temporal feedback. We demonstrate that this barrier can be overcome through hardware-software co-design. Specifically, we identify that Bistable Memory Recurrent Units (BMRUs), a class of Recurrent Neural Networks (RNNs) with discrete-valued outputs and hysteretic dynamics, admit an ultra-low power current-mode analog implementation which we design from first principles. The resulting circuit establishes a one-to-one correspondence between each learned parameter and a circuit element. The discrete outputs suppress analog noise by at least 20-fold at each cell boundary, breaking the noise accumulation that prevents analog recurrence. We reformulate BMRUs for first-quadrant operation with fixed thresholds, enabling the direct correspondence while preserving expressivity and trainability. Transistor-level simulations in 180 nm Complementary Metal-Oxide-Semiconductor (CMOS) show near-perfect agreement between software predictions and circuit-level behavior, with the software model thereby serving as a high-fidelity simulator of the physical hardware at low computational cost. We leverage this fidelity to conduct large-scale noise immunity and power scaling analyses: the power cost of adding recurrence scales linearly with state dimension, while the feedforward layers dominating total power scale quadratically, meaning recurrence is added at linear marginal cost relative to the feedforward backbone. End-to-end keyword spotting achieves sub-microwatt inference at the RNN core.

Arthur Fyon, Julien Brandoit, Loris Mendolia +3

University of Liège Liège, Belgium

Energy-Efficient MLKeyword SpottingRecurrent Neural NetworksHardware-Software Co-DesignNeuromorphic ComputingAnalog Computing
AskPDF
May 12, 2026

Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications

Sequence learning is dominated by Transformers and parallelizable recurrent neural networks (RNNs) such as state-space models, yet learning long-term dependencies remains challenging, and state-of-the-art designs trade power consumption for performance. The Bistable Memory Recurrent Unit (BMRU) was introduced to enable hardware-software co-design of ultra-low power RNNs: quantized states with hysteresis provide persistent memory while mapping directly to analog primitives. However, BMRU performance lags behind parallelizable RNNs on complex sequential tasks. In this paper, we identify gradient blocking during state updates as a key limitation and propose a cumulative update formulation that restores gradient flow while preserving persistent memory, creating skip-connections through time. This leads to the Cumulative Memory Recurrent Unit (CMRU) and its relaxed variant, the ααCMRU. Experiments show that the cumulative formulation dramatically improves convergence stability and reduces initialization sensitivity. The CMRU and ααCMRU match or outperform Linear Recurrent Units (LRUs) and minimal Gated Recurrent Units (minGRUs) across diverse benchmarks at small model sizes, with particular advantages on tasks requiring discrete long-range retention, while the CMRU retains quantized states, persistent memory, and noise-resilient dynamics essential for analog implementation.

Julien Brandoit, Arthur Fyon, Damien Ernst +1

Department of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium.

Energy-Efficient MLRecurrent Neural NetworksNeuromorphic Computing
AskPDF
May 11, 2026

A Random-Matrix Criterion for Initializing Gated Recurrent Neural Networks

Proper weight initialization prior to training has historically been one of the key factors that helped kick off the deep learning revolution. Initialization is even more crucial in "reservoir computing", where the weights of a readout layer are learned linearly while the reservoir weights are fixed and largely determine the richness, stability and memory of the resulting dynamics. In the infinite-width limit it has been shown that meaningful initializations are those sitting at an effective critical point of the randomly initialized model. The phase transition is controlled by the weight variance g2g^2 and separates an ordered phase from a chaotic one where information progressively degrades. Here we derive a simple criterion to estimate the critical gcg_c for a broad class of recurrent architectures and we show that it closely tracks the gain at which a gated-RNN reservoir achieves peak performance on a chaotic forecasting task. Finally, we argue that our criterion can serve as a design principle for future initialization schemes.

Tommaso Fioratti, Riccardo Marcaccioli, Francesco Casola

Capital Fund Management, 75007 Paris, France · Institute of Mathematics, EPFL, 1015 Lausanne, Switzerland

Gated Recurrent UnitNeural Network InitializationRecurrent Neural NetworksReservoir Computing
AskPDF
May 11, 2026

A Single-Layer Model Can Do Language Modeling

Modern language models scale depth by stacking layers, each holding its own state - a per-layer KV cache in transformers, a per-layer matrix in Mamba, Gated DeltaNet (GDN), RWKV, and xLSTM. Biological systems lean heavily on recurrence rather than on stacking. We ask how far that shape can go on language modeling. We propose Grounded Prediction Networks (GPN): one state vector revisited at every step through a single recurrent block - one FFN, one shared matrix memory. At 130M parameters, a 1-layer GPN+M reaches FineWeb-Edu perplexity 18.06, within 13% of a 12-layer Transformer++ (16.05) and 18% of a 10-layer GDN (15.34); a 2-layer variant closes the gap to 6%/11%. We do not match the deep baselines. Because the working context is a single vector, we can directly inspect its geometry: a persistent default-token direction, a content-bearing horizon of tens of tokens, and memory heads that split spontaneously into fast and slow retention pools.

Zanmin Wang

Language Model PretrainingRecurrent Language ModelsRecurrent Neural NetworksAutoregressive Language ModelingNext-Token PredictionLanguage Modeling
AskPDF
May 11, 2026

Explainability of Recurrent Neural Networks for Enhancing P300-based Brain-Computer Interfaces

Brain-Computer Interfaces (BCIs) based on P300 event-related potentials offer promising applications in health, education, and assistive technologies. However, challenges related to inter- and intra-subject variability and the explainability of Deep Learning (DL) models limit their practical deployment. In this work, we present the Post-Recurrent Module (PRM), an additional layer designed to improve both performance and transparency, incorporated into a Recurrent Neural Network (RNN) architecture for classifying P300 signals from EEG data. Our approach enables a dual analysis of spatio-temporal signals through both global and local explainability techniques, allowing us not only to identify the most relevant brain regions and critical time intervals involved in classification, but also to interpret model decisions in terms of spatio-temporal EEG patterns consistent with well-stablished neurophysiological descriptions of the P300. Experimental results show a 9% improvement in performance over state of the art, while also revealing the importance of inter- and intra-subject variability, in alignment with established neuroscience literature. By making model decisions transparent and efficient, we present a framework for explainable EEG-based models. This framework is not limited to more efficient P300 detection, but can be generalized to a wide range of EEG-based tasks. Its ability to identify key spatial and temporal features makes it suitable for applications such as motor imagery, steady-state visual evoked potentials, and even cognitive workload assessment.

Christian Oliva, Vinicio Changoluisa, Francisco B Rodríguez +1

Grupo de Neurocomputación Biológica, Departamento de Ingeniería Informática, Escuela Politécnica Superior, Universidad Autónoma de Madrid, c/ Francisco Tomas y Valiente, 11, Madrid, 28049, Madrid, Spain · Grupo de Investigación en Electrónica y Telemática, Universidad Politécnica Salesiana, Campus el Girón, Ave 12 de Octubre 24-22, Quito, 170143, Quito, Ecuador

Brain-Computer InterfacesEEG DecodingRecurrent Neural NetworksNeural Network Interpretability
AskPDF
May 9, 2026

Structured Recurrent Mixers for Massively Parallelized Sequence Generation

Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput. Here we introduce the Structured Recurrent Mixer, an architecture that allows for algebraic conversion between a sequence parallel representation at train time and a recurrent representation at inference, notably without the need for specialized kernels or device-specific memory management. We show experimentally that this dual representation allows for greater training efficiency, higher input information capacity, and larger inference throughput and concurrency when compared to other linear complexity models. We postulate that recurrent models are poorly suited to extended sequence length scaling for information-rich inputs typical of language, but are well suited to scaling in the sample (batch) dimension due to their constant memory per sample. We provide Mojo/MAX inference implementations of SRMs exhibiting 12x the throughput and 170x the concurrency of similarly powerful Transformers inferenced on vLLM, increases characteristic of Pytorch implementations resulting in a 30% increase in compute-constant GSM8k Pass@k. We conclude by demonstrating that SRMs are effective reinforcement learning training candidates.

Benjamin L. Badger

IBM

Recurrent Neural NetworksAutoregressive Language ModelingEfficient Language Model TrainingLLM Inference AccelerationRecurrent Language Models
AskPDF
May 8, 2026

Transfer Learning for Dead Fuel Moisture Prediction Using Time-Warping Recurrent Neural Networks

This paper proposes a time-warping transfer learning method, a technique for temporally rescaling the learned dynamics of a recurrent neural network (RNN) with a Long Short-Term Memory (LSTM) layer to enable task transfer across fuel moisture classes. Fuel moisture content (FMC) is divided into idealized classes based on characteristic lag time. Large quantities of real-time data are available for 10h fuels from sensors on weather stations, but observations of other fuel classes are sparse in space and time. We use transfer learning to adapt an RNN pretrained on 10h FMC to predict FMC for 1h, 100h, and 1000h fuels. We validate this method using data from a landmark field study conducted in Oklahoma that was used to calibrate the state-of-the-art Nelson fuel moisture model.

Jonathon Hirschi, Jan Mandel, Adam Kochanski

Department of Mathematical and Statistical Sciences, University of Colorado Denver, Colorado · Department of Meteorology and Climate Science, Wildfire Interdisciplinary Research Center, San Jose State University, San Jose, California

Recurrent Neural NetworksTransfer Learning
AskPDF
May 8, 2026

Convex Optimization of Spiking Neural Networks with Global and Approximation Guarantees

Spiking neural networks (SNNs) have been proposed as energy-efficient alternatives to deep neural networks. Their training is non-convex and involves a non-differentiable spike function, so existing methods offer no guarantee of reaching a global optimum. We address this issue by formulating SNN optimization as an equivalent convex optimization problem. Specifically, we show that the weight-decay-regularized training problem for a class of threshold RNNs, including leaky integrate-and-fire SNNs, has zero duality gap and the same optimal value as a convex program. Since the resulting convex programs can have exponentially many variables in the amount of data, we introduce a practical randomized surrogate and establish a high-probability guarantee on its approximation quality. Empirically, the best convex variant outperforms surrogate-gradient training on addition and first-last-XOR, matches its performance on sequential MNIST, and trails a reservoir baseline on base-3 and base-5 addition.

Himanshu Udupi, Calvin Ang, Sungyoon Kim +2

University of Illinois Urbana-Champaign

Neural Network OptimizationRecurrent Neural NetworksSpiking Neural Networks
AskPDF
May 8, 2026

Rethinking State Tracking in Recurrent Models Through Error Control Dynamics

The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite-horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.

Jiwan Chung, Heechan Choi, Seon Joo Kim

Yonsei University

Recurrent Neural NetworksState TrackingState Space Models
AskPDF
May 7, 2026

MinMax Recurrent Neural Cascades

We introduce MinMax Recurrent Neural Cascades (MinMax RNCs), a class of recurrent neural networks built from a novel form of recurrence over the MinMax algebra. We show that MinMax RNCs enjoy key properties that are difficult to obtain simultaneously: strong formal expressivity, efficient evaluation, stable dynamics, and non-vanishing state gradients. First, their formal expressivity corresponds to the regular languages, arguably the maximal expressivity for finite-memory systems. Second, in addition to evaluation in recurrent form, they also admit parallel-scan evaluation with logarithmic depth and linear work in the input length. Third, their states and activations are uniformly bounded for all sequence lengths. Fourth, their loss gradients exist almost everywhere and are uniformly bounded for all sequence lengths. Fifth, they do not exhibit vanishing state gradients: the gradient of a state with respect to a past state can retain norm one independently of the temporal distance between the states. Empirically, we find that these theoretical properties translate into strong practical performance. MinMax RNCs solve the considered synthetic tasks perfectly, generalise to long sequences, and outperform the recurrent baselines considered in our experiments. We also train a 112M-parameter MinMax RNC for next-token prediction, obtaining competitive performance for its size and providing initial evidence that MinMax recurrence can scale to real-world sequence-modelling tasks.

Alessandro Ronca

IRIS-AI

Recurrent Neural NetworksNeural Network Training DynamicsRecurrent Language Models
AskPDF
May 7, 2026

Understanding Cross-Language Transfer Improvements in Low-Resource HTR: The Role of Sequence Modeling

Handwritten Text Recognition (HTR) for Arabic-script languages benefits from cross-language joint training under low-resource conditions, particularly when using CRNN-based models that combine convolutional encoders with sequence modeling. However, it remains unclear whether these improvements are better explained by shared visual representations or sequence-level dependencies. In this work, we conduct a controlled architectural study of line-level Arabic-script HTR, comparing CNN-only models with CTC decoding and CRNN models under identical single-script and multi-script training regimes. Experiments are performed on Arabic (KHATT), Urdu (NUST-UHWR), and Persian (PHTD) datasets under low-resource settings (K in {100, 500, 1000}). Our results show a clear divergence in transfer behavior: while CNN-only models exhibit limited or unstable improvements, CRNN models achieve better performance under multi-script training, particularly in the most data-constrained regimes. Focusing on transfer improvements (delta CER) rather than absolute performance, we find that cross-language improvements are associated with sequence-level modeling, while sharing visual representations learned by the CNN encoder, corresponding to similarities in character shapes across scripts, alone appears to be insufficient. This finding suggests that contextual modeling plays an important role in enabling effective transfer in low-resource scenarios, and that similar behavior may extend to other low-resource language settings.

Sana Al-azzawi, Chang Liu, Nudrat Habib +2

Luleå University of Technology, Department of Computer Science, Electrical and Space Engineering , Luleå, Sweden · Luleá University of Technology, Department of Computer Science, Electrical and Space Engineering , Luleå, Sweden

Convolutional RNNsRecurrent Neural NetworksOnline Handwriting RecognitionOptical Character RecognitionSequence ModelingCross-Lingual Transfer
AskPDF
May 6, 2026

How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

We study signal propagation in linear recurrent models at finite width. While existing signal propagation theory relies predominantly on the infinite-width limit, it remains unclear for how long that approximation remains accurate when recurrent depth tt grows jointly with width nn. This question is especially relevant for modern recurrent sequence models, whose natural operating regime involves long input sequences, i.e., large tt. We derive exact finite-width formulas for the hidden state signal energies in linear recurrences under complex Gaussian initialization. Using these formulas, we identify the joint depth-width scaling regimes that govern signal propagation: (i) a subcritical regime t=o(n)t=o(\sqrt n), in which the infinite-width approximation remains valid; (ii) a critical regime t∼cnt\sim c\sqrt n, in which non-negligible deviations from infinite-width predictions appear and a nontrivial joint scaling limit emerges; and (iii) a supercritical regime t≫nt\gg \sqrt n, in which finite-width effects dominate. Thus, our results pinpoint the precise recurrent depth scale at which infinite-width theory breaks down in long-range linear recurrences. In turn, this shows when standard initialization schemes, such as Glorot, become unstable. More broadly, our results demonstrate that finite-width effects accumulate more rapidly with depth in recurrent models than in feedforward ones, leading to qualitatively different signal propagation behavior.

Mariia Seleznova

Ludwig-Maximilians-Universität München · Max Planck Institute of Quantum Optics

Neural Network InitializationRecurrent Neural NetworksLinear RNNs
AskPDF
May 6, 2026

A Comparative Analysis of Machine Learning and Deep Learning Models for Tweet Sentiment Classification: A Case Study on the Sentiment140 Dataset

The exponential growth of social media has created an urgent need for automated systems to analyze unstructured public sentiment in real time. This study compares a traditional Logistic Regression model using TF-IDF features with a deep learning Bidirectional Long Short-Term Memory (BiLSTM) architecture on a 10,000-tweet subset of the Sentiment140 dataset. Experimental results show that Logistic Regression outperformed BiLSTM, achieving an accuracy of 73.5% compared with 69.17%, while the deep learning model exhibited mild overfitting. These findings suggest that for medium-scale informal text data, classical machine learning with robust feature extraction can outperform more complex deep learning approaches. Finally, the trained models were integrated into an interactive web application using Streamlit and deployed on Hugging Face Spaces for public access.

Vita Anggraini, Cintya Bella, Bastian +3

Faculty of Science · Faculty of Science[cs.CL]

Social Media AnalysisSentiment AnalysisRecurrent Neural NetworksText Classification
AskPDF
May 5, 2026

Unifying Dynamical Systems and Graph Theory to Mechanistically Understand Computation in Neural Networks

Understanding how biological and artificial neural networks implement computation from connectivity is a central problem in neuroscience and machine learning. In neural systems, structural and functional connectivity are known to diverge, motivating approaches that move beyond direct connections alone. Here, we show that the spatial and temporal function of recurrent neural networks (RNNs) trained on hierarchically modular tasks can be recovered by modelling the network as a graph and analysing the multi-hop pathways between input and output units. In particular, decomposing these pathways by hop length reveals how the network temporally routes information. This perspective reframes regularisation: if function is implemented through multi-hop communication, then standard penalties such as L1 regularisation, which act only on individual weights, constrain single-hop structure rather than the multi-hop pathways that support computation. Motivated by this view, we introduce resolvent-RNNs (R-RNNs), which constrain multi-hop pathways and thereby induce temporal sparsity beyond that achieved by standard L1 regularisation. Compared with L1 regularisation, R-RNNs achieve improved performance by inducing temporal sparsity that matches the task structure, even when the task signal is sparse. Moreover, R-RNNs exhibit stronger sparsity-function alignment, reflected in their increased robustness under strong regularisation. Together, our results identify multi-hop communication as a key principle linking structure to function in recurrent networks, and suggest that sparsity should be defined over functional pathways rather than individual parameters.

Jatin Sharma, Dan F. M Goodman, Danyal Akarca

Department of Electrical and Electronic Engineering Imperial College London

Recurrent Neural Networks
AskPDF
May 5, 2026

Learning reveals invisible structure in low-rank RNNs

Learning in neural systems arises from synaptic changes that reshape the representations underlying behavior. While low-rank recurrent neural networks (RNNs) have emerged as a powerful framework for linking connectivity to function, a theoretical understanding of their learning process remains elusive. Here, we extend the low-rank framework from activity to learning by deriving gradient-descent dynamics directly in a reduced overlap space. We formulate a closed-form, low-dimensional system of ODEs that governs learning in this space, exact for linear RNNs and asymptotically exact for nonlinear RNNs in the large-NN Gaussian limit. Central to our analysis is a distinction between two classes of overlaps: loss-visible overlaps, which fully determine network activity, output, and loss, and loss-invisible overlaps, which do not affect function but are required to describe learning. We illustrate the consequences of this decomposition through two phenomena. First, we show that learning can serve as a perturbation that exposes differences in connectivity between functionally equivalent networks. Second, we show that loss-invisible overlaps can act as memory variables that encode training history, and characterize the conditions under which this occurs. Finally, we present several testable predictions for biological learning experiments derived from our theory.

Yoav Ger, Omri Barak

Ruth and Bruce Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion – Israel Institute of Technology, Haifa, Israel

Neural ProcessesRecurrent Neural NetworksNeural Network Training DynamicsGradient Descent Dynamics
AskPDF
May 5, 2026

Symmetry, Defects, and Diffusion in Continuous-memory Recurrent Networks

Continuous-memory recurrent networks must preserve phase, position, or orientation despite model imperfections and state noise. We develop a geometric framework that separates three questions: how many memory coordinates are neutrally transported, how deterministic perturbations alter their finite-horizon stability, and how ambient noise is decoded along them. Exact per-input equivariance transports analytical group tangents pathwise and, on a compact nondegenerate orbit stratum, yields at least q=dim⁡(G/H)q=\dim(G/H) zero group-tangent Lyapunov exponents under stationary ergodic driving. For imperfect dynamics, a four-block tangent/normal decomposition gives local and finite-horizon bounds on tangent growth and subspace rotation, distinguishing first-order direct damage from second-order leakage through contracting normal directions. For noisy dynamics, a specified decoder maps ambient covariance QQ to coordinate covariance ZQZ⊤ZQZ^\top; under isotropic noise and fixed tangent energy, least-squares decoding and scaled-isometric action geometry minimize local diffusion. Local and finite-horizon evaluations include cases both within and outside the sufficient conditions. In a fresh twenty-seed T2T^2 replication, a decoder-covariance objective improves noisy horizon-256 memory in every pair while meeting a prespecified clean-error equivalence margin. Direct noise training also improves noisy memory but incurs a clean-error tradeoff. In coupled T4/T8T^4/T^8 integrators, the advantage of anisotropic-covariance over isotropic regularization reverses when evaluation noise becomes isotropic. These results connect continuous symmetry to measurable limits and design choices for recurrent memory under explicit dynamical, decoder, and noise assumptions.

Hanson Hanxuan Mo

Equivariant Neural NetworksRecurrent Neural NetworksContinuous Attractor Neural Networks
AskPDF
May 4, 2026

Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning

Building and maintaining state to learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs) have become a key point of interest for the state-building problem, and several large-scale reinforcement learning agents incorporate recurrent networks. While RNNs have become a mainstay in many RL applications, many key design choices and implementation details responsible for performance improvements are often not reported. In this work, we discuss one axis on which RNN architectures can be (and have been) modified for use in RL. Specifically, we look at how action information can be incorporated into the state update function of a recurrent cell. We discuss several choices in using action information and empirically evaluate the resulting architectures on a set of illustrative domains. Finally, we discuss future work in developing recurrent cells and discuss challenges specific to the RL setting.

Matthew Schlegel, Volodymyr Tkachuk, Adam White +1

University of Alberta

Reinforcement LearningRecurrent Neural Networks
AskPDF