Recurrent Neural Networks

Also known as RNN

Ask about this topic

Overview

Recurrent neural networks study how a model can process ordered inputs by repeatedly updating an internal state, which is useful for streaming data, long-term memory, and decisions under partial observability. Common approaches include gated LSTMs and GRUs, linear and state-space recurrences, continuous-time and spiking networks, and weight-shared loops for iterative reasoning; training ranges from backpropagation through time to online learning and parallel scans. Researchers test these models on state-tracking and recall tasks, language modeling, time-series forecasting, control, and physical-system prediction. A central tension is how to preserve useful information and stable learning over long sequences while keeping computation efficient and matching the accuracy and flexibility of Transformers.

Latest papers

CardsList

303 papers47 papers in the last four weeks, up 15% on the four weeks before. 0.3% of all new papers.

Aug 26, 2026

Scalable Multi-GPU Simulation of 3D Multicellular Growth with RNN-Based Workload Balancing

Detailed multicellular growth simulations based on subcellular element models (SEMs) can capture complex tissue development, but their element-level interactions impose substantial computational cost. This work presents a scalable multi-GPU framework for 3D multicellular growth simulation that combines GPU acceleration, spatial binning, domain decomposition, and workload-aware partitioning. Cell movement, growth, and division continuously reshape the spatial workload distribution, causing initially balanced partitions to become inefficient over time. To address this, we introduce an RNN-based load-balancing controller that observes recent per-rank execution times and partition states and learns residual corrections to a reactive boundary-adjustment rule. The controller is trained offline in a differentiable surrogate of the load-balancing loop with randomized workload dynamics, requiring no measured execution traces for training. We evaluate the framework in terms of single-GPU acceleration, multi-GPU computation scaling, controller-level load-balancing behavior, and end-to-end simulation performance, with comparisons against static partitioning, reactive load balancing, and conventional time-series prediction baselines. A representative embryonic epidermal development use case further demonstrates the type of spatially and temporally evolving workload targeted by the framework. In our evaluation, GPU acceleration with spatial binning accelerates the interaction computation by roughly three orders of magnitude over a serial CPU baseline. RNN-guided load balancing reduces the mean global imbalance from 11.3% under static partitioning to 3.5%, lowers end-to-end runtime by 9.0% relative to static partitioning, and reduces slice migration by 7.7x compared with the reactive baseline, showing that history-aware control can improve workload balance while avoiding unnecessary repartitioning.

Matvey Moisseyev, Huijing Du, Dandan Zheng +2

School of Computing, University of Nebraska–Lincoln · Department of Mathematics, University of Nebraska–Lincoln · Department of Radiation Oncology, University of Rochester Medical Center +2

Domain Decomposition MethodsRecurrent Neural Networks
AskPDF
Aug 25, 2026

Steering Recurrent Reasoners at Inference Time with Readout Feedback

Recurrent models, which repeatedly update latent states with shared computation blocks, have emerged as powerful architectures for solving complex reasoning tasks. Existing inference-time methods scale computation by running more steps or sampling more trajectories, but ignore information revealed within each trajectory. Here we show that recurrent models can be improved at inference time by using their own readout probabilities to steer latent dynamics without retraining. We introduce Readout Feedback (RoFB), a test-time intervention that converts intermediate predictions into token-wise pairwise coupling forces injected into the latent dynamics. Across three recurrent models (AKOrN, ItrSA++, TRM) on Sudoku and Maze, RoFB yields clear gains in four of six model-task pairs and a small gain in one. The four positive pairs achieve performance unattainable by merely running more steps or selecting from multiple trajectories, at comparable or lower computational cost. These results suggest that closed-loop steering of latent dynamics can serve as a complementary inference-time control mechanism for recurrent reasoning models.

Shunsuke Kamiya, Masanori Koyama, Seongcheol Jeong +5

Graduate School of Engineering, The University of Tokyo

Inference-Time GuidanceRecurrent Neural Networks
AskPDF
Aug 24, 2026

Free-Energy-Gated Plasticity for Real-Time Online Motor Learning in Physical Human-Robot Interaction

Fully online embodied learning requires synaptic adaptation to acquire new behaviors while preserving previously learned dynamics during ongoing interaction. We extend the Predictive-Coding-inspired Variational Recurrent Neural Network (PV-RNN) to continuously adapt its synaptic weights and propose Free-Energy-Gated Plasticity (FEGP), which regulates the effective learning rate according to variational free energy. In real-time physical human-robot interaction, a randomly initialized network acquired three cyclic motor patterns without offline pretraining, replay, or task-boundary signals, with all three patterns emerging in autonomous rollouts. Controlled experiments over ten randomized teaching streams and five network initializations per stream showed that FEGP substantially improved repertoire coverage and retention of previously acquired patterns after they left the recent observation window. Neither a constant learning rate matched to the gate's time-averaged effective rate nor replay of the same gain values with disrupted temporal organization reproduced these improvements. These results indicate that the temporal allocation of plasticity relative to model-environment mismatch, rather than simply its average magnitude or distribution, is critical for maintaining previously acquired behaviors during continued online learning.

Hiroki Sawada, Jun Tani

Cognitive Neurorobotics Research Unit, Okinawa Institute of Science and Technology Graduate University, Okinawa, Japan 904-0302

Online Continual LearningContinual Robot LearningRecurrent Neural NetworksHuman-Robot InteractionPredictive Coding
AskPDF
Aug 23, 2026

SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Delta-Rule recurrent models maintain a fixed-size state, enabling O(1)O(1) inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: localized norm explosion atop a relatively sparse substrate, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose State Anomaly Neutralization (SANE), which applies adaptive tanh⁡\tanh compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range (3≤α≤53 \le α\le 5), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over 24,000×24{,}000\times, SANE retains functional reasoning (33.4633.46--35.5635.56) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds (α≥8α\ge 8) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

Qingwen Lin, Boyan Xu, Xiao Liu +2

School of Computer Science, Guangdong University of Technology · Yuanshi Intelligence · College of Science, Shantou University

Recurrent Neural NetworksGated Delta Networks
AskPDF
Aug 19, 2026

DevGRU: Depth-guided Visual Navigation using a Collision-aware Recurrent Model

Existing visual navigation models often aim to develop foundation models that can generalize robot navigation across diverse platforms. However, many of these models are prone to collisions when deployed in complex indoor environments, particularly in structured layouts and narrow passages. To address this problem, we propose a depth image- and point-goal-conditioned navigation system, DevGRU. The proposed system employs an action predictor (AP) that generates collision-aware future trajectories, enabling effective avoidance of immediate obstacles. In conjunction with a collision predictor, the AP further compensates for errors accumulated in the goal pose estimation and proactively mitigates future deviations. To evaluate our method, we conducted experiments across nine different scenes and three state-of-the-art approaches - ViNT, NoMaD, and NavDP - as well as four additional variants of ViNT and NoMaD. In terms of navigation performance, DevGRU significantly outperforms ViNT and NoMaD by a large margin. In addition, the proposed model has a relatively small number of trainable parameters, resulting in the fastest inference time among the baselines, particularly outperforming NavDP by 7x in model size and 17x in inference time.

Kyung Min Han, Eunsom Kim, Young J. Kim

Gated Recurrent UnitRobot NavigationRobot Collision AvoidanceRecurrent Neural Networks
AskPDF
Aug 18, 2026

Think Shallow, Solve Deep: Controlling Recurrent Dynamics for Reliable Test-Time Depth

Recurrent-depth reasoners aim to solve harder problems by iterating their update longer at test time, but additional iterations can improve, preserve, or degrade an answer. We show that a measurable property of the trained operator, its finite-time dynamical regime (estimated as settling, marginal, or drifting), indicates which of these occurs. We give a sufficient condition for depth-safety: once an operator's per-step displacement is small relative to the decoder margin, the decoded answer cannot change under further iterations. Empirically, on algorithmic tasks trained from 800800 unaugmented examples per difficulty tier, settling operators do not degrade with added depth, and on some tasks convert it into higher accuracy on harder unseen instances (Sudoku, 0.190.19 to 0.340.34 past the training horizon). A single terminal fixed-point objective moves the regime and the depth behavior together: removing it induces drift and removes the gains, and adding it to a generic recurrence yields depth-safe extrapolation on carry propagation. We give four operational criteria for useful test-time depth, use them to catalogue failure modes, and, as a consistency check, apply the same measurements to Huginn-3.5B, which falls in the non-settling family.

Ivan Viakhirev, Kirill Borodin, Amirah Almutairi +3

FRC RAS, Saint-Petersburg, Russia · ITMO, Saint-Petersburg, Russia · lab260, Moscow, Russia +4

Test-Time ScalingRecurrent Neural NetworksNeural Algorithmic ReasoningAlgorithmic Reasoning
AskPDF
Aug 18, 2026

Dynamic Compression in Recurrent Networks

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise across possible future demands. We introduce dynamic compression, which allows a recurrent model to selectively revisit past tokens and revise its fixed-size state through additional recurrent updates. The model need not preserve every part of the history at uniformly high fidelity in its recurrent state, because lower-fidelity information can be revisited from the retained raw sequence when it becomes relevant. We study this in a controlled setting where the model first learns multiple functions in-context and, later in the same sequence, encounters a series of few-shot tasks that each require it to identify and reuse one of those functions. A single-pass model must preserve every function at sufficient fidelity for any future task, whereas selective re-scanning allows the model to revisit and refine only the function currently needed. We find that dynamic compression substantially reduces the recurrent state required for accurate reuse and scales more favorably as the number of stored functions grows. These results demonstrate a computation--memory tradeoff in which recurrent models can spend more computation revisiting their history to make more effective use of a fixed-size state.

Jyothish Pari, Ryan Bahlous-Boldi, Pulkit Agrawal

Improbable AI Lab, Massachusetts Institute of Technology

Recurrent Neural NetworksModel Compression
AskPDF
Aug 11, 2026

Predictive Allostatic Organization in Recurrent and Spiking Agents Under Partial Observability

Adaptive behavior under partial observability depends on internal organization that carries information beyond the current observation. Drawing on Barrett and Miller's account of categorization as predictive, compressive, functionally organized, and allostatically constrained, we test whether recurrent and spiking agents develop internal states with corresponding computational properties. Agents operate in an energy-constrained foraging task requiring resource acquisition, threat avoidance, contact-dependent consumption, and regulation of an internal energy variable. In a frozen benchmark, learned agents outperform random and heuristic baselines; the trace-augmented recurrent policy is strongest overall, while spiking variants show stress-specific differences. Early internal dynamics predict later full-safe-efficient success above permutation baseline, reaching a maximum ROC-AUC of 0.802. Reduced PCA subspaces retain behaviorally relevant information. Feature-family controls show that predictive signal is distributed across trace, policy-head, internal-dynamics, observation, and allostatic variables, and low-energy state remains strongly decodable after explicit energy-related features are removed. Evaluation-time perturbations to temporal state, sensory information, operating conditions, and allostatic mechanisms alter behavior and/or internal prediction. Seed-balanced event probes show weaker but measurable information about future contact, successful consumption, and threat events, alongside strong low-energy decoding. We interpret this pattern as a computational analogue of predictive allostatic organization: distributed control regimes that are predictive, energy-sensitive, action-relevant, and partly causally involved, without claiming biological validation or discrete symbolic categories.

Frederick Hayes

Independent Researcher San Antonio, Texas, USA

Partially Observable RLRecurrent Neural NetworksState Representation Learning
AskPDF
Aug 10, 2026

Recurrent Neural Networks Beyond Time: Learning from Multiple Ordered Projections

Recurrent neural networks (RNNs) are widely used for sequence learning, yet their application is commonly associated with temporal data, although recurrent computation fundamentally operates on ordered sequences rather than on time itself. Building on this observation, we introduce the Ordered Structural Dependency Hypothesis (OSDH), which proposes that multiple admissible orderings of the same observations may reveal complementary structural dependencies inaccessible through a single sequential organization. To operationalize this hypothesis, we propose the Independent Structural Expert Principle (ISEP), whereby projection-specific sequence models are trained independently before their learned representations are integrated through a dedicated fusion model. As a concrete realization, we present Structural Evolution RNNs (SE-RNNs), which employ conventional RNNs as projection-specific structural experts while preserving the underlying recurrent computation unchanged. Proof-of-concept experiments on three synthetic datasets with substantially different levels of structural complexity demonstrate that the proposed architecture consistently benefits from multiple ordered projections when hidden structural dependencies are present, while remaining competitive on simpler datasets. Since OSDH is independent of the underlying sequence-processing model, the proposed framework naturally extends beyond recurrent networks and may be instantiated using alternative architectures. The results suggest a general computational perspective for exploiting complementary ordered representations across diverse structured learning problems.

Vagan Terziyan, Artur Terziian, Oleksandra Vitko

Faculty of Information Technology, University of Jyväskylä, FINLAND · Faculty of Informatics and Statistics, Prague University of Economics and Business, Czech Republic · Department of Artificial Intelligence, Kharkiv National University of Radio Electronics, Ukraine

Multi-View LearningRecurrent Neural NetworksSequence Modeling
AskPDF
Aug 5, 2026

Effective pruning of task-trained recurrent neural networks using noisy fluctuations and connection rescaling

The pruning of network connections is key to brain function but, despite its importance, there exist few biologically-plausible pruning rules with demonstrated good performance. In this work we evaluate noise-prune, a recently introduced unsupervised local pruning rule for recurrent networks that uses noisy fluctuations to determine the importance of connections. Noise-prune has previously only been empirically tested on random networks without a specific computational function. We show that noise-prune preserves task-performance in task-trained recurrent neural networks, greatly outperforming a strategy that only uses the magnitude of connections and performing on par with or exceeding a non-local strategy that uses second-order information. Rather than deterministically removing connections that fall below a certain threshold importance, noise-prune samples connections to preserve based on their importance and strengthens retained connections to preserve average synaptic strength. We show that this sampling and rescaling is essential to good performance, but that the optimal empirical degree of rescaling is lower than that predicted by the original theoretical argument. Our work thus validates noise-prune as a biologically-plausible pruning rule for functional recurrent network architectures and characterizes its optimal parameter settings.

Sanjith Senthil, Rishidev Chaudhuri

1The Harker School, San Jose, CA 95129, USA · Department of Neurobiology, Physiology and Behavior, University of California, Davis, Davis CA 95616 · Department of Mathematics, University of California, Davis, Davis CA 95616 +1

Recurrent Neural NetworksBiologically Plausible LearningNeural Network Pruning
AskPDF
Aug 5, 2026

Benchmarking Deep Learning Models for Dense Event Classification of Offshore Wind Infrastructure in Sentinel-1 Time Series

Monitoring of offshore wind energy infrastructure life cycles, especially during the deployment phase, is an important contribution for stakeholders to make informed decisions in a phase of increasing deployment activities. ESA's Sentinel-1 Synthetic Aperture Radar (SAR) mission produces large data archives that enable the global monitoring of offshore wind infrastructure. Turning these high-volume archives into information requires algorithms that automatically extract single event labels from dense time series at a global scale. In this study, we present a structured comparison of ten deep learning model-training variants for the dense classification of Sentinel-1 based offshore wind infrastructure time series, aiming to advance rule-based event classification of this task. We trained LSTM, Transformer, and fully connected model variants with monotemporal, unidirectional, and bidirectional context awareness, each with and without self-supervised pretraining. Among these, the supervised BiLSTM performs best, raising the target AUC score from 0.7853 for the rule-based baseline to 0.8509, and the perfect match rate from 0.3508 to 0.5063. Combining the BiLSTM predictions with the existing baseline labels in a label-transition-minimising ensemble further improves agreement with the test data. Using these improved labels, we isolate the deployment phase of individual turbines at a global scale and conduct a regional and subregional analysis covering 2016-01-01 to 2025-03-31, reporting median deployment durations of 84 d (China), 242 d (EU), and 258 d (UK). Deployment-related drivers, including legal regulations such as subsidies, and environmental conditions, emerge clearly from the analysed results across multiple spatial scales.

Thorsten Hoeser, Felix Bachofer, Claudia Kuenzer

Earth Observation Center (EOC), German Aerospace Center (DLR), Oberpfaffenhofen, 82234 Wessling, Germany · Institute for Geography and Geology, University of Wuerzburg, 97074 Wuerzburg, Germany

Time Series ClassificationRecurrent Neural NetworksDense Prediction
AskPDF
Aug 4, 2026

State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking

Transformer-based architectures have dominated sequence modeling, largely due to the expressive power of attention mechanisms. However, for a class of deterministic state tracking tasks---such as parity checking, modular counting, and parenthesis matching---attention may be overkill. In this paper, we show that state propagation alone is sufficient. We propose the Complex State Propagator (CSP), a minimalistic recurrent architecture that only propagates hidden states across layers without output projections at intermediate steps. The state is represented as a complex-valued vector, updated via input-dependent rotations in the complex domain. To enable deep propagation without gradient vanishing or degradation, we introduce a block-level skip connection alongside element-wise complex normalization and SiLU activation at sequence boundaries. Applied with Focal Loss, CSP achieves 100% accuracy with perfect F1 scores across canonical tasks.

Xiaohe Li, Yang Lu

GuangDong Police College · Department of Computer Science and Technology, School of Informatics, Xiamen University

Recurrent Neural NetworksState TrackingState Space Models
AskPDF
Aug 3, 2026

Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

The ability to robustly maintain and update continuous variables is a hallmark of working memory. While classical continuous attractor networks suffer from severe fine-tuning fragility, standard artificial recurrent neural networks (RNNs) like GRUs and LSTMs typically fail to stably learn continuous manifolds, instead shattering the state space into discretized point attractors. To bridge this gap, we draw inspiration from divisive normalization, a canonical neural computation widely observed across cortical circuits, and propose the Recurrent Divisive Normalization Network (RDNN), a minimal and algebraically isolated model of dynamic division. Through dynamical systems analysis on canonical working memory tasks, we demonstrate that this biophysical constraint allows the network to converge to robust, high-fidelity slow manifolds. Furthermore, we analyze the gradient dynamics of divisive normalization during Backpropagation Through Time (BPTT), showing that it introduces an activity-dependent local gradient scaling. This scaling dampens parameter updates in highly active regimes, which empirically aligns with a significant self-compression of the network's effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization. Finally, ablations demonstrate that while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs. Our findings identify divisive normalization not merely as a biological artifact, but as a critical computational mechanism for learning high-fidelity continuous representations.

Zhaotian Gu, Jie Su, Weiwei Wang +3

School of System Science, Beijing Normal University · Qiyuan Laboratory · State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University

Recurrent Neural NetworksContinuous Attractor Neural NetworksNeural Network Training Dynamics
AskPDF
Aug 2, 2026

Interpretable Machine Learning for Traffic Congestion Prediction: Unveiling the Impact of Different COVID-19 Periods

Traffic congestion prediction is essential for congestion mitigation, but the COVID-19 pandemic and related control measures altered travel behavior and increased prediction complexity. This study predicts congestion in Alameda County, California, during pre-lockdown, lockdown, and post-lockdown periods. Weather, seasonality, and COVID-19 variables are incorporated, and Recursive Feature Elimination with Cross-Validation is used to select important features and reduce overfitting. Support vector regression, multiple linear regression, recurrent neural networks, and long short-term memory networks are trained and optimized. Because LSTM is more sensitive to hyperparameter settings, an adaptive parameter selection approach is used, while SVR and RNN are manually tuned. Performance is evaluated using Normalized Root Mean Square Error. Bidirectional LSTM consistently performs best across all periods because it captures temporal dependence in both directions. Integrated Gradients is used to interpret Bi-LSTM predictions, and SHapley Additive exPlanations is applied to SVR. New COVID-19 cases have a mainly negative effect on congestion during lockdown and post-lockdown, likely due to greater risk awareness, voluntary travel reduction, and compliance with mobility restrictions. In the post-pandemic period, higher hospitalization reduces travel and congestion, while higher fuel prices do not prevent a shift toward private vehicles and therefore increase congestion.

Dan Zhu, Chi Sin Ng, Litian Xie +1

Department of Civil and Environmental Engineering, National University of Singapore, Singapore · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore

Time Series ForecastingRecurrent Neural NetworksTraffic Forecasting
AskPDF
Jul 29, 2026

Phoneme- vs. Character-Level Targets and Selective State-Space Models for Intracortical Brain-to-Text

State-of-the-art intracortical brain-to-text systems pair a neural-sequence phone decoder with an external language model. Two design axes remain underexplored: whether selective state-space models (Mamba) improve on recurrent decoders, and how the output target (phonetic vs.\ character) interacts with that choice. On the public Brain-to-Text '25 benchmark, we study a controlled 2x2 grid (GRU vs.\ hybrid Mamba decoder; phonetic vs.\ character targets) trained with a CTC objective under one reproducible protocol. The recurrent baseline remains strongest: the best phonetic GRU reaches 12.62% PER and 21.19% WER, while the best textual GRU after LM rescoring reaches 13.39% CER and 26.28% WER. The Mamba hybrid is competitive but does not surpass it. Ablations isolate architectural contributions, and error analysis shows representation-dependent failures: articulatory-like phoneme confusions vs.\ lexical and word-boundary errors.

Lucas Zamora Vera, Jose A. Gonzalez-Lopez

Universitat Oberta de Catalunya (UOC), Spain · Dpt. of Signal Theory, Telematics and Communications, University of Granada, Spain · Research Centre for Information and Communication Technologies (CITIC-UGR), University of Granada, Spain

Selective SSMsNeural DecodingBrain-to-Speech DecodingRecurrent Neural Networks
AskPDF
Jul 29, 2026

The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy

Spiking neural networks (SNNs) are promoted as an energy-efficient substrate because sparse, event-driven activity replaces dense multiply-accumulates with cheap accumulates. We argue the energy dividend of sparsity is not a property of SNNs but of the task. Holding architecture fixed and swapping only the hidden unit (continuous vs. leaky-integrate-and-fire), plus a two-sided target-firing-rate probe, we measure how far activity can be pushed down before quality breaks. Low-load feed-forward perception sparsifies to 5% firing at no accuracy cost; a recurrent language model cannot go below ~50% -- the recurrent state must stay active to carry information. A spiking Transformer, by contrast, sparsifies freely to 2% (3 seeds) -- so the ceiling is a property of recurrent compression, not sequence modeling. Attention escapes the floor only by storing the full key-value cache, trading a firing floor for a memory wall: on neuromorphic hardware, recurrence and attention pay on different axes, neither escapes. We formalize the ceiling with an information-theoretic bound rho >= H_b^{-1}(log2 M / H) and confirm its predictions: the floor rises with memory load, falls with state width, and (refuting a naive memory-only reading) rises with task difficulty. A layer-wise input floor further caps op reduction under dense input, isolating event-driven perception as where neuromorphic hardware wins.

Zeyu Wang

Georgia Institute of Technology

Efficient Neural Network InferenceSpiking TransformersActivation SparsityRecurrent Neural NetworksSpiking Neural NetworksNeuromorphic Computing
AskPDF
Jul 25, 2026

Sensitivity Analysis of GRU, LSTM and Transformer Encoder in Classification of Automated Driving Systems

Automated driving systems (ADSs) are becoming ubiquitous. Future Software Defined Vehicles (SDVs) may be able to run multiple ADSs, both native and aftermarket such as Comma.ai's Openpilot. Monitoring systems to independently verify which automated driving system is active are important for safety monitoring, regulatory compliance, insurance assessment, and anomaly detection. In this paper, we first evaluate the effectiveness of three sequence-based classification models: Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM) networks, and a Transformer encoder model for identifying Level 2 automated driving systems using vehicle telematics data alone: Comma Openpilot, Tesla Autopilot, and Cadillac Super Cruise, along with manual driving. All three models achieve strong clean-data performance with macro F1-scores of 0.92 (GRU), 0.90 (LSTM), and 0.93 (Transformer encoder model) when trained on clean data; threat-matched training yields 0.904-0.916 macro F1 with only a modest clean-data penalty. Second, we introduce a modular robustness evaluation framework that simulates realistic telematics degradation through five corruption families at five severity levels (L1-L5). Continuous channels are perturbed using additive white Gaussian noise with cumulative drift, correlated cross-channel noise, and temporal jitter. Binary event signals are subjected to burst loss, delayed transitions, spurious toggles and cross-feature inconsistencies inspired by communication errors. Robustness is measured using macro-F1, which gives equal weight to each class and is suitable for imbalanced multiclass evaluation. Our evaluation reveals a sharp failure-mode split: event-level corruptions reduce macro-F1 only slightly (greater than equal to 0.87 at L5), while temporal jitter collapses macro-F1 to 0.44-0.50 across GRU, LSTM, and Transformer encoder model.

Bidhya Shrestha, Christos Papadopoulos

University of Memphis

TransformerTime Series ClassificationRecurrent Neural Networks
AskPDF
Jul 22, 2026

Memoir: Should a Model Write to Its Memory While It Thinks?

Memoir combines per-sample fast memory, shared slow parameters, variable-depth latent recurrence, and a future-latent energy objective. We test its riskiest coupling: each pondering iteration may rewrite the fast tier that the same iteration reads. On procedural associative recall with key interference, we compare a coupled arm against an otherwise identical read-only pondering arm. Both arms contain 81,738 parameters, including 76,362 trainable parameters, and use matched declared forward multiply-accumulate counts, data, optimizer, schedule, and seeds. After 240 training steps across 12 seeds, coupled recall is 0.5203 with a 95 percent interval of [0.4522, 0.5883], while read-only recall is 0.6557 with [0.5953, 0.7160]. The arms are paired per seed, and the read-only lead of 0.1354 gives a paired t of 3.23 on 11 degrees of freedom with a 95 percent interval of [0.0431, 0.2277] on the difference, winning on 10 of 12 seeds. After 960 steps across 8 seeds, both arms reach 1.0000, so the measured effect is a learning-speed penalty at a fixed budget, not a demonstrated capability penalty. That longer control is ceiling limited, leaving convergence on a non-saturating task unmeasured. A predicted failure in which memory rewriting corrupts the energy signal did not occur: the energy margin grew and held. Kernel restructuring also reduced delta-rule forward time from 0.907 ms to 0.351 ms on the stated device. Code and evidence are available at https://github.com/RightNow-AI/Memoir

Jaber Jaber, Osama Jaber

RightNow AI

Memory-Augmented Neural NetworksRecurrent Neural NetworksAssociative Memory
AskPDF
Jul 22, 2026

Cumsum-Composable Phase Transport for Low-Cost Streaming Keyword Spotting

State-space sequence models are attractive for streaming speech because they maintain compact recurrent state, but scan-style training kernels can have unfavorable constants for short audio tasks. We study cumsum-composable phase transport, a streaming-native temporal layer for keyword spotting. Each layer projects acoustic frames to complex channels, transports them by learned unitary rotations, accumulates a finite window using prefix differences, and applies a gated residual update. The same prefix representation gives exact batched training with ordinary cumulative sums and exact online inference with one prefix update per frame. Unitary transport is the key constraint: inverse rotations have norm one, keeping prefix terms well conditioned while memory is supplied by windows or block readouts. On Google Speech Commands v2 with 12 labels, mel+cumsum models retain competitive accuracy with compact baselines. The strongest single-seed run reaches 97.3% test accuracy; a 51.6K-parameter tied model also reaches 97.3%, and a 24.8K tied model reaches 96.8% versus 97.1% for a 25.6K MelCNNMaxPool baseline. In a matched cumsum-versus-scan benchmark, cumsum+window gives comparable accuracy, 94.82% versus 94.33%, while training 1.07x faster and reducing single-example latency from 7.09 ms to 5.01 ms on a Tesla T4. These results support cumsum phase transport as a simple low-cost temporal primitive for streaming keyword spotting.

Mahesh Godavarti

A Carrot, Inc

Keyword SpottingRecurrent Neural NetworksSpeech ProcessingSequence Modeling
AskPDF
Jul 22, 2026

Time Series Network Utilization KPI Forecasting Using Advanced AI/ML Models

The rapid proliferation of data-intensive applications, cloud infrastructure, and IoT ecosystems has made proactive resource provisioning critical for maintaining optimal network performance. However, network administrators face a constant battle against capacity constraints, where traditional reactive approaches fail to accurately anticipate traffic fluctuations. This inability to foresee demand leads to costly over-provisioning, unexpected downtime, and degraded quality of service directly impacting operational budgets and business continuity. To achieve efficient capacity planning, accurate forecasting of bandwidth utilization is essential. This study addresses the challenge by evaluating a diverse spectrum of models including seasonal decomposition, Prophet, Random Forest, XGBoost, Support Vector Regression, and advanced deep learning architectures like bidirectional and Convolutional LSTMs - using a common interface dataset benchmarked across MAPE, NRMSE, and R-square metrics. Ultimately, this research delivers actionable insights into the trade-offs between model accuracy and computational efficiency, empowering engineers, operators, and business owners to select the optimal forecasting model for their specific infrastructure needs.

Niraj Gadhe, Kirti Bhardwaj, Moulik Jain +2

Independent Researcher

Time Series ForecastingForecasting BenchmarksRecurrent Neural NetworksTraffic Forecasting
AskPDF
Jul 21, 2026

Real-time optimal control with shallow recurrent decoder networks

Controlling dynamical systems in real-time across multiple scenarios is critical to enabling adaptive control strategies, ensuring stability and efficiency. However, to tailor control actions in response to varying scenarios, traditional optimal control problems typically require several system simulations, which are often computationally demanding due to the high-dimensionality of the underlying spatio-temporal dynamics. In this work, we exploit SHallow REcurrent Decoder networks-based Reduced Order Modeling (SHRED-ROM) to synthesize a real-time closed-loop controller for high-dimensional and parametric dynamics, relying solely on limited state sensor readings. After training the model on a few optimal examples given by an expert demonstrator, SHRED-ROM mimics the expert behavior with effective distributed control actions in new scenarios, alleviating the curse of dimensionality. Moreover, a sensor forecaster is synthesized and used to close the loop at the latent level, thus efficiently mitigating possible sensor failures or delays. The performance of the proposed optimal control strategy is finally assessed on three challenging high-dimensional cases dealing with either parametric density control or fluid flow control.

Matteo Tomasetto, Francesco Braghin, J. Nathan Kutz +1

Department of Mechanical Engineering, Politecnico di Milano, Milano, Italy · Autodesk Research, London, UK · MOX - Department of Mathematics, Politecnico di Milano, Milano, Italy

Recurrent Neural NetworksReduced-Order ModelingOptimal Control
AskPDF
Jul 15, 2026

A Comparative Analysis of Machine Learning Models for Long and Short-Term Forecasting of the Egyptian Stock Market: A Focus on EGX30

This study concentrates on predicting stock prices in the Egyptian market, focusing on the EGX30, an influential financial hub in the Middle East. While most research focuses on global stocks, there's a growing need to understand stock trends in developing countries like Egypt. The study compares different machine learning models for forecasting EGX30 trends, covering short and long-term predictions. Using historical EGX30 data, including metrics like root mean squared error, Mean Absolute Percentage Error, and coefficient of determination, models like K-Nearest Neighbours, random forest, extreme gradient boosting, long short-term memory networks, and gated recurrent unit networks were evaluated. The goal is to determine the most effective models for EGX30 prediction, considering Egypt's unique market dynamics. Insights from this study aid investors in making informed decisions. Results show that the Gated Recurrent Unit (GRU) outperformed the other models in the one-week, one-month, and two-months while the eXtreme Gradient Boosting (XGBoost) model outperformed others in the one-day predictions, highlighting their usefulness in predictive analysis for financial markets. The study also showed the importance of using the ensemble techniques, especially in the long-term predictions which proved better results reaching 5 times the GRU in the two-month predictions. Additionally, the study notes the surprisingly good performance of K-Nearest Neighbours (KNN) on long-term predictions, suggesting its enduring relevance and potential for future applications in the fintech domains.

Muhammed Walid, Ahmed El-Naeimy, Hosam Moubarak +1

Cyber Physical Lab, Department of Computer Science & Engineering · Egypt-Japan University Of Science and Technology, Alexandria, Egypt · Accounting Department +1

Gated Recurrent UnitLong-Term Time Series ForecastingRecurrent Neural NetworksFinancial Time Series
AskPDF
Jul 13, 2026

FAD-SA-GRU: Enhancing Hate Speech Detection in Algerian Dialect Through Feature-Augmented Self-Attention GRU Networks

The widespread adoption of social media platforms has transformed online communication by enabling users to exchange information and opinions instantly. However, these platforms have also facilitated the dissemination of abusive and hateful content, posing major social, psychological, and ethical challenges. Hate speech can incite discrimination, harassment, and violence against individuals or communities based on attributes such as ethnicity, religion, gender, nationality, or political affiliation. Consequently, automatic hate speech detection has become a major research topic in natural language processing (NLP) and an essential component of content moderation systems. This paper investigates automatic hate speech detection in the Algerian Arabic dialect (Darija) on social media. This task remains challenging because of the dialect's linguistic diversity, characterized by the coexistence of Arabic, French, and Arabizi (Arabic written using the Latin alphabet). We compare four categories of text classification approaches: (1) traditional machine learning models using TF-IDF features, (2) deep learning models based on recurrent neural networks, (3) Transformer-based language models, including DziriBERT and multilingual BERT, and (4) a novel hybrid architecture, FAD-SA-GRU, which combines semantic representations from DZ FastText, DZ AraVec, and DziriBERT through multi-embedding fusion, followed by a self-attention-enhanced GRU encoder. Experiments on an annotated dataset of Algerian Darija social media comments for binary hate speech classification show that FAD-SA-GRU outperforms all baselines, achieving 93.2% accuracy, 93.4% precision, 91.0% recall, 92.1% F1-score, and 97.0% ROC-AUC. Results demonstrate the effectiveness of combining complementary embedding representations with attention-based sequence modeling for robust hate speech detection in low-resource dialectal Arabic.

Sara Yakoubi, Ikram Khalfallah, Kenza Khelkhal +1

USTHB · ATM Mobilis

Recurrent Neural NetworksHate Speech DetectionLow-Resource Language Processing
AskPDF
Jul 11, 2026

Emergent Generalization by Representation Learning in Artificial Neural Networks

Dimensionality reduction has proven powerful for identifying neural manifolds, which are low-dimensional structures underlying high-dimensional neural activity. These low-dimensional representations have improved the interpretability of population-level coding. Yet whether such low-dimensional representations are biologically relevant and confer functional advantages in learning systems, or merely reflect neuron-level activity, remains contested in neuroscience. We show that an explicit information bottleneck forcing a recurrent neural network to learn a low-dimensional representation is necessary for rotational and out-of-distribution generalisation in a time-series prediction task. Using information-theoretic measures of causal emergence, we characterise the dynamics of this representation across the memorisation-to-generalisation transition, finding a non-monotonic trajectory which shows an initial decrease, a minimum, and a subsequent rise to a maximum, even as prediction loss falls monotonically. This trajectory scales with task complexity, and the magnitude of emergent structure reliably predicts generalisation performance. Analysis of CA1 hippocampal activity in mice learning an alternating maze task reveals analogous non-monotonic emergence dynamics that track behavioural performance. Together, these findings indicate that the ability of neural networks to learn compact, distributed and emergent representations confers a functional advantage for generalisation, supporting a causal role for learned representations in cognition.

Hardik Rajpal, Dan Goodman

1I-X Centre for AI in Science, Imperial College London, W12 0BZ, UK · Department of Electrical and Electronic Engineering, Imperial College London, SW7 2AZ, UK

Neural Network GeneralizationRepresentation LearningInformation BottleneckRecurrent Neural NetworksNeural Encoding
AskPDF
Jul 11, 2026

Exploratory Analysis of Deep Learning Models for Forecasting Meteorological Parameters in the Agricultural Sector

Accurate meteorological forecasting is essential for agricultural planning, irrigation management, and environmental decision support. This study conducts a comparative evaluation of recurrent and hybrid deep learning architectures for multivariate forecasting of reference evapotranspiration (ET0ET_0), vapour pressure deficit (VPD), wind speed, and the sine and cosine components of wind direction. The analysis utilizes 134,376 hourly observations from Ioannina, Greece, spanning January 2011 to April 2026, sourced from ERA5 via the OpenMeteo Historical Weather API. Single and multi-layer GRU and LSTM networks are compared with hybrid 1D-CNN-GRU and 1D-CNN-LSTM models for two forecasting tasks: a 24-hour next-day forecast and a 168-hour week-ahead forecast. Performance is evaluated using normalized root mean squared error, the coefficient of determination, and a composite Weighted Quotient Score (WQS). The most effective purely recurrent models are a 64-unit LSTM for the 24-hour horizon, with a WQS of 0.816755, and a 1024-unit GRU for the 168-hour horizon, with a WQS of 0.779465. The hybrid CNN-GRU models achieved the highest overall scores of 0.827535 and 0.782863 for the 24-hour and 168-hour horizons, but with additionally more number of units respectively to LSTM models, while the CNN-LSTM models yield nearly identical results with substantially fewer parameters. Compared to the corresponding recurrent baselines, the hybrid models improve WQS by 1.22--1.63% at 24 hours and by 0.44--0.45% at 168 hours, indicating that convolutional feature extraction is more beneficial for short-term forecasting.

Piotr Sikora, Sotirios Kontogiannis

Faculty of Technical Physics, Information Technology and Applied Mathematics Lodz Universtiy of Technlogy · MicroComputer Systems Laboratory Dept. of Mathematics University of Ioannina

Convolutional RNNsMultivariate Time Series ForecastingRecurrent Neural NetworksWeather Forecasting
AskPDF
Jul 10, 2026

Robustly Invertible Nonlinear Dynamics and the BiLipREN: From Inversion-Based Control to Generative Trajectory Modelling

This paper proposes a new notion of robust invertibility for nonlinear dynamical systems, and introduces constructive parameterizations of recurrent neural network which are robustly invertible by design. We define robust invertibility as the existence of a causal inverse system such that both the forward and inverse systems are contracting and have bounded incremental input-output gains (the system is bi-Lipschitz), implying that both forward prediction and input reconstruction are robust to signal perturbations and initial-state mismatch. We construct robustly invertible recurrent models via series composition of static orthogonal layers and dynamic layers satisfying a strong input-output monotonicity property, and provide a differentiable neural network parameterizations in the form of the bi-Lipschitz recurrent equilibrium network (BiLipREN). Additionally, composition with dynamic orthogonal layers yields a nonlinear minimum-phase/all-pass (a.k.a. inner--outer) factorization. We illustrate the utility of the framework through a series of application examples in data-driven internal model control, dynamic surrogate loss learning, and signal-space normalizing flows, illustrating its utility for robust control, trajectory optimization, and generative modeling of complex trajectory distributions.

Yurui Zhang, Ruigang Wang, Ian R. Manchester

Australian Centre for Robotics (ACFR), and the School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney, Australia

Invertible Neural NetworksRecurrent Neural NetworksRobust Control
AskPDF
Jul 10, 2026

Letter Lemmatization: One-to-one and Banded RNNs for Reversing Character-Set Simplification and Abbreviation in Medieval Text

Medieval document transcribers have very different practices; on top of that, heterogeneous digitization policies have resulted in corpora where the character-set must be viewed as fluid. In this paper we address the problem of changing between character-sets in a flexible manner. We focus on one-to-one character mappings and train characterlevel one-to-one RNNs to undo them with self-supervision; recovering half the CER even with 20 text lines. We analyse the use of these one-to-one networks for HTR post-correction and we see that they obtain significant improvements while totally ignoring ins-dels. We then use the exact same networks with character-level alignment groundtruth compiled from parallel corpora in a training and inference mode we call Banded RNNs. We use such networks to successfully expand abbreviations in medieval charter transcriptions. Finally we introduce an elaborate heuristic which takes the characters of two arbitrary character-sets and defines a metric encapsulating what we consider to be semantic similarity of characters. We call the construction of such mappings letter lemmatization and present a rich Python library that efficiently performs all presented methods.

Anguelos Nicolaou, Maria Pia Tiseo, Tamas Kovacs +2

University of Graz, Graz, Austria · University of Basel, Basel, Switzerland · Universit`a degli Studi di Napoli Federico II, Naples, Italy

Recurrent Neural NetworksOptical Character RecognitionHistorical Document Processing
AskPDF
Jul 9, 2026

LEXIC: Lightweight On-Device Decoding of Reading Comprehension from Eye Movements

Predicting comprehension from eye movements could support adaptive reading interfaces. We present LEXIC, a compact recurrent model that predicts response correctness from fixation sequences, word frequency, and character length. It has 41.6K parameters and requires no language-model inference. Mean area under the receiver operating characteristic curve (AUROC) reaches 0.529 for Unseen Text and 0.554 for Unseen Reader on OneStop. Matched comparisons with AhnCNN show gains from both encoder redesign and lexical augmentation in these settings. Separate training and evaluation on SB-SAT also yield higher mean AUROC than AhnCNN. LEXIC occupies only 166 KB of model weights and requires 1.4 ms per trial on a single CPU core, supporting lightweight on-device inference.

Sumin Lee, Kyeonghun Kim, Subeen Lee +5

Seoul National University · OUTTA · NVIDIA

Recurrent Neural NetworksEye TrackingOn-Device Inference
AskPDF
Jul 8, 2026

Evaluating the Effect of Frame Rate in Sequence-Based Classification of Autism-Related Self-Stimulatory Hand Idiosyncrasies

Autism spectrum disorder (ASD) affects over 75 million individuals worldwide, yet scalable computational methods for remote behavioral screening remain limited. This study addresses two complementary challenges in automated detection of autism-related self-stimulatory behaviors from video: (1) identifying the optimal sequence-based neural network architecture and temporal sampling rate, and (2) characterizing data augmentation strategies for training on small behavioral datasets. For the first objective, long short-term memory (LSTM) and gated recurrent unit (GRU) models were trained on pose-derived features from the Self-Stimulatory Behavior Diagnosis (SSBD) dataset at frame sampling intervals of 1, 5, 15, 30, 45, and 90 frames. Both architectures exceeded prior convolutional neural network (CNN) baselines (62-76% accuracy), with peak accuracies of 97.5% (LSTM) and 98.75% (GRU) at a sampling interval of every 15 frames. For the second objective, ten data augmentation strategies were applied to an I3D transfer learning pipeline, with an ablation study quantifying the marginal contribution of each technique. Horizontal flip achieved the highest standalone accuracy (48.78%), while exclusion of upsampling from the augmentation pipeline produced the largest performance degradation, indicating its necessity for complex behavioral video augmentation. A personalized machine learning approach, in which per-subject models were trained and tested on temporally split segments of each video, produced consistent predictions (mean loss 1.84, SD 0.79). These results provide practitioners with concrete guidance on architecture selection, sampling rate, and augmentation strategy for video-based behavioral classification in data-scarce clinical domains.

Raunak Mondal, Peter Washington

School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Information and Computer Sciences, University of Hawaiʻi at Mānoa, Honolulu, HI, USA

Data AugmentationHuman Activity RecognitionAutism Spectrum DisorderRecurrent Neural NetworksVideo Action Recognition
AskPDF
Jul 8, 2026

Unveiling Public Opinion: A Study of Sentiment Analysis Using LSTM and Traditional Models

In this age of social media, sites like Twitter have become meeting places for people to share their views and feelings on a wide range of issues and current events as they unfold in real time. Sentiment analysis, a critical application of NLP, has become indispensable due to the massive influx of user-generated content, enabling the extraction of meaningful insights from the opinions and emotions expressed in textual data. Sentiment analysis on Twitter employs sophisticated computational techniques to categorize tweets into positive, negative, or neutral sentiments. This method not only examines individual expressions but also analyzes vast databases related to specific subjects or events. By spotting these emotions, machine learning models help improve public opinion interpretation and trend forecasting. This paper examines the effectiveness of various machine learning and deep learning approaches. Designed for this use, the system evaluates logistic regression, random forest, naïve bayes, gradient boosting, and LSTM networks, among other algorithms applied in sentiment classification. This work identifies the optimal sentiment analysis model using a Kaggle Twitter dataset that has been preprocessed through tokenization, lemmatization, and stopword elimination. Emphasizing the better performance of the LSTM approach, the model attained a training accuracy of 90.98%, a testing accuracy of 80.00%, and a micro-average ROC- AUC score of 0.92. These results show that the model outperforms conventional machine learning techniques in capturing contextual and sequential textual aspects.

Atiq Ur Rehman

Department of Economics and Decision Sciences, University of South Dakota, USA

Social Media AnalysisSentiment AnalysisRecurrent Neural Networks
AskPDF