cs.LGAug 31, 2026

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

Authors: Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi Dawson

Organizations: Independent Researcher, United Kingdom · Department of Electrical and Electronic Engineering, University of Buea, Cameroon · minoHealth AI Labs, Ghana

Abstract

The linear recurrent neural network (LRNN) is a simple model for studying how much memory a network builds up as it trains. For uncorrelated inputs, earlier work found that training itself settles the network between keeping the past and reacting only to the present. Real sequences are correlated, and we solve the learning dynamics exactly for correlated inputs. In the solution, keeping the past carries a cost. The whole effect of correlation lands on that cost. This cost reduces to the earlier one when inputs are uncorrelated and grows once they are positively correlated. Three findings follow. (1) Correlation reshapes the course of learning, not only its end. Memory builds, overshoots, and is partly removed, and the settled network keeps less of the past. (2) Memory switches off at a threshold set by one number, how much each input resembles the one just before it. Neither sequence length nor longer-range correlation moves this threshold. Memory is worth keeping only when the task needs the previous input more than the current input already supplies it through correlation with the past. (3) The best network changes too. Zero error demands a feedthrough, a path that passes the current input straight to the network's output and remembers nothing, and training builds it unprompted when given one spare hidden dimension. Our work turns one property of the input into a prediction of whether a network learns memory and explains why correlated data turns recurrent networks into change detectors.

Explore similar work

CardsList
May 5, 2026cs.LG

Learning reveals invisible structure in low-rank RNNs

Learning in neural systems arises from synaptic changes that reshape the representations underlying behavior. While low-rank recurrent neural networks (RNNs) have emerged as a powerful framework for linking connectivity to function, a theoretical understanding of their learning process remains elusive. Here, we extend the low-rank framework from activity to learning by deriving gradient-descent dynamics directly in a reduced overlap space. We formulate a closed-form, low-dimensional system of ODEs that governs learning in this space, exact for linear RNNs and asymptotically exact for nonlinear RNNs in the large-NN Gaussian limit. Central to our analysis is a distinction between two classes of overlaps: loss-visible overlaps, which fully determine network activity, output, and loss, and loss-invisible overlaps, which do not affect function but are required to describe learning. We illustrate the consequences of this decomposition through two phenomena. First, we show that learning can serve as a perturbation that exposes differences in connectivity between functionally equivalent networks. Second, we show that loss-invisible overlaps can act as memory variables that encode training history, and characterize the conditions under which this occurs. Finally, we present several testable predictions for biological learning experiments derived from our theory.
Yoav Ger, Omri Barak
Recurrent Neural NetworksSynaptic PlasticityLearning Dynamics
AskPDF
Sep 16, 2026cs.LG

Learning-Induced Dynamical Transition in Recurrent Neural Networks

Learning in recurrent neural networks can fundamentally reshape their underlying dynamics, transforming initially chaotic activity into stable task-dependent behavior. We develop a non-equilibrium dynamical mean-field theory(DMFT) to describe this transition during learning. We show that a slow feedback-driven learning process generates an evolving effective feedback strength that drives the network through a transition from chaotic to stable dynamics defined by a bifurcation of the DMFT solution. By deriving the two-time correlation function throughout learning, we identify a critical feedback strength and a corresponding learning rate dependent critical time separating these regimes. The transition arises from the progressive deformation of an effective dynamical landscape by the growing learned feedback structure. Starting from the untrained state, the theory predicts the time evolution of the network output during training and shows quantitative agreement with numerical simulations.
Varun Vaidya
Learning DynamicsRecurrent Neural NetworksMean-Field LimitTransitions
AskPDF
Dec 5, 2025cs.LG

Learnability Window in Gated Recurrent Neural Networks

We develop a statistical theory of temporal learnability in recurrent neural networks, quantifying the maximal temporal horizon HN\mathcal{H}_N over which gradient-based learning can recover lag-dependent structure at finite sample size NN. The theory is built on the effective learning rate envelope f(ℓ)f(\ell), a function that captures how gating mechanisms and adaptive optimizers jointly shape the coupling between state-space dynamics and parameter updates during Backpropagation Through Time. Under heavy-tailed (αα-stable) fluctuations, where empirical averages concentrate at rate N−1/καN^{-1/κ_α} with κα=α/(α−1)κ_α= α/(α-1), the interplay between envelope decay and statistical concentration yields explicit scaling laws for the growth of HN\mathcal{H}_N: logarithmic, polynomial, and exponential temporal learning regimes emerge according to the decay law of f(ℓ)f(\ell). These results identify envelope decay as the key determinant of temporal learnability. Slower attenuation of f(ℓ)f(\ell) enlarges HN\mathcal{H}_N, while heavy-tailed fluctuations compress it by weakening statistical concentration. Moreover, envelope geometry outweighs dataset size: slowing the envelope's decay enlarges HN\mathcal{H}_N more than adding data, so more complex architectures that realize slower-decaying envelopes can be more data-efficient than simpler ones. Experiments across multiple gated architectures and optimizers corroborate these structural predictions.
Lorenzo Livi
Recurrent Neural NetworksGated Recurrent UnitLearning RatesNeural Tangent KernelTemporal Difference
AskPDF