Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks

Lorenzo Livi

cs.LGJun 28, 2026

Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks

Authors: Lorenzo Livi

Organizations: OPIT – Open Institute of Technology

Abstract

Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$ , and if it fades too fast the dependence cannot be learned from finite data. This fade is captured by an envelope $f(\ell)$ . An exponential fade makes the data needed to learn a lag- $\ell$ dependence grow exponentially, putting long horizons out of reach; a power-law fade keeps the cost polynomial. We show that the asymptotic decay class of $f(\ell)$ is not fixed by the architecture. Instead, it emerges from the coupling between the state dynamics and parameter dynamics, settling into either a collapsed regime (fast, exponential forgetting) or an extended, anti-collapsed regime (slow, power-law forgetting). The intuition is a competition within these coupled dynamics. Training drives the network's effective time scales toward short ones, while rare, heavy-tailed fluctuations of the learning dynamics push a few of them to very long values. The extended regime survives only when these heavy-tailed pushes are strong enough to balance the pull. We make this mathematically precise with a coarse-grained stochastic process and prove exactly when the extended regime exists. A single exponent, the spectral exponent~ $β$ , then governs both the spread of time scales and how slowly the network forgets. Realizing the regime in practice needs one more ingredient: the joint action of the architecture and the optimizer must be able to hold such a broad spread. A network whose capacity to generate broad time-scale spectra is severely constrained still collapses, even when supplied with strong heavy-tailed forcing. Heavy-tailed fluctuations thus act not as noise to be suppressed, but as the mechanism that sustains long-range learning.

Explore similar work

Dec 5, 2025cs.LG

Learnability Window in Gated Recurrent Neural Networks

We develop a statistical theory of temporal learnability in recurrent neural networks, quantifying the maximal temporal horizon

\mathcal{H}_N

over which gradient-based learning can recover lag-dependent structure at finite sample size

N

. The theory is built on the effective learning rate envelope

f(\ell)

, a function that captures how gating mechanisms and adaptive optimizers jointly shape the coupling between state-space dynamics and parameter updates during Backpropagation Through Time. Under heavy-tailed (

α

-stable) fluctuations, where empirical averages concentrate at rate

N^{-1/κ_α}

with

κ_α= α/(α-1)

, the interplay between envelope decay and statistical concentration yields explicit scaling laws for the growth of

\mathcal{H}_N

: logarithmic, polynomial, and exponential temporal learning regimes emerge according to the decay law of

f(\ell)

. These results identify envelope decay as the key determinant of temporal learnability. Slower attenuation of

f(\ell)

enlarges

\mathcal{H}_N

, while heavy-tailed fluctuations compress it by weakening statistical concentration. Moreover, envelope geometry outweighs dataset size: slowing the envelope's decay enlarges

\mathcal{H}_N

more than adding data, so more complex architectures that realize slower-decaying envelopes can be more data-efficient than simpler ones. Experiments across multiple gated architectures and optimizers corroborate these structural predictions.

Lorenzo Livi

Recurrent Neural Networks Gated Recurrent Unit Learning Rates Neural Tangent Kernel Temporal Difference

Ask PDF

Sep 16, 2026cs.LG

Learning-Induced Dynamical Transition in Recurrent Neural Networks

Learning in recurrent neural networks can fundamentally reshape their underlying dynamics, transforming initially chaotic activity into stable task-dependent behavior. We develop a non-equilibrium dynamical mean-field theory(DMFT) to describe this transition during learning. We show that a slow feedback-driven learning process generates an evolving effective feedback strength that drives the network through a transition from chaotic to stable dynamics defined by a bifurcation of the DMFT solution. By deriving the two-time correlation function throughout learning, we identify a critical feedback strength and a corresponding learning rate dependent critical time separating these regimes. The transition arises from the progressive deformation of an effective dynamical landscape by the growing learned feedback structure. Starting from the untrained state, the theory predicts the time evolution of the network output during training and shows quantitative agreement with numerical simulations.

Varun Vaidya

Learning Dynamics Recurrent Neural Networks Mean-Field Limit Transitions

Ask PDF

Sep 8, 2026cs.LG

Learning Length-Extrapolatable Recurrent Models

Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.

Hanwen Jiang

Recurrent Model Catastrophic Forgetting Backpropagation Credit Scoring

Ask PDF