A theory of noisy linear recurrent neural network learning dynamics
Authors: John J. Vastola, Kanaka Rajan
Organizations: Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.
Figures & tables
Figure 1: Theoretical setting. ( a ) We study a noisy linear RNN which receives a scalar white noise signal x(t) as input and produces an output y(t) . The dynamics of individual ri are noisy (bottom). ( b ) On average, the network filters its input. ( c ) Output variance is input-independent and eventually saturates (assuming the network is stable). ( d ) We assume the network is trained to optimize an objective with three terms: a filter-matching term that incentivizes matching its average behavior to that of a target filter f∗(t) , a noise reduction term, and a regularization term.
Figure 2: Noise produces representational pressure. ( a ) Two representations with the same filter but different readout vectors differ in noise level if S is anisotropic. Pictured is a schematic 2D example with a high-noise direction (horizontal) and a low-noise direction (vertical). One can reduce noise by rotating v (white arrow). ( b ) The noise term Jnoise in the schematic example as a function of v orientation. Jfilter is unaffected as long as the filter remains the same. ( c ) Jnoise and Jfilter as a function of α (see main text); a zero readout scale is preferred given noise. ( d ) Jfilter and Jnoise vs c (see main text) generally have different optima, and their competition can change where the minimum of J is, or whether it has a finite minimum at all.
Figure 3: Noise determines alignment. ( a ) The dominant contribution to ui(τ) and vi(τ) grows exponentially quickly at a rate λi+ that decreases as the corresponding S eigenvalue (i.e., noise level) si increases, so higher-noise modes grow more slowly. ( b ) Although dynamics involves both growing and decaying contributions, after enough learning time has elapsed, mode amplitudes (solid lines) are well-approximated by only their growing contribution (dashed lines). ( c ) The dot product between u and v , and the least noisy eigenvector of S , increases during this phase. W aligns in the same direction and becomes approximately rank-one. ( d ) The discrepancy between the scales of u and v grows with learning, and is higher in directions with larger noise levels.
Figure 4: Rich regime learning dynamics. For plotting, we take c∗=1 , σ2=1 , and κ=0 . ( a ) In the absence of regularization, learning dynamics drive c toward its target, c∗ , and the amplitude N:=uv toward 1 . ( b ) Unlike the noise-free ( σ2=0 ) dynamics, noisy dynamics produce asymmetric trajectories for u and v , even if they are initialized to the same value. Asymptotically, uv≈1 , u continues to increase, and v continues to decrease. ( c ) The filter-matching term and noise term are both driven to zero, although in this case the latter occurs on a slower time scale.
Figure 5: Excessive noise causes amplitude collapse. For plotting, we take c∗=1 and κ=0.2 . ( a ) Below a critical noise level σcrit2 , learning dynamics drives the filter amplitude N:=uv toward a nontrivial fixed point. Above that noise level, N is driven to zero. ( b ) The optimal amplitude Nopt as a function of noise level. For σ2≥σcrit2 , the trivial solution with f(t)=0 is optimal. ( c ) Below σcrit2 , J is minimized at a specific c value. Above it, J is insensitive to c since N=0 . ( d ) The phase boundary σcrit2(κ) . The noise threshold separating trivial and nontrivial solutions depends on the regularization strength; it is higher when κ is small, and σcrit2→∞ as κ→0 .
Figure 6: Excessive noise produces an infinitely fast filter. c∗=1 assumed throughout. ( a ) When the read-in and readout scales are fixed, J is a function of c only, and for low enough noise it has a single attractor basin at c=copt . As σ2 increases, this basin becomes increasingly shallow. ( b ) For σ2≥σcrit2=3 , J is a decreasing function of c with no finite minimum. ( c ) Below the noise threshold, dynamics drive c toward the unique stable fixed point; above it, c grows without bound. ( d ) copt diverges like 1/(σcrit2−σ2) just below the noise threshold.
Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through unrolling in time. How does this structural similarity fit with the distinct functional properties these networks exhibit? To address this question, we here develop a unified mean-field theory for RNNs and DNNs in terms of representational kernels, describing fully trained networks in the feature learning (μP) regime. This theory casts training as Bayesian inference over sequences and patterns, directly revealing the functional implications induced by the RNNs' weight sharing. In DNN-typical tasks, we identify a phase transition when the learning signal overcomes the noise due to randomness in the weights: below this threshold, RNNs and DNNs behave identically; above it, only RNNs develop correlated representations across timesteps. For sequential tasks, the RNNs' weight sharing furthermore induces an inductive bias that aids generalization by interpolating unsupervised time steps. Overall, our theory offers a way to connect architectural structure to functional biases.
Jan P. Bauer, Kirsten Fischer, Moritz Helias +1
Gatsby Computational Neuroscience Unit, UCL, London, UK · IAS-6, Forschungszentrum Juelich, Juelich, Germany
Learning in recurrent neural networks can fundamentally reshape their underlying dynamics, transforming initially chaotic activity into stable task-dependent behavior. We develop a non-equilibrium dynamical mean-field theory(DMFT) to describe this transition during learning. We show that a slow feedback-driven learning process generates an evolving effective feedback strength that drives the network through a transition from chaotic to stable dynamics defined by a bifurcation of the DMFT solution. By deriving the two-time correlation function throughout learning, we identify a critical feedback strength and a corresponding learning rate dependent critical time separating these regimes. The transition arises from the progressive deformation of an effective dynamical landscape by the growing learned feedback structure. Starting from the untrained state, the theory predicts the time evolution of the network output during training and shows quantitative agreement with numerical simulations.
Varun Vaidya
Department of Physics, University of South Dakota, Vermillion 57069, USA
Learning in neural systems arises from synaptic changes that reshape the representations underlying behavior. While low-rank recurrent neural networks (RNNs) have emerged as a powerful framework for linking connectivity to function, a theoretical understanding of their learning process remains elusive. Here, we extend the low-rank framework from activity to learning by deriving gradient-descent dynamics directly in a reduced overlap space. We formulate a closed-form, low-dimensional system of ODEs that governs learning in this space, exact for linear RNNs and asymptotically exact for nonlinear RNNs in the large-N Gaussian limit. Central to our analysis is a distinction between two classes of overlaps: loss-visible overlaps, which fully determine network activity, output, and loss, and loss-invisible overlaps, which do not affect function but are required to describe learning. We illustrate the consequences of this decomposition through two phenomena. First, we show that learning can serve as a perturbation that exposes differences in connectivity between functionally equivalent networks. Second, we show that loss-invisible overlaps can act as memory variables that encode training history, and characterize the conditions under which this occurs. Finally, we present several testable predictions for biological learning experiments derived from our theory.
Yoav Ger, Omri Barak
Ruth and Bruce Rappaport Faculty of Medicine and Network Biology Research Laboratory Technion – Israel Institute of Technology, Haifa, Israel