A theory of noisy linear recurrent neural network learning dynamics
Authors: John J. Vastola, Kanaka Rajan
Organizations: Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.
Figures & tables
Figure 1: Theoretical setting. ( a ) We study a noisy linear RNN which receives a scalar white noise signal x(t) as input and produces an output y(t) . The dynamics of individual ri are noisy (bottom). ( b ) On average, the network filters its input. ( c ) Output variance is input-independent and eventually saturates (assuming the network is stable). ( d ) We assume the network is trained to optimize an objective with three terms: a filter-matching term that incentivizes matching its average behavior to that of a target filter f∗(t) , a noise reduction term, and a regularization term.
Figure 2: Noise produces representational pressure. ( a ) Two representations with the same filter but different readout vectors differ in noise level if S is anisotropic. Pictured is a schematic 2D example with a high-noise direction (horizontal) and a low-noise direction (vertical). One can reduce noise by rotating v (white arrow). ( b ) The noise term Jnoise in the schematic example as a function of v orientation. Jfilter is unaffected as long as the filter remains the same. ( c ) Jnoise and Jfilter as a function of α (see main text); a zero readout scale is preferred given noise. ( d ) Jfilter and Jnoise vs c (see main text) generally have different optima, and their competition can change where the minimum of J is, or whether it has a finite minimum at all.
Figure 3: Noise determines alignment. ( a ) The dominant contribution to ui(τ) and vi(τ) grows exponentially quickly at a rate λi+ that decreases as the corresponding S eigenvalue (i.e., noise level) si increases, so higher-noise modes grow more slowly. ( b ) Although dynamics involves both growing and decaying contributions, after enough learning time has elapsed, mode amplitudes (solid lines) are well-approximated by only their growing contribution (dashed lines). ( c ) The dot product between u and v , and the least noisy eigenvector of S , increases during this phase. W aligns in the same direction and becomes approximately rank-one. ( d ) The discrepancy between the scales of u and v grows with learning, and is higher in directions with larger noise levels.
Figure 4: Rich regime learning dynamics. For plotting, we take c∗=1 , σ2=1 , and κ=0 . ( a ) In the absence of regularization, learning dynamics drive c toward its target, c∗ , and the amplitude N:=uv toward 1 . ( b ) Unlike the noise-free ( σ2=0 ) dynamics, noisy dynamics produce asymmetric trajectories for u and v , even if they are initialized to the same value. Asymptotically, uv≈1 , u continues to increase, and v continues to decrease. ( c ) The filter-matching term and noise term are both driven to zero, although in this case the latter occurs on a slower time scale.
Figure 5: Excessive noise causes amplitude collapse. For plotting, we take c∗=1 and κ=0.2 . ( a ) Below a critical noise level σcrit2 , learning dynamics drives the filter amplitude N:=uv toward a nontrivial fixed point. Above that noise level, N is driven to zero. ( b ) The optimal amplitude Nopt as a function of noise level. For σ2≥σcrit2 , the trivial solution with f(t)=0 is optimal. ( c ) Below σcrit2 , J is minimized at a specific c value. Above it, J is insensitive to c since N=0 . ( d ) The phase boundary σcrit2(κ) . The noise threshold separating trivial and nontrivial solutions depends on the regularization strength; it is higher when κ is small, and σcrit2→∞ as κ→0 .
Figure 6: Excessive noise produces an infinitely fast filter. c∗=1 assumed throughout. ( a ) When the read-in and readout scales are fixed, J is a function of c only, and for low enough noise it has a single attractor basin at c=copt . As σ2 increases, this basin becomes increasingly shallow. ( b ) For σ2≥σcrit2=3 , J is a decreasing function of c with no finite minimum. ( c ) Below the noise threshold, dynamics drive c toward the unique stable fixed point; above it, c grows without bound. ( d ) copt diverges like 1/(σcrit2−σ2) just below the noise threshold.