Organizations: Department of Neurobiology, Harvard Medical School · Department of Psychology and Center for Brain Science, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University
Learning algorithms are usually analyzed under the assumption that informative gradients are continuously available. But this is not always true---learning signals in biological neural networks, for example, may be sparse, local, and only intermittently available at any given synapse. In this work, we show that if gradient information is noisy and only intermittently available, learning may be unstable. In particular, even if average learning dynamics converge to the correct optimum, variance in learning may be asymptotically infinite. We explore this point in a number of analytically tractable settings, including single-layer and multi-layer linear networks, and derive explicit conditions under which learning is stable. These conditions complement, but are conceptually distinct from, existing conditions related to the stability of stochastic gradient descent. Our work demonstrates that unbiased credit assignment is not enough for reliable learning; sufficiently frequent and informative feedback is also required, which yields an additional constraint on models of biologically plausible learning.
Figures & tables
Figure 1: Sparse gradient arrivals: motivation and intuition. ( a ) A simple example of sparse gradient arrivals: a weight receives intermittent updates, in between which the regularization term produces relaxation toward zero. Three example stochastic simulations are shown. ( b ) Error as a function of number of arrivals in a toy model. If s/ρ<1 , error always decreases (left); if 1<s/ρ<2 , error alternates in sign but decreases in magnitude (center); and if s/ρ>2 , learning becomes unstable, and error both alternates in sign and increases in magnitude (right).
Figure 2: Instability in a linear model. ( a ) Average dynamics are unconditionally stable and do not depend on the arrival rate ρ . ( b ) Output variance converges more slowly as ρ decreases and diverges to infinity when ρ<ρc . ( c ) The steady state output variance diverges like 1/(ρ−ρc) . ( d ) ρc decreases as γ increases, so one way to improve stability is to increase the regularization strength.
Figure 3: Instability in a two-layer network. ( a ) We trained a two-layer network to reproduce the output of an identical teacher network. To ensure our linear model provides a (locally) reasonable description of its dynamics, we initialized it near the teacher’s weights. ( b ) Normalized objective and output variance throughout learning. Note that for ρ<ρc (dark red), J blows up. For ρ somewhat above the threshold (red, orange), learning still looks unstable on this finite time scale. Learning only appears stable well beyond the threshold (green). ( c ) Instantaneous growth rate of the covariance dynamics near the fixed point. The black line indicates the linear model’s prediction and the red dots indicate numerical measurements. Below ρ=ρc , this local growth rate can be positive, which indicates instability.
Figure 4: Depth benignly modulates stability in a deep linear network. ( a ) ρc can increase, decrease, or remain constant with L depending on the value of the target W∗ . In all cases, the difference between ρc(L) and its asymptotic value ρc(∞) is not substantial. ( b ) ρc is non-monotonic as a function of γ ( L=10 shown), but this is mainly because sufficiently high regularization destabilizes the nontrivial fixed point.
Figure 5: Pooling parameters can improve stability. ( a ) If nearby synapses store similar information, neurons may benefit from sharing information with their spatial neighbors. Pictured is a line of synaptic weights; arrows indicate bidirectional information sharing. ( b ) The eigenvalues dq of D are indexed by spatial frequencies ∣q∣ , and dq decreases as ∣q∣ increases. ( c ) The q -dependence of dq causes higher spatial frequencies of the fixed point to be damped, and the effect size increases as the amount of diffusion k increases (more purple → higher k ). ( d ) One effect of this frequency-dependence is to smooth the fixed point of the average dynamics. In the pictured example, wi∗ is highly oscillatory (black), but dynamics converge to a smoothed version of it (shades of purple). ( e ) ρc decreases as k increases, so although parameter-pooling produces bias, it also improves stability.
Figure 6: Averaging over gradient arrivals can improve stability. ( a ) Intuitively, one can reduce variance by performing a temporal average over recent gradient arrivals. ( b ) Mean dynamics are linear but not monotonic. As the gradient-averaging time scale τz increases, they converge more slowly. (black: τz=0 ; lightest green: τz=100 ) ( c , d ) Increasing τz improves stability since ρc decreases as long as γ>0 . But if γ=0 , there is no stability benefit.
Stable training is often treated as evidence that learning is succeeding, but stability characterizes optimization behavior rather than correctness relative to an external objective. We study what happens when the signal being optimized remains persistently biased. We define Stable but Wrong (SBW) as a learning state in which the learning process remains stable under a task-appropriate operational criterion while the learned outcome remains systematically displaced from an independently defined objective. A minimal strongly convex model shows that a persistent bias in the update direction can shift the unique convergence point away from the true optimum. Controlled experiments in reinforcement learning, supervised learning, and continual fine-tuning of a large language model reveal a recurring separation between apparently normal optimization and correctness under static and feedback-coupled biases. Recovery-stage clean-data access and exploration interventions further show that subsequent trajectories can remain modifiable, although the interventions differ in protocol and do not imply a shared mechanism. The results expose a basic limit of optimization stability as a reliability signal in persistent and feedback-coupled learning systems.
Zhipeng Zhang
China Mobile Research Institute, Beijing 100053, China · China Mobile GBA (Greater Bay Area) Innovation Institute, Guangzhou 510656, China
We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the trajectory, which has been linked to better generalization performance. We introduce the mean--fluctuation dynamics, a tractable continuous-time model coupling the window-averaged trajectory to its fluctuation covariance. Among our contributions, we rigorously derive this model from gradient descent in a sharp-valley framework, characterize its stationary states and their linear stability, and establish precise connections with other effective dynamics. Numerical experiments illustrate these predictions and their finite-time limitations. We also study our model in the overparametrized regime of wide two-layer networks at a fixed learning rate, where we rigorously derive a kinetic equation describing weights and their fluctuations as a Wasserstein-2 gradient flow, for which we prove well-posedness, a mean-field limit, and conditional convergence results.
Antonin Chodron de Courcel
Ecole Normale Sup´erieure, CNRS, 45 rue d’Ulm, 75005 Paris, France
Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.
John J. Vastola, Kanaka Rajan
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University