We prove that in a coupled Kuramoto oscillator network at stable equilibrium, the physical phase displacement under weak output nudging is the gradient of the loss with respect to natural frequencies, with equality as the nudging strength beta tends to zero. Prior oscillator equilibrium propagation work explicitly set aside natural frequency as a learnable parameter; we show that on sparse layered architectures, frequency learning outperforms coupling-weight learning among converged seeds (96.0% vs. 83.3% at matched parameter counts, p = 1.8e-12). The approximately 50% convergence failure rate under random initialization is a loss-landscape property, not a gradient error; topology-aware spectral seeding eliminates it in all settings tested (46/100 to 100/100 seeds on the primary task; 50/50 on a second task, K-only training, and a larger architecture).
Figures & tables
Figure 1: The two-phase gradient readout protocol. (a) Solve for the free equilibrium θ∗ . (b) Add a weak nudging force −β(θi−θitgt) at output oscillators and re-solve for θβ . (c) The per-node phase displacement (θkβ−θk∗)/β approximates the negative gradient −∂L/∂ωkc at finite β , with equality in the limit β→0 (Theorem 6 ). Blue: input nodes (frequencies set by data). Gray: hidden nodes. Amber: output nodes. Dashed line in (b): target phase.
N
Freq. params
Cosine (TP vs FD)
Cosine (AG vs TP)
Eq. residual
6
5
1.000000
1.000000
8.4×10−14
10
9
1.000000
1.000000
2.2×10−16
15
14
1.000000
1.000000
3.4×10−14
20
19
1.000000
1.000000
3.0×10−15
30
29
1.000000
1.000000
1.1×10−15
50
49
1.000000
1.000000
1.1×10−16
Table 1: Phase-gradient identity verification. Cosine similarity between gradient vectors computed by the scipy-based two-phase (TP) and finite-difference (FD) methods, and by the computationally independent PyTorch autograd (AG) method.
Figure 2: Gradient identity verification. (a) Per-node scatter of the two-phase gradient vs. the analytical gradient at N=15 , β=10−3 : all points fall on y=x (cosine similarity =1.000000 ). (b) Equilibrium residual across network sizes from N=6 to N=200 ; all residuals are at or below machine ϵ . The cosine similarity is 1.000000 at every scale.
Table 3: Parameter ablation (100 seeds, 150 epochs, lr = 0.001). Convergence defined by final-epoch training accuracy >60% ; reported accuracy is final-epoch test accuracy. Features normalized using training-set statistics only.
Figure 3: Parameter ablation on /a/ vs /i/ (100 seeds, sparse layered 2+5+2 topology). (a) Violin plots of all 100 seeds showing the bimodal distribution: 47/100 ω -only seeds and 48/100 K -only seeds training-converge (final-epoch training accuracy above 60%; convergence rate difference not significant, Fisher p=1.0 ). Purple: joint ( ω+K ) training. (b) Training-converged seeds only: ω -only achieves 94.3% vs. K -only 82.3% ( p=1.5×10−10 , Welch’s t -test).
Condition
Params
Conv. rate
Conv. test acc.
ω -only
7
47% (47/100)
96.0%
K -matched
7
46% (46/100)
83.3%
K -full
24
47% (47/100)
83.0%
Table 4: Parameter-matched ablation (100 seeds, 200 epochs, lr = 0.001). Convergence defined by final-epoch training accuracy >60% . At equal parameter counts (7 vs. 7), ω -only outperforms K -matched by 12.7 percentage points. Tripling K ’s parameter budget does not close the gap.
N
ω params
K params
ω conv. acc.
K conv. acc.
Gap
7
5
14
93.6%
83.9%
+9.7
9
7
24
95.5%
81.8%
+13.7
11
9
34
89.6%
82.5%
+7.1
14
12
49
90.5%
82.9%
+7.6
19
17
74
88.2%
86.7%
+1.5
Table 5: Architecture sweep (50 seeds per size, 200 epochs). ω -only outperforms K -only at every network size despite having 3–4 × fewer parameters.
Converged (18/40)
Failed (22/40)
Init cond( J~ )
15.2±1.9
15.4±1.8
Final cond( J~ )
14.5±1.6
14.7±1.5
Skip rate
0.0
0.0
Table 6: Convergence diagnosis (40 seeds, 150 epochs). The gradient identity holds for all seeds, including those that fail to learn.
Figure 4: Convergence failure is a landscape problem, not a gradient problem. A converging seed (green) and a failing seed (amber) show different accuracy trajectories but indistinguishable diagnostics: the Jacobian condition number stays ≈15 and the skip rate is zero for both seeds.
Initialization
Success rate
Success accuracy
Overall
Random
46/100 (46%)
96.8%±5.1%
48.5%
Spectral seed
100/100 (100%)
97.6%±2.8%
97.6%
Table 7: Spectral seeding vs. random initialization (100 seeds, ω -only, 200 epochs). Success defined as final test accuracy exceeding 60%.
Figure 5: Spectral seeding eliminates the random-initialization basin failure (100 seeds, ω -only, 200 epochs). (a) Random initialization produces a bimodal distribution: 46/100 seeds exceed 60% test accuracy. Spectral seeding places all 100 seeds above 90%. (b) Per-seed accuracy change: 54 seeds rescued (green), 0 lost.
We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the trajectory, which has been linked to better generalization performance. We introduce the mean--fluctuation dynamics, a tractable continuous-time model coupling the window-averaged trajectory to its fluctuation covariance. Among our contributions, we rigorously derive this model from gradient descent in a sharp-valley framework, characterize its stationary states and their linear stability, and establish precise connections with other effective dynamics. Numerical experiments illustrate these predictions and their finite-time limitations. We also study our model in the overparametrized regime of wide two-layer networks at a fixed learning rate, where we rigorously derive a kinetic equation describing weights and their fluctuations as a Wasserstein-2 gradient flow, for which we prove well-posedness, a mean-field limit, and conditional convergence results.
Antonin Chodron de Courcel
Ecole Normale Sup´erieure, CNRS, 45 rue d’Ulm, 75005 Paris, France
Neural networks trained with gradient-based methods exhibit a strong simplicity bias: they learn simpler statistical features of their data before moving to more complex features. Previous analyses of this phenomenon have largely focused on settings with (quasi-)isotropic inputs. In this work, we study the simplicity bias from a Fourier perspective, which allows us to include two key features of natural images in the analysis: approximate translation-invariance and power-law spectra. We first show experimentally that simple neural networks trained on image classification tasks first rely on amplitude information -- related to pair-wise correlations between pixels -- before exploiting phase information, which encodes edges and higher-order correlations. In view of this, we introduce a synthetic data model for translation-invariant inputs that allows precise control over amplitudes and phases while remaining tractable. We rigorously establish that for isotropic and high-dimensional inputs, classification based on phase information alone is a genuinely hard task: online stochastic gradient descent (SGD) cannot distinguish the structured inputs from noise within n≪N3 steps, but needs at least n≫N3log2N steps. In contrast, we show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information, even if the spectra do not help with classification. Simulations with two-layer networks trained on textures and with deep convolutional networks on ImageNet and CIFAR100 confirm this non-trivial interaction between amplitudes and phases, providing mechanistic insights into how deep neural networks can learn natural image distributions efficiently.
Fabiola Ricci, Claudia Merger, Sebastian Goldt
International School of Advanced Studies (SISSA), Trieste, Italy
Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.
Yanyang Xiao
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences