Organizations: Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Thomas van Aquinostraat 4, Nijmegen, 6525GD, The Netherlands
Natural gradient descent, and the preconditioned optimizers which have followed from it, correct gradient descent by estimating curvature and applying its inverse to the gradient. For any single layer of a deep neural network, this correction can be shown to approximately separate into two factors: one which accounts for correlations in the gradient signal, and one which accounts for correlations in the data arriving at that layer. Herein we show that this second factor can be expressed as a change of basis rather than of curvature and, considered thus, it can be removed from the optimizer entirely and placed within the network itself. Specifically, we show that, once forward weights are corrected, right-sided natural gradient descent can be implemented in a form which has three exactly equivalent forms: a gradient preconditioner, a decorrelating linear transform in the forward pass, or a set of recurrent dynamics in which all updates are Hebbian and rely only upon locally available signals. The (recurrent) decorrelation method we discover aligns closely with proposed methods for inhibitory learning and homeostatic plasticity in the brain. Notably, our transform requires no matrix inverse, no matrix decomposition, and no information beyond that which is available at an individual synapse. It can therefore be applied in distributed and physically constrained systems, including neuromorphic hardware and nervous systems, where existing preconditioners cannot. We demonstrate its effectiveness empirically in convolutional and residual network models while significantly improving a range of approximate biologically plausible methods for credit assignment.
Figures & tables
Figure 1: Left, the gradient descent vector is shown for a linear regression problem, though now the input data x has a correlation structure ( ⟨xx⊤⟩=Σ ). Right, if the correlation matrix is integrated into the parameters, alignment with the direction of steepest descent in this landscape is returned.
Figure 2: Left, our proposed method can be equivalently represented as either a single linear transformation or the fixed point of a recurrent set of dynamics (see the main text). Right, randomly sampled directions on a skewed quadratic loss function landscape that fall within 90 degrees of the natural/preconditioned gradient direction can be improved by our method.
Figure 3: Computing right-sided natural gradient updates massively helps in generalisation performance of approximate backpropagation algorithms. Results are shown for backpropagation (BP), feedback alignment (FA), and sign-concordant-fa (SC) trained upon the CIFAR10/100 in a LeNet-like and a five hidden-layer convolutional networks. Network architectures and training hyperparameters are described in detail in Appendix G . Envelopes show the standard deviation in accuracy levels across five randomly seeded networks.
Figure 4: Our method compared against Shampoo ( Gupta et al., 2018 ) and Muon ( Jordan et al., 2024 ) , for each credit assignment method, in the five hidden-layer convolutional network trained upon CIFAR-100. Envelopes show the standard deviation across five randomly seeded networks. The same comparison for every network and dataset is given in Appendix H . Note that our correction is computed during the forward pass and from locally available quantities alone, whereas both baselines require statistics of the gradient together with a matrix decomposition or orthogonalization carried out outside of the network.
Model
Dataset
ePC
ePC + CRGD
BP
VGG-7
CIFAR-10
88.84±0.31
90.59±0.22
89.60±0.16
CIFAR-100
58.62±0.20
64.24±0.30
65.14±0.29
VGG-9
CIFAR-10
86.81±0.09
88.85±0.27
89.76±0.20
CIFAR-100
60.65±0.25
63.46±0.45
61.11±0.45
ResNet-18
CIFAR-10
91.73±0.21
92.41±0.08
91.85±0.24
CIFAR-100
69.47±0.32
71.26±0.23
71.46±0.32
Table 1: Top-1 test accuracy for error-based predictive coding (ePC), for ePC combined with our method, and for backpropagation, across deeper architectures. Mean and standard deviation across seeds. Bold marks the better of the two ePC variants. Backpropagation is given in grey as a reference rather than as a competing method, and is italicised where it exceeds both.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Left is shown a loss function landscape (in parameter space) which is a well-conditioned convex loss. In red is shown a model being optimised and the gradient descent direction (grey arrow) points toward the minimum of the function. Right, the loss function is modified to include a positive semi-definite scaling matrix ( L=∑(x,y)∈X(Wx−y)⊤A(Wx−y) ), which causes a skew and stretch of the loss landscape. Gradient descent no longer points toward the minimum but instead in the direction of steepest descent (a detour), whereas natural gradient descent (orange arrow) points directly once more at the loss functions minimum.
Network
Layer Types
Layer size
LeNet Architecture
Input
32×32×3
Conv
5×5×32,(1,2)
MaxPool
3×3,(2,1)
Conv
5×5×32,(1,2)
MaxPool
3×3,(2,1)
Linear
512
Appendix
Table 2: The neural network architectures for the experiments in Figure 3 . For convolutional and pooling layers, shapes are organised as ‘Kernel Width × Kernel Height × Output Channels (Stride, Padding)’. Each parameterized layer is followed by an exponential linear unit (ELU) activation function ( Clevert et al., 2015 ) .
Optimizer
Learning rate
Weight decay
Additional
SGD
[10−5,5×10−1]
[10−5,10−3]
–
CRGD (ours)
[10−5,5×10−1]
[10−5,10−3]
ηM∈[10−7,10−4]
Shampoo
[10−5,5×10−1]
[10−5,10−3]
–
Muon
[10−4,5×10−1]
[10−6,10−2]
AdamW LR ∈[10−4,10−1]
Appendix
Table 3: Ranges searched for each optimizer. Every hyperparameter was drawn from a log-uniform distribution over the interval given, with 100 trials for each combination of network, credit assignment method and optimizer. A dash marks a hyperparameter which does not apply.
Dataset, Network
Method
Optimizer
Learning rate
Weight decay
Decor. LR
AdamW LR
CIFAR-10, LeNet
BP
SGD
2.12×10−2
4.90×10−4
–
–
CRGD (ours)
3.86×10−2
2.14×10−4
5.51×10−6
–
Shampoo
3.89×10−2
7.34×10−4
–
–
Muon
1.14×10−2
1.97×10−3
–
2.12×10−3
SC
SGD
2.77×10−2
5.91×10−4
–
–
CRGD (ours)
1.80×10−2
5.74×10−5
4.05×10−5
–
Appendix
Table 4: Hyperparameters selected for the simulations of Figures 3 and 4 , for every combination of network, credit assignment method and optimizer. A dash marks a hyperparameter which does not apply to that optimizer: the decorrelation learning rate ηM exists only for our method, and the AdamW learning rate only for Muon, which applies AdamW to those parameters it does not orthogonalize. Values are given to three significant figures.
Model
Dataset
Loss
Learning rate
Weight decay
Decor. LR ηM
VGG-7
CIFAR-10
Cross-entropy
1.56×10−3
5.46×10−4
2.50×10−4
CIFAR-10
MSE
4.26×10−4
2.16×10−6
2.50×10−4
CIFAR-100
Cross-entropy
3.13×10−4
7.99×10−4
2.50×10−4
CIFAR-100
MSE
4.02×10−4
1.47×10−5
1.00×10−4
VGG-9
CIFAR-10
Cross-entropy
5.36×10−4
6.88×10−4
7.50×10−5
CIFAR-10
MSE
6.61×10−4
4.01×10−5
2.50×10−4
Appendix
Table 5: Hyperparameters for the error-based predictive coding experiments of Table 1 , for both the cross-entropy and mean-squared-error objectives. The learning rate and weight decay are those reported by Goemaere et al. (2025) and are shared by both the ePC baseline and ePC + CRGD; the decorrelation learning rate ηM applies to ePC + CRGD alone. Values are given to three significant figures.
Figure 6: Our method compared against Shampoo ( Gupta et al., 2018 ) and Muon ( Jordan et al., 2024 ) , for each credit assignment method (rows) and each network and dataset (columns). Envelopes show the standard deviation across five randomly seeded networks. Note that our correction is computed during the forward pass and from locally available quantities alone, whereas both baselines require statistics of the gradient together with a matrix decomposition or orthogonalization carried out outside of the network.
Credit assignment
SGD
CRGD (ours)
Shampoo
Muon
Backpropagation
4.04±0.17
17.60±0.13
23.21±0.10
5.79±0.16
Sign-concordance
4.16±0.24
17.96±0.13
23.42±0.08
5.72±0.43
Appendix
Table 6: Wall-clock time per training epoch, in seconds, upon CIFAR-10 (Conv). Mean and standard deviation across five randomly seeded networks. Ordinary gradient descent is given in grey as a reference point rather than as a competing method.
We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point's residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to find the minimum-norm parameter update that best satisfies all conditions at once. In practice, this pseudoinverse is approximated via a truncated singular value decomposition, retaining only the k most significant directions. We show that Sven can be understood as a natural gradient method generalized to the overparametrized regime, recovering natural gradient descent in the underparametrized limit. We test Sven on a variety of regression and classification tasks, including small-scale language modeling with transformers, and find that it is competitive with leading baselines such as Adam, Muon, and K-FAC. We also discuss Sven's memory overhead, which presents a barrier to scaling under a naive implementation, and introduce an optimized implementation that keeps memory usage on par with standard baselines under mild restrictions on model architecture. Beyond standard machine learning benchmarks, we anticipate that Sven will find natural application in scientific computing settings where custom loss functions decompose into several conditions.
Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1
Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford +2
We consider the problem of approximating a function by an element of a nonlinear manifold which admits a differentiable parametrization, typical examples being neural networks with differentiable activation functions or tensor networks. Natural gradient descent (NGD) for the optimization of a loss function can be seen as a preconditioned gradient descent where updates in the parameter space are driven by a functional perspective. In a spirit similar to Newton's method, a NGD step uses, instead of the Hessian, the Gram matrix of the generating system of the tangent space to the approximation manifold at the current iterate, with respect to a suitable metric. This corresponds to a locally optimal update in function space, following a projected gradient onto the tangent space to the manifold. Still, both gradient and natural gradient descent methods get stuck in local minima. Furthermore, when the model class is a nonlinear manifold or the loss function is not ideally conditioned (e.g., the KL-divergence for density estimation, or a norm of the residual of a partial differential equation in physics informed learning), even the natural gradient might yield non-optimal directions at each step. This work introduces a natural version of classical inertial dynamic methods like Heavy-Ball or Nesterov and show how it can improve the learning process when working with nonlinear model classes.
Anthony Nouy, Agustín Somacal
Centrale Nantes, Nantes Universit´e, Laboratoire de Math´ematiques Jean Leray UMR CNRS 6629
Predictive coding (PC) trains neural networks through local energy-minimization dynamics without depending on a global error signal that backpropagation (BP) requires. Although PC is a promising alternative algorithm for efficient learning, the computational cost of inference dynamics currently outweighs the resulting learning benefits in practice. In this work, we take inspiration from optimization theory to extract more learning benefit from each inference phase. We introduce Natural PC, which augments standard PC with a lateral circuit, enabling exact and efficient natural gradient learning locally in space and time. Our lateral circuit learns the activity correlations that determine the Fisher information matrix and implicitly computes its inverse, avoiding expensive matrix inversion. Across all tested image reconstruction benchmarks, Natural PC achieves state of the art in terms of sample and compute efficiency compared to existing PC approaches. Most notably, Natural PC scales to ImageNet and matches the performance of BP trained with AdamW and various second-order optimizers at an order of magnitude lower computational cost while also using two orders of magnitude fewer training samples. Our results demonstrate that local natural gradient learning makes PC competitive with BP in both sample efficiency and computational cost in reconstruction tasks.
Lhea Beumer, Sander de Haan, Niclas Pokel +2
Institute of Neuroinformatics, UZH & ETH Zurich · ETH AI Center