cs.LGApr 1, 2026
SaveSven: Singular Value Descent as a Computationally Efficient Natural Gradient Method
Organizations: Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford · Institut des Hautes Études Scientifiques · Institut de Physique Théorique, CEA Paris-Saclay
Abstract
We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point's residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to find the minimum-norm parameter update that best satisfies all conditions at once. In practice, this pseudoinverse is approximated via a truncated singular value decomposition, retaining only the most significant directions. We show that Sven can be understood as a natural gradient method generalized to the overparametrized regime, recovering natural gradient descent in the underparametrized limit. We test Sven on a variety of regression and classification tasks, including small-scale language modeling with transformers, and find that it is competitive with leading baselines such as Adam, Muon, and K-FAC. We also discuss Sven's memory overhead, which presents a barrier to scaling under a naive implementation, and introduce an optimized implementation that keeps memory usage on par with standard baselines under mild restrictions on model architecture. Beyond standard machine learning benchmarks, we anticipate that Sven will find natural application in scientific computing settings where custom loss functions decompose into several conditions.
Figures & tables
Figure 1: Validation loss against epoch (top) and against wall time (bottom) for Sven and four best-performing baselines on the three headline tasks: random polynomial regression (left), MNIST classification with label regression (middle), and tiny-Shakespeare language modeling with nanoGPT (right). Curves are means over 5 seeds, with optimizer parameter settings chosen according to lowest mean validation loss at the end of training.
Figure 2: Mean paired differences in test set loss relative to Sven for all baseline optimizers on the synthetic regression, MNIST, and nanoGPT tasks. Error bars are Student- 95% intervals on the mean difference over the seed pairs (indicated in each plot); a filled marker is an interval that excludes zero. An interval wider than the panel is clipped to an arrow.
Figure 3: Normalized singular value spectra at the beginning (a) and end (b) of training. Horizontal lines mark the selected rtol and vertical lines the selected , with colored dots indicating the corresponding datasets. The dotted black line is the float32 round-off floor. (c) The fraction of the batch residual the truncation discards over the course of training.
Appendix figures & tables45 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Loss | Train | Val. | Test | Construction |
|---|---|---|---|---|---|
| 1D regression | 10k | 10k | 10k | independent draws; pool-standardized | |
| Random polynomial | 10k | 10k | 10k | independent draws; pool-standardized | |
| MNIST | label reg./CE | 50k | 10k | 10k | official train split; official test set |
| CIFAR-10 | label reg./CE | 45k | 5k | 10k | official train split; official test set |
| Shakespeare (char) | token CE | 6,971 | 871 | 871 | contiguous 80/10/10 by position |
| FineWeb-edu (BPE) | token CE | 210k | 200 | 200 | val/test from disjoint documents |
Table 1: Dataset splits. Sizes are examples, or blocks for the two language-modeling corpora.
| Task | Model | Configuration | |
|---|---|---|---|
| 1D regression | MLP | , GeLU | 593 |
| Random polynomial | MLP | , GeLU | 673 |
| MNIST | MLP | , GeLU | 27,562 |
| CIFAR-10 | ResNet18 | 10 classes, functional BatchNorm | 11,181,642 |
| Shakespeare (char) | nanoGPT | 4 layers, 4 heads, , block 128 | 826,368 |
| FineWeb-edu | GPT-2 small | 12 layers, 12 heads, , block 1024 | 163,109,376 |
Table 2: Models and parameter counts . The MLP notation lists input, hidden and output widths.
| Scan | rtol | Runs | |||
|---|---|---|---|---|---|
| 1D regression | 32 | 1,2,4,8,16,32 | .01,.02,.05,.1,.5,1 | (5) | 900 |
| Random polynomial | 32 | 1,2,4,8,16,32 | .05,.1,.5,1 | (6) | 720 |
| MNIST label reg. | 64 | 1,2,4,8,16,32,48,64 | .05,.1,.5,1 | (4) | 640 |
| MNIST CE | 64 | 1,2,4,8,16,32,48,64 | .05,.1,.5,1 | (5) | 800 |
| CIFAR-10 label reg. | 128 | 64,128 | .1,.5,1 | 90 | |
| CIFAR-10 CE | 128 | 64,128 | .02,.05,.1,.5,1 | 150 |
Table 4: Sven’s hyperparameter grids. is the training batch size, and “runs” counts the full grid including seeds. All grids use . The CIFAR-10 cross-entropy grid was extended at , with , because the optimum sat on the rtol edge.
| Study | rtol | Swept axis | Runs | |||
|---|---|---|---|---|---|---|
| (MNIST lab. reg.) | 64 | 32,64 | .125–1.5 (7) | : 1,2,3 | 210 | |
| Micro-batch (1D, poly.) | 32 | 32 | .05,.1,.5,1 | : 1–32 (6) | 120 | |
| Micro-batch (MNIST) | 64 | 64 | .05,.1,.5,1 | : 1–64 (7) | 140 | |
| Param. frac. (1D, poly.) | 32 | 32 | .05,.1,.5,1 | : .1–1 (5) | 100 | |
| Param. frac. (MNIST) | 64 | 64 | .05,.1,.5,1 | : .1–1 (5) | 100 | |
| Param. frac. (CIFAR-10) | 128 | 128 | .5 | : .05–1 (5) | 15 |
Table 5: Sven’s grids in follow-up studies varying , (micro-)batch size ( ), updated parameter fraction ( ), and overparametrization ( ). For MNIST, the two rtol values separated by a slash are the label-regression and cross-entropy settings. The sweeps on the two synthetic tasks are full-batch ( ).
| Scan | Standard optimizers | L-BFGS | JD | HIG |
|---|---|---|---|---|
| 1D regression | (8) | .01,.03,.1,.5,1 | (6) | (7) |
| Random polynomial | (10) | .1,.5,1,2,4 | (6) | (5) |
| MNIST label reg. | (8) | .1,.5,1 | (6) | (5) |
| MNIST CE | (8) | .1,.5,1,2,4 | (6) | (5) |
| CIFAR-10 label reg. | (7) | .1,.5,1,2,4 | — | — |
| CIFAR-10 CE | (9) | .1,.5,1,2,4 | — | — |
Table 6: Baseline learning-rate grids. The second column is the shared grid used by all standard optimizers; L-BFGS, JD and HIG have their own. L-BFGS additionally sweeps max_iter and history_size ; HIG additionally sweeps its cutoff . Polyak SGD has no learning rate to sweep. A dash means the method was not run on that scan.
Figure 4: Reproduction of Fig. 1 with all baselines shown: validation loss against epoch (top) and against standalone wall time (bottom). Bands are std over the 5 confirmation seeds. SOAP is omitted for MNIST because it does not converge well.
Figure 5: Confirmation trajectories for 1D regression (top), polynomial regression (middle) and MNIST label regression (bottom). We plot validation loss against epoch (left) and against standalone training time (right). Bands are mean std over the 5 confirmation seeds. SOAP is omitted from MNIST because it does not converge well.
Figure 6: Ranking of the best-performing configurations of each optimizer on each scan using validation loss (left) and test loss (right). Numbers indicate rank, and colors are normalized by the number of contenders for each scan (indicated in the bottom row). A gray cell means the method was not run on that scan or produced no finite confirmation value.
| Method | Configuration | Val. loss | Val. (tune) | Test loss | Test acc. | fin./att. | s/ep. | MB |
|---|---|---|---|---|---|---|---|---|
| Random Polynomial | ||||||||
| Sven | =0.5, =16, rtol=0.03 | 0.1388 0.055 | 0.1095 | 0.136 0.055 | – | 15/15 | 1.33 | 18.86 |
| HIG | =0.1, tau=1e-08 | 0.08406 0.024 | 0.07149 | 0.08028 0.02 | – | 15/15 | 1.4 | 21.59 |
| MuonW | =0.01, wd=0.1 | 0.2313 0.07 | 0.2061 | 0.2226 0.06 | – | 15/15 | 0.967 | 18.44 |
| Muon | =0.01 | 0.207 0.062 | 0.1651 | 0.202 0.058 | – | 15/15 | 0.978 | 18.44 |
| SOAP | =0.01 | 0.1818 0.05 | 0.1496 | 0.1756 0.042 | – | 15/15 | 1.15 | 21.29 |
Table 15: Headline results on the confirmation seeds: final validation and test loss (mean std over seeds), test accuracy where defined, finished/attempted runs, seconds per epoch and peak memory. Losses are from the confirmation pass; the two cost columns are the standalone timing pass.
| Method | Val. (pooled) | Val. (inst. mean) | Betw.-inst. std | Within-inst. std | Ratio |
|---|---|---|---|---|---|
| Toy 1D | |||||
| HIG | 0.791 | ||||
| Sven | 0.682 | ||||
| SOAP | 0.318 | ||||
| SGD + momentum | 0.652 | ||||
| AdamW | 0.657 | ||||
Table 16: Data-seed replicates on the two synthetic tasks. Each data seed is a different random problem, so the standard deviation of the three per-instance means is variation of the target and not of the optimizer. A ratio above one says the choice of instance moves the answer more than the seed does.
| Method | Selected configuration | Val. loss | Test loss | Train (eval) |
|---|---|---|---|---|
| HIG | lr=0.005, tau=1e-06 | |||
| Sven | k=32, lr=0.01, rtol=0.0001 | |||
| SOAP | lr=0.001 | |||
| SGD + momentum | lr=0.1 | |||
| AdamW | lr=0.001, weight_decay=0.01 | |||
| KFAC | lr=0.001 |
Table 17: Every method’s selected configuration on the confirmation seeds, 1D regression (top) and random-polynomial regression (bottom). Rows are sorted by validation loss, so the row order is the ranking; the band is std over seeds.
| Method | Selected configuration | Val. loss | Test loss | Test acc. | Train (eval) |
|---|---|---|---|---|---|
| MuonW | lr=0.001, weight_decay=0.1 | 0.04982 0.0013 | 0.04959 0.0025 | 97.03% | 0.02206 0.00041 |
| HIG | lr=0.05, tau=0.03 | 0.05298 0.0026 | 0.05324 0.0023 | 96.60% | 0.02293 0.0019 |
| Sven | k=64, lr=0.5, rtol=0.001 | 0.05328 0.0007 | 0.05363 0.0024 | 96.72% | 0.02804 0.0012 |
| Muon | lr=0.001 | 0.05473 0.0017 | 0.05424 0.0038 | 96.72% | 0.01652 0.0011 |
| SGD | lr=0.1 | 0.05524 0.0025 | 0.05439 0.0019 | 96.89% | 0.02968 0.0019 |
| SGD + momentum | lr=0.01 | 0.05524 0.0015 | 0.05454 0.0017 | 96.95% | 0.02863 0.0011 |
Table 18: Every method’s selected configuration on the confirmation seeds, MNIST label regression (top) and MNIST cross-entropy (bottom). Conventions as in Table 17 . K-FAC is absent from both: it has no eligible configuration on either scan (Appendix Q ).
Figure 7: Full-width online Gram spectra over training (one model seed per scan). Each line is from one logged batch and normalized by its ; the vertical line indicates when , the dotted line the rtol cut, and the dashed line the float32 round-off floor.
Figure 8: Validation loss against epoch over the rank cutoff , with and rtol pinned at each scan’s selected values; the thick curve is the selected .
Figure 9: Mean final validation loss over at fixed rtol (top) and over rtol at fixed (bottom), both at each scan’s selected learning rate; the star is the selection of record and ineligible configurations are omitted.
Figure 10: The rank the update uses as a fraction of the cap , over the tuning grid at each scan’s selected learning rate, final epoch. A cell at is a cell where truncates; anything below it is a cell where rtol truncates. The red square is the selection of record and a hatched cell indicates fewer seeds than the grid intended (failed runs).
Figure 11: (a)–(c) Kept and discarded fraction of the batch residual and the rank actually inverted, against training progress, seed mean over the diagnostics seeds with a 60-step moving average. (d) The five seeds of the MNIST cross-entropy scan drawn separately: the collapse to directions happens on every seed.
| Scan | / | rtol | Binds | Used, first final | Kept, first final |
|---|---|---|---|---|---|
| Toy 1D | / | rtol | 4.2 14.4 | 92.08% 91.53% | |
| Polynomial | / | rtol | 7.0 13.4 | 92.25% 99.59% | |
| MNIST-LR | / | neither ( ) | 64.0 63.6 | 100.00% 100.00% | |
| MNIST-CE | / | rtol | 10.0 2.8 | 31.01% 95.17% | |
| CIFAR-LR | / | rtol | 128.0 81.6 | 100.00% 100.00% | |
| CIFAR-CE | / | rtol | 106.0 36.6 | 90.13% 76.69% |
Table 19: What the truncation does, per scan. “Used” is the number of directions actually inverted, , at the first and last logged step; “Kept” is the fraction of that batch’s residual they account for. “Binds” is read at the last logged step. Seed means over the diagnostics pass.
Figure 12: Where the batch residual sits: against singular-value index over training, one seed per panel.
| Scan | Probe rows | Spectrum width | Resolved rank, final step | |
|---|---|---|---|---|
| Toy 1D | 10,000 | 593 | 593 | 45.0 |
| Random Polynomial | 10,000 | 673 | 673 | 673.0 |
| MNIST (label reg.) | 512 | 27,562 | 512 | 512.0 |
| MNIST (CE) | 512 | 27,562 | 512 | 512.0 |
Table 20: The cached probe sets. The spectrum width is and only one of the two bounds is ever active. “Resolved rank” counts the singular values above the double-precision threshold on Sven’s final checkpoint, averaged over 5 model seeds.
Figure 13: Cumulative fraction of the reachable probe residual against the number of leading probe directions kept, one seed per scan, colored by checkpoint. Two cuts are marked: is the rank a step inverts, the row space a step lives in, so even an untruncated step reaches no more than the curve’s value at . The axis runs to .
Figure 14: Probe-set spectra in double precision along the trajectories of four optimizers (columns) on every cached scan (rows), using the same probe rows at every checkpoint/optimizer. The red line is Sven’s . The spectrum width is .
Figure 15: Resolved rank, effective rank, condition number and of the probe Jacobian along the trajectories of Sven, Adam, MuonW and HIG, seed mean 1 std over five cached seeds. The condition number is taken over the resolved part of the spectrum and cannot exceed , the inverse of the float64 threshold; a hollow marker means index lies outside the resolved rank, so the ratio beside it is a threshold artifact. On the 1D task only 14 of 593 directions are resolved at initialization, which is why that scan’s condition number and are artifacts throughout.
Figure 16: Distance from initialization and parameter norm along the four cached trajectories, seed mean 1 std.
Figure 17: Paired per-seed differences in final validation loss. Differences are taken within a pair (same model seed, and data seed on the two synthetic tasks). Error bars are Student- 95% intervals on the mean difference and a filled marker is an interval that excludes zero. The count beside each name is the pairs Sven won out of the pairs that finished. An interval wider than the panel is clipped and its end drawn as an arrow.
Figure 18: Final loss on the training subset (top), validation (middle) and test (bottom) against for thirteen methods on the 1D, polynomial and MNIST label-regression tasks, with the configuration selected inside each arm on validation loss and a band of std over seeds. SOAP is omitted from the MNIST panels, as in Fig. 1 .
Figure 19: Batch-size sweep on random-polynomial regression: selected final validation loss against for. Error bars are std over seeds.
Figure 20: Confirmation-seed trajectories on MNIST with cross-entropy: validation loss against epoch (left) and against synchronized standalone training time (right); bands of std over seeds. SOAP is omitted, as in Fig. 1 .
Figure 21: Validation loss (left) and test accuracy (right) for CIFAR-10 / ResNet18 with label regression (top) and cross-entropy (bottom).
Figure 22: Validation loss (left) and test accuracy (right) plotted against training wall-time for CIFAR-10 / ResNet18 with label regression (top) and cross-entropy (bottom).
Figure 23: Seconds per epoch and peak GPU memory of each method’s selected configuration on the label-regression scan.
Figure 24: Validation/test loss (left) and accuracy (right) against the actual parameter fraction on ResNet18 / CIFAR-10 with label regression. Quality holds to and collapses at .
Figure 25: Peak memory (left) and time per optimizer step (right) against the actual parameter fraction; error bars are std over seeds.
Figure 26: nanoGPT on character-level Shakespeare. Validation loss against epoch (left) and against training time (center), std over 5 seeds; final validation loss against learning rate (right).
Figure 27: GPT-2-small on FineWeb-edu: validation loss against optimization step for the best learning rate of each method (left) and for each of Sven’s learning rates (center), and the final validation loss against the learning rate for every method (right). Resource constraints prevented a full hyperparameter scan.
Figure 28: on MNIST label regression, plotted against the effective step rather than against , at (truncation active) and (inactive). Left: final validation loss. Center: the spread over at each matched step. Right: the number of singular values the update inverts against effective step. Bands are std over 5 model seeds.
Figure 29: The two memory-mitigation knobs on the four MLP scans, each normalized by its own reference setting. Top: final validation loss against micro-batch size and against parameter fraction. Bottom: peak memory (left) and step time (right) against each knob scaled to its own maximum, relative to the reference setting (solid for micro-batching, dashed for parameter batching).
Figure 30: Left: fraction of Sven runs that diverged versus rtol , on the four headline tuning grids (solid) and on the and batch-size sweeps (dashed). Right: fraction of each grid that diverged, per method. Sven’s grid includes the rtol axis, so the denominators are not equal across methods.
Figure 31: Sven’s diverged runs over the plane, as diverged/attempted per cell: the four headline grids, then the three sweeps and the batch-size sweep.
Figure 32: Time per optimizer step (left) and peak GPU memory (right) against parameter count over 5 model scales spanning 593 to 11,181,642 parameters (ResNet18), at each method’s selected configuration.
Figure 33: Step time relative to Adam (left) and peak memory relative to SGD (right) for every method at every architecture, at each architecture’s profile set point.
Figure 34: Step time against the rank cap , per architecture and backend.
Figure 35: Share of a Sven step spent in the capture (solid) against the solve-plus-apply (hatched), per backend and architecture, with the total step time in ms above each bar.
Figure 36: Step time (top) and peak memory (bottom) against the batch size , per architecture. The dotted line on the memory panels is the memory size of the dense Jacobian.
Figure 37: Step time and peak memory against parameter count over two width families: a perceptron sweep that runs past ResNet18’s parameter count, and a nanoGPT sweep.