cs.LGOct 1, 2026
SaveSoftServe: A Scalable Quasi-Newton Method for Deep Learning
Organizations: University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University
Abstract
Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.
Figures & tables
Algorithm 1 SoftServe -Kron: high-level training iteration (details in Algorithm D.1 )
Figure 1: Training objectives against gradient evaluations: prediction MSE for RNN Adding (left), and reconstruction binary cross-entropy with regularization for full-batch MNIST (middle) and minibatch MNIST (right). Lines show three-seed means; shading spans the seed minimum and maximum. K-BFGS(L) with failed seeds are shown as individual traces.
Figure 2: PINN training objectives for Wave, Convection, and Reaction, with fixed points (top) and resampled points (bottom). Lines show three-seed means; shading spans the seed minimum and maximum. Settings with failed seeds are shown as individual traces. Horizontal extensions show the final loss after an L-BFGS run terminates.
Figure 3: PirateNet on Allen–Cahn (left) and KdV (middle), and the 136M-parameter PIDM (right). PirateNet uses unweighted PDE and initial-condition MSE on fixed points; PIDM uses the training objective on fixed examples and noise. Lines show three-seed means; shading spans the seed minimum and maximum.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | State | Apply | Refresh |
|---|---|---|---|
| Dense | |||
| Diag | |||
| Kron, |
Table D.1: SoftServe method costs for updating the parameters of a linear layer.
| \displaystyle\text{Update~\bm{L}_{t+1}~by~solving~}\bm{L}\,\bm{U}_{L}\,\bm{L}+\bm{L}=\bm{V}_{L} | |||
| \displaystyle\text{Update~\bm{R}_{t+1}~by~solving~}\bm{R}\,\bm{U}_{R}\,\bm{R}+\bm{R}=\bm{V}_{R}, |
Algorithm D.1 SoftServe -Kron: detailed implementation
Figure E.1: RNN Adding learning-rate sweeps on seed 0. Stars mark selected rates; crosses mark numerical failures.
Figure E.2: RNN Adding training minibatch MSE and held-out validation MSE against parameter updates and wall-clock time. Failed cohorts retain their individual traces.
Figure E.3: RNN Adding loss-Hessian singular values at initialization, normalized by the largest value. The dashed line marks the numerical-rank threshold . All four seeds have singular values below this threshold.
Figure E.4: Autoencoder full-budget LR sweeps at selected momentum and fixed . Stars mark selected rates. Crosses mark failed or off-scale runs.
Figure E.5: Autoencoder test reconstruction BCE. Dots are individual seeds; ticks are three-seed means. Full-batch K-BFGS(L) has one failed seed and therefore no three-seed mean.
Figure E.6: Autoencoder training objective against parameter updates and wall-clock time. Shading spans the three seed values.
Figure E.7: Small PINN learning-rate sweeps on seed 0, scored by the mean of the final five training losses. Stars mark selected rates, crosses numerical failures, and triangles off-scale finite scores.
Figure E.8: FP32 PINN relative solution error. Columns: Wave, Convection, Reaction. Rows: fixed and resampled collocation points. The horizontal continuation of a terminated L-BFGS run represents its unchanged solution.
Figure E.9: FP64 PINN training objectives with the FP32-selected settings. The gradient budget includes replay and line-search evaluations.
Figure E.10: FP64 PINN solution errors with the FP32-selected settings. Settings with failed seeds are shown as individual traces.
Figure E.11: FP32 PINN training objectives against parameter updates.
Figure E.12: FP32 PINN training objectives against wall-clock time. K-FAC and K-BFGS(L) are omitted here because their timers also include construction time.
Figure E.13: PirateNet full-budget learning-rate sweeps, scored by unweighted validation loss. Stars mark selected rates; crosses mark failures.
Figure E.14: PirateNet unweighted PDE and initial-condition MSE against updates and wall-clock time. All curves stop at 300,600 gradient evaluations for Allen–Cahn and 100,200 for KdV; Kron therefore uses fewer parameter updates.
Figure E.15: PirateNet relative error over the reference space–time grid, up to the same gradient budgets as the training-loss plots. This metric is not used to select the learning rate.
Figure E.16: PIDM LR selection after 10,000 updates. The score is the EMA validation objective. Stars mark selected rates; crosses mark numerical failures.
Figure E.17: PIDM training objective (unaveraged parameters) and validation objective (EMA parameters) against gradient evaluations, updates, and wall-clock time. Every curve stops at 100,000 gradient evaluations, corresponding to approximately 90,900 parameter updates for Kron and 100,000 for the baselines.
Figure F.1: RNN Adding validation MSE after 40,000 gradient evaluations, with a separately selected learning rate for each . Points show means and bars show seed ranges. The dashed line is the mean of the completed Muon comparison. The cross above the axis denotes three numerical failures at , not a measured loss.
Figure F.2: Relative QME solution error as the root and inverse iteration counts vary. Each cell is the median over three matrix seeds; all panels share a logarithmic color scale. Lower values are better. White outlines mark the default 18 root and 10 inverse iterations.
Figure F.3: RNN validation MSE as root iterations vary (left) and inverse iterations vary (right). Legend entries give root/inverse counts; 18/10 is the default. Each completed setting uses a separately selected learning rate and three fresh seeds. Lines show means and shading spans the seed minimum and maximum. Insets enlarge the last 10,000 gradient evaluations on a linear MSE axis with shared limits. The 12/10 setting failed at all three tested tuning rates and has no learning curve.
Figure F.4: Learning curves for different refresh intervals. Rows show RNN validation MSE and minibatch MNIST training objective; columns show gradient evaluations and wall-clock time. All runs stop at the same task-specific gradient budget, including replay: 40,000 for RNN and 6,000 for MNIST. Lines show three-seed means and shading spans the seed minimum and maximum. Wall-clock time includes evaluation and diagnostic overhead. Orange lines mark the default .