Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.
Figures & tables
Xt+1=Xt−γttr(M^t⊤RtM^tLt)RtM^tLt
Algorithm 1 SoftServe -Kron: high-level training iteration (details in Algorithm D.1 )
Figure 1: Training objectives against gradient evaluations: prediction MSE for RNN Adding (left), and reconstruction binary cross-entropy with ℓ2 regularization for full-batch MNIST (middle) and minibatch MNIST (right). Lines show three-seed means; shading spans the seed minimum and maximum. K-BFGS(L) with failed seeds are shown as individual traces.
Figure 2: PINN training objectives for Wave, Convection, and Reaction, with fixed points (top) and resampled points (bottom). Lines show three-seed means; shading spans the seed minimum and maximum. Settings with failed seeds are shown as individual traces. Horizontal extensions show the final loss after an L-BFGS run terminates.
Figure 3: PirateNet on Allen–Cahn (left) and KdV (middle), and the 136M-parameter PIDM (right). PirateNet uses unweighted PDE and initial-condition MSE on fixed points; PIDM uses the training objective on fixed examples and noise. Lines show three-seed means; shading spans the seed minimum and maximum.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
State
Apply
Refresh
Dense
O(n2m2)
O(n2m2)
O(n2m2)
Diag
O(nm)
O(nm)
O(nm)
Kron, n×m
O(n2+m2)
O(n2m+nm2)
O(n3+m3+n2m+nm2)
Appendix
Table D.1: SoftServe method costs for updating the parameters of a n×m linear layer.
Figure E.1: RNN Adding learning-rate sweeps on seed 0. Stars mark selected rates; crosses mark numerical failures.
Figure E.2: RNN Adding training minibatch MSE and held-out validation MSE against parameter updates and wall-clock time. Failed cohorts retain their individual traces.
Figure E.3: RNN Adding loss-Hessian singular values at initialization, normalized by the largest value. The dashed line marks the numerical-rank threshold 321ε64 . All four seeds have singular values below this threshold.
Figure E.4: Autoencoder full-budget LR sweeps at selected momentum and fixed λ . Stars mark selected rates. Crosses mark failed or off-scale runs.
Figure E.5: Autoencoder test reconstruction BCE. Dots are individual seeds; ticks are three-seed means. Full-batch K-BFGS(L) has one failed seed and therefore no three-seed mean.
Figure E.6: Autoencoder training objective against parameter updates and wall-clock time. Shading spans the three seed values.
Figure E.7: Small PINN learning-rate sweeps on seed 0, scored by the mean of the final five training losses. Stars mark selected rates, crosses numerical failures, and triangles off-scale finite scores.
Figure E.8: FP32 PINN relative solution error. Columns: Wave, Convection, Reaction. Rows: fixed and resampled collocation points. The horizontal continuation of a terminated L-BFGS run represents its unchanged solution.
Figure E.9: FP64 PINN training objectives with the FP32-selected settings. The gradient budget includes replay and line-search evaluations.
Figure E.10: FP64 PINN solution errors with the FP32-selected settings. Settings with failed seeds are shown as individual traces.
Figure E.11: FP32 PINN training objectives against parameter updates.
Figure E.12: FP32 PINN training objectives against wall-clock time. K-FAC and K-BFGS(L) are omitted here because their timers also include construction time.
Figure E.13: PirateNet full-budget learning-rate sweeps, scored by unweighted validation loss. Stars mark selected rates; crosses mark failures.
Figure E.14: PirateNet unweighted PDE and initial-condition MSE against updates and wall-clock time. All curves stop at 300,600 gradient evaluations for Allen–Cahn and 100,200 for KdV; Kron therefore uses fewer parameter updates.
Figure E.15: PirateNet relative L2 error over the reference space–time grid, up to the same gradient budgets as the training-loss plots. This metric is not used to select the learning rate.
Figure E.16: PIDM LR selection after 10,000 updates. The score is the EMA validation objective. Stars mark selected rates; crosses mark numerical failures.
Figure E.17: PIDM training objective (unaveraged parameters) and validation objective (EMA parameters) against gradient evaluations, updates, and wall-clock time. Every curve stops at 100,000 gradient evaluations, corresponding to approximately 90,900 parameter updates for Kron and 100,000 for the baselines.
Figure F.1: RNN Adding validation MSE after 40,000 gradient evaluations, with a separately selected learning rate for each λ . Points show means and bars show seed ranges. The dashed line is the mean of the completed Muon comparison. The cross above the axis denotes three numerical failures at λ=9999 , not a measured loss.
Figure F.2: Relative QME solution error as the root and inverse iteration counts vary. Each cell is the median over three matrix seeds; all panels share a logarithmic color scale. Lower values are better. White outlines mark the default 18 root and 10 inverse iterations.
Figure F.3: RNN validation MSE as root iterations vary (left) and inverse iterations vary (right). Legend entries give root/inverse counts; 18/10 is the default. Each completed setting uses a separately selected learning rate and three fresh seeds. Lines show means and shading spans the seed minimum and maximum. Insets enlarge the last 10,000 gradient evaluations on a linear MSE axis with shared limits. The 12/10 setting failed at all three tested tuning rates and has no learning curve.
Figure F.4: Learning curves for different refresh intervals. Rows show RNN validation MSE and minibatch MNIST training objective; columns show gradient evaluations and wall-clock time. All runs stop at the same task-specific gradient budget, including replay: 40,000 for RNN and 6,000 for MNIST. Lines show three-seed means and shading spans the seed minimum and maximum. Wall-clock time includes evaluation and diagnostic overhead. Orange lines mark the default K=10 .
Sign-based and LMO-inspired optimizers have recently attracted substantial attention in deep learning due to their strong performance and low memory footprint. However, their fixed-magnitude updates can hurt terminal convergence: they decouple update mechanisms from gradient magnitudes and fail to account for parameter heterogeneity, often leading to oscillation rather than convergence. We propose SoftSignum, a smooth relaxation of sign-based optimization that replaces the hard sign map with a temperature-controlled soft-sign transformation, enabling a parameter-wise transition from sign-like updates to magnitude-sensitive SGD-like steps. We complement it with an adaptive quantile-based temperature schedule and extend the same principle to matrix-valued optimizers, obtaining SoftMuon. We also develop a generalized geometry-relaxation framework based on strongly convex regularizers and Fenchel conjugates, proving convergence in stochastic non-convex setting. Experiments on diverse deep learning tasks, including LLM pretraining, show that SoftSignum and SoftMuon consistently improve over their hard sign-based counterparts and standard AdamW.
We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point's residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to find the minimum-norm parameter update that best satisfies all conditions at once. In practice, this pseudoinverse is approximated via a truncated singular value decomposition, retaining only the k most significant directions. We show that Sven can be understood as a natural gradient method generalized to the overparametrized regime, recovering natural gradient descent in the underparametrized limit. We test Sven on a variety of regression and classification tasks, including small-scale language modeling with transformers, and find that it is competitive with leading baselines such as Adam, Muon, and K-FAC. We also discuss Sven's memory overhead, which presents a barrier to scaling under a naive implementation, and introduce an optimized implementation that keeps memory usage on par with standard baselines under mild restrictions on model architecture. Beyond standard machine learning benchmarks, we anticipate that Sven will find natural application in scientific computing settings where custom loss functions decompose into several conditions.
Samuel Bright-Thonney, Thomas R. Harvey, Andre Lukas +1
Department of Physics, Massachusetts Institute of Technology · The NSF Institute for Artificial Intelligence and Fundamental Interactions · Rudolf Peierls Centre for Theoretical Physics, University of Oxford +2
Geometry-aware optimizers such as Newton and natural gradient can improve conditioning in deep learning, but scalable variants such as K-FAC, Shampoo, and related preconditioners usually impose structural approximations early, often discarding cross-layer interactions induced by the network computation. We introduce Layerwise LQR (LLQR), a framework for learning structured inverse preconditioners under a global layerwise optimal-control objective. The starting point is an exact equivalence: the steepest-descent step under a broad class of divergence-induced quadratic models--including Newton, Gauss-Newton, Fisher/natural-gradient, and intermediate-layer metrics--can be written as a finite-horizon Linear Quadratic Regulator (LQR) problem. This formulation serves as a reference that exposes the layerwise dynamics and cost matrices encoding the original dense geometry. We then derive a scalable relaxation that learns diagonal, (E-)Kronecker-factored, or other structured inverse preconditioners by minimizing the LQR objective and reusing them across iterations. The resulting optimizer wraps standard methods while retaining a principled connection to second-order geometry, without forming or inverting the global curvature matrix. Experiments on ResNets and Transformers show that LLQR improves optimization dynamics and often translates these gains into improved final test performance, while adding only modest wall-clock overhead. It establishes LLQR as a practical framework for geometry-aware second-order methods and a reference for evaluating scalable approximations.
Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu +2
Mila, Université de Montréal · Samsung – SAIL Montreal