Quasi-Newton Methods

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 10

All topics

SoftServe: A Scalable Quasi-Newton Method for Deep Learning
Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning Optimization Second-Order Optimization
University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University
lapanda: A Matrix-Free Differentiable Solver for Nonconvex Constrained Optimization Layers
Sep 27, 2026Yuankun Chen, Zifei Nie, Kangyu Lin +2Differentiable Programming Nonconvex Optimization
School of Artificial Intelligence, Jilin University, China · Graduate School of Informatics, Kyoto University, JPN · Department of Civil and Systems Engineering, Johns Hopkins University, USA
From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs
Aug 4, 2026Chenhao Si, Kang An, Shiqian Ma +1Neural PDE Solvers Second-Order Optimization
School of Data Science, The Chinese University of Hong Kong, Shenzhen, 2001 Longxiang Road, Longgang District, Shenzhen, 518172, Guangdong, China · Department of Computational Applied Mathematics and Operations Research, Rice University, 6100 Main Street, MS 134, Houston, 77005-1827, Texas, USA · Data Science and AI Institute, and Department of Applied Mathematics and Statistics, Johns Hopkins University, 3400 N. Charles Street, Baltimore, 21218, Maryland, USA
Optimization Geometrodynamics: Variational Reduction and Interaction Curvature
Jul 7, 2026Zavier LiRiemannian Optimization Second-Order Optimization
Xidian University Xi’an, China
On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism
Jul 7, 2026Don LiSecond-Order Optimization Nonconvex Optimization
Department of Mathematics & Statistics Portland State University Portland, OR 97201, USA
Hybrid deep learning-based phase diversity method for wavefront reconstruction
Jun 24, 2026Y. Rodimkov, A. Kotov, K. Burdonov +4Quasi-Newton Methods Inverse Problems
Lobachevskii State University of Nizhni Novgorod, pr. Gagarina 23, Nizhni Novgorod, 603022 Russia · Institute of Applied Physics, Russian Academy of Sciences, Nizhny Novgorod 603951, Russia
Form and Function: Machine Unlearning as a Problem of Misaligned States
May 17, 2026Kennon StewartQuasi-Newton Methods Machine Unlearning
Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA
cuRegOT: A GPU-Accelerated Solver for Entropic-Regularized Optimal Transport
May 9, 2026Yixuan QiuGPU Kernel Optimization Quasi-Newton Methods
School of Statistics and Data Science & Institute of Big Data Research Shanghai University of Finance and Economics
Quasi-Quadratic Gradient: A New Direction for Accelerating the BFGS Method in Quasi-Newton Optimization
Apr 27, 2026John ChiangSecond-Order Optimization Quasi-Newton Methods
L-SR1: Learned Symmetric-Rank-One Preconditioning
Aug 17, 2025Gal Lifshitz, Shahar Zuler, Ori Fouks +1Learning to Optimize Second-Order Optimization
Tel Aviv University, Tel Aviv, Israel.

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

SoftServe: A Scalable Quasi-Newton Method for Deep Learning

Joohwan Ko, Tetiana Parshakova, Diana Cai, Robert M. Gower

University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University

Deep Learning Optimization Second-Order Optimization Nonconvex Optimization Quasi-Newton Methods

Open paper AskPDF arXiv

Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.

Momentum

Related topics

Latest papers 10