Quasi-Newton Methods

Ask about this topic

Overview

Quasi-Newton methods use changes in gradients to build or update an approximation to the Hessian or its inverse, aiming for faster convergence than first-order methods at lower cost than exact Newton steps. Common families include BFGS, limited-memory L-BFGS, and Broyden methods, often combined with line searches, trust regions, or safeguards that control conditioning. Current work applies these methods to settings such as physics-informed neural networks, large neural models, and differentiable constrained optimization, where memory-efficient and GPU-friendly curvature estimates are important. A central tension is that curvature can improve optimization, especially on ill-conditioned problems, but its cost, stability, and relationship to generalization remain challenging in large nonconvex models.

Latest papers

CardsList

15 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

  1. SoftServe: A Scalable Quasi-Newton Method for Deep Learning

    Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning OptimizationSecond-Order Optimization

    University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University

  2. lapanda: A Matrix-Free Differentiable Solver for Nonconvex Constrained Optimization Layers

    Sep 27, 2026Yuankun Chen, Zifei Nie, Kangyu Lin +2Differentiable ProgrammingNonconvex Optimization

    School of Artificial Intelligence, Jilin University, China · Graduate School of Informatics, Kyoto University, JPN · Department of Civil and Systems Engineering, Johns Hopkins University, USA

  3. A family of spectral conjugate gradient algorithms derived by least-squares approximations based on a modified quasi--Newton update with application to a revised robust binary classification model

    Sep 11, 2026Saman Babaie-Kafaki, Maryam Khoshsimaye-Bargard, Ahmad MousaviRobust ClassificationNonconvex Optimization

    Faculty of Engineering, Free University of Bozen–Bolzano, 39100 Bolzano, Italy · Department of Mathematics, Faculty of Mathematics, Statistics and Computer Science, Semnan University, Semnan, Iran · Department of Mathematics and Statistics, American University, Washington, D.C., USA

  4. From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs

    Aug 4, 2026Chenhao Si, Kang An, Shiqian Ma +1Neural PDE SolversSecond-Order Optimization

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, 2001 Longxiang Road, Longgang District, Shenzhen, 518172, Guangdong, China · Department of Computational Applied Mathematics and Operations Research, Rice University, 6100 Main Street, MS 134, Houston, 77005-1827, Texas, USA · Data Science and AI Institute, and Department of Applied Mathematics and Statistics, Johns Hopkins University, 3400 N. Charles Street, Baltimore, 21218, Maryland, USA

  5. Optimization Geometrodynamics: Variational Reduction and Interaction Curvature

    Jul 7, 2026Zavier LiRiemannian OptimizationSecond-Order Optimization

    Xidian University Xi’an, China

  6. On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

    Jul 7, 2026Don LiSecond-Order OptimizationNonconvex Optimization

    Department of Mathematics & Statistics Portland State University Portland, OR 97201, USA

  7. Hybrid deep learning-based phase diversity method for wavefront reconstruction

    Jun 24, 2026Y. Rodimkov, A. Kotov, K. Burdonov +4Quasi-Newton MethodsInverse Problems

    Lobachevskii State University of Nizhni Novgorod, pr. Gagarina 23, Nizhni Novgorod, 603022 Russia · Institute of Applied Physics, Russian Academy of Sciences, Nizhny Novgorod 603951, Russia

  8. Form and Function: Machine Unlearning as a Problem of Misaligned States

    May 17, 2026Kennon StewartQuasi-Newton MethodsMachine Unlearning

    Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA

  9. cuRegOT: A GPU-Accelerated Solver for Entropic-Regularized Optimal Transport

    May 9, 2026Yixuan QiuGPU Kernel OptimizationQuasi-Newton Methods

    School of Statistics and Data Science & Institute of Big Data Research Shanghai University of Finance and Economics

  10. Quasi-Quadratic Gradient: A New Direction for Accelerating the BFGS Method in Quasi-Newton Optimization

    Apr 27, 2026John ChiangSecond-Order OptimizationQuasi-Newton Methods

  11. Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks

    Apr 6, 2026Anas Jnini, Elham Kiyani, Khemraj Shukla +5Physics-Informed MLNeural PDE Solvers

    Department of Information Engineering and Computer Science, University of Trento, Trento, Italy · Division of Applied Mathematics, Brown University, Providence, RI, USA · Departament de Física, Universitat d’Alacant, Spain +3

  12. Unifying Optimization and Dynamics to Parallelize Sequential Computation: A Guide to Parallel Newton Methods for Breaking Sequential Bottlenecks

    Mar 17, 2026Xavier GonzalezLyapunov Exponent EstimationDynamical Systems

    STANFORD UNIVERSITY

  13. Self-Scaled Broyden Family of Quasi-Newton Methods in JAX

    Mar 11, 2026Ivan Bioli, Mikel Mendibe AbarrategiQuasi-Newton Methods

    Dipartimento di Matematica, Universit`a degli Studi di Pavia, Via A. Ferrata 5, 27100 Pavia, Italy · Dipartimento di Ingegneria Civile e Architettura, Universit`a degli Studi di Pavia, Via A. Ferrata 3, 27100 Pavia, Italy · University of the Basque Country, 48013 Bilbao, Spain +1

  14. Neural network optimization strategies and the topography of the loss landscape

    Feb 24, 2026Jianneng Yu, Alexandre V. MorozovNeural Network OptimizationStochastic Gradient Descent

    Department of Physics & Astronomy, Rutgers, The State University of New Jersey, 136 Frelinghuysen Rd., Piscataway, NJ 08854, U.S.A.

  15. L-SR1: Learned Symmetric-Rank-One Preconditioning

    Aug 17, 2025Gal Lifshitz, Shahar Zuler, Ori Fouks +1Learning to OptimizeSecond-Order Optimization

    Tel Aviv University, Tel Aviv, Israel.