Second-Order Optimization

Ask about this topic

Overview

Second-order optimization studies how curvature can guide updates, often reaching good solutions in fewer steps than gradient-only methods, especially on ill-conditioned problems. Methods include Newton and cubic-regularized steps, Gauss–Newton, quasi-Newton methods such as BFGS, and structured preconditioners such as K-FAC and Shampoo. For neural networks and large models, researchers use Hessian-vector products, low-rank sketches, Kronecker factors, and matrix-function approximations to reduce memory and computation, with evaluations ranging from convex benchmarks and PINNs to language-model pretraining. A central tension is how to obtain reliable curvature cheaply: stale or inaccurate estimates can destabilize training, while exact curvature and its synchronization can be too costly at scale.

Latest papers

CardsList

92 papers12 papers in the last four weeks, up 20% on the four weeks before. 0.1% of all new papers.

  1. Global O(k−2)\mathcal{O}\left(k^{-2}\right) Convergence of Multilevel Gradient-Regularized Newton Methods

    Oct 26, 2026Nick Tsipinakis, Matthias VoigtOptimization Convergence AnalysisSecond-Order Optimization

    Faculty of Mathematics and Computer Science, UniDistance Suisse, Brig, Switzerland

  2. Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity

    Oct 19, 2026Youran Dong, Junfeng Yang, Wei Yao +1Bilevel OptimizationSecond-Order Optimization

    School of Mathematics, Nanjing University, Nanjing, P. R. China. · School of Artificial Intelligence, Southern University of Science and Technology, Shenzhen, P. R. China. · Department of Mathematics and National Center for Applied Mathematics Shenzhen, Southern University of Science and Technology, Shenzhen, P. R. China.

  3. A Statistical Viewpoint on Adam, Shampoo and Muon

    Oct 17, 2026Tetiana Parshakova, Robert M. Gower, Paulo OrensteinStochastic OptimizationAdaptive Gradient Methods

  4. Scalable Levenberg-Marquardt Training Framework for Deep Learning

    Oct 16, 2026Xianghao Wang, Johan Barthélemy, John TaylorNeural Network OptimizationSecond-Order Optimization

    School of Computing Australian National University Canberra, Australia · NVIDIA CA, United States

  5. A Refresh Interval Does Not Fully Specify a Refresh Schedule: Learning-Rate-Dependent Placement in Kronecker-Preconditioned Language Model Training

    Oct 13, 2026Xinyu Wang, Zhiwei Zhai, Yu Gu +6Deep Learning OptimizationEfficient Language Model Training

    McGill University · The Chinese University of Hong Kong · University of Manitoba +1

  6. Calibrated αα-Barycenters for Multiobjective ff-Divergence Optimization

    Oct 12, 2026Dier Tang, Peijie Li, Xiaoyuan ZhangMulti-Objective OptimizationSecond-Order Optimization

    The University of Hong Kong · Zhongguancun Academy

  7. Gauss-Newton Accuracy and Indefinite Hessians: Uniform Coexistence in Low-Cost Sets

    Oct 7, 2026Kihun Rhee, Hanjoon ByunGauss-Newton OptimizationSecond-Order Optimization

  8. Second-order optimization of variable projection SVM models and road abnormality detection

    Oct 7, 2026Andrea Angino, Matthias Voigt, Rolf Krause +1Second-Order OptimizationSupport Vector Machines

    Faculty of Mathematics and Computer Science, UniDistance Suisse, Brig, Switzerland · CEMSE Division, KAUST, Thuwal, Saudi Arabia · Eötvös Loránd University, Faculty of Informatics, Department of Numerical Analysis

  9. Trust-Region Optimization for Smooth Potential-Interaction Energies in Wasserstein Space

    Oct 6, 2026You Wan, Ting Gao, Jinqiao DuanTrust-Region OptimizationSecond-Order Optimization

    School of Mathematics, Nanjing University, Nanjing, 210093, China · School of Sciences, Great Bay University, Dongguan 523000, China · Guangdong Provincial Key Laboratory of Mathematical and Neural Dynamical Systems, Great Bay University, Dongguan 523000, China +3

  10. Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching

    Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi +3Nyström MethodSecond-Order Optimization

    Mila & DIRO, Université de Montréal · Critical ML, University of Waterloo · Independent Researcher +2

  11. Near-Optimal Convex Optimization with Lazy Second-Order Oracles

    Oct 2, 2026Xinliang Zhang, Lesi Chen, Chengchang Liu +1Second-Order Optimization

    IIIS, Tsinghua University · Department of Artificial Intelligence, Westlake University.

  12. SoftServe: A Scalable Quasi-Newton Method for Deep Learning

    Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning OptimizationSecond-Order Optimization

    University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University

  13. Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest

    Sep 29, 2026Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash +1Parameter EstimationSecond-Order Optimization

    Department of Mathematics, Institut Teknologi Sepuluh Nopember, Surabaya, 60111, Indonesia · Department of Mathematics, Khalifa University of Science & Technology, Abu Dhabi, PO Box 127788, United Arab Emirates

  14. Inertia-Corrected Newton Method For Generalized Nash Equilibria in Dynamic Games with Optimality Verification

    Sep 25, 2026Zhiyuan Zhang, Panagiotis TsiotrasNash Equilibrium ComputationGame Theory

    School of Aerospace Engineering, Institute for Robotics and Intelligent Machines, Georgia Institute of Technology, Atlanta, GA 30332, USA

  15. To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationarity

    Sep 24, 2026Zhiyao Zhang, Menglu Yu, Alvaro Velasquez +2Bilevel OptimizationSecond-Order Optimization

    The Ohio State University · Meta · University of Colorado Boulder +1

  16. Anchored Extra-Proximal Methods: Optimal Higher-Order Methods for Monotone Inclusion Problems

    Sep 24, 2026Ruichen Jiang, TaeHo YoonOracle ComplexitySecond-Order Optimization

    Google Research · Johns Hopkins University

  17. On the second-order optimization for spiking neural networks

    Sep 24, 2026Ngoc Phu Doan, Ihsen AlouaniNeural Network OptimizationSecond-Order Optimization

    Centre for Secure Information Technologies Queen’s University Belfast

  18. Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradients

    Sep 19, 2026Ali Beikmohammadi, Sarit Khirirat, Sindri MagnússonFederated RLMeta-Learning

    Department of Computer and Systems Sciences, Stockholm University, Stockholm, Sweden. · Faculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, Thailand.

  19. Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers

    Sep 17, 2026Nilo Schwencke, Roland MaierGauss-Newton OptimizationNeural PDE Solvers

  20. Optimization over covariance matrices with a parameterized metric

    Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian OptimizationSecond-Order Optimization

    Nanyang Technological University, Singapore · Microsoft India · Indian Institute of Technology Bombay, India

  21. Newton Deep Unfolding for Compressed Sensing

    Sep 13, 2026Changhua He, Xianchao XiuCompressed SensingDeep Unfolding

    School of Mechatronic Engineering and Automation, Shanghai University

  22. Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling

    Sep 11, 2026Xinchen Du, Elizaveta Rebrova, Michał Dereziński +1Stochastic ApproximationSecond-Order Optimization

    School of Industrial and Systems Engineering, Georgia Institute of Technology · Department of Operations Research and Financial Engineering, Princeton University

  23. Beyond the Matrix Sign: Quadratic Spectral Descent

    Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based OptimizationMuon Optimizer

    School of Electronic Information and Communications Huazhong University of Science and Technology

  24. Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View

    Aug 23, 2026Rodion PodorozhnyLoss LandscapeAdaptive Gradient Methods

    Texas State University

  25. Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

    Aug 22, 2026Rodion PodorozhnySecond-Order OptimizationNeural Network Optimization

    Texas State University

  26. Primal Acceleration of Newton's Method

    Aug 21, 2026Nikita DoikovSecond-Order Optimization

    School of Operations Research and Information Engineering (ORIE), Cornell University, Ithaca, NY, USA.

  27. Dion3: Full-Stack Orthogonal Updates

    Aug 12, 2026Noah Amsel, Jack Zhang, Kwangjun Ahn +5Newton-Schulz IterationGPU Kernel Optimization

    New York University · Microsoft Research · Princeton University +3

  28. Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature

    Aug 10, 2026Tong CheGauss-Newton OptimizationInformation Geometry

    NVIDIA Research

  29. Halpern Iteration Achieves O~(ε−1/p)\tilde{\mathcal{O}}(ε^{-1/p}) ppth-Order Oracle Complexity for Monotone Variational Inequalities

    Aug 9, 2026Lesi Chen, Xinliang Zhang, Hengyu Wang +3Second-Order Optimization

    1IIIS, Tsinghua University · Apex Intelligence · School of Mathematical Sciences, Tongji University +2

  30. Newton-Schulz Retraction-Based Inference Enables Hidden Quantum Markov Models to Outperform Classical HMMs

    Aug 6, 2026Ning NingNewton-Schulz IterationMarkov Models

    Department of Statistics, Texas A&M University College Station, Texas, USA