Overview

Second-order optimization studies how curvature can guide updates, often reaching good solutions in fewer steps than gradient-only methods, especially on ill-conditioned problems. Methods include Newton and cubic-regularized steps, Gauss–Newton, quasi-Newton methods such as BFGS, and structured preconditioners such as K-FAC and Shampoo. For neural networks and large models, researchers use Hessian-vector products, low-rank sketches, Kronecker factors, and matrix-function approximations to reduce memory and computation, with evaluations ranging from convex benchmarks and PINNs to language-model pretraining. A central tension is how to obtain reliable curvature cheaply: stale or inaccurate estimates can destabilize training, while exact curvature and its synchronization can be too costly at scale.

Latest papers

CardsList

96 papers5 papers in the last four weeks, down 67% on the four weeks before. 0.0% of all new papers.

  1. AG-BFGS: Foundations for Accelerated Geometry Learning

    Nov 18, 2026Regina Carla Sousa, Denis Vargas, Elizabeth Wanner +1Quasi-Newton MethodsSecond-Order Optimization

  2. CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion

    Nov 10, 2026Maryam Arshad, Muhammad Khuram ShahzadNeural Network OptimizationGPU Kernel Optimization

  3. Global O(k−2)\mathcal{O}\left(k^{-2}\right) Convergence of Multilevel Gradient-Regularized Newton Methods

    Oct 26, 2026Nick Tsipinakis, Matthias VoigtOptimization Convergence AnalysisSecond-Order Optimization

  4. Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity

    Oct 19, 2026Youran Dong, Junfeng Yang, Wei Yao +1Bilevel OptimizationSecond-Order Optimization

  5. Scalable Levenberg-Marquardt Training Framework for Deep Learning

    Oct 16, 2026Xianghao Wang, Johan Barthélemy, John TaylorNeural Network OptimizationSecond-Order Optimization

  6. Calibrated αα-Barycenters for Multiobjective ff-Divergence Optimization

    Oct 12, 2026Dier Tang, Peijie Li, Xiaoyuan ZhangMulti-Objective OptimizationSecond-Order Optimization

  7. Second-order optimization of variable projection SVM models and road abnormality detection

    Oct 7, 2026Andrea Angino, Matthias Voigt, Rolf Krause +1Second-Order OptimizationSupport Vector Machines

  8. Trust-Region Optimization for Smooth Potential-Interaction Energies in Wasserstein Space

    Oct 6, 2026You Wan, Ting Gao, Jinqiao DuanTrust-Region OptimizationSecond-Order Optimization

  9. Clean: Second-order LLM Training at Linear Memory Cost via Nystr{ö}m Sketching

    Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi +3Nyström MethodOptimizer State Compression

  10. Near-Optimal Convex Optimization with Lazy Second-Order Oracles

    Oct 2, 2026Xinliang Zhang, Lesi Chen, Chengchang Liu +1Second-Order Optimization

  11. SoftServe: A Scalable Quasi-Newton Method for Deep Learning

    Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning OptimizationSecond-Order Optimization

  12. Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest

    Sep 29, 2026Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash +1Parameter EstimationSecond-Order Optimization

  13. Inertia-Corrected Newton Method For Generalized Nash Equilibria in Dynamic Games with Optimality Verification

    Sep 25, 2026Zhiyuan Zhang, Panagiotis TsiotrasNash Equilibrium ComputationGame Theory

  14. To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationarity

    Sep 24, 2026Zhiyao Zhang, Menglu Yu, Alvaro Velasquez +2Bilevel OptimizationSecond-Order Optimization

  15. On the second-order optimization for spiking neural networks

    Sep 24, 2026Ngoc Phu Doan, Ihsen AlouaniNeural Network OptimizationSecond-Order Optimization

  16. Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradients

    Sep 19, 2026Ali Beikmohammadi, Sarit Khirirat, Sindri MagnússonFederated RLMeta-Learning

  17. Optimization over covariance matrices with a parameterized metric

    Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian OptimizationSecond-Order Optimization

  18. Newton Deep Unfolding for Compressed Sensing

    Sep 13, 2026Changhua He, Xianchao XiuCompressed SensingDeep Unfolding

  19. Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling

    Sep 11, 2026Xinchen Du, Elizaveta Rebrova, Michał Dereziński +1Stochastic ApproximationSecond-Order Optimization

  20. Beyond the Matrix Sign: Quadratic Spectral Descent

    Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based OptimizationMuon Optimizer

  21. Primal Acceleration of Newton's Method

    Aug 21, 2026Nikita DoikovSmooth Convex OptimizationSecond-Order Optimization