Second-Order Optimization

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 63

All topics

Accelerating SAV-based optimization via randomized low-rank Hessian approximation
Jun 9, 2026Ryo Sagawa, Daisuke Furihata, Yuto MiyatakeOptimization Convergence Analysis Second-Order Optimization
Department of Pure and Applied Mathematics, Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarNewton-Schulz Iteration Momentum Methods
MIT
FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
Jun 1, 2026Kyunghun Nam, Sumyeong AhnDeep Learning Optimization Optimization Convergence Analysis
KENTECH, Naju, Republic of Korea.
Rethinking Bregman Divergences in Kronecker-Factored Optimizers
May 30, 2026Bing Liu, Wenjie Zhou, Chengcheng ZhaoSecond-Order Optimization Adaptive Gradient Methods
College of Control Science and Engineering, Zhejiang University · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences
Exploiting weight-space symmetries for approximating curvature
May 30, 2026Artem Artemev, Rui Xia, Benjamin M. Boyd +4Neural Network Optimization Second-Order Optimization
MediaTek, Cambridge, UK · University of Cambridge, Cambridge, UK · Vector Institute, Toronto, Canada
How Much Orthogonalization Does Muon Need?
May 29, 2026Hua HuangNewton-Schulz Iteration Neural Network Optimization
NVIDIA
Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens
May 29, 2026Junbin Qiu, Zhaowei Hong, Renzhe Xu +1Zeroth-Order Optimization Second-Order Optimization
Hong Kong University of Science and Technology (Guangzhou) · Shanghai University of Finance and Economics
Geometry-Correct Diffusion Posterior Sampling with Denoiser-Pullback Curvature Guidance and Manifold-Aligned Damping
May 27, 2026Seunghyeok Shin, Minwoo Kim, Dabin Kim +1Gauss-Newton Optimization Bayesian Inverse Problems
Department of Electrical and Computer Engineering, Inha University, Incheon, 22212, South Korea.
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
May 22, 2026Anherutowa CalvoNeural Network Optimization Second-Order Optimization
19D Labs
Real-Time Auto-Optimization in Unknown Environments via Structure-Exploiting Dual Control for Exploration and Exploitation
May 21, 2026Shiying Dong, Haoyang Yang, Qiwei Liu +1Second-Order Optimization Optimal Control
Department of Aeronautical and Aviation Engineering and Research Centre for Low Altitude Economy (RCLAE), the Hong Kong Polytechnic University, Hong Kong, China
AMO: Operator-level Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Language Model Pretraining Newton-Schulz Iteration
The Chinese University of Hong Kong · Shopee · MBZUAI +3
Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training
May 15, 2026Yishun Lu, Junhao Zhang, Zeyu Yang +1Second-Order Optimization LLM Training
Engineering Science Department University of Oxford Oxford, United Kingdom
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
May 15, 2026Yishun Lu, Wes ArmourSecond-Order Optimization Multimodal Optimization
University of Oxford, Oxford, United Kingdom
A Non-Monotone Preconditioned Trust-Region Method for Neural Network Training
May 14, 2026Andrea Angino, Bindi Çapriqi, Shega Likaj +2Neural Network Optimization Second-Order Optimization
UniDistance Suisse · King Abdullah University of Science and Technology · Universit`a della Svizzera italiana
Error whitening: Why Gauss-Newton outperforms Newton
May 11, 2026Maricela Best McKay, Nathan P. Lawrence, Brian Wetton +1Gauss-Newton Optimization Deep Learning Optimization
University of British Columbia · University of California, Berkeley
Optimality of Sub-network Laplace Approximations: New Results and Methods
May 9, 2026Swarnali Raha, Kshitij Khare, Rohit K PatraBayesian Neural Networks Uncertainty Quantification
Department of Statistics University of Florida Gainesville, FL · Linkedin Inc. New York, NY
Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit
May 8, 2026Konstantin Riedl, Konstantinos Spiliopoulos, Justin SirignanoNeural Network Optimization Spectral Bias
Mathematical Institute University of Oxford · Department of Mathematics & Statistics Boston University
Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
May 7, 2026Ruotong Sun, Ermin WeiLanguage Model Pretraining Deep Learning Optimization
†Department of Industrial Engineering & Management Sciences, Northwestern University · ∗Department of Electrical & Computer Engineering, Northwestern University
Fast Gauss-Newton for Multiclass Cross-Entropy
May 7, 2026Mikalai Korbit, Mario ZanonGauss-Newton Optimization Neural Network Optimization
IMT School for Advanced Studies Lucca
Layerwise LQR for Geometry-Aware Optimization of Deep Networks
May 5, 2026Simon Dufort-Labbé, Pierre-Luc Bacon, Razvan Pascanu +2Deep Learning Optimization Second-Order Optimization
Mila, Université de Montréal · Samsung – SAIL Montreal
A second-order method landing on the Stiefel manifold via Newton $\unicode{x2013}$ Schulz iteration
May 4, 2026Xinhui Xiong, Bin Gao, P. -A. AbsilNewton-Schulz Iteration Stiefel Manifold Optimization
State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, and University of Chinese Academy of Sciences, China · State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, China · ICTEAM Institute, UCLouvain, Belgium
Exact Higher-Order Derivatives for SE(3) via Analytical/AD Methods
May 4, 2026Frank O. KuehnelAutomatic Differentiation Lie Group Methods
Sigma Point Labs, Denver, CO, USA
Gradient Regularized Newton Boosting Trees with Global Convergence
May 1, 2026Nikita Zozoulenko, Daniel Falkowski, Thomas Cass +1Gradient-Boosted Decision Trees Optimization Convergence Analysis
Department of Mathematics, Imperial College London, UK · Department of Mathematics, Stockholm University, Sweden · School of Computer Science, University of St. Gallen, Switzerland
Quasi-Quadratic Gradient: A New Direction for Accelerating the BFGS Method in Quasi-Newton Optimization
Apr 27, 2026John ChiangSecond-Order Optimization Quasi-Newton Methods
Inference of Online Newton Methods with Nesterov's Accelerated Sketching
Apr 25, 2026Haoxuan Wang, Xinchen Du, Sen NaStochastic Approximation Second-Order Optimization
School of Industrial and Systems Engineering, Georgia Institute of Technology.
Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
Apr 24, 2026Kennon StewartSecond-Order Optimization Machine Unlearning
Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA
Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks
Apr 16, 2026Kang An, Chenhao Si, Shiqian Ma +1Neural Network Optimization Second-Order Optimization
School of Data Science The Chinese University of Hong Kong, Shenzhen Shenzhen, China
DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
Feb 2, 2026Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis +2Deep Learning Optimization GPU Acceleration
Institute of Science and Technology Austria · Lancaster University · Red Hat AI
Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates
Dec 4, 2025Thibaut Boissin, Thomas Massena, Franck Mamalet +1Newton-Schulz Iteration Neural Network Optimization
IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2
Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimization
Sep 3, 2025Wu Lin, Scott C. Lowe, Felix Dangel +3Deep Learning Optimization Neural Network Optimization
1Vector Institute, Toronto, Canada · University of Cambridge, Cambridge, United Kingdom · 3Microsoft, United States +1

Jun 9, 2026, math.OCJ/K move · Enter open · S save

Accelerating SAV-based optimization via randomized low-rank Hessian approximation

Ryo Sagawa, Daisuke Furihata, Yuto Miyatake

Department of Pure and Applied Mathematics, Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka

Optimization Convergence Analysis Second-Order Optimization

Open paper AskPDF arXiv

We propose a new optimization method, the Nyström-enhanced relaxed scalar auxiliary variable method (N-RSAV), which incorporates curvature information into the RSAV framework to accelerate convergence while preserving an unconditional modified energy dissipation law. Existing RSAV-based methods rely solely on first-order information and often suffer from slow convergence, particularly for ill-conditioned problems such as those arising in physics-informed neural networks (PINNs). To address this limitation, we design the linear operator in the RSAV scheme using approximate Hessian information obtained from a randomized low-rank Nyström approximation. To preserve the dissipation structure, we enforce positive semidefiniteness through eigenvalue truncation. Furthermore, we introduce an adaptive strategy that reuses the approximate Hessian based on the deviation between the original and modified energies, significantly reducing computational cost. We also provide a convergence analysis of the RSAV scheme with a general positive semidefinite operator under the Polyak-Lojasiewicz (PL) condition and establish corresponding convergence guarantees for N-RSAV under the PL condition and an additional convexity assumption. Numerical experiments on ill-conditioned problems with effectively low-rank structure, including convex quadratic problems and training of PINNs, demonstrate that the proposed methods achieve substantially faster convergence than conventional RSAV-based approaches.

Momentum

Related topics

Latest papers 63