Second-Order Optimization

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 63

All topics

Gauss-Newton Accuracy and Indefinite Hessians: Uniform Coexistence in Low-Cost Sets
Oct 7, 2026Kihun Rhee, Hanjoon ByunGauss-Newton Optimization Second-Order Optimization
Second-order optimization of variable projection SVM models and road abnormality detection
Oct 7, 2026Andrea Angino, Matthias Voigt, Rolf Krause +1Second-Order Optimization Support Vector Machines
Faculty of Mathematics and Computer Science, UniDistance Suisse, Brig, Switzerland · CEMSE Division, KAUST, Thuwal, Saudi Arabia · Eötvös Loránd University, Faculty of Informatics, Department of Numerical Analysis
Trust-Region Optimization for Smooth Potential-Interaction Energies in Wasserstein Space
Oct 6, 2026You Wan, Ting Gao, Jinqiao DuanTrust-Region Optimization Second-Order Optimization
School of Mathematics, Nanjing University, Nanjing, 210093, China · School of Sciences, Great Bay University, Dongguan 523000, China · Guangdong Provincial Key Laboratory of Mathematical and Neural Dynamical Systems, Great Bay University, Dongguan 523000, China +3
Clean: Second-order LLM Training at Linear Memory Cost via Nyström Sketching
Oct 3, 2026Beheshteh T. Rakhshan, Sahar Rajabi, Maziar Sargordi Shikai Fang +2Second-Order Optimization Memory-Efficient Training
Mila & DIRO, Université de Montréal · Critical ML, University of Waterloo · Independent Researcher +1
SoftServe: A Scalable Quasi-Newton Method for Deep Learning
Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Deep Learning Optimization Second-Order Optimization
University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University
Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest
Sep 29, 2026Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash +1Parameter Estimation Second-Order Optimization
Department of Mathematics, Institut Teknologi Sepuluh Nopember, Surabaya, 60111, Indonesia · Department of Mathematics, Khalifa University of Science & Technology, Abu Dhabi, PO Box 127788, United Arab Emirates
Anchored Extra-Proximal Methods: Optimal Higher-Order Methods for Monotone Inclusion Problems
Sep 24, 2026Ruichen Jiang, TaeHo YoonOracle Complexity Second-Order Optimization
Google Research · Johns Hopkins University
On the second-order optimization for spiking neural networks
Sep 24, 2026Ngoc Phu Doan, Ihsen AlouaniNeural Network Optimization Second-Order Optimization
Centre for Secure Information Technologies Queen’s University Belfast
Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers
Sep 17, 2026Nilo Schwencke, Roland MaierGauss-Newton Optimization Neural PDE Solvers
Optimization over covariance matrices with a parameterized metric
Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian Optimization Second-Order Optimization
Nanyang Technological University, Singapore · Microsoft India · Indian Institute of Technology Bombay, India
Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling
Sep 14, 2026Xinchen Du, Elizaveta Rebrova, Micha\l Dereziński +1Stochastic Approximation Second-Order Optimization
School of Industrial and Systems Engineering, Georgia Institute of Technology · Department of Operations Research and Financial Engineering, Princeton University · Computer Science and Engineering, University of Michigan
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based Optimization Muon Optimizer
School of Electronic Information and Communications Huazhong University of Science and Technology
Dion3: Full-Stack Orthogonal Updates
Aug 12, 2026Noah Amsel, Jack Zhang, Kwangjun Ahn +5Newton-Schulz Iteration GPU Kernel Optimization
New York University · Microsoft Research · Princeton University +3
Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature
Aug 10, 2026Tong CheGauss-Newton Optimization Information Geometry
NVIDIA Research
Halpern Iteration Achieves $\tilde{\mathcal{O}}(ε^{-1/p})$ $p$ th-Order Oracle Complexity for Monotone Variational Inequalities
Aug 9, 2026Lesi Chen, Xinliang Zhang, Hengyu Wang +3Second-Order Optimization
1IIIS, Tsinghua University · Apex Intelligence · School of Mathematical Sciences, Tongji University +2
Newton-Schulz Retraction-Based Inference Enables Hidden Quantum Markov Models to Outperform Classical HMMs
Aug 6, 2026Ning NingNewton-Schulz Iteration Markov Models
Department of Statistics, Texas A&M University College Station, Texas, USA
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Aug 5, 2026Thang Do, Steffen Dereich, Arnulf JentzenStochastic Optimization Newton-Schulz Iteration
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistics, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam · Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany +2
From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs
Aug 4, 2026Chenhao Si, Kang An, Shiqian Ma +1Neural PDE Solvers Second-Order Optimization
School of Data Science, The Chinese University of Hong Kong, Shenzhen, 2001 Longxiang Road, Longgang District, Shenzhen, 518172, Guangdong, China · Department of Computational Applied Mathematics and Operations Research, Rice University, 6100 Main Street, MS 134, Houston, 77005-1827, Texas, USA · Data Science and AI Institute, and Department of Applied Mathematics and Statistics, Johns Hopkins University, 3400 N. Charles Street, Baltimore, 21218, Maryland, USA
Generalized Quadratic Gradient: A New Direction in Optimization via the Fusion of Positive-Definite Curvature Matrices and Gradients into A Unified Framework
Aug 3, 2026John ChiangSecond-Order Optimization
Variational Boosting for Physics-Informed Neural Networks
Jul 27, 2026Kaylee Vo, Pavlos ProtopapasEnsemble Learning PDE Solving
Harvard University
A Defense of the Quadratic Model
Jul 23, 2026Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3Language Model Pretraining Edge of Stability
1Kempner Institute at Harvard University · †: Work done while at Harvard. · 2MIT
On Optimization Complexity of Second-Order Certified Unlearning
Jul 22, 2026Nikita Doikov, Anastasia KoloskovaSecond-Order Optimization Machine Unlearning
Cornell University, Ithaca, USA · University of Zurich, Zurich, Switzerland
An Efficient Newton Algorithm for Nonnegative Matrix Factorization with the Kullback-Leibler Divergence
Jul 15, 2026Damien Lesens, Jérémy E. Cohen, Bora UçarNon-Negative Matrix Factorization Low-Rank Matrix Decomposition
ENS de Lyon · Univ. Lyon, INSA-Lyon, UCBL, UJM, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621, Lyon, France. · CNRS and LIP (UMR5668, Université de Lyon - ENS de Lyon - UCBL - CNRS - Inria), 46 allée d'Italie, Lyon, France.
Optimization Geometrodynamics: Variational Reduction and Interaction Curvature
Jul 7, 2026Zavier LiRiemannian Optimization Second-Order Optimization
Xidian University Xi’an, China
On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism
Jul 7, 2026Don LiSecond-Order Optimization Nonconvex Optimization
Department of Mathematics & Statistics Portland State University Portland, OR 97201, USA
One More Time: Revisiting Neural Quantum States from a Reinforcement Learning Perspective
Jul 2, 2026Juan Agustín Duque, Sergio García Heredia, Vinicius Hernandes +4Neural Quantum States Second-Order Optimization
Mila Quebec AI Institute, Université de Montréal, Montréal, Canada · Applied Quantum Algorithms ⟨aQaL⟩, LIACS & LION, Leiden University, Leiden, Netherlands · QuTech and Kavli Institute of Nanoscience, Delft University of Technology, Delft, Netherlands +1
An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks
Jul 2, 2026Joseph Webb, Sadok Jerad, Coralia CartisGauss-Newton Optimization PDE Solving
Mathematical Institute, University of Oxford, Oxford, United Kingdom
Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization
Jun 25, 2026Ziyuan Tang, Tianshi Xu, Yousef Saad +1Newton-Schulz Iteration Neural Network Optimization
Department of Computer Science and Engineering, University of Minnesota, Minneapolis · Department of Mathematics, Emory University, Atlanta, GA 30322
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Edge of Stability Transformer
Pluralis Research
A Riemannian Approach to Low-Rank Optimal Transport
Jun 10, 2026Pratik Jawanpuria, Bamdev MishraRiemannian Optimization Second-Order Optimization
Centre for Machine Intelligence and Data Science, IIT Bombay · Microsoft India

Oct 7, 2026, cs.LGJ/K move · Enter open · S save

Gauss-Newton Accuracy and Indefinite Hessians: Uniform Coexistence in Low-Cost Sets

Kihun Rhee, Hanjoon Byun

Gauss-Newton Optimization Second-Order Optimization Nonconvex Optimization Ridge Regression

Open paper AskPDF arXiv

We study the accuracy of Gauss-Newton curvature in ridge-regularized nonlinear least squares. Under local regularity and persistence of level-set curvature magnitude along an exact-fit section, we prove uniform coexistence of two curvature regimes. Global minimizers exist, and every global minimizer has relative Hessian error below

(1+\sqrt2)/8

, while the same low-cost set contains a point with an indefinite Hessian and relative error at least

15/8

. One positive ridge cap works for all independent center and label perturbations in fixed neighborhoods and every positive ridge weight up to the cap. These neighborhoods do not shrink as the ridge weight tends to zero. A pointwise certificate based on the current prediction level set controls the normal, mixed, and tangent parts of the Hessian correction. We prove a sharp relative-error bound over the stated pointwise class when the prediction map and ridge vary. Analytic examples describe the roles of output alignment, curvature orientation, and persistence. A separate structural result gives full Jacobian row rank throughout low-cost sets and exact interpolation near a rank-deficient reference.

Momentum

Related topics

Latest papers 63