Newton

Follow this topic Ask about this topic

Momentum

12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 80

All topics

SoftServe: A Scalable Quasi-Newton Method for Deep Learning
Oct 1, 2026Joohwan Ko, Tetiana Parshakova, Diana Cai +1Neural Network Optimization Newton
University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University
Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentum
Sep 30, 2026Hanyng Peng, Hui Wang, Yue YuMomentum Stochastic Gradient Descent Newton
Pengcheng Laboratory, Shenzhen, China
Hybrid Joint-Selective Optimization: Reduced-Space Levenberg-Marquardt Refinement of Low-Dimensional Parameters of Interest
Sep 29, 2026Muhammad Luthfi Shahab, Gabriella Alfa Indahsari, Imam Mukhlash +1Newton Fixed-Point Iteration
Department of Mathematics, Institut Teknologi Sepuluh Nopember, Surabaya, 60111, Indonesia · Department of Mathematics, Khalifa University of Science & Technology, Abu Dhabi, PO Box 127788, United Arab Emirates
Learning Conditional Expectation Operators via Functional Newton Updates
Sep 28, 2026Thiago Ramos, Alek Fröhlich, Daniel Perazzo +1Density Ratio Estimation Spectral Representation Method
Federal University of São Carlos · CSML, Istituto Italiano di Tecnologia and University of Genoa · CSML, Istituto Italiano di Tecnologia and University College London
IMC-CLINIC: Coupled Loss-Informed Newton Iterations for Clipping in Analog In-Memory Computing
Sep 28, 2026Yung-Chin Chen, Chia-Yu Chen, Naveen VermaIn-Memory Computing Gradient Clipping
Princeton University, NJ, USA · EnCharge AI, CA, USA
NS-ATTENTION: Newton-Schulz Transformations of Attention Outputs in Vision Transformers
Sep 23, 2026Xiaohe Jiang, Guoqiang Zhang, Tianjin Huang +1Transformer Attention Linear Attention
University of Exeter · University of Exeter, Exeter, United Kingdom
Beyond PINNs: A Unified Gauss--Newton and Petrov--Galerkin Framework for Neural and Hybrid PDE Solvers
Sep 17, 2026Nilo Schwencke, Roland MaierVariational Formulation Neural Partial Differential Equation Solvers
SOL-SLAM: Inverse Compositional Gauss-Newton Direct Registration for Fast Sonar-Only Local SLAM
Sep 16, 2026Kalvik Jakkala, Jason O'KaneForward-Looking Sonar Simultaneous Localization And Mapping
Texas A&M University
Hybrid coupling with numerics-informed neural networks and the overlapping Schwarz alternating method
Sep 15, 2026George Chumbipuma, Irina Tezaur, Alejandro Diaz +1One-Dimensional Viscous Burgers Equation Newton
Rice University · Sandia National Laboratories
Regularized Least Squares Training of Quadratic Neural Networks with Applications to System Identification
Sep 15, 2026Luis Rodrigues, Zachary Yetman Van Egmond, Mohammad R. Amiri FardLeast Squares System Identification
Department of Electrical and Computer Engineering Concordia University Montr´eal, QC, Canada
Inference for Newton Methods with Accelerated Sketch-and-Project via Random Scaling
Sep 14, 2026Xinchen Du, Elizaveta Rebrova, Micha\l Dereziński +1Newton Covariance
School of Industrial and Systems Engineering, Georgia Institute of Technology · Department of Operations Research and Financial Engineering, Princeton University · Computer Science and Engineering, University of Michigan
Online Inverse Integer Linear Optimization via Small-Gradient Skipping: Constant Regret and Finite Mistakes
Sep 9, 2026Akira KitaokaLinear Programming Nonconvex
HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation
Sep 1, 2026Alessandro Zirilli, Davide Marincione, Evgenios M. Kornaropoulos +2Homomorphic Encryption Newton
Sapienza University of Rome · George Mason University · Paradigma
Gradient-Update Mismatch: Rethinking Conflict-Free Training of Physics-Informed Neural Networks
Sep 1, 2026Jing Xiao, Xinhai Chen, Qinglin Wang +5Parametric Physics-Informed Neural Network Newton
Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology, Changsha 410073, China · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology, Changsha 410073, China · School of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China
Subspace Levenberg Marquardt Algorithms in Training Neural Networks
Sep 1, 2026M. Duc HoangNeural Network Training Neural Network
Department of Mathematics, University of California, Davis, Davis, CA, USA
Data-driven Koopman mode approximation: A neural power iteration algorithm
Aug 27, 2026Guillaume O. Berger, Raphaël M. JungersKoopman Operator Nonlinear Dynamics
ICTEAM, UCLouvain, Louvain-la-Neuve, Belgium
Dion3: Full-Stack Orthogonal Updates
Aug 12, 2026Noah Amsel, Jack Zhang, Kwangjun Ahn +5Muon Newton
New York University · Microsoft Research · Princeton University +3
Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature
Aug 10, 2026Tong CheMuon Newton
NVIDIA Research
Halpern Iteration Achieves $\tilde{\mathcal{O}}(ε^{-1/p})$ $p$ th-Order Oracle Complexity for Monotone Variational Inequalities
Aug 9, 2026Lesi Chen, Xinliang Zhang, Hengyu Wang +3Monotone Variational Inequalities First Order Oracle Complexity
1IIIS, Tsinghua University · Apex Intelligence · School of Mathematical Sciences, Tongji University +2
Newton-Schulz Retraction-Based Inference Enables Hidden Quantum Markov Models to Outperform Classical HMMs
Aug 6, 2026Ning NingQuantum Learning Hidden Markov Models
Department of Statistics, Texas A&M University College Station, Texas, USA
Alternating Levenberg-Marquardt Training of Physics-Informed Neural Networks with Fourier-Enhanced Features
Aug 6, 2026Yulun Wu, Matthieu Barreau, Miguel Aguiar +1Parametric Physics-Informed Neural Network Partial Differential Equations
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, 100 44, Sweden
NewtonGS: Physics-Structured Object-Level Neural Newtonian Dynamics for Gaussian Scene Animation
Aug 6, 2026Lianlei Shan, Feiyang Ye, Yan Chen +1Motion Newton
Lixiang
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
Aug 5, 2026Thang Do, Steffen Dereich, Arnulf JentzenStochastic Gradient Descent Newton
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · Department of Probability and Statistics, Institute of Mathematics, Vietnam Academy of Science and Technology, Vietnam · Institute for Mathematical Stochastics, Faculty of Mathematics and Computer Science, University of Münster, Germany +2
From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs
Aug 4, 2026Chenhao Si, Kang An, Shiqian Ma +1Parametric Physics-Informed Neural Network Newton
School of Data Science, The Chinese University of Hong Kong, Shenzhen, 2001 Longxiang Road, Longgang District, Shenzhen, 518172, Guangdong, China · Department of Computational Applied Mathematics and Operations Research, Rice University, 6100 Main Street, MS 134, Houston, 77005-1827, Texas, USA · Data Science and AI Institute, and Department of Applied Mathematics and Statistics, Johns Hopkins University, 3400 N. Charles Street, Baltimore, 21218, Maryland, USA
Generalized Quadratic Gradient: A New Direction in Optimization via the Fusion of Positive-Definite Curvature Matrices and Gradients into A Unified Framework
Aug 3, 2026John ChiangHessian Newton
Learning features from Newton's algorithm: a way to accelerate nonlinear parametrized PDE solvers
Jul 30, 2026Rémy Vallot, Florian de Vuyst, Thibault Dairay +1Newton
CB, Michelin · Centre Borelli UMR 9010, ENS Paris-Saclay, Université Paris-Saclay, CNRS, France · Michelin, Centre de Recherche de Ladoux, Cébazat, France +4
Barzilai-Borwein Fails Superlinear Convergence on an Open Set of Quadratics for Every Dimension $n\geq 4$
Jul 23, 2026Dawei Li, Xiaotian Jiang, Mingyi HongConvergence Newton
University of Minnesota
An Efficient Newton Algorithm for Nonnegative Matrix Factorization with the Kullback-Leibler Divergence
Jul 15, 2026Damien Lesens, Jérémy E. Cohen, Bora UçarNonnegative Matrix Factorization Kullback-Leibler Divergence
ENS de Lyon · Univ. Lyon, INSA-Lyon, UCBL, UJM, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621, Lyon, France. · CNRS and LIP (UMR5668, Université de Lyon - ENS de Lyon - UCBL - CNRS - Inria), 46 allée d'Italie, Lyon, France.
Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates
Jul 8, 2026Jianing Liu, Dong H. ZhangNewton Geodesic
State Key Laboratory of Chemical Reaction Dynamics and Department of Chemical Physics, University of Science and Technology of China, Hefei, 230026, China · State Key Laboratory of Chemical Reaction Dynamics, Dalian Institute of Chemical Physics, Chinese Academy of Sciences, Dalian, 116023, China. · University of Chinese Academy of Sciences, Beijing, 100049, China +1
On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism
Jul 7, 2026Don LiHessian Newton
Department of Mathematics & Statistics Portland State University Portland, OR 97201, USA
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Jul 5, 2026Siyu Ding, Mingchuan Ma, Jiabo Tong +3Fp8 Large Language Model Pretraining
Institute of Automation, Chinese Academy of Sciences · Sichuan University · Beijing Zhongguancun Academy +3
An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks
Jul 2, 2026Joseph Webb, Sadok Jerad, Coralia CartisParametric Physics-Informed Neural Network Newton
Mathematical Institute, University of Oxford, Oxford, United Kingdom
Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization
Jun 25, 2026Ziyuan Tang, Tianshi Xu, Yousef Saad +1Muon Newton
Department of Computer Science and Engineering, University of Minnesota, Minneapolis · Department of Mathematics, Emory University, Atlanta, GA 30322
Taming Curvature: Architecture Warm-Up for Stable Transformer Training
Jun 15, 2026Sameera Ramasinghe, Ajanthan Thalaiyasingam, Hadi Mohaghegh Dolatabadi +6Curvature-Aware Spectral Framework Transformer Architectures
Pluralis Research
CacheMuon: Using Temporal Preconditioning To Approximate Polar Factor
Jun 15, 2026Bishnu Dev, Sushil Bohara, Martin Takáč +1Muon Spectral Preconditioning
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Mohamed bin Zayed University of Artificial Intelligence
Redesign Mixture-of-Experts Routers with Manifold Power Iteration
Jun 10, 2026Songhao Wu, Ang Lv, Ruobing Xie +1Mixture-Of-Experts Expert Demonstrations
Gaoling School of Artificial Intelligence, Renmin University of China · Large Language Model Department, Tencent
Accelerating SAV-based optimization via randomized low-rank Hessian approximation
Jun 9, 2026Ryo Sagawa, Daisuke Furihata, Yuto MiyatakeHessian Newton
Department of Pure and Applied Mathematics, Graduate School of Information Science and Technology, The University of Osaka · D3 Center, The University of Osaka
From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video
Jun 4, 2026Jessy LauerBiomechanics Force
Harvard
Preserving Data Privacy in Learning Causal Structure with Fully Homomorphic Encryption
Jun 3, 2026Jian Yang, Yuan Tong, Qinbin Li +2Homomorphic Encryption Privacy
Hong Kong University of Science and Technology (Guangzhou) · Hong Kong University of Science and Technology · University of California, Berkeley
Loss-Conditional PINNs for Parametric PDE Families
Jun 3, 2026Anna Lazareva, Alexander TarakanovParametric Physics-Informed Neural Network Partial Differential Equations
Faculty of Computer Science HSE University, Moscow, Russia · VK and HSE University Moscow, Russia
NewtPhys: Do Foundation Models Understand Newtonian Physics?
Jun 2, 2026Sebastian Cavada, Soumava Paul, Tuan-Hung Vu +2Physics-Aware Models Weak Visual Grounding
MBZUAI · Inria
Finite-Iteration Local Dynamics and Warm Starts for Alternating Power Iteration in Spiked Tensor PCA
Jun 2, 2026Yanjin Xiang, Zhihua ZhangNewton Warm Starts
Peking University
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarMuon Scaling Laws
MIT
Tiny Recursive Models for Solving the J2-Perturbed Lambert Problem
May 30, 2026Minduli Wijayatunga, Roberto ArmellinNewton Lagrangian Methods
Department of Aerospace Engineering University of Illinois Urbana-Champaign Urbana-Champaign, IL, USA · Te Pūnaha Ātea – Space Institute University of Auckland Auckland, New Zealand
Graph Transfer Learning via Shared Latent Geometry: Theory and Applications
May 30, 2026Tong Wu, Andrew Campbell, Anna ScaglioneDiscriminative Congruence Transform Physics-Informed Learning
University of Central Florida, USA · Cornell University, USA
How Much Orthogonalization Does Muon Need?
May 29, 2026Hua HuangMuon Orthogonality
NVIDIA
A Unifying View of Anchoring via Operator-Side Tikhonov Regularization
May 29, 2026Zihao ChenMonotone Variational Inequalities Proximal Operators
UC Berkeley
Geometry-Correct Diffusion Posterior Sampling with Denoiser-Pullback Curvature Guidance and Manifold-Aligned Damping
May 27, 2026Seunghyeok Shin, Minwoo Kim, Dabin Kim +1Diffusion Sampling Diffusion Models
Department of Electrical and Computer Engineering, Inha University, Incheon, 22212, South Korea.
Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent
May 22, 2026Anherutowa CalvoHessian Local Curvature
19D Labs
Hinge Regression Trees and HRT-Boost: Newton-Optimized Oblique Learning for Compact Tabular Models
May 22, 2026Hongyi Li, Jun Xu, Hong YanDecision Trees Tabular Learning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, 518055, China and with the Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments, Shenzhen, 518055, China · Department of Electrical Engineering, City University of Hong Kong, Kowloon, Hong Kong
Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVR
May 19, 2026Chongyu Fan, Gaowen Liu, Mingyi Hong +2Muon Reinforcement Learning With Verifiable Reward
Michigan State University · Cisco · University of Minnesota +1
Statistical Limits and Efficient Algorithms for Differentially Private Federated Learning
May 18, 2026Arnab Auddy, Xiangni Peng, Subhadeep PaulFederated Learning Federated Averaging
Department of Statistics · The Ohio State University
NEWTON: Agentic Planning for Physically Grounded Video Generation
May 18, 2026Yuxiang Feng, Juncheng Wang, Chao Xu +7Video Generation Iterative Re-Planning
Zhejiang University · The Hong Kong Polytechnic University · IROOTECH TECHNOLOGY +1
SNLP: Layer-Parallel Inference via Structured Newton Corrections
May 18, 2026Ligong Han, Kai Xu, Hao Wang +1Transformer Architectures Transformer Residual Streams
1Red Hat AI Innovation · 2MIT-IBM Watson AI Lab · 3Core AI, IBM
AMO: Operator-level Adaptive Muon Orthogonalization
May 18, 2026Xinlin Zhuang, Panyi Ouyang, Yichen Li +7Muon Orthogonality
The Chinese University of Hong Kong · Shopee · MBZUAI +3
Form and Function: Machine Unlearning as a Problem of Misaligned States
May 17, 2026Kennon StewartMachine Unlearning Learning-Augmented Algorithms
Second Street Labs, Detroit, MI, USA · Department of Statistics, University of Michigan, Ann Arbor, MI, USA
Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
May 13, 2026Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3Muon Newton
Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia
MetaColloc: Optimization-Free PDE Solving via Meta-Learned Basis Functions
May 12, 2026Zichuan YangNeural Partial Differential Equation Solvers Partial Differential Equations
School of Mathematical Sciences Tongji University Shanghai, P.R. China
Neural-Schwarz Tiling for Geometry-Universal PDE Solving at Scale
May 12, 2026Paolo Secchi, Daniel S. Balint, Marco MauriziNeural Partial Differential Equation Solvers Partial Differential Equations
Imperial College London · Italian Institute of Artificial Intelligence
Error whitening: Why Gauss-Newton outperforms Newton
May 11, 2026Maricela Best McKay, Nathan P. Lawrence, Brian Wetton +1Newton Whitening
University of British Columbia · University of California, Berkeley

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

SoftServe: A Scalable Quasi-Newton Method for Deep Learning

Joohwan Ko, Tetiana Parshakova, Diana Cai, Robert M. Gower

University of Massachusetts Amherst · CCM, Flatiron Institute · Cornell University

Neural Network Optimization Newton Convex Optimization Deep Learning Local Curvature

Open paper AskPDF arXiv

Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.

Momentum

Related topics

Latest papers 80