Newton Method

Newton's method, a powerful iterative technique for finding roots or optimizing functions, remains a central focus in optimization research, particularly within machine learning and scientific computing. Current research emphasizes improving its efficiency and robustness, especially for high-dimensional problems, through techniques like cubic regularization, variance reduction, and subspace methods (e.g., Krylov subspace methods), as well as addressing challenges in federated learning and privacy. These advancements enhance the applicability of Newton-type methods to large-scale problems, impacting fields ranging from neural network training and PDE solving to multi-objective optimization and robust statistics.

Papers

February 23, 2023

Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods
El Mahdi Chayti, Nikita Doikov, Martin Jaggi
Second Order Stochastic Way Non Convex Optimization Newton Method

September 2, 2022

Cubic-Regularized Newton for Spectral Constrained Matrix Optimization and its Application to Fairness
Casey Garner, Gilad Lerman, Shuzhong Zhang
Application Proficiency Procedural Fairness Newton Method Smooth Convex Optimization Matrix Function

June 17, 2022

FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning
Anis Elgabli, Chaouki Ben Issaid, Amrit S. Bedi, Ketan Rajawat, Mehdi Bennis, Vaneet Aggarwal
Hessian Matrix Gradient Information Newton Method Fast Convergence Hessian Vector Product Newton Type Method

June 7, 2022

Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation
Rustem Islamov, Xun Qian, Slavomír Hanzely, Mher Safaryan, Peter Richtárik
Hessian Matrix Approximate Curvature Newton Method Communication Compression Bidirectional Compression Optimal Aggregation Gradient Communication

June 6, 2022

Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches
Michał Dereziński
Second Order Stochastic Way Variance Reduction Subsampling Method Large Batch Newton Method Finite Sum Minimization Variance Reduced Stochastic

May 25, 2022

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function
Saeed Masiha, Saber Salehkaleybar, Niao He, Negar Kiyavash, Patrick Thiran
Stochastic Gradient Descent Sample Complexity Second Order Global Optimality Newton Method Stochastic Policy Gradient Gradient Dominance Property

May 23, 2022

Augmented Newton Method for Optimization: Global Linear Rate and Momentum Interpretation
Md Sarowar Morshed
Optimization Purpose Newton Method Unconstrained Optimization Penalty Method Cross Sectional Momentum

May 18, 2022

On the efficiency of Stochastic Quasi-Newton Methods for Deep Learning
Mahsa Yousefi, Angeles Martinez
Deep Learning High Efficiency Adam Optimizer Newton Method Quasi Newton Method Quasi Newton

April 20, 2022

Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergence
Sen Na, Michał Dereziński, Michael W. Mahoney
Hessian Matrix Newton Method Superlinear Convergence

December 2, 2021

Newton methods based convolution neural networks using parallel processing
Ujjwal Thakur, Anuj Sharma
Neural Network Convolutional Neural Network Deep Neural Network Direct Convolution Hessian Matrix Parallel Processing Newton Method

November 2, 2021

Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning
Xun Qian, Rustem Islamov, Mher Safaryan, Peter Richtárik
Newton Method Shallow Learning Second Order Method Bidirectional Compression

Newton Method

Papers

Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods

Cubic-Regularized Newton for Spectral Constrained Matrix Optimization and its Application to Fairness

FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning

Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation

Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Augmented Newton Method for Optimization: Global Linear Rate and Momentum Interpretation

On the efficiency of Stochastic Quasi-Newton Methods for Deep Learning

Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergence

Newton methods based convolution neural networks using parallel processing

Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning