Gradient Descent

May 23, 2024

The Vital Role of Gradient Clipping in Byzantine-Resilient Distributed Learning
Youssef Allouah, Rachid Guerraoui, Nirupam Gupta, Ahmed Jellouli, Geovani Rizk, John Stephan
Native Robustness Gradient Descent Robust Version Distributed Learning Robust Aggregation Truly Robust Case State of the Art Robustness

May 22, 2024

Transformers Learn Temporal Difference Methods for In-Context Reinforcement Learning
Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, Shangtong Zhang
Reinforcement Learning Transformer Megatron Decepticons Context Learning Gradient Descent Temporal Difference Instance Level Pseudo Label Temporal Difference Algorithm
Thermodynamic Natural Gradient Descent
Kaelan Donatella, Samuel Duffield, Maxwell Aifer, Denis Melanson, Gavin Crooks, Patrick J. Coles
Gradient Descent Natural Gradient Second Order Natural Gradient Descent Analog Computing Virtual Analog
Almost sure convergence rates of stochastic gradient methods under gradient domination
Simon Weissmann, Sara Klein, Waïss Azizian, Leif Döring
Gradient Descent Stochastic Gradient Convergence Rate Strong Convexity Gradient Dominance Property
Deep linear networks for regression are implicitly regularized towards flat minima
Pierre Marion, Lénaïc Chizat
Gradient Descent Learning Rate Novel Regression Gradient Flow Implicit Regularization Deep Linear Flat Minimum

May 19, 2024

May 16, 2024

Random Scaling and Momentum for Non-smooth Non-convex Optimization
Qinzi Zhang, Ashok Cutkosky
Gradient Descent Convergence Guarantee Online Convex Optimization Non Convex Residual Momentum Non Smooth Random Variable Uniform Scaling

May 13, 2024

Analysis of the rate of convergence of an over-parametrized convolutional neural network image classifier learned by gradient descent
Michael Kohler, Adam Krzyzak, Benjamin Walter
Neural Network Deep Learning General Analysis Gradient Descent Image Classification Early Stage Convergence Universal Rate Global Pooling

May 10, 2024

Efficient Federated Low Rank Matrix Completion
Ahmed Ali Abbasi, Namrata Vaswani
Gradient Descent Sample Complexity Sample Complexity Bound Low Rank Matrix Completion

May 6, 2024

April 30, 2024

Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent
Pingzhi Li, Junyu Liu, Hanrui Wang, Tianlong Chen
Neural Network Gradient Descent Quantum Computing Hybrid Quantum Classical Quantum Linear

April 29, 2024

Learning a Sparse Neural Network using IHT
Saeed Damadi, Soroush Zolfaghari, Mahdi Rezaie, Jinglai Shen
Gradient Descent Sparse Neural Network Sparse Optimization Hard Thresholding Sparse Solution Technique

April 27, 2024

Make the Most of Your Data: Changing the Training Data Distribution to Improve In-distribution Generalization Performance
Dang Nguyen, Paymon Haddad, Eric Gan, Baharan Mirzasoleiman
Raw Data Gradient Descent Generalization Performance Distribution Generalization Sharpness Aware Minimization Tiny ImageNet Training Data Distribution

April 23, 2024

FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent
Cameron Smith, David Charatan, Ayush Tewari, Vincent Sitzmann
Gradient Descent Optical Flow Large Depth Camera Pose Stable Pose Dense Depth Map Camera Intrinsics Intrinsic Method Flow Map

April 19, 2024

The Positivity of the Neural Tangent Kernel
Luís Carvalho, João L. Costa, José Mourão, Gonçalo Oliveira
Neural Network Gradient Descent Neural Tangent Kernel Wide Neural Network Positive Emotional Language Polynomial Activation

April 18, 2024

Singular-limit analysis of gradient descent with noise injection
Anna Shalova, André Schlichting, Mark Peletier
Gradient Descent Stochastic Gradient Descent Noise Injection Limit Cycle Noisy Gradient Descent

April 17, 2024

Distributed Fractional Bayesian Learning for Adaptive Optimization
Yaqun Yang, Jinlong Lei, Guanghui Wen, Yiguang Hong
Gradient Descent Optimization Problem Adaptive Optimization Superlinear Convergence Fractional Posterior

April 14, 2024

Momentum-based gradient descent methods for Lie groups
Cédric M. Campos, David Martín de Diego, José Torrente
Gradient Descent Lie Group Accelerated Gradient Cross Sectional Momentum Momentum Gradient Descent Momentum Acceleration

Papers

The Vital Role of Gradient Clipping in Byzantine-Resilient Distributed Learning

Transformers Learn Temporal Difference Methods for In-Context Reinforcement Learning

Thermodynamic Natural Gradient Descent

Almost sure convergence rates of stochastic gradient methods under gradient domination

Deep linear networks for regression are implicitly regularized towards flat minima

Approximation and Gradient Descent Training with Neural Networks

Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method

Random Scaling and Momentum for Non-smooth Non-convex Optimization

Analysis of the rate of convergence of an over-parametrized convolutional neural network image classifier learned by gradient descent

Efficient Federated Low Rank Matrix Completion

$\epsilon$-Policy Gradient for Online Pricing

Optimisation challenge for superconducting adiabatic neural network implementing XOR and OR boolean functions

Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent

Learning a Sparse Neural Network using IHT

Make the Most of Your Data: Changing the Training Data Distribution to Improve In-distribution Generalization Performance

FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent

The Positivity of the Neural Tangent Kernel

Singular-limit analysis of gradient descent with noise injection

Distributed Fractional Bayesian Learning for Adaptive Optimization

Momentum-based gradient descent methods for Lie groups