Gradient Prediction

Gradient prediction focuses on accelerating optimization algorithms by estimating gradients, thereby reducing the computational burden of iterative methods like those used in training deep neural networks. Current research explores various approaches, including using kernelized gradient estimation, joint-embedding predictive architectures, and small auxiliary networks to improve prediction accuracy and efficiency, often within the context of specific applications like federated learning and sparse training. These advancements offer significant potential for improving the speed and energy efficiency of machine learning and other computationally intensive tasks, impacting fields ranging from computer vision to robotics.

Papers

October 17, 2024

Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations
Kexuan Shi, Hai Chen, Leheng Zhang, Shuhang Gu
Meaningful Representation Implicit Neural Network Spectral Bias Gradient Prediction Empirical Interpolation Method

February 18, 2024

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations
Yao Shu, Jiongfeng Fang, Ying Tiffany He, Fei Richard Yu
Optimization Purpose Stochastic Gradient Descent Gradient Estimation First Order Optimization Iteration Head Gradient Prediction

November 26, 2023

Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture
Weijie Li, Yang Wei, Tianpeng Liu, Yuenan Hou, Yuxuan Li, Zhen Liu, Yongxiang Liu, Li Liu
Self Supervised Learning Synthetic Aperture Radar Synthetic Aperture Radar Data Target Recognition Joint Embedding Predictive Architecture Gradient Prediction

June 12, 2023

Can Forward Gradient Match Backpropagation?
Louis Fournier, Stéphane Rivaud, Eugene Belilovsky, Michael Eickenberg, Edouard Oyallon
Gradient Backpropagation Forward Gradient Gradient Prediction Gradient Based Solver

May 22, 2023

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction
Vahid Janfaza, Shantanu Mandal, Farabi Mahmud, Abdullah Muzahid
Deep Neural Network DNN Accelerator Deep Neural Network Training Gradient Prediction

January 17, 2023

Expected Gradients of Maxout Networks and Consequences to Parameter Initialization
Hanna Tseran, Guido Montúfar
Convolutional Neural Network Natural Gradient Jacobian Matrix Experiential Consequence Initial Condition Gradient Prediction

January 9, 2023

Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction
Bowen Lei, Dongkuan Xu, Ruqi Zhang, Shuren He, Bani K. Mallick
Balancing Strategy Sparse Training Sparsity Constraint Gradient Prediction Training Epoch

August 28, 2022

Asynchronous Training Schemes in Distributed Learning with Time Delay
Haoxiang Wang, Zhanhong Jiang, Chao Liu, Soumik Sarkar, Dongxiang Jiang, Young M. Lee
Convergence Rate Objective Function Distributed Learning Time Delay Asynchronous Training Algorithm Performance Gradient Prediction Resilient Algorithm

August 17, 2022

NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data
Xin Zhang, Minghong Fang, Zhuqing Liu, Haibo Yang, Jia Liu, Zhengyuan Zhu
Heterogeneous Data Decentralized Federated Learning Gradient Prediction FLEET Management

July 25, 2022

DeepFusion: Real-Time Dense 3D Reconstruction for Monocular SLAM using Single-View Depth and Gradient Predictions
Tristan Laidlow, Jan Czarnowski, Stefan Leutenegger
Simultaneous Localization and Mapping Dense Depth Map Dense 3D Reconstruction Depth Camera Single View Depth Dense Reconstruction Gradient Prediction Real Time Dense 3D Reconstruction

May 1, 2022

Adaptive Composite Online Optimization: Predictions in Static and Dynamic Environments
Pedro Zattoni Scroccaro, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani
Human Prediction Dynamic Environment Online Convex Optimization Online Optimization Portfolio Optimization Dynamic Regret Convex Cost Gradient Prediction

Gradient Prediction

Papers

Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations

OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations

Predicting Gradient is Better: Exploring Self-Supervised Learning for SAR ATR with a Joint-Embedding Predictive Architecture

Can Forward Gradient Match Backpropagation?

ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction

Expected Gradients of Maxout Networks and Consequences to Parameter Initialization

Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction

Asynchronous Training Schemes in Distributed Learning with Time Delay

NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Data

DeepFusion: Real-Time Dense 3D Reconstruction for Monocular SLAM using Single-View Depth and Gradient Predictions

Adaptive Composite Online Optimization: Predictions in Static and Dynamic Environments