Early Stage Convergence

Early stage convergence in machine learning focuses on understanding and improving the initial phases of training algorithms, aiming to accelerate convergence speed and enhance generalization performance. Current research investigates this through the lens of various optimization algorithms (e.g., Adam, SGD, FedProx), model architectures (e.g., transformers, diffusion models), and specific problem domains (e.g., federated learning, collaborative filtering). These studies leverage techniques from dynamical systems theory and optimal transport to establish convergence guarantees and bounds, ultimately contributing to more efficient and robust machine learning systems across diverse applications.

Papers

March 19, 2023

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing
Vishnu Pandi Chellapandi, Antesh Upadhyay, Abolfazl Hashemi, Stanislaw H /. Zak
Early Stage Convergence Decentralized Learning Decentralized Federated Learning Information Sharing Noisy Communication

March 15, 2023

SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence
Heung-Chang Lee, Jeonggeun Song
Neural Network Early Stage Convergence Back Propagation Forward Forward Algorithm Plausible Learning Intrinsic Class

February 23, 2023

Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers
Minsoo Kim, Kyuhong Shim, Seongmin Park, Wonyong Sung, Jungwook Choi
Early Stage Convergence Pre Trained Transformer Quantization Aware Training Quantization Technique Precision Transformer

February 15, 2023

On Rank Energy Statistics via Optimal Transport: Continuity, Convergence, and Change Point Detection
Matthew Werenski, Shoaib Bin Masud, James M. Murphy, Shuchin Aeron
Optimal Transport Early Stage Convergence Convergence Analysis Change Point Detection Continuous Speech

February 13, 2023

Implications of the Convergence of Language and Vision Model Geometries
Jiaang Li, Yova Kementchedjhieva, Anders Søgaard
Language Model Human Language Early Stage Convergence Meaningful Representation Vision Model Computer Vision Model Future Implication Language Space LLM Representation

February 10, 2023

On the Convergence of Stochastic Gradient Descent for Linear Inverse Problems in Banach Spaces
Z. Kereta, B. Jin
Gradient Descent Stochastic Gradient Descent Inverse Problem Early Stage Convergence Ill Posed Inverse Problem Linear Inverse Problem Banach Space

February 8, 2023

On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes
Yashaswini Murthy, Mehrdad Moharrami, R. Srikant
Early Stage Convergence Policy Iteration

February 6, 2023

On the Convergence of Federated Averaging with Cyclic Client Participation
Yae Jee Cho, Pranay Sharma, Gauri Joshi, Zheng Xu, Satyen Kale, Tong Zhang
Early Stage Convergence Convergence Rate Federated Averaging FedAvg Converges Client Participation

February 1, 2023

Weight Prediction Boosts the Convergence of AdamW
Lei Guan
Deep Neural Network Early Stage Convergence Deep Neural Network Training Deep Neural Network Parameter Weight Prediction

January 30, 2023

Convergence of uncertainty estimates in Ensemble and Bayesian sparse model discovery
L. Mars Gao, Urban Fasel, Steven L. Brunton, J. Nathan Kutz
Uncertainty Estimation Early Stage Convergence Sparse Identification Sparse Model Discovery

January 26, 2023

On the Convergence of No-Regret Learning Dynamics in Time-Varying Games
Ioannis Anagnostides, Ioannis Panageas, Gabriele Farina, Tuomas Sandholm
Early Stage Convergence Dynamic Regret General Sum Game Regret Dynamic Dependent Regret

January 25, 2023

Graph Neural Tangent Kernel: Convergence on Large Graphs
Sanjukta Krishnagopal, Luana Ruiz
Graph Neural Network Early Stage Convergence Graph Data Graphon Estimation Graph Neural Tangent Kernel

January 23, 2023

January 19, 2023

Convergence beyond the over-parameterized regime using Rayleigh quotients
David A. R. Robin, Kevin Scaman, Marc Lelarge
Loss Function Early Stage Convergence Gradient Flow Loss Minimization Parameterized Regime Parametric Learning Rayleigh Regression

January 17, 2023

Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes
Konstantin Mishchenko, Slavomír Hanzely, Peter Richtárik
Early Stage Convergence Hessian Matrix Convergence Guarantee First Order Algorithm Agnostic Meta learnIng Gradient Step Moreau Envelope

January 5, 2023

Beyond spectral gap (extended): The role of the topology in decentralized learning
Thijs Vogels, Hadrien Hendrikx, Martin Jaggi
Integral Role Early Stage Convergence Sparse Graph Topology Problem Parallel Optimization Spectral Gap

January 4, 2023

On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats
Matteo Cacciola, Antonio Frangioni, Masoud Asgharian, Alireza Ghaffari, Vahid Partovi Nia
Stochastic Gradient Descent Early Stage Convergence Low Precision

January 1, 2023

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
Hongru Yang, Ziyu Jiang, Ruizhe Zhang, Yingbin Liang, Zhangyang Wang
Strong Generalization Early Stage Convergence Neural Tangent Kernel ReLU Network Sparse Network Wide Neural Network Sparse Activation Relative Stance Bias

December 28, 2022

On the Convergence of Discounted Policy Gradient Methods
Chris Nota
Reinforcement Learning Policy Gradient Early Stage Convergence Estimation Bias Gradient Ascent

Early Stage Convergence

Papers

On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing

SymBa: Symmetric Backpropagation-Free Contrastive Learning with Forward-Forward Algorithm for Optimizing Convergence

Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers

On Rank Energy Statistics via Optimal Transport: Continuity, Convergence, and Change Point Detection

Implications of the Convergence of Language and Vision Model Geometries

On the Convergence of Stochastic Gradient Descent for Linear Inverse Problems in Banach Spaces

On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes

On the Convergence of Federated Averaging with Cyclic Client Participation

Weight Prediction Boosts the Convergence of AdamW

Convergence of uncertainty estimates in Ensemble and Bayesian sparse model discovery

On the Convergence of No-Regret Learning Dynamics in Time-Varying Games

Graph Neural Tangent Kernel: Convergence on Large Graphs

On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation

On the Convergence of the Gradient Descent Method with Stochastic Fixed-point Rounding Errors under the Polyak-Lojasiewicz Inequality

Convergence beyond the over-parameterized regime using Rayleigh quotients

Convergence of First-Order Algorithms for Meta-Learning with Moreau Envelopes

Beyond spectral gap (extended): The role of the topology in decentralized learning

On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats

Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK

On the Convergence of Discounted Policy Gradient Methods