Non Asymptotic Convergence
Non-asymptotic convergence analysis focuses on determining precise, finite-time error bounds for iterative algorithms, rather than just asymptotic behavior. Current research emphasizes establishing such bounds for various machine learning models and optimization methods, including transformers, diffusion models, and stochastic gradient-based algorithms like AdaGrad and SGHMC, often addressing challenges posed by non-convexity and discontinuous gradients. These analyses provide crucial insights into algorithm efficiency and generalization capabilities, leading to improved algorithm design and more reliable performance guarantees in diverse applications such as generative modeling and neural network training.
Papers
Non-asymptotic Convergence of Training Transformers for Next-token Prediction
Ruiquan Huang, Yingbin Liang, Jing Yang
Non-asymptotic convergence analysis of the stochastic gradient Hamiltonian Monte Carlo algorithm with discontinuous stochastic gradient with applications to training of ReLU neural networks
Luxu Liang, Ariel Neufeld, Ying Zhang