Stochastic Approximation

Stochastic approximation (SA) is an iterative method for finding the root of an operator using noisy observations, crucial for solving optimization problems where exact gradients are unavailable. Current research emphasizes improving SA's efficiency and robustness, focusing on variance reduction techniques, adaptive step sizes, and handling Markovian noise and delays in various settings, including distributed and federated learning, reinforcement learning, and temporal difference learning. These advancements are significant for tackling large-scale optimization problems in diverse fields, leading to improved algorithms with stronger convergence guarantees and enhanced applicability to real-world scenarios.

Papers

February 27, 2024

Stochastic Approximation with Biased MCMC for Expectation Maximization
Samuel Gruffaz, Kyurae Kim, Alain Oliviero Durmus, Jacob R. Gardner
Langevin Dynamic Expectation Maximization Stochastic Approximation Metropolis Adjusted Langevin Algorithm

February 20, 2024

Stochastic Approximation Approach to Federated Machine Learning
Srihari P, Bharath Bhikkaji
Neural Network Model Stochastic Approximation Federated Machine Learning Sample Gradient Iterative Stochastic

February 19, 2024

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling
Arman Adibi, Nicolo Dal Fabbro, Luca Schenato, Sanjeev Kulkarni, H. Vincent Poor, George J. Pappas, Hamed Hassani, Aritra Mitra
Stochastic Approximation Prompt Optimization Finite Time Finite Time Convergence Significant Delay Change Markovian Sampling

February 6, 2024

SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning
Paul Mangold, Sergey Samsonov, Safwan Labbi, Ilya Levin, Reda Alami, Alexey Naumov, Eric Moulines
Sample Complexity Stochastic Approximation Linear Function Approximation Communication Complexity Heterogeneous Agent TD Learning Federated Automatic Differentiation

January 23, 2024

Fast Nonlinear Two-Time-Scale Stochastic Approximation: Achieving $O(1/k)$ Finite-Sample Complexity
Thinh T. Doan
Sample Complexity Stochastic Approximation Finite Time Convergence Non Linear Operator Nonlinear Function

January 18, 2024

Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks
Jie Hu, Vishwaraj Doshi, Do Young Eun
Stochastic Optimization Markov Chain Random Walk Stochastic Approximation

January 17, 2024

Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications
Jie Hu, Vishwaraj Doshi, Do Young Eun
Financial Application Theoretical Understanding Stochastic Optimization Stochastic Approximation Markovian Sampling Iterative Stochastic Central Limit Theorem

January 15, 2024

The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise
Shuze Liu, Shuhang Chen, Shangtong Zhang
Reinforcement Learning Stochastic Gradient Descent Stochastic Approximation Iterative Stochastic

December 31, 2023

Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise
Shaan Ul Haque, Sajad Khodadadian, Siva Theja Maguluri
Reinforcement Learning Algorithm Convergence Rate Stochastic Approximation Finite Time

December 18, 2023

Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inference
Dongyan Huo, Yudong Chen, Qiaomin Xie
Stochastic Approximation Step Size Statistical Inference Inference Strategy

December 16, 2023

A Concentration Bound for TD(0) with Function Approximation
Siddharth Chandak, Vivek S. Borkar
Markov Chain Stochastic Approximation Linear Function Approximation Function Approximation Concentration Bound Concentration Inequality

December 8, 2023

Two-Timescale Q-Learning with Function Approximation in Zero-Sum Stochastic Games
Zaiwei Chen, Kaiqing Zhang, Eric Mazumdar, Asuman Ozdaglar, Adam Wierman
Stochastic Approximation Function Approximation Stochastic Game Q$ Learning Zero Sum Markov Game Q$ Learning Algorithm Two Timescale

December 5, 2023

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications
Rajeeva L. Karandikar, M. Vidyasagar
Financial Application Stochastic Gradient Descent Stochastic Gradient Convergence Rate Stochastic Approximation Convex Function Unbounded Variance Convergence Property

October 22, 2023

Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability
Sergey Samsonov, Daniil Tiapkin, Alexey Naumov, Eric Moulines
Markov Decision Process Stochastic Approximation Linear Function Approximation Temporal Difference Learning Temporal Difference Type Recurrence High Probability Bound Polyak Ruppert Nonlinear Stability

October 17, 2023

Resampling Stochastic Gradient Descent Cheaply for Efficient Uncertainty Quantification
Henry Lam, Zitong Wang
Stochastic Gradient Descent Uncertainty Quantification Stochastic Optimization Stochastic Approximation Confidence Interval

August 25, 2023

SGMM: Stochastic Approximation to Generalized Method of Moments
Xiaohong Chen, Sokbae Lee, Yuan Liao, Myung Hwan Seo, Youngki Shin, Myunghyun Song
Stochastic Way Stochastic Approximation Temporal Moment Conditional Moment Restriction

August 2, 2023

Compressed and distributed least-squares regression: convergence rates with applications to Federated Learning
Constantin Philippenko, Aymeric Dieuleveut
Financial Application Stochastic Gradient Convergence Rate Least Square Stochastic Approximation Noise Covariance Unbiased Compression

July 11, 2023

Stochastic Nested Compositional Bi-level Optimization for Robust Feature Learning
Xuxing Chen, Krishnakumar Balasubramanian, Saeed Ghadimi
Stochastic Gradient Stochastic Approximation Compositional Optimization Robust Feature Compositional Nature Stochastic First Order Oracle

July 5, 2023

Stability of Q-Learning Through Design and Optimism
Sean Meyn
Product Design Q Learning Core Stability Stochastic Approximation Linear Function Approximation Reinforcement Learning Library

July 4, 2023

Accelerated stochastic approximation with state-dependent noise
Sasila Ilandarideva, Anatoli Juditsky, Guanghui Lan, Tianjiao Li
Stochastic Gradient Stochastic Approximation Accelerated Gradient Descent Dependent Noise Smooth Stochastic Convex Optimization

Stochastic Approximation

Papers

Stochastic Approximation with Biased MCMC for Expectation Maximization

Stochastic Approximation Approach to Federated Machine Learning

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling

SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learning

Fast Nonlinear Two-Time-Scale Stochastic Approximation: Achieving $O(1/k)$ Finite-Sample Complexity

Accelerating Distributed Stochastic Optimization via Self-Repellent Random Walks

Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications

The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise

Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise

Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inference

A Concentration Bound for TD(0) with Function Approximation

Two-Timescale Q-Learning with Function Approximation in Zero-Sum Stochastic Games

Convergence Rates for Stochastic Approximation: Biased Noise with Unbounded Variance, and Applications

Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability

Resampling Stochastic Gradient Descent Cheaply for Efficient Uncertainty Quantification

SGMM: Stochastic Approximation to Generalized Method of Moments

Compressed and distributed least-squares regression: convergence rates with applications to Federated Learning

Stochastic Nested Compositional Bi-level Optimization for Robust Feature Learning

Stability of Q-Learning Through Design and Optimism

Accelerated stochastic approximation with state-dependent noise