Markovian Sampling

Markovian sampling explores the challenges and opportunities of using Markov chain-generated data in various machine learning algorithms, particularly focusing on improving the efficiency and convergence guarantees of stochastic approximation methods. Current research emphasizes analyzing the impact of Markovian sampling on algorithms like temporal difference learning, actor-critic methods, and stochastic gradient descent, often employing continuous normalizing flows or operator-valued stochastic gradient descent for model training. This research is crucial for advancing the theoretical understanding and practical application of reinforcement learning, federated learning, and other areas where data is inherently sequential or generated by a stochastic process, leading to more efficient and robust algorithms.

Papers

September 12, 2024

Theoretical guarantees in KL for Diffusion Flow Matching
Marta Gentiloni Silveri, Giovanni Conforti, Alain Durmus
Flow Matching K Content Theoretical Guarantee Brownian Motion Markovian Sampling

May 23, 2024

Markovian Flow Matching: Accelerating MCMC with Continuous Normalizing Flows
Alberto Cabezas, Louis Sharrock, Christopher Nemeth
Generative Modeling Normalizing Flow Markov Chain Monte Carlo Flow Matching Markovian Sampling Conditional Flow Matching

May 13, 2024

Compressed Online Learning of Conditional Mean Embedding
Boya Hou, Sina Sanjari, Alec Koppel, Subhonmesh Bose
Operator Learning Kernel Hilbert Space Transition Kernel Markovian Sampling

May 3, 2024

Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization
Mudit Gaur, Amrit Singh Bedi, Di Wang, Vaneet Aggarwal
Neural Network Global Convergence State Action Space Sample Complexity Bound Critic Model Markovian Sampling Local Optimality Gradient Dominance Property

March 4, 2024

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation
Aritra Mitra
Stochastic Approximation Linear Function Approximation Finite Time Finite Time Convergence Multidimensional Projection TD Learning Markovian Sampling

February 19, 2024

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling
Arman Adibi, Nicolo Dal Fabbro, Luca Schenato, Sanjeev Kulkarni, H. Vincent Poor, George J. Pappas, Hamed Hassani, Aritra Mitra
Stochastic Approximation Prompt Optimization Finite Time Finite Time Convergence Significant Delay Change Markovian Sampling

January 17, 2024

Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications
Jie Hu, Vishwaraj Doshi, Do Young Eun
Financial Application Theoretical Understanding Stochastic Optimization Stochastic Approximation Iterative Stochastic Markovian Sampling Central Limit Theorem

October 4, 2023

Hoeffding's Inequality for Markov Chains under Generalized Concentrability Condition
Hao Chen, Abhishek Gupta, Yin Sun, Ness Shroff
Markov Chain Stochastic Bandit Soft Inequality Markovian Sampling Hoeffding Tree

August 3, 2023

Online covariance estimation for stochastic gradient descent under Markovian sampling
Abhishek Roy, Krishnakumar Balasubramanian
Gradient Descent Stochastic Gradient Descent Stochastic Way Covariance Matrix Markovian Sampling Covariance Regularization

May 14, 2023

Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Sampling
Nicolò Dal Fabbro, Aritra Mitra, George J. Pappas
Reinforcement Learning Linear Speedup Erasure Method Markovian Sampling Erasure Channel

February 28, 2023

Stochastic Gradient Descent under Markovian Sampling Schemes
Mathieu Even
Stochastic Gradient Descent Stochastic Gradient Vanilla Gradient Descent SGD Style Markovian Sampling

February 4, 2023

Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneity
Han Wang, Aritra Mitra, Hamed Hassani, George J. Pappas, James Anderson
Markov Decision Process Linear Function Approximation Federated Reinforcement Learning Finite Time Markovian Sampling MDP Model Heterogeneity Environment

October 5, 2022

Extending Conformal Prediction to Hidden Markov Models with Exact Validity via de Finetti's Theorem for Markov Chains
Buddhika Nettasinghe, Samrat Chatterjee, Ramakrishna Tipireddy, Mahantesh Halappanavar
Conformal Prediction Markov Chain Hidden Markov Model Markovian Sampling De Finetti

June 21, 2022

Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning
Sajad Khodadadian, Pranay Sharma, Gauri Joshi, Siva Theja Maguluri
Q Learning Federated Reinforcement Learning Linear Speedup Slow Convergence Markovian Sampling Federated Automatic Differentiation

Markovian Sampling

Papers

Theoretical guarantees in KL for Diffusion Flow Matching

Markovian Flow Matching: Accelerating MCMC with Continuous Normalizing Flows

Compressed Online Learning of Conditional Mean Embedding

Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization

A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

Stochastic Approximation with Delayed Updates: Finite-Time Rates under Markovian Sampling

Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications

Hoeffding's Inequality for Markov Chains under Generalized Concentrability Condition

Online covariance estimation for stochastic gradient descent under Markovian sampling

Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Sampling

Stochastic Gradient Descent under Markovian Sampling Schemes

Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneity

Extending Conformal Prediction to Hidden Markov Models with Exact Validity via de Finetti's Theorem for Markov Chains

Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning