stat.MLFeb 5, 2026

Variance Reduction Based Experience Replay for Policy Optimization

Authors: Hua Zheng, Wei Xie, M. Ben Feng, Keilung Choy

Organizations: Department of Mechanical and Industrial Engineering, Northeastern University, Boston, USA · Department of Statistics and Actuarial Science, University of Waterloo, Waterloo, Canada

Abstract

Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To address this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce the variance of policy gradient estimates. VRER is algorithm-agnostic and can be integrated with existing policy optimization methods, yielding the sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). To provide rigorous theoretical guarantees, we develop a novel analysis framework for experience replay that explicitly accounts for dependencies induced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and characterize a fundamental bias-variance trade-off: reusing older samples reduces gradient variance but may introduce greater estimation bias. Extensive experiments show that VRER consistently accelerates learning and outperforms state-of-the-art policy optimization algorithms

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Variance-Reduced Policy Gradient is Nearly Minimax-Optimal

    Oct 9, 2026Leonardo Cesani, Matteo Papini, Marcello RestelliPolicy GradientPolicy Gradient Methods

  2. Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds

    Oct 2, 2026Gabor Paczolay, Matteo Papini, Alberto Maria Metelli +2Policy GradientPolicy Gradient Methods