Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces

Ji Gao; Caleb Ju; Guanghui Lan; Zhaohui Tong

cs.LGMar 10, 2026

Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces

Authors: Ji Gao, Caleb Ju, Guanghui Lan, Zhaohui Tong

Organizations: School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, GA, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, GA, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, GA, USA

Abstract

Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose actor-accelerated PDA, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.

Figures & tables

Figure 1: Visualization of optimum tracking with the actor in the Pendulum-v1 environment. The landscape evolution of the scaled optimization sub-problem $\tilde{\Psi}^{\prime}$ over epochs 5, 8, and 11, at a fixed $\dot{\theta}=0.2$ is shown in the first three plots. The pink dotted line represents the true optimum of $\tilde{\Psi}^{\prime}$ , and the solid red line represents the output of the actor network. The final plot tracks the mean absolute error (MAE) between the true optimum and the actor output averaged across a range of states and actions at each epoch for an extended training process.

Figure 2: Performance comparison of PDA, PPO, TRPO, and NPG across MuJoCo and Box2D environments. The curves and the shaded areas represent the mean and standard deviation across 100 test evaluations (10 seeds per environment and 10 tests per seed), respectively.

Figure 3: OR-Gym Benchmark for PDA and PPO. Episodic reward distributions of agents after training for 3 million (Newsvendor) and 1 million (PortfolioOpt) environment steps, aggregated over 10 random seeds with $10^{3}$ random trials per seed. A lower threshold is applied for the Newsvendor environment to maintain readability of the plot due to the existence of extreme negative values for PPO. The training curves are shown in Appendix B.2 .

Backlog	PDA	PPO	SHLP	DFO	MIP	Oracle
Mean	491.6	496.4	508.0	360.9	388.0	546.8
Std	11.8	6.8	28.1	39.9	30.8	30.3
Ratio	1.1	1.1	1.1	1.5	1.4	1.0
LS	PDA	PPO	SHLP	DFO	MIP	Oracle
Mean	472.3	447.4	485.4	364.3	378.5	542.7
Std	10.3	16.8	29.1	33.8	26.1	29.9

Table 1: Performance comparison of PDA, PPO, and classical OR methods on the InvManagement environments with Backlog and Lost Sales (LS) settings. We report the mean episodic return, standard deviation (Std), and performance ratio relative to the Oracle benchmark (lower is better). Results for the OR methods and the Oracle are taken from the original study Hubbs et al. (2020) .

Figure 4: Sensitivity on exploration noise parameter $\sigma_{0}$ and step size $\lambda$ . The heatmap shows the testing episodic reward averaged for the last 5 epochs. The final plot shows the sensitivity study with constant noise (3 seeds per environment and 10 tests per seed), while the rest of the plots show the default decreasing noise (5 seeds per environment and 10 tests per seed)

Figure 5: Performance comparison between the PDA with scaled sum advantage update (labeled as PDA) and an exponential smoothing scheme (labeled with smoothing parameter $\alpha$ ) across two environments. Results are averaged over 5 random seeds. The curve and shaded region represent the mean and standard deviation of the test results.

Figure 6: Comparison of optimizer effects on PDA and PPO in the Ant and Hopper environments. across two environments. Results are averaged over 5 random seeds. The curve and shaded region represent the mean and standard deviation of the test results.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	NPG	PPO	TRPO	PDA
HalfCheetah-v4	3556.4 ± 837.9	4067.5 ± 572.3	4496.8 ± 969.7	5174.6 ± 686.7
Ant-v4	2002.5 ± 300.6	2589.5 ± 445.8	2807.1 ± 645.5	3568.2 ± 449.7
Hopper-v4	1650.8 ± 726.4	2329.7 ± 572.9	2017.0 ± 849.0	2944.3 ± 787.4
Walker2d-v4	2923.2 ± 707.2	3277.0 ± 762.6	4128.8 ± 638.1	4367.1 ± 915.9
InvertedPendulum-v4	1000.0 ± 0.0	993.3 ± 20.2	380.3 ± 391.6	1000.0 ± 0.0
InvertedDoublePendulum-v4	7967.2 ± 1205.3	7926.8 ± 1241.8	2723.4 ± 2393.5	9167.5 ± 552.6

Appendix

Table 2: Benchmark results of PDA, PPO, TRPO, and NPG for MuJoCo-v4, Box2d, and OR-Env environments in 1M (or 3M) environment steps (mean ± std). The results are calculated with the average episodic reward of the last 5 epochs using 10 seeds per eenvironment and 10 tests per epoch. The Newsvendor-v0 environment uses 500 tests per epoch due to its high variance.

Figure 7: Performance comparison of PDA, PPO, TRPO, and NPG for OR-Env environments. The curves and the shaded areas represent the mean and standard deviation across 100 test evaluations (10 seeds per environment and 10 tests per seed), respectively.

Environment	PPO	PDA
HalfCheetah-v5	4067.5 ± 572.3	5174.6 ± 686.7
Ant-v5	1115.4 ± 302.8	2625.1 ± 368.2
Hopper-v5	2397.8 ± 344.2	2693.9 ± 867.0
Walker2d-v5	3796.7 ± 908.3	3642.3 ± 1285.8
InvertedPendulum-v5	975.8 ± 72.6	916.0 ± 221.1
InvertedDoublePendulum-v5	6782.6 ± 1691.0	9349.2 ± 9.6

Appendix

Table 3: Benchmark results of PDA, PPO for MuJoCo-v5 environments in 1M (or 3M) environment steps (mean ± std) . The results are calculated with the average episodic reward of the last 5 epochs using 10 seeds per environment and 10 tests per epoch.

Figure 8: Performance comparison of PDA and PPO for MuJoCo environments. The curves and the shaded areas represent the mean and standard deviation across 100 test evaluations (10 seeds per environment and 10 tests per seed), respectively.

Hyperparameter	NPG	PDA	PPO	TRPO
Hidden sizes	$[64,64]$	$[64,64]$	$[64,64]$	$[64,64]$
Activation	Tanh	Tanh	Tanh	Tanh
Learning rate	$1\times 10^{-3}$	$1\times 10^{-3}$	$3\times 10^{-4}$	$1\times 10^{-3}$
Batch size	1024	1000	64	1024
Discount factor	0.99	0.99	0.99	0.99
GAE $\lambda$	0.95	0.95	0.95	0.95

Appendix

Table 4: Hyperparameters for NPG, PDA, PPO, and TRPO algorithms

Algorithm	MuJoCo-v4 (w/o humanoid)	MuJoCo-v4 (humanoid variants)
PDA	$423.4\pm 104.8$	$1480.7\pm 178.2$
PPO	$677.8\pm 278.2$	$1720.8\pm 295.8$
NPG	$566.5\pm 283.3$	$1484.3\pm 420.0$
TRPO	$512.3\pm 268.7$	$1365.6\pm 422.6$

Appendix

Table 5: Average runtime (5 parallel runs) on Intel i7-14700K for 1M environment steps of training and testing. Times are reported in seconds (mean ± standard deviation).

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression
Oct 3, 2026Nianli Peng, Geoffrey J. Gordon, Kianté BrantleyActor-Critic Methods Experience Replay
Harvard University · Carnegie Mellon University
Policy Gradient for Continuous-Time Robust Markov Decision Processes
Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaRobust Markov Decision Processes Markov Decision Processes
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR) · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR) · College of Computing and Data Science, Nanyang Technological University
Align and Filter: Improving Performance in Asynchronous On-Policy RL
Mar 2, 2026Homayoun Honari, Roger Creus Castanyer, Michael Przystupa +3Asynchronous RL Policy Gradient Methods
Mila – Quebec AI Institute, Canada · Université de Montréal, Canada · University of Alberta, Canada

Oct 3, 2026J/K move · Enter open · S save

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Nianli Peng, Geoffrey J. Gordon, Kianté Brantley

Harvard University · Carnegie Mellon University

Actor-Critic Methods Experience Replay Off-Policy RL

Open paper AskPDF arXiv

Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model

Z_θ

to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin-

Q

labels, RDA2C matches SAC at matched batch size and update frequency.

Abstract

Reinforcement Learning

Explore similar work

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Policy Gradient for Continuous-Time Robust Markov Decision Processes

Align and Filter: Improving Performance in Asynchronous On-Policy RL