Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Nianli Peng; Geoffrey J. Gordon; Kianté Brantley

cs.LGOct 3, 2026

Policy as Data: Replay-Based Policy Dual Averaging via Advantage Regression

Authors: Nianli Peng, Geoffrey J. Gordon, Kianté Brantley

Organizations: Harvard University · Carnegie Mellon University

Abstract

Actor-critic methods reuse past experience to improve sample efficiency. However, historical data are typically regarded as off-policy samples for the current policy-improvement update. This work introduces Regularized Dual Averaging Actor Critic (RDA2C), which assigns a distinct role to replay. In regularized dual averaging, the subsequent policy is determined by accumulated policy-improvement feedback, so historical advantage estimates contribute directly to the actor objective rather than solely to the most recent update. RDA2C stores state-action samples with critic-estimated advantage labels, fits a dual score model $Z_θ$ to the aggregated dataset, and derives the current policy from the accumulated score model using the entropy mirror map. In this way, replay defines an empirical dual objective from which the policy is computed. To analyze RDA2C, we establish a finite-time value-gap decomposition, separating the regularized dual-averaging term from errors due to stale-replay supervised fitting, critic bias, finite-buffer variance, and replay coverage, and stating the assumptions under which each error is bounded. RDA2C accepts advantage labels from any critic. With GAE labels, RDA2C outperforms PPO on six of eight MuJoCo tasks and eight of twelve Atari games. RDA2C also outperforms AAPDA, the closest dual-averaging baseline, on six of eight MuJoCo tasks. With twin- $Q$ labels, RDA2C matches SAC at matched batch size and update frequency.

Figures & tables

Figure 1: Left Standard actor updates compress policy-improvement history into the current parameters. Right In RDA2C, the policy is determined by the replayed data. The buffer stores accumulated advantage evidence, the dual score summarizes that evidence, and the mirror map converts it into the next policy.

Figure 2: RDA2C improves over matched PPO on six of eight MuJoCo tasks and over AAPDA on six of eight. Bars show per-environment relative improvement,

100\cdot(\bar{R}_{\mathrm{ours}}-\bar{R}_{\mathrm{base}})/|\bar{R}_{\mathrm{base}}|

. Left: PPO matched to RDA2C’s architecture and hyperparameters (GAE labels, online return). Right: AAPDA at its released defaults (deterministic evaluation). Exact values: Appendix B .

Figure 3: RDA2C improves over PPO on eight of twelve Atari games. Bars show relative improvement over PPO,

100\cdot(\bar{R}_{\mathrm{RDA2C}}-\bar{R}_{\mathrm{PPO}})/|\bar{R}_{\mathrm{PPO}}|

, after 10M frames. The value axis is broken between

+95\%

and

+235\%

. The bars for BeamRider-v5 , Enduro-v5 , and Seaquest-v5 continue in the right segment. Exact mean

\pm

std values are reported in Appendix B , Table 9 .

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Shared Rollout / Optimization Settings
Parameter	RDA2C (Ours)	PPO (Baseline)
Optimizer	Adam	Adam
Discount Factor ( $\gamma$ )	0.99	0.99
Total Timesteps	$1{,}000{,}000$	$1{,}000{,}000$
Total Batch Size	2048	2048
Minibatch Size	64	64

Appendix

Table 1: MuJoCo Hyperparameters (GAE labels). Comparison of the GAE RDA2C configuration reported in Table 4 and the PPO baseline. RDA2C uses a replay buffer of size

W=50{,}000

and enables optional clipped prediction regression with

\epsilon_{\mathrm{clip}}=0.2

. The target-advantage clip is inactive for this configuration.

Common Backbone / Evaluation Settings
Parameter	RDA2C with twin- $Q$ labels (Ours)	SAC (Baseline)
Actor Trunk	$2\times 256$ ReLU MLP	$2\times 256$ ReLU MLP
Actor Heads	Mean, LogStd, bias	Mean, LogStd
Twin- $Q$ Networks	$2\times 256$ ReLU MLP	$2\times 256$ ReLU MLP
Total Timesteps	$1{,}000{,}000$	$1{,}000{,}000$
Num. Environments	5	5

Appendix

Table 2: MuJoCo Hyperparameters (twin-

Q

critic labels). Comparison of the SAC baseline at matched batch size and policy update frequency and the RDA2C with twin-

Q

labels instantiation reported in Table 5 .

Parameter	RDA2C (Ours)	PPO (Baseline)
Shared Settings
CNN Backbone	Nature CNN, 512-unit output	Nature CNN, 512-unit output
Actor–Critic Trunks	Independent policy and value trunks	Shared trunk
Optimizer	Adam	Adam
Learning Rate	$2.5\times 10^{-4}$	$2.5\times 10^{-4}$
LR Schedule	Linear Decay	Linear Decay

Appendix

Table 3: Atari Hyperparameters. Final clipped RDA2C setting compared with PPO.

Environment	PPO (Baseline)	RDA2C (Ours)	$\Delta\%$
Ant-v5	$1953\pm 372$	$\mathbf{2831\pm 583}$	$\mathbf{+45.0\%}$
HalfCheetah-v5	$1864\pm 78$	$\mathbf{7099\pm 645}$	$\mathbf{+280.9\%}$
Hopper-v5	$2333\pm 590$	$\mathbf{2607\pm 272}$	$\mathbf{+11.7\%}$
Humanoid-v5	$2370\pm 976$	$\mathbf{3768\pm 1290}$	$\mathbf{+59.0\%}$
HumanoidStandup-v5	$\mathbf{170412\pm 8967}$	$156068\pm 9614$	$-8.4\%$
InvertedPendulum-v5	$989\pm 36$	$\mathbf{999.2\pm 2.6}$	$\mathbf{+1.1\%}$

Appendix

Table 4: MuJoCo Continuous Control Results. Mean episodic return (

\pm

std) over 10 seeds. PPO is matched to RDA2C’s architecture and optimization settings (Appendix A ); the CleanRL-default comparison appears in Appendix B . RDA2C uses clipped dual regression with

\epsilon_{\mathrm{clip}}=0.2

and value-loss clipping enabled. Last column reports relative change

\Delta\%\triangleq 100\cdot(\text{RDA2C}-\text{PPO})/(|\text{PPO}|)

(higher is better).

Environment	SAC (Baseline)	RDA2C with twin- $Q$ labels (Ours)	$\Delta\%$
Ant-v5	$\mathbf{4389.60\pm 1100.19}$	$4240.94\pm 877.82$	$-3.4\%$
HalfCheetah-v5	$\mathbf{10118.79\pm 894.32}$	$9524.52\pm 1388.40$	$-5.9\%$
Hopper-v5	$3345.29\pm 204.84$	$\mathbf{3408.27\pm 121.76}$	$\mathbf{+1.9\%}$
Humanoid-v5	$\mathbf{5376.44\pm 175.72}$	$5195.96\pm 314.71$	$-3.4\%$
HumanoidStandup-v5	$158309.50\pm 4088.25$	$\mathbf{158784.44\pm 6868.90}$	$\mathbf{+0.3\%}$
InvertedPendulum-v5	$\mathbf{1000.00\pm 0.00}$	$\mathbf{1000.00\pm 0.00}$	$0.0\%$

Appendix

Table 5: MuJoCo twin-

Q

critic results. Best deterministic evaluation return (

\pm

seed std) within the 1M-step budget over 10 seeds. Both methods use batch size

512

, policy update frequency

1

, 5 parallel environments, and 20 evaluation episodes; the CleanRL-default SAC comparison appears in Appendix B . Last column reports relative percent change against SAC (higher is better).

Environment	AAPDA	RDA2C (Ours)	$\Delta\%$
Ant-v5	$2562\pm 374$	$\mathbf{4043\pm 691}$	$\mathbf{+57.8\%}$
HalfCheetah-v5	$4948\pm 705$	$\mathbf{7808\pm 629}$	$\mathbf{+57.8\%}$
Hopper-v5	$2594\pm 794$	$\mathbf{3218\pm 196}$	$\mathbf{+24.0\%}$
Humanoid-v5	$1209\pm 1335$	$\mathbf{3653\pm 1459}$	$\mathbf{+202.1\%}$
HumanoidStandup-v5	$\mathbf{162208\pm 2025}$	$156485\pm 9003$	$-3.5\%$
InvertedPendulum-v5	$980.7\pm 49.6$	$\mathbf{1000.0\pm 0.0}$	$\mathbf{+2.0\%}$

Appendix

Table 6: MuJoCo results against AAPDA. Mean of the final five aligned deterministic checkpoints (

\pm

seed std) over 10 paired seeds. Last column reports relative change

\Delta\%\triangleq 100\cdot(\text{RDA2C}-\text{AAPDA})/(|\text{AAPDA}|)

Environment	PPO (default)	RDA2C (Ours)	$\Delta\%$
Ant-v5	$1855\pm 519$	$\mathbf{2831\pm 583}$	$\mathbf{+52.6\%}$
HalfCheetah-v5	$1884\pm 78$	$\mathbf{7099\pm 645}$	$\mathbf{+276.8\%}$
Hopper-v5	$2588\pm 831$	$\mathbf{2607\pm 272}$	$\mathbf{+0.7\%}$
Humanoid-v5	$1100\pm 388$	$\mathbf{3768\pm 1290}$	$\mathbf{+242.5\%}$
HumanoidStandup-v5	$\mathbf{158572\pm 12341}$	$156068\pm 9614$	$-1.6\%$
InvertedPendulum-v5	$\mathbf{1000.0\pm 0.0}$	$999.2\pm 2.6$	$-0.1\%$

Appendix

Table 7: MuJoCo results against CleanRL-default PPO. Mean episodic return (

\pm

std) over 10 seeds. Same RDA2C runs as Table 4 .

Environment	SAC (default)	RDA2C with twin- $Q$ labels (Ours)	$\Delta\%$
Ant-v5	$4048.10\pm 859.12$	$\mathbf{4240.94\pm 877.82}$	$\mathbf{+4.8\%}$
HalfCheetah-v5	$\mathbf{9844.07\pm 914.73}$	$9524.52\pm 1388.40$	$-3.2\%$
Hopper-v5	$3348.58\pm 157.32$	$\mathbf{3408.27\pm 121.76}$	$\mathbf{+1.8\%}$
Humanoid-v5	$\mathbf{5228.44\pm 241.40}$	$5195.96\pm 314.71$	$-0.6\%$
HumanoidStandup-v5	$156360.89\pm 7197.51$	$\mathbf{158784.44\pm 6868.90}$	$\mathbf{+1.5\%}$
InvertedPendulum-v5	$\mathbf{1000.00\pm 0.00}$	$\mathbf{1000.00\pm 0.00}$	$0.0\%$

Appendix

Table 8: MuJoCo twin-

Q

results against CleanRL-default SAC (batch

256

, policy frequency

2

). Best deterministic evaluation return (

\pm

seed std) within the 1M-step budget over 10 seeds. Same RDA2C runs as Table 5 .

Environment	PPO (Baseline)	RDA2C (Ours)	$\Delta\%$
Asterix	$4061\pm 1014$	$\mathbf{6024\pm 1336}$	$\mathbf{+48.3\%}$
BeamRider	$2540\pm 316$	$\mathbf{9142\pm 786}$	$\mathbf{+259.9\%}$
Breakout	$\mathbf{422\pm 24}$	$408\pm 29$	$-3.5\%$
Enduro	$980\pm 135$	$\mathbf{5348\pm 1720}$	$\mathbf{+445.6\%}$
Freeway	$\mathbf{32.7\pm 0.6}$	$23.6\pm 16.3$	$-27.8\%$
Gravitar	$686\pm 187$	$\mathbf{835\pm 376}$	$\mathbf{+21.8\%}$

Appendix

Table 9: Atari results. Mean running return (

\pm

seed std) over 10 seeds on twelve Atari games with clipped dual regression,

\epsilon_{\mathrm{clip}}=1.0

\beta=0.6

, and

\lambda=0.005

. Last column reports relative change

\Delta\%\triangleq 100\cdot(\text{RDA2C}-\text{PPO})/(|\text{PPO}|)

when PPO

\neq 0

Figure 4: PPO versus RDA2C MuJoCo learning curves. Blue denotes RDA2C and red denotes PPO. Shaded regions show standard deviation over 10 seeds. The clipped-dual-regression configuration yields large gains on HalfCheetah , Humanoid , and Ant , while remaining close to PPO on the other environments.

Environment	SAC (Baseline)	RDA2C with twin- $Q$ labels (Ours)	$\Delta\%$
Ant-v5	$3414.91\pm 873.59$	$\mathbf{3683.40\pm 842.94}$	$\mathbf{+7.9\%}$
HalfCheetah-v5	$\mathbf{9182.51\pm 888.07}$	$8538.56\pm 1167.47$	$-7.0\%$
Hopper-v5	$2289.03\pm 637.75$	$\mathbf{2948.40\pm 357.63}$	$\mathbf{+28.8\%}$
Humanoid-v5	$4771.82\pm 320.91$	$\mathbf{4979.12\pm 215.64}$	$\mathbf{+4.3\%}$
HumanoidStandup-v5	$152428.66\pm 7472.47$	$\mathbf{155100.32\pm 5994.39}$	$\mathbf{+1.8\%}$
InvertedPendulum-v5	$935.64\pm 72.85$	$\mathbf{969.61\pm 33.70}$	$\mathbf{+3.6\%}$

Appendix

Table 10: MuJoCo online training return with twin-

Q

critic labels. Final logged running reward (

\pm

seed std) over 10 seeds at

1{,}000{,}000

environment steps. This is an online data collection metric, distinct from the deterministic evaluation return in Table 5 . Last column reports relative percent change against SAC (higher is better).

Figure 5: MuJoCo online training curves with twin-

Q

critic labels. Blue denotes RDA2C twin-

Q

and red denotes SAC. Shaded regions show standard deviation over 10 seeds. The curves report the online running reward metric logged during data collection. RDA2C twin-

Q

achieves higher final running reward on Ant, Hopper, Humanoid, HumanoidStandup, InvertedPendulum, Pusher, and Walker2d, while SAC remains stronger on HalfCheetah.

Figure 6: Full Training Dynamics (10M Frames). Comparison of RDA2C and PPO across the twelve Atari games used in the main comparison.

Environment	$W=300,000$	$W=500,000$	$W=700,000$	$W=1,000,000$
Asterix	$\mathbf{5448\pm 1063}$	$5305\pm 204$	$4758\pm 929$	$4122\pm 505$
BeamRider	$9768\pm 429$	$\mathbf{10509\pm 874}$	$9823\pm 317$	$9831\pm 832$
Breakout	$\mathbf{369\pm 17}$	$353\pm 6$	$319\pm 22$	$314\pm 31$
Enduro	$\mathbf{3474\pm 372}$	$3374\pm 273$	$3077\pm 520$	$3076\pm 340$
Freeway	$33.8\pm 0.0$	$33.8\pm 0.0$	$33.8\pm 0.0$	$\mathbf{33.9\pm 0.0}$
Gravitar	$690\pm 93$	$\mathbf{830\pm 169}$	$675\pm 40$	$670\pm 65$

Appendix

Table 11: Atari Window Size Sweep Results. Mean episodic return at 10M frames, plain squared loss. Bold indicates the best performing configuration.

Figure 7: Effect of Window Size ( $W$ ) on Atari. Training curves for $W\in\{300\text{k},500\text{k},700\text{k},1\text{M}\}$ on the twelve Atari games used in the main comparison. $W=500\text{k}$ offers the most consistent performance across tasks.

Environment	$W=1024$	$W=2048$	$W=4096$	$W=8192$	$W=12{,}800$	$W=25{,}600$
Ant-v5	$4755\pm 985$	$4008\pm 953$	$4314\pm 909$	$\mathbf{4798\pm 747}$	$4241\pm 878$	$4672\pm 946$
HalfCheetah-v5	$7068\pm 3653$	$8800\pm 2565$	$9387\pm 1542$	$8081\pm 3410$	$9525\pm 1388$	$\mathbf{9819\pm 1029}$
Hopper-v5	$3404\pm 195$	$3330\pm 111$	$3365\pm 207$	$3351\pm 133$	$3408\pm 122$	$\mathbf{3430\pm 150}$
Humanoid-v5	$5280\pm 110$	$5237\pm 159$	$5279\pm 200$	$\mathbf{5294\pm 127}$	$5196\pm 315$	$5218\pm 107$
Walker2d-v5	$4444\pm 459$	$3810\pm 1291$	$\mathbf{4724\pm 522}$	$3566\pm 1302$	$4505\pm 788$	$4179\pm 977$

Appendix

Table 12: MuJoCo window size sweep with twin-

Q

critic labels. Mean best deterministic evaluation return (

\pm

seed std) over 10 seeds. Here

W

is the FIFO label buffer size in stored

(s,a,g)

tuples. With batch size

512

and

K=2

label samples,

W=1024

retains approximately one actor trigger’s fresh labels and therefore acts as a current label control. Bold indicates the best window per environment.

Environment	current ( $W=2048$ )	history ( $W=50{,}000$ )	$\Delta\%$
Ant-v5	$884\pm 412$	$\mathbf{2831\pm 583}$	$\mathbf{+220.1\%}$
HalfCheetah-v5	$5162\pm 1298$	$\mathbf{7099\pm 645}$	$\mathbf{+37.5\%}$
Hopper-v5	$2112\pm 802$	$\mathbf{2607\pm 272}$	$\mathbf{+23.4\%}$
Humanoid-v5	$500\pm 75$	$\mathbf{3768\pm 1290}$	$\mathbf{+654.0\%}$
HumanoidStandup-v5	$94062\pm 24669$	$\mathbf{156068\pm 9614}$	$\mathbf{+65.9\%}$
InvertedPendulum-v5	$\mathbf{1000.0\pm 0.0}$	$999.2\pm 2.6$	$-0.1\%$

Appendix

Table 13: Current-label control (GAE labels, MuJoCo). Mean running episodic return (

\pm

seed std) over 10 paired seeds. Last column reports relative change

\Delta\%\triangleq 100\cdot(\text{history}-\text{current})/(|\text{current}|)

(higher is better).

Figure 8: Sensitivity to the RDA stabilizer $\beta$ . Returns are normalized independently within each environment across the swept $\beta$ values before aggregation. Error bars in the aggregate panels denote task SEM. For the reason in footnote 2 , they do not denote confidence intervals. They illustrate the scale of variation across tasks.

Figure 9: Sensitivity to the entropy floor $\lambda$ . Returns are min-max normalized independently within each environment across the swept $\lambda$ values before aggregation. Error bars in the aggregate panels denote task SEM. For the reason in footnote 2 , they do not denote confidence intervals. They illustrate the scale of variation across tasks.

Panel of Figure 10 — Figure 10: Advantage label loss ablation. Atari results are shown above and MuJoCo results below. Returns are min-max normalized within each environment across loss types before aggregation. Error bars in the aggregate panels denote task SEM. For the reason in footnote 2 , they do not denote confidence intervals. They illustrate the scale of variation across tasks. Prediction clipping gives the best aggregate, while Huber alone is weaker.

Figure 11: Sensitivity to standard SAC replay buffer size. We vary the ordinary SAC replay buffer and keep the RDA label window fixed at $12{,}800$ . Returns are min-max normalized independently within each environment across all method and buffer configurations before aggregation. Error bars denote task SEM. InvertedPendulum is omitted from the normalized aggregate because it is saturated for every setting.

Environment	PPO $2\times 64$	PPO $2\times 128$	PPO $2\times 128$	RDA2C $2\times 128$
	lr $3\times 10^{-4}$	lr $3\times 10^{-4}$	lr $2\times 10^{-4}$	lr $2\times 10^{-4}$
Ant-v5	$1855\pm 519$	$1260\pm 441$	$1537\pm 757$	$\mathbf{2831\pm 583}$
HalfCheetah-v5	$1884\pm 78$	$1940\pm 69$	$2190\pm 1059$	$\mathbf{7099\pm 645}$
Hopper-v5	$2588\pm 831$	$2154\pm 719$	$2363\pm 464$	$\mathbf{2607\pm 272}$
Humanoid-v5	$1100\pm 388$	$1187\pm 529$	$1573\pm 901$	$\mathbf{3768\pm 1290}$
HumanoidStandup-v5	$158572\pm 12341$	$\mathbf{161394\pm 6265}$	$159009\pm 8458$	$156068\pm 9614$

Appendix

Table 14: PPO network capacity control (MuJoCo, GAE labels). Mean running episodic return (

\pm

seed std) over 10 seeds. All PPO variants use global log-standard-deviation. Bold marks the best mean in each row.

Figure 12: PPO network capacity control on MuJoCo (GAE labels). Returns are min–max normalized within each environment across the four configurations before aggregation. Error bars in the aggregate panel denote task SEM. Matching PPO’s width to RDA2C does not reproduce the RDA2C gains.

Parameter	CartPole-v1	Acrobot-v1
Total timesteps	$1{,}000{,}000$	$1{,}000{,}000$
Num envs / steps	$8$ / $256$	$8$ / $256$
Batch size	$2048$	$2048$
$\gamma$ / GAE $\lambda_{\mathrm{GAE}}$	$0.99$ / $0.8$	$0.99$ / $0.8$
Value LR / epochs	$2.5{\times}10^{-3}$ / $4$	$2.5{\times}10^{-3}$ / $4$
Anneal LR	True	False

Appendix

Table 15: XGBoost dataset aggregation ( xgb_dagger ) configurations.

Parameter	CartPole-v1	Acrobot-v1
Total timesteps	$1{,}000{,}000$	$1{,}000{,}000$
Num envs / steps	$8$ / $256$	$8$ / $256$
Batch size	$2048$	$2048$
$\gamma$ / GAE $\lambda_{\mathrm{GAE}}$	$0.99$ / $0.8$	$0.99$ / $0.8$
Value LR / epochs	$3{\times}10^{-3}$ / $1$	$10^{-2}$ / $1$
Anneal LR	True	True

Appendix

Table 16: XGBoost incremental boosting ( xgb_boosting ) configurations.

Environment	Method	Final return
CartPole-v1	xgb_dagger	$500.00\pm 0.00$
CartPole-v1	xgb_boosting	$500.00\pm 0.00$
Acrobot-v1	xgb_dagger	$-84.15\pm 4.57$
Acrobot-v1	xgb_boosting	$-88.98\pm 2.43$

Appendix

Table 17: RDA2C gradient boosting results. Final episodic return (mean

\pm

std over 5 seeds).

Panel of Figure 13 — Figure 13: RDA2C gradient boosting learning curves. On CartPole-v1 (left), both xgb_dagger and xgb_boosting reach $500$ within $10^{6}$ steps. On Acrobot-v1 (right), xgb_dagger is slightly more sample efficient early, and both methods converge similarly by $10^{6}$ steps.

Actor-Critic Methods

Actor-critic methods train a policy alongside a value estimator to guide decisions in complex control and decision-making tasks.

See all 104 papers

Mar 10, 2026

Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces

Policy Dual Averaging (PDA) offers a principled Policy Mirror Descent (PMD) framework that more naturally admits value function approximation than standard PMD, enabling the use of approximate advantage (or Q-) functions while retaining strong convergence guarantees. However, applying PDA in continuous state and action spaces remains computationally challenging, since action selection involves solving an optimization sub-problem at each decision step. In this paper, we propose actor-accelerated PDA, which uses a learned policy network to approximate the solution of the optimization sub-problems, yielding faster runtimes while maintaining convergence guarantees. We provide a theoretical analysis that quantifies how actor approximation error impacts the convergence of PDA under suitable assumptions. We then evaluate its performance on several benchmarks in robotics, control, and operations research problems. Actor-accelerated PDA achieves superior performance compared to popular on-policy baselines such as Proximal Policy Optimization (PPO). Overall, our results bridge the gap between the theoretical advantages of PDA and its practical deployment in continuous-action problems with function approximation.

Ji Gao, Caleb Ju, Guanghui Lan +1

School of Chemical and Biomolecular Engineering, Georgia Institute of Technology, GA, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, GA, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, GA, USA

Reinforcement Learning Policy Optimization Policy Learning

Ask PDF

Mar 6, 2026

Optimistic Policy Regularization

Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through directional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments despite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance.

Mai Pham, Vikrant Vaze, Peter Chin

Thayer School of Engineering, Dartmouth College, USA

Sample-Efficient RL Proximal Policy Optimization

Ask PDF

Oct 17, 2026

Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics

Reinforcement learning with general utilities optimizes nonlinear functionals of policy-induced occupancy measures. We develop a natural actor-critic method for concave RLGU objectives admitting an exact variational dual representation, with explicit constructions for a broad class of separable occupancy utilities. Rather than estimating the occupancy measure and then forming the policy-gradient pseudo-reward, our method learns this pseudo-reward directly as a function of the state-action pair using a neural critic. The critic is trained by projected stochastic updates, while multilevel Monte Carlo combines outputs across training horizons to control its bias at logarithmic expected cost. Our analysis separates conditional actor bias from estimator second moments and accounts explicitly for dual optimization, critic approximation, and neural linearization errors. Under policy-transfer, dual-stability, and local critic-regularity conditions, the method achieves

\widetilde{O}(T^{-1/2})

global convergence above policy, critic-class, and finite-width approximation errors using

\widetilde{O}(T)

expected environment transitions and reference samples. For a fixed critic representation, this yields

\widetilde{O}(ε^{-2})

sampling complexity above the approximation floor, covering infinite state-action spaces and occupancy utilities beyond fixed finite vectors of returns.

Ankur Naskar, Vaneet Aggarwal

Indian Institute of Science, India · Purdue University, USA

Reinforcement Learning Sample-Efficient RL Actor-Critic Methods Function Approximation in RL

Ask PDF

Abstract

Actor-Critic Methods

Explore similar work

Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces

Optimistic Policy Regularization

Sample-Efficient Reinforcement Learning with General Utilities via Neural Dual Critics