Optimistic Policy Regularization

Mai Pham; Vikrant Vaze; Peter Chin

cs.LGMar 6, 2026

Optimistic Policy Regularization

Authors: Mai Pham, Vikrant Vaze, Peter Chin

Organizations: Thayer School of Engineering, Dartmouth College, USA

Abstract

Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through directional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments despite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance.

Figures & tables

Figure 1: The Optimistic Policy Regularization Architecture. Top-performing trajectories are conditionally stored in a FIFO Good-Episode Buffer. During optimization, these historical successes provide both a directional log-ratio reward shaping signal on the primary policy and an auxiliary Behavioral Cloning objective, shielding the agent against pessimistic exploration collapse.

Panel of Figure 2 — Figure 2: Performance dominance across the Atari suite. The left panel compares the total number of environments where each evaluated configuration achieved the highest peak score at the 10M-step benchmark across the full 49-game suite. The right panel shows the same metric for the 14-game subset evaluated after 50M environment steps. OPR maintains a clear performance advantage across both evaluation horizons.

Game	DQN	A2C	PPO	ACPER	SIL	OPR (Ours)
Alien	227.1	1859.2	1475.6	390.2	2242.2	3389.0
Amidar	163.6	739.9	757.2	424.8	1362.0	2571.2
Assault	537.5	1981.4	4970.6	818.2	1812.0	5285.7
Asterix	855.0	16083.3	4531.0	3533.1	17984.2	3410.0
Asteroids	1007.8	2056.0	2195.1	1780.1	2259.4	2632.0
Atlantis	58012.5	3032444.2	3010196.0	58012.5	3084781.7	880040.0

Table 1: Detailed Performance across 49 Atari games. OPR is evaluated at 10M steps (100 Epochs), while ACPER, SIL, and DQN baselines are evaluated at the standard 50M step (200M frame) mark. OPR outperforms or ties these high-resource baselines in 22 environments.

Game	DQN	A2C	PPO	ACPER	A2C+SIL	OPR (Ours)
Alien	227.1	1859.2	1475.6	390.2	2242.2	4888.0
Amidar	163.6	739.9	757.2	424.8	1362.0	7155.3
Assault	537.5	1981.4	4970.6	818.2	1812.0	14508.2
Asterix	855.0	16083.3	4531.0	3533.1	17984.2	35110.0
BankHeist	421.1	1333.7	1118.8	1203.2	1137.8	375.0
BeamRider	2896.7	3931.7	5046.1	2602.4	2366.2	9663.6

Table 2: Detailed Performance at 50M Steps (500 Epochs) across the subset of 14 Atari games. Baseline scores correspond to standard 50M-step benchmark results reported in prior work.

Figure 3: Learning dynamics of OPR on representative Atari environments where the method demonstrates strong improvements. The green curve shows the performance of PPO augmented with OPR during training, with shaded regions indicating variability across evaluations. Dashed horizontal lines denote benchmark scores achieved by baseline methods (DQN, A2C, PPO, ACPER, and A2C+SIL) after 50M environment interactions . The vertical dashed line marks the 10M-step evaluation point used for the primary comparison in this work. In many environments, OPR approaches or surpasses the final baseline performance within the first 10M interactions and continues improving thereafter. Notably, several curves show no clear performance plateau even near the 50M interaction horizon, suggesting that OPR does not prematurely saturate and continues improving policies over extended training.

Figure 4: Performance of OPR on the CAGE Challenge 2 attacker resilience environment. The right figure shows a zoomed view of the final phase, where OPR surpasses the Cardiff’s winning solution.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
PPO Base
Learning Rate	$2.5\times 10^{-4}$ (Linear Decay)
Gamma ( $\gamma$ )	0.99
GAE Lambda ( $\lambda$ )	0.95
Entropy Coefficient	0.01
Value Function Coefficient	0.25

Appendix

Table 3: Hyperparameters for Atari 2600 experiments (PPO + OPR).

Hyperparameter	Value
Learning Rate	0.002
Gamma	0.99
Betas	[0.9, 0.990]
K Epochs	6
EPS Clip	0.2
Max Episodes	100,000

Appendix

Table 4: Hyperparameters for CAGE Challenge 2 experiments.

Figure 5: Breakthrough Learning Dynamics (Balanced Environments): Reward escalation for the remaining 6 environments. In these settings, OPR remains highly competitive with high-resource baselines, demonstrating stable strategy refinement even in environments where dense-reward imitation (SIL) or large-scale prioritized replay (ACPER) provide strong specialized advantages.

Sample-Efficient RL

Sample-efficient reinforcement learning studies how agents learn effective policies from fewer environment interactions, demonstrations, or costly rollouts.

See all 245 papers

Reflective Policy Optimization
Jun 6, 2024Yaozhong Gan, Renye Yan, Zhe Wu +1Policy Optimization Policy Gradient Methods
QiYuan Lab.
Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
Mar 13, 2026Zelal Su, Mustafaoglu, Sungyoung Lee +3Policy Optimization Proximal Policy Optimization
Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience Replay Proximal Policy Optimization
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy

Jun 6, 2024J/K move · Enter open · S save

Reflective Policy Optimization

Yaozhong Gan, Renye Yan, Zhe Wu, Junliang Xing

QiYuan Lab.

Policy Optimization Policy Gradient Methods Sample-Efficient RL

Open paper AskPDF arXiv

On-policy reinforcement learning methods, like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand extensive data per update, leading to sample inefficiency. This paper introduces Reflective Policy Optimization (RPO), a novel on-policy extension that amalgamates past and future state-action information for policy optimization. This approach empowers the agent for introspection, allowing modifications to its actions within the current state. Theoretical analysis confirms that policy performance is monotonically improved and contracts the solution space, consequently expediting the convergence procedure. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks, culminating in superior sample efficiency. The source code of this work is available at https://github.com/Edgargan/RPO.

Abstract

Sample-Efficient RL

Explore similar work

Reflective Policy Optimization

Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization

Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?