Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

Alessandro Montenegro; Riccardo Venturelli; Marco Mussi; Matteo Papini; Alberto Maria Metelli

cs.LGOct 1, 2026

Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

Authors: Alessandro Montenegro, Riccardo Venturelli, Marco Mussi, Matteo Papini, Alberto Maria Metelli

Organizations: Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy

Abstract

Among on-policy deep reinforcement learning methods, Proximal Policy Optimization (PPO) has become the de facto standard, due to its consistently strong empirical performance across diverse application domains. However, on-policy methods are inherently sample inefficient: fresh data collected under the current policy is used for just a few updates before being discarded. Off-policy methods avoid this inefficiency via experience replay, achieving notable sample efficiency gains, but at the cost of training instabilities or extensive tuning. This motivated the rise of hybrid strategies that augment PPO with off-policy data reuse. Existing sample-reuse variants of PPO demonstrated improved sample efficiency over vanilla PPO, yet a systematic study of when reuse helps, in which scenarios, and to what extent remains missing. In this work, we study the effectiveness of sample reuse in PPO by instantiating two variants within a multiple importance weighting framework. Both retain the core PPO mechanics, reusing only samples from a window of recent iterations, thereby isolating the effect of data reuse from other factors. The variants, termed wPPO-U and wPPO-BH, employ vanilla importance weights or balance-heuristic-corrected ones, respectively. For both, we derive policy improvement lower bounds providing theoretical grounding for their respective losses. We use them to empirically study when and how data reuse improves sample efficiency or final performance of PPO across continuous control tasks.

Figures & tables

Panel of Figure 1 — Figure 1 : PPO , $\omega$ PPO-U , and $\omega$ PPO-BH in the fixed $B$ setting (10 runs, mean $\pm$ 95% C.I.).

		Reacher-v5		Walker2d-v5		Hopper-v5		HalfCheetah-v5
Algorithm		Fixed $B$	Fixed $N_{\mathcal{B}}$	Fixed $B$	Fixed $N_{\mathcal{B}}$	Fixed $B$	Fixed $N_{\mathcal{B}}$	Fixed $B$	Fixed $N_{\mathcal{B}}$
PPO		$-4.35\pm 0.063$		$2430\pm 443$		$2530\pm 137$		$4330\pm 725$
PPO ( $\omega\widetilde{K}$ epochs)	$2\times$	$-4.70\pm 0.069$	—	$2510\pm 556$	—	$2790\pm 305$	—	$4500\pm 1120$	—
	$4\times$	$-6.76\pm 0.308$	—	$682\pm 138$	—	$1190\pm 442$	—	$4840\pm 861$	—
	$8\times$	$-7.97\pm 0.531$	—	$392\pm 32.4$	—	$415\pm 317$	—	$3940\pm 810$	—
$\omega$ PPO-U	$\omega=2$	$-4.29\pm 0.065$	$-4.22\pm 0.060$	$\mathbf{3350\pm 556}$	$\mathbf{3340\pm 535}$	$\mathbf{3240\pm 94.3}$	$\mathbf{2680\pm 163}$	$\mathbf{5560\pm 335}$	$\mathbf{5160\pm 515}$

Table 1 : Mean final performance (average of last 10 evaluations, 10 runs

\pm 95\%

C.I.) for the settings of Figures 1 – 2 . Colored cells: best mean performance; bold: methods whose C.I. intersects the best.

Panel of Figure 2 — Figure 2 : PPO , $\omega$ PPO-U , and $\omega$ PPO-BH in the fixed $N_{\mathcal{B}}$ setting (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 3 — Figure 3 : Clip ratio diagnostics (10 runs, mean $\pm$ 95% C.I.).

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

\widehat{L}^{\texttt{PPO}}(\bm{\theta},\bm{w},\mathcal{B})=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}L^{\text{clip}}_{i}(\bm{\theta})-c_{1}L^{\text{vf}}_{i}(\bm{w})+c_{2}\mathcal{H}(\pi_{\bm{\theta}}(\cdot\mid\bm{s}_{i})),

Appendix

Algorithm 2 PPO [ 52 ] .

\widehat{L}^{\mathrm{actor}}(\bm{\theta},\mathcal{B})=\frac{1}{|\mathcal{B}|}\sum_{(i,j)\in\mathcal{B}}\min\left\{\rho_{i,j}\,A_{i,j},\ \mathrm{clip}\left(\rho_{i,j},1-\epsilon,1+\epsilon\right)A_{i,j}\right\}

Appendix

Algorithm 3

\omega

PPO-U and

\omega

PPO-BH .

Hyperparameter	Ant-v5	HalfCheetah-v5	Hopper-v5	Reacher-v5	Swimmer-v5	Walker2d-v5
Environment
Total timesteps	$3\times 10^{6}$	$1\times 10^{6}$	$1\times 10^{6}$	$1\times 10^{6}$	$1\times 10^{6}$	$1\times 10^{6}$
Number of environments ( $N_{E}$ )	1	1	1	1	4	1
PPO
Learning rate	$1.91\times 10^{-5}$	$2.06\times 10^{-5}$	$3\times 10^{-4}$	$1.04\times 10^{-4}$	$6\times 10^{-4}$	$5.05\times 10^{-5}$
Steps per rollout ( $H$ )	512	512	2048	512	1024	512

Appendix

Table 2 : Employed hyperparameters values for conducting the experimental validation shown in Section 4 and D .

Setting	Environment Steps	Update Epochs	Mini-Batch Size	Mini-Batch Count	Updates per Iteration
PPO	$H$	$\widetilde{K}$	$B$	$N_{\mathcal{B}}$	$\widetilde{K}N_{\mathcal{B}}$
PPO ( $\omega\widetilde{K}$ epochs)	$H$	$\omega\widetilde{K}$	$B$	$N_{\mathcal{B}}$	$\omega\widetilde{K}N_{\mathcal{B}}$
$\omega$ PPO- $\Box$ : Fixed $B$	$H$	$\widetilde{K}$	$B$	$\omega N_{\mathcal{B}}$	$\omega\widetilde{K}N_{\mathcal{B}}$
$\omega$ PPO- $\Box$ : Fixed $N_{\mathcal{B}}$	$H$	$\widetilde{K}$	$\omega B$	$N_{\mathcal{B}}$	$\widetilde{K}N_{\mathcal{B}}$
$\omega$ PPO- $\Box$ : $H/\omega$ Steps	$H/\omega$	$\widetilde{K}$	$B$	$N_{\mathcal{B}}$	$\widetilde{K}N_{\mathcal{B}}$

Appendix

Table 3 : Summary of the main features of the considered comparison setting. Highlighted cells mark values that differ from the PPO baseline.

Panel of Figure 4 — Figure 4 : Comparison among PPO , $\omega$ PPO-BH , and $\omega$ PPO-U in various MuJoCo environments under different window values in the fixed $B$ setting, showing the PPO version with increased epochs to match the number of updates performed by the reuse variants (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 5 — Figure 5 : Comparison among PPO , $\omega$ PPO-BH , and $\omega$ PPO-U in various MuJoCo environments under different window values in the fixed $B$ setting, showing the PPO version with increased epochs to match the number of updates performed by the reuse variants (10 runs, mean $\pm$ 95% C.I.).

Algorithm		Reacher-v5	Walker2d-v5	Hopper-v5	HalfCheetah-v5	Swimmer-v5	Ant-v5
PPO		$-4.35\pm 0.063$	$2430\pm 443$	$2530\pm 137$	$4330\pm 725$	$322\pm 13.6$	$\mathbf{3640\pm 204}$
PPO ( $\omega\widetilde{K}$ epochs)	$2\times$	$-4.70\pm 0.069$	$2510\pm 556$	$2790\pm 305$	$4500\pm 1120$	$\mathbf{340\pm 17.5}$	$\mathbf{3610\pm 203}$
	$4\times$	$-6.76\pm 0.308$	$682\pm 138$	$1190\pm 442$	$4840\pm 861$	$309\pm 28.3$	$2740\pm 266$
	$8\times$	$-7.97\pm 0.531$	$392\pm 32.4$	$415\pm 317$	$3940\pm 810$	$256\pm 49.3$	$1890\pm 236$
$\omega$ PPO-U	$\omega=2$	$-4.29\pm 0.065$	$\mathbf{3350\pm 556}$	$\mathbf{3240\pm 94.3}$	$\mathbf{5560\pm 335}$	$\mathbf{334\pm 16.2}$	$\mathbf{3650\pm 227}$
	$\omega=4$	$-4.25\pm 0.075$	$\mathbf{3740\pm 546}$	$\mathbf{3220\pm 192}$	$\mathbf{5960\pm 729}$	$\mathbf{339\pm 15.0}$	$\mathbf{3840\pm 254}$

Appendix

Table 4 : Average of the last 10 final performance values of the curves shown in the fixed

B

setting (Figures 4 and 5 ). Each cell shows the mean

\pm 95\%

C.I. edges (10 runs).

Algorithm		Reacher-v5	Walker2d-v5	Hopper-v5	HalfCheetah-v5	Swimmer-v5	Ant-v5
PPO		$-4.35\pm 0.063$	$2430\pm 443$	$2530\pm 137$	$4330\pm 725$	$\mathbf{322\pm 13.6}$	$\mathbf{3640\pm 204}$
$\omega$ PPO-U	$\omega=2$	$-4.22\pm 0.060$	$\mathbf{3340\pm 535}$	$\mathbf{2680\pm 163}$	$\mathbf{5160\pm 515}$	$\mathbf{327\pm 9.56}$	$\mathbf{3910\pm 130}$
	$\omega=4$	$-4.17\pm 0.063$	$\mathbf{2820\pm 485}$	$\mathbf{2810\pm 157}$	$\mathbf{5760\pm 569}$	$\mathbf{327\pm 11.2}$	$\mathbf{3820\pm 491}$
	$\omega=8$	$\mathbf{-4.05\pm 0.057}$	$\mathbf{3030\pm 430}$	$\mathbf{2750\pm 131}$	$\mathbf{5360\pm 384}$	$\mathbf{334\pm 10.5}$	$\mathbf{3970\pm 301}$
$\omega$ PPO-BH	$\omega=2$	$-4.12\pm 0.052$	$\mathbf{3250\pm 530}$	$\mathbf{2740\pm 128}$	$\mathbf{4580\pm 949}$	$\mathbf{327\pm 7.69}$	$3650\pm 136$
	$\omega=4$	$\mathbf{-4.01\pm 0.045}$	$\mathbf{3810\pm 642}$	$\mathbf{2920\pm 159}$	$\mathbf{5780\pm 558}$	$\mathbf{334\pm 7.51}$	$\mathbf{4030\pm 197}$

Appendix

Table 5 : Average of the last 10 final performance values of the curves shown in the fixed

N_{\mathcal{B}}

setting (Figures 6 and 7 ). Each cell shows the mean

\pm 95\%

C.I. edges (10 runs).

Panel of Figure 6 — Figure 6 : Comparison among PPO , $\omega$ PPO-BH , and $\omega$ PPO-U in various MuJoCo environments under different window values in the fixed $N_{\mathcal{B}}$ setting (10 runs, mean $\pm$ 95%).

Panel of Figure 7 — Figure 7 : Comparison among PPO , $\omega$ PPO-BH , and $\omega$ PPO-U in various MuJoCo environments under different window values in the Fixed $N_{\mathcal{B}}$ setting (10 runs, mean $\pm$ 95%).

Algorithm		Reacher-v5	Walker2d-v5	Hopper-v5	HalfCheetah-v5	Swimmer-v5	Ant-v5
PPO		$\mathbf{-4.35\pm 0.063}$	$\mathbf{2430\pm 443}$	$2530\pm 137$	$\mathbf{4330\pm 725}$	$\mathbf{322\pm 13.6}$	$\mathbf{3640\pm 204}$
$\omega$ PPO-U	$\omega=2$	$-4.86\pm 0.113$	$\mathbf{2300\pm 629}$	$\mathbf{2920\pm 154}$	$\mathbf{3960\pm 994}$	$\mathbf{292\pm 51.6}$	$\mathbf{3520\pm 222}$
	$\omega=4$	$-6.30\pm 0.342$	$1420\pm 300$	$1920\pm 553$	$3830\pm 373$	$\mathbf{251\pm 62.1}$	$2670\pm 262$
	$\omega=8$	$-7.78\pm 0.366$	$686\pm 157$	$438\pm 198$	$2710\pm 750$	$137\pm 70.1$	$1600\pm 216$
$\omega$ PPO-BH	$\omega=2$	$\mathbf{-4.48\pm 0.084}$	$\mathbf{2270\pm 463}$	$\mathbf{3020\pm 163}$	$\mathbf{5040\pm 711}$	$\mathbf{269\pm 52.0}$	$\mathbf{3500\pm 116}$
	$\omega=4$	$-5.06\pm 0.191$	$1160\pm 286$	$1770\pm 555$	$\mathbf{4070\pm 1080}$	$207\pm 43.5$	$2150\pm 459$

Appendix

Table 6 : Average of the last 10 final performance values of the curves shown in the

H/\omega

steps setting (Figures 8 and 9 ). Each cell shows the mean

\pm 95\%

C.I. edges (10 runs).

Panel of Figure 8 — Figure 8 : Comparison among PPO , $\omega$ PPO-BH , and $\omega$ PPO-U in various MuJoCo environments under different window values in the $H/\omega$ steps setting (10 runs, mean $\pm$ 95%).

Panel of Figure 9 — Figure 9 : Comparison among PPO , $\omega$ PPO-BH , and $\omega$ PPO-U in various MuJoCo environments under different window values in the $H/\omega$ steps setting (10 runs, mean $\pm$ 95%).

Panel of Figure 10 — Figure 10 : Clip Fraction and ESS diagnostics in Reacher-v5 (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 11 — Figure 11 : Clip Fraction and ESS diagnostics in Swimmer-v5 (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 12 — Figure 12 : Clip Fraction and ESS diagnostics in Hopper-v5 (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 13 — Figure 13 : Clip Fraction and ESS diagnostics in Walker2d-v5 (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 14 — Figure 14 : Clip Fraction and ESS diagnostics in HalfCheetah-v5 (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 15 — Figure 15 : Clip Fraction and ESS diagnostics in Ant-v5 (10 runs, mean $\pm$ 95% C.I.).

Panel of Figure 16 — Figure 16 : Ablation on training the critic on on-policy data only vs. on off-policy data too. Evaluation conducted in the Reacher-v5 environment, considering all the presented comparison settings (10 runs, mean $\pm$ 95%).

Panel of Figure 17 — Figure 17 : Ablation on training the critic on on-policy data only vs. on off-policy data too. Evaluation conducted in the Swimmer-v5 environment, considering all the presented comparison settings (10 runs, mean $\pm$ 95%).

Panel of Figure 18 — Figure 18 : Ablation on training the critic on on-policy data only vs. on off-policy data too. Evaluation conducted in the Hopper-v5 environment, considering all the presented comparison settings (10 runs, mean $\pm$ 95%).

		Reacher			Hopper			Swimmer
Algorithm		Fixed $B$	Fixed $N_{\mathcal{B}}$	$H/\omega$ Steps	Fixed $B$	Fixed $N_{\mathcal{B}}$	$H/\omega$ Steps	Fixed $B$	Fixed $N_{\mathcal{B}}$	$H/\omega$ Steps
PPO		$-4.35\pm 0.063$	$-4.35\pm 0.063$	$\mathbf{-4.35\pm 0.063}$	$2530\pm 137$	$2530\pm 137$	$2530\pm 137$	$322\pm 13.6$	$\mathbf{322\pm 13.6}$	$\mathbf{322\pm 13.6}$
PPO ( $\omega\widetilde{K}$ epochs)	$2\times$	$-4.70\pm 0.069$	—	—	$2790\pm 305$	—	—	$\mathbf{340\pm 17.5}$	—	—
	$4\times$	$-6.76\pm 0.308$	—	—	$1190\pm 442$	—	—	$309\pm 28.3$	—	—
	$8\times$	$-7.97\pm 0.531$	—	—	$415\pm 317$	—	—	$256\pm 49.3$	—	—
$\omega$ PPO-U	$\omega=2$	$-4.29\pm 0.065$	$-4.22\pm 0.060$	$-4.86\pm 0.113$	$\mathbf{3240\pm 94.3}$	$2680\pm 163$	$\mathbf{2920\pm 154}$	$\mathbf{334\pm 16.2}$	$\mathbf{327\pm 9.56}$	$\mathbf{292\pm 51.6}$

Appendix

Table 7 : Average of the last 10 final performance values of the curves shown in the critic fitting ablation (Figures 16 , 17 , and 18 ). Each cell shows the mean

\pm 95\%

C.I. edges (10 runs).

Explore similar work

Sep 30, 2026cs.LG

Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reuse

Common policy improvement methods, including TRPO, PPO, and GRPO, estimate policy improvement under the behavioral policy's state-visitation distribution rather than the improved policy's own. The substitution makes the objective estimable from the behavioral policy's rollouts but adds a bias growing with policy divergence, hence the trust region or clip, and hence no reuse of a batch far off-policy. We show that under history-injective dynamics, where each state is reached by exactly one history, the dropped state-visitation ratio equals the product of per-step policy ratios along the sampled prefix, on every trajectory and not only in expectation. The ratio is therefore restored exactly, from log-probabilities PPO already computes. Autoregressive generation and canonical-order constructive optimization are both history-injective. The exact correction pays importance-sampling variance that grows with the horizon, so we generalize it to a one-parameter family with PPO (

α{=}0

) and the full correction (

α{=}1

) as endpoints: a single bias--variance knob. A gradient-level analysis of the unclipped surrogate identifies two channels the correction acts through and three conditions under which it carries signal; an enumerable testbed confirms the conditions' predictions. On hard credit-assignment scheduling tasks, a short corrected warmup with aggressive early sample reuse learns faster than PPO and than the same reuse uncorrected; the marginal gain grows with task difficulty (

+0.02

+0.09

learning-curve AUC), and the early win over PPO tracks the prefix bias that reuse incurs. A correction held throughout, or applied where clipping already contains the reuse bias, is null to harmful.

Nima H. Siboni

Juna.ai, Kastanienallee 32, 10435 Berlin, Germany

Reuse Step Accuracy

Ask PDF

Jun 6, 2024cs.LG

Transductive Off-policy Proximal Policy Optimization

Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.

Yaozhong Gan, Renye Yan, Xiaoyang Tan +2

Independent Researcher · Peking University · Nanjing University of Aeronautics and Astronautics +1

Proximal Policy Optimization Off-Policy Learning Model-Free Reinforcement Learning Transductive Learning

Ask PDF

Jun 6, 2024cs.LG

Reflective Policy Optimization

On-policy reinforcement learning methods, like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand extensive data per update, leading to sample inefficiency. This paper introduces Reflective Policy Optimization (RPO), a novel on-policy extension that amalgamates past and future state-action information for policy optimization. This approach empowers the agent for introspection, allowing modifications to its actions within the current state. Theoretical analysis confirms that policy performance is monotonically improved and contracts the solution space, consequently expediting the convergence procedure. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks, culminating in superior sample efficiency. The source code of this work is available at https://github.com/Edgargan/RPO.

Yaozhong Gan, Renye Yan, Zhe Wu +1

QiYuan Lab.

Proximal Policy Optimization Frictive Policy Optimization

Ask PDF