cs.LGOct 1, 2026

Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

Authors: Alessandro Montenegro, Riccardo Venturelli, Marco Mussi, Matteo Papini, Alberto Maria Metelli

Organizations: Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Among on-policy deep reinforcement learning methods, Proximal Policy Optimization (PPO) has become the de facto standard, due to its consistently strong empirical performance across diverse application domains. However, on-policy methods are inherently sample inefficient: fresh data collected under the current policy is used for just a few updates before being discarded. Off-policy methods avoid this inefficiency via experience replay, achieving notable sample efficiency gains, but at the cost of training instabilities or extensive tuning. This motivated the rise of hybrid strategies that augment PPO with off-policy data reuse. Existing sample-reuse variants of PPO demonstrated improved sample efficiency over vanilla PPO, yet a systematic study of when reuse helps, in which scenarios, and to what extent remains missing. In this work, we study the effectiveness of sample reuse in PPO by instantiating two variants within a multiple importance weighting framework. Both retain the core PPO mechanics, reusing only samples from a window of recent iterations, thereby isolating the effect of data reuse from other factors. The variants, termed wPPO-U and wPPO-BH, employ vanilla importance weights or balance-heuristic-corrected ones, respectively. For both, we derive policy improvement lower bounds providing theoretical grounding for their respective losses. We use them to empirically study when and how data reuse improves sample efficiency or final performance of PPO across continuous control tasks.

Figures & tables

Figure 1 : PPO , ω\omega PPO-U , and ω\omega PPO-BH in the fixed BB setting (10 runs, mean ±\pm 95% C.I.).
  Reacher-v5Walker2d-v5Hopper-v5HalfCheetah-v5
Algorithm Fixed BBFixed NBN_{\mathcal{B}}Fixed BBFixed NBN_{\mathcal{B}}Fixed BBFixed NBN_{\mathcal{B}}Fixed BBFixed NBN_{\mathcal{B}}
PPO −4.35±0.063-4.35\pm 0.0632430±4432430\pm 4432530±1372530\pm 1374330±7254330\pm 725
PPO ( ωK~\omega\widetilde{K} epochs)2×2\times−4.70±0.069-4.70\pm 0.069—2510±5562510\pm 556—2790±3052790\pm 305—4500±11204500\pm 1120—
4×4\times−6.76±0.308-6.76\pm 0.308—682±138682\pm 138—1190±4421190\pm 442—4840±8614840\pm 861—
8×8\times−7.97±0.531-7.97\pm 0.531—392±32.4392\pm 32.4—415±317415\pm 317—3940±8103940\pm 810—
ω\omega PPO-Uω=2\omega=2−4.29±0.065-4.29\pm 0.065−4.22±0.060-4.22\pm 0.0603350±556\mathbf{3350\pm 556}3340±535\mathbf{3340\pm 535}3240±94.3\mathbf{3240\pm 94.3}2680±163\mathbf{2680\pm 163}5560±335\mathbf{5560\pm 335}5160±515\mathbf{5160\pm 515}
Table 1 : Mean final performance (average of last 10 evaluations, 10 runs ±95%\pm 95\% C.I.) for the settings of Figures 1 – 2 . Colored cells: best mean performance; bold: methods whose C.I. intersects the best.
Figure 2 : PPO , ω\omega PPO-U , and ω\omega PPO-BH in the fixed NBN_{\mathcal{B}} setting (10 runs, mean ±\pm 95% C.I.).
Figure 3 : Clip ratio diagnostics (10 runs, mean ±\pm 95% C.I.).
Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

 L^PPO(θ,w,B)=1∣B∣∑i∈BLiclip(θ)−c1Livf(w)+c2H(πθ(⋅∣si)),\widehat{L}^{\texttt{PPO}}(\bm{\theta},\bm{w},\mathcal{B})=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}L^{\text{clip}}_{i}(\bm{\theta})-c_{1}L^{\text{vf}}_{i}(\bm{w})+c_{2}\mathcal{H}(\pi_{\bm{\theta}}(\cdot\mid\bm{s}_{i})), 
Appendix
Algorithm 2 PPO [ 52 ] .
 L^actor(θ,B)=1∣B∣∑(i,j)∈Bmin⁡{ρi,j Ai,j, clip(ρi,j,1−ϵ,1+ϵ)Ai,j}\widehat{L}^{\mathrm{actor}}(\bm{\theta},\mathcal{B})=\frac{1}{|\mathcal{B}|}\sum_{(i,j)\in\mathcal{B}}\min\left\{\rho_{i,j}\,A_{i,j},\ \mathrm{clip}\left(\rho_{i,j},1-\epsilon,1+\epsilon\right)A_{i,j}\right\} 
Appendix
Algorithm 3 ω\omega PPO-U and ω\omega PPO-BH .
HyperparameterAnt-v5HalfCheetah-v5Hopper-v5Reacher-v5Swimmer-v5Walker2d-v5
Environment
Total timesteps3×1063\times 10^{6}1×1061\times 10^{6}1×1061\times 10^{6}1×1061\times 10^{6}1×1061\times 10^{6}1×1061\times 10^{6}
Number of environments ( NEN_{E} )111141
PPO
Learning rate1.91×10−51.91\times 10^{-5}2.06×10−52.06\times 10^{-5}3×10−43\times 10^{-4}1.04×10−41.04\times 10^{-4}6×10−46\times 10^{-4}5.05×10−55.05\times 10^{-5}
Steps per rollout ( HH )51251220485121024512
Appendix
Table 2 : Employed hyperparameters values for conducting the experimental validation shown in Section 4 and D .
SettingEnvironment StepsUpdate EpochsMini-Batch SizeMini-Batch CountUpdates per Iteration
PPOHHK~\widetilde{K}BBNBN_{\mathcal{B}}K~NB\widetilde{K}N_{\mathcal{B}}
PPO ( ωK~\omega\widetilde{K} epochs)HHωK~\omega\widetilde{K}BBNBN_{\mathcal{B}}ωK~NB\omega\widetilde{K}N_{\mathcal{B}}
ω\omega PPO- □\Box : Fixed BBHHK~\widetilde{K}BBωNB\omega N_{\mathcal{B}}ωK~NB\omega\widetilde{K}N_{\mathcal{B}}
ω\omega PPO- □\Box : Fixed NBN_{\mathcal{B}}HHK~\widetilde{K}ωB\omega BNBN_{\mathcal{B}}K~NB\widetilde{K}N_{\mathcal{B}}
ω\omega PPO- □\Box : H/ωH/\omega StepsH/ωH/\omegaK~\widetilde{K}BBNBN_{\mathcal{B}}K~NB\widetilde{K}N_{\mathcal{B}}
Appendix
Table 3 : Summary of the main features of the considered comparison setting. Highlighted cells mark values that differ from the PPO baseline.
Figure 4 : Comparison among PPO , ω\omega PPO-BH , and ω\omega PPO-U in various MuJoCo environments under different window values in the fixed BB setting, showing the PPO version with increased epochs to match the number of updates performed by the reuse variants (10 runs, mean ±\pm 95% C.I.).
Figure 5 : Comparison among PPO , ω\omega PPO-BH , and ω\omega PPO-U in various MuJoCo environments under different window values in the fixed BB setting, showing the PPO version with increased epochs to match the number of updates performed by the reuse variants (10 runs, mean ±\pm 95% C.I.).
Algorithm Reacher-v5Walker2d-v5Hopper-v5HalfCheetah-v5Swimmer-v5Ant-v5
PPO −4.35±0.063-4.35\pm 0.0632430±4432430\pm 4432530±1372530\pm 1374330±7254330\pm 725322±13.6322\pm 13.63640±204\mathbf{3640\pm 204}
PPO ( ωK~\omega\widetilde{K} epochs)2×2\times−4.70±0.069-4.70\pm 0.0692510±5562510\pm 5562790±3052790\pm 3054500±11204500\pm 1120340±17.5\mathbf{340\pm 17.5}3610±203\mathbf{3610\pm 203}
4×4\times−6.76±0.308-6.76\pm 0.308682±138682\pm 1381190±4421190\pm 4424840±8614840\pm 861309±28.3309\pm 28.32740±2662740\pm 266
8×8\times−7.97±0.531-7.97\pm 0.531392±32.4392\pm 32.4415±317415\pm 3173940±8103940\pm 810256±49.3256\pm 49.31890±2361890\pm 236
ω\omega PPO-Uω=2\omega=2−4.29±0.065-4.29\pm 0.0653350±556\mathbf{3350\pm 556}3240±94.3\mathbf{3240\pm 94.3}5560±335\mathbf{5560\pm 335}334±16.2\mathbf{334\pm 16.2}3650±227\mathbf{3650\pm 227}
ω=4\omega=4−4.25±0.075-4.25\pm 0.0753740±546\mathbf{3740\pm 546}3220±192\mathbf{3220\pm 192}5960±729\mathbf{5960\pm 729}339±15.0\mathbf{339\pm 15.0}3840±254\mathbf{3840\pm 254}
Appendix
Table 4 : Average of the last 10 final performance values of the curves shown in the fixed BB setting (Figures 4 and 5 ). Each cell shows the mean ±95%\pm 95\% C.I. edges (10 runs).
Algorithm Reacher-v5Walker2d-v5Hopper-v5HalfCheetah-v5Swimmer-v5Ant-v5
PPO −4.35±0.063-4.35\pm 0.0632430±4432430\pm 4432530±1372530\pm 1374330±7254330\pm 725322±13.6\mathbf{322\pm 13.6}3640±204\mathbf{3640\pm 204}
ω\omega PPO-Uω=2\omega=2−4.22±0.060-4.22\pm 0.0603340±535\mathbf{3340\pm 535}2680±163\mathbf{2680\pm 163}5160±515\mathbf{5160\pm 515}327±9.56\mathbf{327\pm 9.56}3910±130\mathbf{3910\pm 130}
ω=4\omega=4−4.17±0.063-4.17\pm 0.0632820±485\mathbf{2820\pm 485}2810±157\mathbf{2810\pm 157}5760±569\mathbf{5760\pm 569}327±11.2\mathbf{327\pm 11.2}3820±491\mathbf{3820\pm 491}
ω=8\omega=8−4.05±0.057\mathbf{-4.05\pm 0.057}3030±430\mathbf{3030\pm 430}2750±131\mathbf{2750\pm 131}5360±384\mathbf{5360\pm 384}334±10.5\mathbf{334\pm 10.5}3970±301\mathbf{3970\pm 301}
ω\omega PPO-BHω=2\omega=2−4.12±0.052-4.12\pm 0.0523250±530\mathbf{3250\pm 530}2740±128\mathbf{2740\pm 128}4580±949\mathbf{4580\pm 949}327±7.69\mathbf{327\pm 7.69}3650±1363650\pm 136
ω=4\omega=4−4.01±0.045\mathbf{-4.01\pm 0.045}3810±642\mathbf{3810\pm 642}2920±159\mathbf{2920\pm 159}5780±558\mathbf{5780\pm 558}334±7.51\mathbf{334\pm 7.51}4030±197\mathbf{4030\pm 197}
Appendix
Table 5 : Average of the last 10 final performance values of the curves shown in the fixed NBN_{\mathcal{B}} setting (Figures 6 and 7 ). Each cell shows the mean ±95%\pm 95\% C.I. edges (10 runs).
Figure 6 : Comparison among PPO , ω\omega PPO-BH , and ω\omega PPO-U in various MuJoCo environments under different window values in the fixed NBN_{\mathcal{B}} setting (10 runs, mean ±\pm 95%).
Figure 7 : Comparison among PPO , ω\omega PPO-BH , and ω\omega PPO-U in various MuJoCo environments under different window values in the Fixed NBN_{\mathcal{B}} setting (10 runs, mean ±\pm 95%).
Algorithm Reacher-v5Walker2d-v5Hopper-v5HalfCheetah-v5Swimmer-v5Ant-v5
PPO −4.35±0.063\mathbf{-4.35\pm 0.063}2430±443\mathbf{2430\pm 443}2530±1372530\pm 1374330±725\mathbf{4330\pm 725}322±13.6\mathbf{322\pm 13.6}3640±204\mathbf{3640\pm 204}
ω\omega PPO-Uω=2\omega=2−4.86±0.113-4.86\pm 0.1132300±629\mathbf{2300\pm 629}2920±154\mathbf{2920\pm 154}3960±994\mathbf{3960\pm 994}292±51.6\mathbf{292\pm 51.6}3520±222\mathbf{3520\pm 222}
ω=4\omega=4−6.30±0.342-6.30\pm 0.3421420±3001420\pm 3001920±5531920\pm 5533830±3733830\pm 373251±62.1\mathbf{251\pm 62.1}2670±2622670\pm 262
ω=8\omega=8−7.78±0.366-7.78\pm 0.366686±157686\pm 157438±198438\pm 1982710±7502710\pm 750137±70.1137\pm 70.11600±2161600\pm 216
ω\omega PPO-BHω=2\omega=2−4.48±0.084\mathbf{-4.48\pm 0.084}2270±463\mathbf{2270\pm 463}3020±163\mathbf{3020\pm 163}5040±711\mathbf{5040\pm 711}269±52.0\mathbf{269\pm 52.0}3500±116\mathbf{3500\pm 116}
ω=4\omega=4−5.06±0.191-5.06\pm 0.1911160±2861160\pm 2861770±5551770\pm 5554070±1080\mathbf{4070\pm 1080}207±43.5207\pm 43.52150±4592150\pm 459
Appendix
Table 6 : Average of the last 10 final performance values of the curves shown in the H/ωH/\omega steps setting (Figures 8 and 9 ). Each cell shows the mean ±95%\pm 95\% C.I. edges (10 runs).
Figure 8 : Comparison among PPO , ω\omega PPO-BH , and ω\omega PPO-U in various MuJoCo environments under different window values in the H/ωH/\omega steps setting (10 runs, mean ±\pm 95%).
Figure 9 : Comparison among PPO , ω\omega PPO-BH , and ω\omega PPO-U in various MuJoCo environments under different window values in the H/ωH/\omega steps setting (10 runs, mean ±\pm 95%).
Figure 10 : Clip Fraction and ESS diagnostics in Reacher-v5 (10 runs, mean ±\pm 95% C.I.).
Figure 11 : Clip Fraction and ESS diagnostics in Swimmer-v5 (10 runs, mean ±\pm 95% C.I.).
Figure 12 : Clip Fraction and ESS diagnostics in Hopper-v5 (10 runs, mean ±\pm 95% C.I.).
Figure 13 : Clip Fraction and ESS diagnostics in Walker2d-v5 (10 runs, mean ±\pm 95% C.I.).
Figure 14 : Clip Fraction and ESS diagnostics in HalfCheetah-v5 (10 runs, mean ±\pm 95% C.I.).
Figure 15 : Clip Fraction and ESS diagnostics in Ant-v5 (10 runs, mean ±\pm 95% C.I.).
Figure 16 : Ablation on training the critic on on-policy data only vs. on off-policy data too. Evaluation conducted in the Reacher-v5 environment, considering all the presented comparison settings (10 runs, mean ±\pm 95%).
Figure 17 : Ablation on training the critic on on-policy data only vs. on off-policy data too. Evaluation conducted in the Swimmer-v5 environment, considering all the presented comparison settings (10 runs, mean ±\pm 95%).
Figure 18 : Ablation on training the critic on on-policy data only vs. on off-policy data too. Evaluation conducted in the Hopper-v5 environment, considering all the presented comparison settings (10 runs, mean ±\pm 95%).
  ReacherHopperSwimmer
Algorithm Fixed BBFixed NBN_{\mathcal{B}}H/ωH/\omega StepsFixed BBFixed NBN_{\mathcal{B}}H/ωH/\omega StepsFixed BBFixed NBN_{\mathcal{B}}H/ωH/\omega Steps
PPO −4.35±0.063-4.35\pm 0.063−4.35±0.063-4.35\pm 0.063−4.35±0.063\mathbf{-4.35\pm 0.063}2530±1372530\pm 1372530±1372530\pm 1372530±1372530\pm 137322±13.6322\pm 13.6322±13.6\mathbf{322\pm 13.6}322±13.6\mathbf{322\pm 13.6}
PPO ( ωK~\omega\widetilde{K} epochs)2×2\times−4.70±0.069-4.70\pm 0.069——2790±3052790\pm 305——340±17.5\mathbf{340\pm 17.5}——
4×4\times−6.76±0.308-6.76\pm 0.308——1190±4421190\pm 442——309±28.3309\pm 28.3——
8×8\times−7.97±0.531-7.97\pm 0.531——415±317415\pm 317——256±49.3256\pm 49.3——
ω\omega PPO-Uω=2\omega=2−4.29±0.065-4.29\pm 0.065−4.22±0.060-4.22\pm 0.060−4.86±0.113-4.86\pm 0.1133240±94.3\mathbf{3240\pm 94.3}2680±1632680\pm 1632920±154\mathbf{2920\pm 154}334±16.2\mathbf{334\pm 16.2}327±9.56\mathbf{327\pm 9.56}292±51.6\mathbf{292\pm 51.6}
Appendix
Table 7 : Average of the last 10 final performance values of the curves shown in the critic fitting ablation (Figures 16 , 17 , and 18 ). Each cell shows the mean ±95%\pm 95\% C.I. edges (10 runs).

Explore similar work

CardsList

  1. Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reuse

    Sep 30, 2026Nima H. SiboniReuseStep Accuracy

    Juna.ai, Kastanienallee 32, 10435 Berlin, Germany

  2. Transductive Off-policy Proximal Policy Optimization

    Jun 6, 2024Yaozhong Gan, Renye Yan, Xiaoyang Tan +2Proximal Policy OptimizationOff-Policy Learning

    Independent Researcher · Peking University · Nanjing University of Aeronautics and Astronautics +1

  3. Reflective Policy Optimization

    Jun 6, 2024Yaozhong Gan, Renye Yan, Zhe Wu +1Proximal Policy OptimizationFrictive Policy Optimization

    QiYuan Lab.