Efficient RL Training for LLMs with Experience Replay

Charles Arnal; Vivien Cabannes; Taco Cohen; Julia Kempe; Remi Munos

cs.LGApr 9, 2026

Efficient RL Training for LLMs with Experience Replay

Authors: Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe, Remi Munos

Organizations: FAIR at Meta · NYU Courant Institute and CDS

Abstract

While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.

Figures & tables

Figure 1: Experience Replay improves LLM RL Training. Accuracy on MATH as a function of compute spent when training Qwen2.5-7B on OpenR1-Math-220k for the no-buffer baseline (orange curve) and a buffer of size

84

with

(W,T)=(5,3)

. We report the median and IQR over

10

seeds. Compute is calibrated so that a single weight update for the baseline costs

1

unit. Baseline runs display increased instability.

Figure 2: Effect of Experimental Design on Off-Policiness and Diversity Statistics. Top row : Distribution of off-policiness, replay ratio and steps-since-last-use over all samples and uses of samples during a training run for buffer size

N\in\{84,252,756,2268\}

and

(W,T)=(6,2)

. See also Appendix D.3 for details on the steps-since-last-use metric. Bottom row : Same statistics for

N=252

and

(W,T)\in\{(6,2),(5,3),(4,4)\}

. The average replay ratio is

1.78

3.42

and

7.0

for

(W,T)

equal to

(6,2),(5,3)

and

(4,4)

respectively.

(W, T)	(7,1)	(6,2)	(5,3)	(4,4)	(2,6)	(1,7)
$\gamma$	$1.29$	$0.65$	$0.43$	$0.32$	$0.22$	$0.18$

Table 1:

\gamma

for various values of

(W,T)

and an estimated

\mu=5.28

for Qwen2.5-7B.

Panel of Figure 3 — Figure 3: Accuracy and Pass@ $k$ with respect to Buffer Size. Left : Test accuracy as a function of compute spent when training Qwen3-0.6B on OpenR1-Math-220k for $(W,T)=(6,2)$ and various buffer sizes $N\in\{64,128,256,512,768,1536,2304,6912,20736\}$ , as well as for a no-buffer baseline. We report the median and IQR over more than $4$ seeds. Compute is normalized so that each weight update costs $1$ unit for buffer configurations and $1.96$ for the baseline. Middle : Pass@ $k$ increase after training for a representative subset of these buffer configurations, relative to the baseline. Right : Best Accuracy achieved over entire runs for various buffer sizes and $W/T$ ratios. The compute needed to reach those accuracies is reported in Figure 14 in Appendix E .

Figure 4: Pareto Frontier across Hyperparameters Sweep. Test accuracy as a function of compute spent when training Qwen3-0.6B on OpenR1-Math-220k for various learning rates (

\{1.5^{i}\cdot 10^{-7}\}_{i=0}^{5}

) and buffer configurations: no buffer (blue curves), buffer of size

\{64,128,256,512,768,2304,6912,20736\}

with

(W,T)\in\{(6,2),(5,3),(4,4)\}

(orange curves). Each curve is the median over at least

4

seeds. The two boldfaced curves delineate the Pareto frontier of each family of runs. Compute is normalized so that each weight update costs

1

unit for baseline configurations and and

0.51

for buffer configurations.

Figure 5: Alternative Loss, Positive-Bias Sampling Rule. Test accuracy as a function of training steps when training Qwen3-0.6B on OpenR1-Math-220k with a buffer of size

N=4608

and

(W,T)=(6,2)

. We use either GRPO or AsymRE, and apply positive-bias sampling with coefficient

\delta\in\{0,0.2,0.5\}

(note:

\delta=0

corresponds to standard uniform sampling).

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Optimal Staleness and Replay Ratio as a function of Rollout Cost ( $\mu$ ) . As $\mu$ increase, we see that it is better to increase the staleness horizon $x_{*}=N/R$ , and the replay ratio ( $y_{*}=B/R)$ . This also the case when the variance $\alpha$ or the correlation $\rho$ decreases.

Figure 7: Function

x\mapsto{\mathcal{K}}(x)

(which is defined in Eq. ( 5 ) and corresponds to

{\mathcal{J}}

for the specific choice

\sigma(x)=(x/\tau)^{\alpha}

) as a function of the staleness horizon

x=N/R

, for different values of

\alpha\in[0,1/2]

, and the corresponding optimal values of

x_{*}

Model	Median $\mu$	IQR	# Independent runs
Qwen3-0.6B	$6.84$	$[6.45,7.07]$	$242$
Qwen2.5-7B	$5.28$	$[5.12,5.48]$	$129$

Appendix

Table 2: Estimates of

\mu

for various models, computed as the median over several independent runs. We also provide the interquartile range over those runs.

(W, T)	(7,1)	(6,2)	(5,3)	(4,4)	(2,6)	(1,7)
$\gamma$	$1.02$	$0.41$	$0.34$	$0.26$	$0.17$	$0.15$

Appendix

Table 3:

\gamma

for various values of

(W,T)

and an estimated

\mu=6.84

for Qwen3-0.6B.

Panel of Figure 8 — Figure 8: Learning Rate Ablations for Qwen3-0.6B. Test accuracy as a function of the number of steps when training Qwen3-0.6B on OpenR1-Math-220k with various learning rates LR with at least $4$ seeds per configuration. We show the median and IQR over the seeds on the left, and all seeds separately on the right.

Panel of Figure 9 — Figure 9: Learning Rate Ablations for Qwen2.5-7B. Test accuracy on MATH as a function of the number of steps when training Qwen2.5-7B on OpenR1-Math-220k with various learning rates LR with at least $4$ seeds per configuration. We show the median and IQR over the seeds on the left, and all seeds separately on the right. Note the frequent crashes when $\text{LR}>6.8\cdot 10^{-8}$ .

Figure 10: Wall-time efficiency for Qwen3-0.6B Test accuracy as a function of wall-time when training Qwen3-0.6B on OpenR1-Math-220k for the no-buffer baseline (orange curve) and various buffer configurations. We report the median and the IQR over at least

4

seeds per curve.

Figure 11: Wall-time efficiency for Qwen2.5-7B Test accuracy on MATH as a function of wall-time when training Qwen2.5-7B on OpenR1-Math-220k for the no-buffer baseline (orange curve) and various buffer configurations. We report the median and the IQR over at least

4

seeds per curve.

Figure 12: Impact of off-policiness. We train Qwen3-0.6B on OpenR1-Math-220k without a buffer and we artificially introduce various levels of off-policiness by reducing the frequency at which the model’s weights used by the inference workers to generate rollouts are updated. We label each curve with the median level of off-policiness over all rollouts used, and plot the median test accuracy and its IQR as a function of the number of training steps over at least

4

seeds per curve.

Panel of Figure 13 — Figure 13: Test, Train and Entropy Dynamics. We train Qwen3-0.6B on OpenR1-Math-220k with a buffer for $(W,T)\in\{(6,2),(5,3),(4,4)\}$ and various buffer sizes. We report the test accuracy (top), the training accuracy (middle, smoothed using a sliding window), and the training entropy (bottom) as a function of the number of training steps. Note that the training entropy is computed over the batches used by the trainers to compute gradient updates; as using a buffer implies reusing samples generated by outdated policies, it is expected that the reported entropy would be much higher. We also report two baseline curves, corresponding to non-buffer configurations: one is plotted with respect to the number of steps, while the other is rescaled to be at compute-parity with the buffer configurations (i.e. so that an x-axis unit represents the same amount of compute). Each curve is the median (along with its IQR) over at least $4$ seeds.

Figure 14: Accuracy and Speed with respect to Design Choices. We train Qwen3-0.6B on OpenR1-Math-220k for $(W,T)\in\{(6,2),(5,3),(4,4)$ and various buffer sizes. We report on the right the median best test accuracy achieved over each training run (in other words, the median over the seeds of the best accuracy achieved for each seed), and on the left the median amount of compute that was needed to first reach $98\%$ of that score.

Figure 15: Additional results for Qwen2.5-7B. Accuracy on MATH as a function of compute spent when training Qwen2.5-7B on OpenR1-Math-220k for the no-buffer baseline (orange curve) and a buffer of size

N\in\{84,512,2268,20412\}

with

(W,T)

equal to

(6,2)

(left) or

(5,3)

(right). We report the median and IQR over more than

4

seeds. Compute is calibrated so that a single weight update for the baseline costs

1

unit.

Figure 16: Accuracy with respect to Buffer Size for Qwen3-8B on Lean coding tasks. Test accuracy as a function of compute spent when training Qwen3-8B on miniF2F for

(W,T)=(6,2)

and various buffer sizes

N\in\{128,10000\}

, as well as for a no-buffer baseline. We report the median and IQR over

4

seeds. Compute is normalized so that each weight update costs

0.55

unit for buffer configurations and

1

for the baseline.

Figure 17: Accuracy with respect to Buffer Size for Llama 3.2 3B. Test accuracy as a function of compute spent when training Llama 3.2 3B on OpenR1-Math-220k for

(W,T)=(6,2)

and various buffer sizes

N\in\{128,2048,16384\}

, as well as for a no-buffer baseline. We report the median and IQR over

4

seeds. Compute is normalized so that each weight update costs

0.58

unit for buffer configurations and

1

for the baseline.

Figure 18: We compared our standard buffer implementation, in which the buffer is sampled uniformly by the trainer GPUs (”vanilla”), with two variants: one in which the sampling is done uniformly without replacement (”No replacement”), and one in which samples that have never been used are sampled in priority, after what the remainder of the batch is filled without replacement (”No replacement, at least once”). We did not find any strong signal, as exemplified by these two representative buffer configurations (with which we trained Qwen3-0.6B on OpenR1-Math-220k).

Experience Replay

Experience replay stores and reuses past examples or trajectories to improve learning efficiency, stability, and retention across tasks.

See all 45 papers

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
Apr 18, 2026Weiyu Ma, Yongcheng Zeng, Yan Song +4Reinforcement Learning RL Post-Training
King Abdullah University of Science and Technology (KAUST) · Chinese Academy of Sciences, Institute of Automation (CASIA) · AI Centre, Department of Computer Science, University College London +1
VLM-Guided Experience Replay
Feb 2, 2026Elad Sharony, Tom Jurgenson, Orr Krupnik +2Reinforcement Learning Prioritized Experience Replay
Technion · ForSight Robotics · Nvidia Research
Rollout-Level Advantage-Prioritized Experience Replay for GRPO
Jun 3, 2026Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang +2Sample-Efficient RL Group Relative Policy Optimization
Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University · AIIS, ASRI, INMC, and ISRC, Seoul National University

Apr 18, 2026J/K move · Enter open · S save

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning

Weiyu Ma, Yongcheng Zeng, Yan Song, Xinyu Cui+3

King Abdullah University of Science and Technology (KAUST) · Chinese Academy of Sciences, Institute of Automation (CASIA) · AI Centre, Department of Computer Science, University College London+1

Reinforcement Learning RL Post-Training Prioritized Experience Replay Policy Learning Sample-Efficient RL RL for VLMs+1

Open paper AskPDF arXiv

Reinforcement Learning (RL) has achieved impressive success in post-training Large Language Models (LLMs) and Vision-Language Models (VLMs), with on-policy algorithms such as PPO, GRPO, and REINFORCE++ serving as the dominant paradigm. However, these methods discard all collected trajectories after a single gradient update, resulting in poor sample efficiency, particularly wasteful for agentic tasks where multi-turn environment interactions are expensive. While Experience Replay drives sample efficiency in classic RL by allowing agents to reuse past trajectories and prioritize informative ones, directly applying Prioritized Experience Replay (PER) to LLMs fails. The rapid policy evolution of billion-parameter models renders stored priorities stale, causing old high-priority trajectories to dominate sampling long after they have become uninformative. We propose Freshness-Aware PER, which addresses this priority staleness problem by augmenting any PER-based priority with a multiplicative exponential age decay grounded in effective sample size analysis. To the best of our knowledge, Freshness-Aware PER is the first work to successfully apply PER to LLM/VLM reinforcement learning. We evaluate on eight multi-step agentic, reasoning, and math competition tasks with 0.5B, 3B, and 7B models. Freshness-Aware PER significantly outperforms on-policy baselines, achieving +46% on NQ Search, +367% on Sokoban, and +133% on VLM FrozenLake, while standard PER without age decay consistently degrades performance. Our code is publicly available at https://github.com/Vision-CAIR/Freshness-Aware-PER.

Abstract

Experience Replay

Explore similar work

Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning

VLM-Guided Experience Replay

Rollout-Level Advantage-Prioritized Experience Replay for GRPO