Organizations: Griffith University, Australia · Fudan University, China · Shanghai Innovation Institute, China · Beijing University of Post and Telecommunications · INFLY TECH, China · Shanghai Academy of Artificial Intelligence for Science, China
While Reinforcement Learning (RL) enhances Large Language Model reasoning, on-policy algorithms like GRPO are sample-inefficient as they discard past rollouts. Existing experience replay methods address this by reusing accurate samples for direct policy updates, but this can incur additional storage costs and concentrate the policy on a narrow set of solutions. We argue that historical data should prioritize sustaining local policy plasticity rather than simply reinforcing accuracy. We define local policy plasticity as a persistent ability to remain backtrackable and switchable with respect to recent successful trajectories: throughout training, the policy maintains non-negligible support on recently reachable alternatives, so it can smoothly reallocate probability mass when needed without being locked into a single dominant path. To this end, we propose Dynamic Jensen-Shannon Replay (DyJR), a simple yet effective regularization framework using a dynamic reference distribution from recent trajectories. DyJR introduces two innovations: (1) A Time-Sensitive Dynamic Buffer that uses FIFO and adaptive sizing to retain only temporally proximal samples, synchronizing with model evolution; and (2) Jensen-Shannon Divergence Regularization, which replaces direct replay policy-gradient updates with a regularization signal that discourages premature over-commitment to a single Rank-1 trajectory. Experiments on mathematical reasoning, Text-to-SQL, and coding benchmarks demonstrate that DyJR improves over GRPO as well as baselines such as RLEP and Ex-GRPO, while maintaining training efficiency comparable to the original GRPO. Furthermore, Rank-k token probability evolution and semantic-mode analyses show reduced concentration and greater retention of alternative continuations. Matched admission-objective comparisons and repeated runs provide additional evidence for the roles of DyJR's components.
Figures & tables
Figure 1: Conceptual illustration of local replay. DyJR retains recent successful branches as regularization targets when current sampling becomes concentrated. The tree illustrates accessible alternatives; the algorithm does not perform explicit tree search or a backtracking procedure.
Method
AIME25
AMC23
Beyond AIME
BRUMO25
HMMT25
Minerva
Avg
Qwen3-4B-Base
Base model
8.2
51.5
4.3
16.6
1.9
29.0
18.6
GRPO
19.1
64.6
10.7
29.6
9.8
45.2
29.8
DAPO
21.5
65.5
10.6
30.4
9.2
47.8
30.8
DPH-RL
22.8
65.2
10.8
30.1
10.5
48.1
31.3
RLEP
22.6
70.9
10.9
28.7
10.7
47.3
31.8
Table 1: Mathematical reasoning accuracy (%). Minerva uses Mean@16; the other benchmarks use Mean@256. The 4B GRPO, DyJR, RLEP, and Ex-GRPO rows show three-run means; other rows are single runs. Run-level variability is reported in Appendix H.1 . Bold marks the best main-method result within each model size.
Admission rule
PG+IS
JS regularization
Perfect-only
30.1
30.4
Confidence-based
32.6
34.1
Table 2: Matched admission–objective comparison at M=8 . Values are six-benchmark averages (%). The training budget, replay batch size, schedule, and freshness constraint are fixed. PG+IS denotes off-policy policy gradients with importance ratios.
Setting
AIME25
AMC23
Beyond AIME
BRUMO25
HMMT25
Minerva
Avg
DyJR (default)
23.1
72.0
12.7
33.4
12.7
50.4
34.1
Divergence and reference construction
Forward KL
22.8
70.7
11.3
32.6
10.5
47.3
32.5
Static Ref + JS
22.7
66.3
11.2
29.9
10.7
49.4
31.7
Reverse-KL (recent)
18.7
64.3
10.1
28.8
10.1
44.3
29.4
Regularization coefficient and recency window
Table 3: Ablations of the replay reference, divergence, regularization coefficient, and recency window. Values are mathematical accuracy (%); other settings follow DyJR. The default is αJS=0.05 and M=8 . Controls are single runs; the DyJR row shows its three-run mean.
BIRD
Spider
Coding (Pass@16)
Method
Pass@1
Pass@16
Pass@1
Pass@16
LiveCodeBench V2
CodeContests
GRPO
59.4
68.4
72.5
80.3
30.9
25.5
DAPO
60.1
69.2
71.5
79.4
30.7
26.4
DPH-RL
62.8
72.4
76.0
84.1
31.2
26.7
DyJR
62.7
72.9
77.5
87.3
31.8
27.2
Table 4: Cross-domain results (%). SQL columns report execution-based Pass@1 and Pass@16. Coding columns report Pass@16 under unit-test verification. These are single-run comparisons. Bold marks the best value in each column.
Figure 2: Qwen3-4B-Base training dynamics: top- k -normalized entropy, average ranked-token probabilities, and evaluation accuracy. Token concentration is an indirect diagnostic of the local-plasticity interpretation, rather than a measurement of semantic modes.
Figure 3: Rank- k probability trends for RLEP, RLEP with dynamic reference construction, and complete DyJR.
Model
1 mode
2 modes
3 modes
≥4 modes
≥2 modes
Mean modes
Qwen3-4B-Base
22%
24%
34%
20%
78%
2.6
GRPO
88%
4%
6%
2%
12%
1.3
DyJR
42%
18%
24%
16%
58%
2.2
Table 5: Mathematical strategy diversity on the same 50 problems, with 16 complete derivations per problem and model. Columns 2–6 give the percentage of problems. Modes include coherent incorrect strategies; the final column is the reported mean number of modes per problem.
Figure 4: (a) Pass@ k performance on the Beyond AIME dataset across varying sampling budgets k . (b) and (c) Computational latency comparison between GRPO (w/o KL) and DyJR during the Rollout phase and the Actor Update phase (encompassing log-probability calculation, forward passes, and gradient updates), respectively.
Storage configuration
Q=10K
Q=30K
Q=50K
Q=100K
RLEP: 64 trajectories/query
19.91
59.31
99.03
195.86
DyJR: 1,638 trajectories total
0.41
0.41
0.41
0.41
Table 6: Measured resident-memory increase (GiB), summed over four H100 GPUs, in an isolated token-ID storage benchmark. Each trajectory has 8,192 int32 token IDs. RLEP is tested at maximum per-query capacity; DyJR uses a fixed total capacity. Values include allocation/runtime overhead and exclude other replay fields and training state.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Parameters
GRPO
DAPO
DPH-RL
DyJR
RLEP
Inference
G (Group Size)
8
8
8
8
8
Prompt Batch Size
512
512
512
512
512
Temperature
1.0
1.0
1.0
1.0
1.0
Top- p / Top- k
1.0 / -1
1.0 / -1
1.0 / -1
1.0 / -1
1.0 / -1
Evaluation Temp
0.7
0.7
0.7
0.7
0.7
Evaluation Top- p
0.95
0.95
0.95
0.95
0.95
Appendix
Table 7: Experimental settings adopted in this work. We employ the AdamW optimizer with β1=0.9,β2=0.95 and a weight decay of 0.1 . The gradient clipping threshold is set to 1.0 for all models.
Parameter
Value
Description
Replay ratio ( ρ )
0.5
Fraction of replay samples in each mini-batch
Delayed replay threshold
0.35
Minimum batch Pass@1 required to activate experience replay
Difficulty bucket range
[0,7]
Bounds for difficulty-based bucketing ( k/K )
Replay metric
Entropy
Criterion for evaluating trajectory quality
Trajectory selection mode
Argmin
Select the lowest-entropy successful trajectory per prompt
Appendix
Table 8: ExGRPO-specific hyperparameters for experience replay.
Model
AIME25
AMC23
Beyond AIME
BRUMO25
HMMT25
Minerva *
Avg
DyJR ( init=20%,η=20% )
22.9
72.5
12.5
33.8
12.6
51.0
34.2
DyJR ( init=20%,η=5% )
23.1
72.0
12.7
33.4
12.7
50.4
34.1
DyJR ( init=10%,η=5% )
22.5
69.2
11.4
31.5
11.5
48.2
32.4
DyJR ( init=5%,η=5% )
21.8
68.5
10.1
29.2
10.3
47.8
31.3
Appendix
Table 9: Performance of DyJR on various mathematical benchmarks. init denotes the percentage of rollout samples retained per step during the initial 20 steps, while η represents the retention rate thereafter. The maximum age of the FIFO buffer is consistently set to 8 steps.
Replay Selection
Max Age M
Avg
Perfect-only
8
30.4
Perfect-only
32
31.3
Confidence-based
8
34.1
Confidence-based
32
31.7
Appendix
Table 10: Replay selection and staleness. Confidence-based selection reduces buffer starvation, while overly stale replay remains harmful.
Method
Avg
GRPO (1.7B, sparse)
17.3
DyJR (1.7B, sparse)
19.5
Appendix
Table 11: Sparse-reward setting with Qwen3-1.7B-Base.
Method
Avg
KL-to-ref (static reference)
30.8
Dynamic entropy bonus
30.3
Replay + off-policy PG w/ IS ratio
32.6
DyJR
34.1
Appendix
Table 12: Comparison with alternative regularization and off-policy correction strategies under the same compute budget.
Benchmark
GRPO
RLEP
Ex-GRPO
DyJR
AIME25
19.1±1.8
22.6±0.5
22.2±0.4
23.1±1.6
AMC23
64.6±1.5
70.9±1.7
69.9±0.7
72.0±1.1
Beyond AIME
10.7±0.9
10.9±0.3
11.4±1.7
12.7±0.7
BRUMO25
29.6±1.6
28.7±0.8
33.0±0.2
33.4±1.4
HMMT25
9.8±0.8
10.7±1.6
10.0±1.3
12.7±0.8
Minerva
45.2±1.4
47.3±1.5
49.2±0.9
50.4±2.6
Appendix
Table 13: Variability across three independent training runs. Entries are reported means ± sample standard deviations, in percentage points. Minerva uses Mean@16; the other benchmarks use Mean@256. Avg is the six-benchmark average. The aggregate standard deviation for GRPO was not reported, so only its mean is shown.
Method
Run
AIME25
AMC23
Beyond AIME
BRUMO25
HMMT25
Minerva*
Avg
Ex-GRPO
1
21.7
70.2
11.6
32.8
10.4
50.1
32.8
2
22.5
70.3
12.9
33.0
8.5
48.4
32.6
3
22.4
69.1
9.6
33.2
11.1
49.0
32.4
RLEP
1
23.0
68.9
10.6
28.9
10.8
48.4
31.8
2
22.7
72.2
11.1
29.3
9.1
47.8
32.0
3
22.0
71.5
11.1
27.8
12.3
45.6
31.7
Appendix
Table 14: Individual runs of replay baselines. Values are percentages; the final column is the average over six benchmarks. Run 1 is the originally reported run. Reported rounded values are preserved; their three-run summaries appear in Table 13 .
Admission
Replay utilization
Avg
Perfect-only
JS regularization
30.4
Confidence-based
JS regularization (DyJR)
34.1
Perfect-only
Off-policy PG+IS
30.1
Confidence-based
Off-policy PG+IS
32.6
Appendix
Table 15: Matched admission–utilization comparison. All settings use M=8 and the same replay-buffer configuration and training budget. Avg is the six-benchmark average accuracy (%).
Method
Avg
GRPO
29.8
GRPO + standard entropy regularization
29.7
DyJR
34.1
Appendix
Table 16: Standard token-level entropy regularization. Avg is the six-benchmark average accuracy (%).
Protocol family
Identifying response prefix
Divide-and-conquer
“Divide and Conquer …”
Query-plan-first
“Query Plan …”
Direct step-by-step
“Let’s break down and think step by step …”
DeepSeek-R1-distilled
“Okay, to answer the question …”
GPT-4o-distilled
“The original question can …”
Appendix
Table 17: Five protocol families used in the controlled NL2SQL study. Distinct prefixes provide deterministic family labels.
Model
Observed protocol counts
SFT base
Approximately 80% of questions exhibit at least five protocols.
GRPO
95% of questions exhibit one protocol.
DyJR
32% exhibit two; 30% three; 27% four; 89% exhibit at least two.
Appendix
Table 18: Observed protocol diversity in NL2SQL with 32 samples per question. Percentages describe questions, not individual responses.
Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capabilities of large language models, but often reduces the diversity of their reasoning behaviors. We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively suppress semantically distinct but equally rewarded reasoning modes. Once a mode becomes sufficiently under-sampled, standard on-policy RLVR rarely recovers it. Motivated by this observation, we propose TAR, a Targeted Adaptive Replay framework that preserves diverse correct reasoning behaviors already present in the base policy. TAR constructs a compact memory bank of diverse base-model solutions and selectively replays them through an adaptive mechanism that activates only when stored behaviors are being suppressed during RLVR. Across six mathematical reasoning benchmarks and two Qwen2.5 model scales, TAR consistently achieves a better balance between reasoning performance and diversity preservation than existing diversity-preserving post-training methods, while substantially reducing the forgetting of correct base-model solutions. Code is available at https://github.com/wenzhilics/tar.
Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs. It remains sample inefficient. Each rollout is used for a single gradient update and then discarded. Naive replay is not well suited in this setting because LLM policies drift quickly per gradient step. Stored rollouts therefore become stale and can destabilize training. We propose a rollout-level replay buffer for GRPO that stores and samples individual rollouts rather than whole groups. The buffer bounds staleness through age eviction. Any rollout older than tau_max training steps is removed. The buffer also preserves on-policy data via fresh-anchored composition. Each batch keeps its fresh on-policy rollouts and then concatenates replay rollouts drawn separately from the buffer. We prioritize replay by per-rollout advantage magnitude and recycle individual rollouts whose advantages are large. Across three Qwen3-Base scales on five math benchmarks, our method outperforms GRPO and naive replay baselines. Gains are positive at every scale and reach +1.66 pp on the five-benchmark average at 4B. Under an AES metric that jointly measures accuracy and token efficiency, our method is the only condition with a positive margin over GRPO at every scale.
Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang +2
Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in AI, Seoul National University · AIIS, ASRI, INMC, and ISRC, Seoul National University
Reinforcement Learning with Verifiable Rewards (RLVR) has achieved substantial gains in single-attempt accuracy (Pass@1) on reasoning tasks, yet often suffers from reduced multi-sample coverage (Pass@K), indicating diversity collapse. We identify a structural cause for this degradation: common RLVR objectives, such as GRPO, are indifferent to how probability mass is distributed among correct solutions. Combined with stochastic training dynamics, this indifference induces a self-reinforcing collapse, in which probability mass concentrates on a narrow subset of correct outputs while alternative valid solutions are suppressed. We formalize this collapse mechanism and further characterize the optimal policy structure under two complementary criteria: robustness and entropy-regularized optimality, which identify the Uniform-Correct Policy as uniquely optimal. Motivated by this analysis, we propose Uniform-Correct Policy Optimization (UCPO), a modification to GRPO that adds a conditional uniformity penalty on the policy's distribution over correct solutions. The penalty redistributes gradient signal toward underrepresented correct responses, encouraging uniform allocation of probability mass within the correct set. Across three models (1.5B-7B parameters) and five mathematical reasoning benchmarks, UCPO improves Pass@K and diversity while maintaining competitive Pass@1, achieving up to +10% absolute improvement on AIME24 at Pass@64 and up to 45% higher equation-level diversity within the correct set. The code is available at https://github.com/AnamikaLochab/UCPO.
Anamika Lochab, Bolian Li, Ruqi Zhang
Department of Computer Science Purdue University, West Lafayette