Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

Wen-Zhi Li; Sainyam Galhotra; Claire Cardie

cs.CLOct 16, 2026

Preserving Reasoning Diversity in Reinforcement Learning with Verifiable Rewards via Adaptive Replay

Authors: Wen-Zhi Li, Sainyam Galhotra, Claire Cardie

Organizations: Cornell University

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves the reasoning capabilities of large language models, but often reduces the diversity of their reasoning behaviors. We study this phenomenon from the perspective of finite-sample learning dynamics and show that limited on-policy rollouts can progressively suppress semantically distinct but equally rewarded reasoning modes. Once a mode becomes sufficiently under-sampled, standard on-policy RLVR rarely recovers it. Motivated by this observation, we propose TAR, a Targeted Adaptive Replay framework that preserves diverse correct reasoning behaviors already present in the base policy. TAR constructs a compact memory bank of diverse base-model solutions and selectively replays them through an adaptive mechanism that activates only when stored behaviors are being suppressed during RLVR. Across six mathematical reasoning benchmarks and two Qwen2.5 model scales, TAR consistently achieves a better balance between reasoning performance and diversity preservation than existing diversity-preserving post-training methods, while substantially reducing the forgetting of correct base-model solutions. Code is available at https://github.com/wenzhilics/tar.

Figures & tables

Figure 1: Negative log-likelihood distribution of ground truth external expert demonstrations and internal LLM self-generated demonstrations with Qwen2.5-0.5B-Instruct.

Panel of Figure 2 — Figure 2: Toy analysis of mode imbalance and recovery under finite-sample RLVR. (a) Mode imbalance between two equally rewarded modes for different rollout-group sizes. (b) Mode coverage over the 200 updates after the first crossing of $C_{k,t}<1$ , aligned at the crossing step. Shaded regions denote interquartile ranges.

	Model	GSM8K	MATH-500	AMC	College Math	Minerva	OlympiadBench	Avg.
Qwen-2.5-0.5B	Base	46.9 / 77.6	30.8 / 62.0	7.5 / 35.0	27.4 / 45.0	5.5 / 18.4	7.1 / 27.4	20.9 / 44.2
	GRPO	52.7 / 75.5	36.6 / 62.0	10.0 / 40.0	30.6 / 43.3	8.8 / 22.8	9.8 / 28.9	24.8 / 45.4
	On-Policy Diversity
	Unlikely	35.1 / 73.0	24.6 / 56.8	5.0 / 27.5	23.3 / 42.5	4.0 / 21.0	6.5 / 24.7	16.4 / 40.9
	R1-Zero-Div	5.4 / 22.1	7.6 / 25.8	5.0 / 25.0	7.2 / 17.1	3.7 / 10.3	3.1 / 13.0	5.3 / 18.9
	DQO	30.8 / 60.1	22.6 / 41.2	2.5 / 17.5	27.0 / 42.3	2.6 / 7.7	2.4 / 10.2	14.7 / 29.8

Table 1: Main results on Qwen-2.5-Instruct with 0.5B and 1.5B backbones. Each cell reports pass@1 / pass@8.

	Model	Div-Eq	EAD	Emb-Div
Qwen-2.5-0.5B	Base	0.5128	0.4802	0.0804
	GRPO	0.4418	0.3880	0.0574
	Unlikely	0.3967	0.3169	0.1327
	R1-Zero-Div	0.9871	0.5938	0.2018
	DQO	0.7215	0.6757	0.4566
	DPH-RL	0.4708	0.4236	0.0664

Table 2: Diversity results using pass@8 rollouts averaged over six benchmarks. Per-dataset results are provided in Appendix E .

Figure 3: Relationship between trajectory suppression and gated replay loss during TAR training. Each point represents one training step; the x-axis shows the fraction of suppressed replay trajectories ( $b(\tau)>0$ ), and the y-axis shows the gated replay loss.

Figure 4: Forgetting of base-model’s correct solutions after training. A solution is counted as forgotten if the final policy assigns the complete solution at least

10\times

lower sequence-level probability than the base model. We evaluate replay trajectories used during training (light blue), other correct base-model solutions to the same training questions (medium blue), and correct base-model solutions to unseen MATH-500 questions (dark blue).

Model	Reasoning	Diversity
TAR	41.2 / 61.1	0.4348 / 0.4263 / 0.0639
w/o token gate	40.4 / 59.0	0.4179 / 0.4004 / 0.0590
w/o rollout gate	38.5 / 58.7	0.4763 / 0.4755 / 0.0745
hard rollout gate	39.3 / 60.2	0.4714 / 0.4703 / 0.0731

Table 3: Ablation study on Qwen-2.5-1.5B-Instruct, averaged over six benchmarks. Reasoning reports pass@1 / pass@8; diversity reports Div-Eq / EAD / Emb-Div. Per-dataset results are provided in Appendix E .

Figure 5: Training dynamics for TAR and different ablated variants.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Shared across scales
Hyperparameter	0.5B	1.5B
Training epochs	5	3
GPUs	2 $\times$ A6000	2 $\times$ A100
Train batch size	256
PPO mini batch size	128
PPO micro batch size per GPU	8

Appendix

Table 4: Shared training configuration.

Method	Method-specific settings
Unlikely	Likelihood-rank penalty $\beta=0.25$
R1-Zero-Div	Entropy regularization weight $\omega=0.01$ ; applied only to responses with $\hat{A}>0$
DQO	Diversity reward weight $\alpha=1.0$ ; encoder: all-MiniLM-L6-v2 Reimers and Gurevych (2019) ;
DPH-RL	JS-divergence weight $0.0125$
LUFFY	Policy-shaping parameter $\gamma=0.1$
CalibRL	Expert-guidance coefficient $c=0.1$ ; LeakyReLU negative slope $0.5$

Appendix

Table 5: Method-specific hyperparameters.

	0.5B	1.5B
Problems covered	5,978	6,554
(of 7,500)	79.7%	87.4%
Problems with 3 / 2 / 1 traj.	5363/226/389	6186/139/229
Stored trajectories	16,930	19,065

Appendix

Table 6: Memory bank statistics (

N=128

samples per problem, FPS with

k=3

Figure 6: Pairwise distance distributions of FPS-selected and randomly selected correct trajectories. Gray shows the mean pairwise cosine distance of three randomly selected correct trajectories, while blue and orange show the closest and farthest pair among

k=3

FPS-selected trajectories. Dashed lines indicate medians.

Figure 7: Training dynamics comparison in terms of generation entropy.

	Model	GSM8K	MATH-500	AMC	College Math	Minerva	OlympiadBench
Qwen-2.5-0.5B	Base	0.6578 / 0.4214 / 0.0424	0.4513 / 0.4600 / 0.0756	0.4891 / 0.4949 / 0.0969	0.4451 / 0.4679 / 0.0665	0.5791 / 0.5456 / 0.0989	0.4543 / 0.4914 / 0.1018
	GRPO	0.4636 / 0.2823 / 0.0230	0.3780 / 0.3573 / 0.0523	0.4789 / 0.4361 / 0.0712	0.3806 / 0.3570 / 0.0470	0.5280 / 0.4683 / 0.0755	0.4219 / 0.4271 / 0.0755
	Unlikely	0.4117 / 0.2421 / 0.1109	0.3699 / 0.2900 / 0.1272	0.4660 / 0.3397 / 0.1441	0.3397 / 0.3112 / 0.1238	0.3940 / 0.3760 / 0.1447	0.3988 / 0.3426 / 0.1456
	R1-Zero-Div	0.9913 / 0.5851 / 0.2146	0.9877 / 0.5202 / 0.1873	0.9754 / 0.4931 / 0.1874	0.9895 / 0.7051 / 0.1943	0.9838 / 0.7298 / 0.2292	0.9948 / 0.5294 / 0.1982
	DQO	0.6989 / 0.5464 / 0.3579	0.6757 / 0.6541 / 0.4435	0.8300 / 0.7982 / 0.5568	0.5126 / 0.5041 / 0.2224	0.7860 / 0.7470 / 0.5532	0.8257 / 0.8043 / 0.6055
	DPH-RL	0.5402 / 0.3303 / 0.0294	0.4081 / 0.3993 / 0.0611	0.4794 / 0.4552 / 0.0817	0.4045 / 0.3966 / 0.0544	0.5481 / 0.5036 / 0.0848	0.4445 / 0.4566 / 0.0869

Appendix

Table 7: Diversity results on Qwen-2.5-Instruct using pass@8 rollouts. Each cell reports Div-Eq / EAD-Distinct-N / Emb-Div.

Model	GSM8K	MATH-500	AMC	College Math	Minerva	OlympiadBench
TAR	76.1 / 91.7	59.0 / 79.0	30.0 / 67.5	39.9 / 50.7	19.5 / 36.0	22.4 / 41.6
w/o token gate	74.0 / 91.8	58.4 / 77.4	30.0 / 60.0	40.0 / 50.4	19.5 / 34.2	20.7 / 40.4
w/o rollout gate	72.7 / 92.7	53.0 / 77.0	30.0 / 57.5	37.3 / 51.7	18.0 / 33.1	19.7 / 40.3
hard rollout gate	72.5 / 92.7	53.6 / 77.0	35.0 / 67.5	38.6 / 51.1	17.6 / 33.5	18.7 / 39.1

Appendix

Table 8: Ablation study on Qwen-2.5-1.5B-Instruct. Each cell reports pass@1 / pass@8.

Model	GSM8K	MATH-500	AMC	College Math	Minerva	OlympiadBench
TAR	0.4986 / 0.3621 / 0.0317	0.3726 / 0.3988 / 0.0576	0.4399 / 0.4518 / 0.0831	0.3621 / 0.3957 / 0.0483	0.5293 / 0.5018 / 0.0834	0.4065 / 0.4478 / 0.0793
w/o token gate	0.4554 / 0.3162 / 0.0250	0.3504 / 0.3691 / 0.0525	0.4321 / 0.4467 / 0.0752	0.3386 / 0.3530 / 0.0430	0.5260 / 0.4827 / 0.0825	0.4046 / 0.4349 / 0.0755
w/o rollout gate	0.5805 / 0.4337 / 0.0436	0.4139 / 0.4609 / 0.0709	0.4819 / 0.4901 / 0.0900	0.4021 / 0.4576 / 0.0591	0.5503 / 0.5317 / 0.0928	0.4293 / 0.4787 / 0.0904
hard rollout gate	0.5729 / 0.4250 / 0.0423	0.4113 / 0.4527 / 0.0699	0.4731 / 0.4862 / 0.0876	0.3970 / 0.4483 / 0.0576	0.5488 / 0.5332 / 0.0924	0.4251 / 0.4765 / 0.0889

Appendix

Table 9: Ablation study of diversity on Qwen-2.5-1.5B-Instruct using pass@8 rollouts. Each cell reports Div-Eq / EAD / Emb-Div.

	Model	GSM8K	MATH-500	AMC	College Math	Minerva	OlympiadBench
Qwen-2.5-0.5B	Base	0.5915 / 0.4463 / 0.0320	0.4208 / 0.4847 / 0.0488	0.5773 / 0.5681 / 0.0571	0.3877 / 0.4784 / 0.0444	0.5569 / 0.6017 / 0.0804	0.4758 / 0.5470 / 0.0745
	GRPO	0.3818 / 0.2913 / 0.0159	0.3235 / 0.3567 / 0.0328	0.5216 / 0.4712 / 0.0419	0.3234 / 0.3516 / 0.0283	0.4934 / 0.5008 / 0.0545	0.4396 / 0.4724 / 0.0450
	Unlikely	0.4116 / 0.2439 / 0.1003	0.3490 / 0.2644 / 0.1064	0.5451 / 0.3641 / 0.1333	0.3016 / 0.2753 / 0.1093	0.3548 / 0.3218 / 0.1362	0.4226 / 0.3322 / 0.1291
	R1-Zero-Div	0.9862 / 0.3991 / 0.1253	0.9934 / 0.3268 / 0.0891	1.0000 / 0.2523 / 0.0744	0.9968 / 0.4935 / 0.0840	0.8562 / 0.4910 / 0.0936	0.9891 / 0.2892 / 0.0853
	DQO	0.5213 / 0.4181 / 0.0332	0.3855 / 0.4319 / 0.0487	0.6512 / 0.5253 / 0.0431	0.3714 / 0.4211 / 0.0397	0.4534 / 0.4830 / 0.0528	0.5509 / 0.6006 / 0.1010
	DPH-RL	0.4594 / 0.3456 / 0.0217	0.3585 / 0.4070 / 0.0368	0.5611 / 0.5498 / 0.0511	0.3397 / 0.3931 / 0.0341	0.5296 / 0.5381 / 0.0567	0.4606 / 0.5101 / 0.0608

Appendix

Table 10: Diversity of correct rollouts on Qwen-2.5-Instruct using pass@8 sampling. Only problems with at least two correct responses are included. Each cell reports Div-Eq / EAD / Emb-Div.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,661 papers

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
May 1, 2026Anamika Lochab, Bolian Li, Ruqi ZhangRL for Language Model Reasoning Group Relative Policy Optimization
Department of Computer Science Purdue University, West Lafayette
Diversifying RLVR Rollouts via First-Token Exploration
May 27, 2026Soeun Kim, Albert NoRL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Artificial Intelligence, Yonsei University
Measuring and Mitigating Solution Mode Collapse in RLVR
Oct 8, 2026Liv G. d'Aliberti, Marwa Abdulhai, Sofiia Druchyna +2Mode Collapse RL Post-Training
Princeton University

May 1, 2026J/K move · Enter open · S save

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity

Anamika Lochab, Bolian Li, Ruqi Zhang

Department of Computer Science Purdue University, West Lafayette

RL for Language Model Reasoning Group Relative Policy Optimization Reinforcement Learning with Verifiable Rewards Behavioral Diversity in RL

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has achieved substantial gains in single-attempt accuracy (Pass@1) on reasoning tasks, yet often suffers from reduced multi-sample coverage (Pass@K), indicating diversity collapse. We identify a structural cause for this degradation: common RLVR objectives, such as GRPO, are indifferent to how probability mass is distributed among correct solutions. Combined with stochastic training dynamics, this indifference induces a self-reinforcing collapse, in which probability mass concentrates on a narrow subset of correct outputs while alternative valid solutions are suppressed. We formalize this collapse mechanism and further characterize the optimal policy structure under two complementary criteria: robustness and entropy-regularized optimality, which identify the Uniform-Correct Policy as uniquely optimal. Motivated by this analysis, we propose Uniform-Correct Policy Optimization (UCPO), a modification to GRPO that adds a conditional uniformity penalty on the policy's distribution over correct solutions. The penalty redistributes gradient signal toward underrepresented correct responses, encouraging uniform allocation of probability mass within the correct set. Across three models (1.5B-7B parameters) and five mathematical reasoning benchmarks, UCPO improves Pass@K and diversity while maintaining competitive Pass@1, achieving up to +10% absolute improvement on AIME24 at Pass@64 and up to 45% higher equation-level diversity within the correct set. The code is available at https://github.com/AnamikaLochab/UCPO.

Abstract

Reinforcement Learning

Explore similar work

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity

Diversifying RLVR Rollouts via First-Token Exploration

Measuring and Mitigating Solution Mode Collapse in RLVR