Measuring and Mitigating Solution Mode Collapse in RLVR

Liv G. d'Aliberti; Marwa Abdulhai; Sofiia Druchyna; Peter Henderson; Manoel Horta Ribeiro

cs.LGOct 8, 2026

Measuring and Mitigating Solution Mode Collapse in RLVR

Authors: Liv G. d'Aliberti, Marwa Abdulhai, Sofiia Druchyna, Peter Henderson, Manoel Horta Ribeiro

Organizations: Princeton University

Abstract

A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.

Figures & tables

Figure 1: Motivation. Consider the Graph coloring problem shown in A (left), with six valid solutions. We evaluate Qwen2.5-3B across training, tracking the number and diversity of correct solutions. At each training pass, we show 32 sampled questions, eight from each of four matched runs. Training with Dr. GRPO ( B ) improves accuracy and leads to a single solution dominating the output. In contrast, Re:Max ( C ) increases both accuracy and diversity of correct solutions.

Figure 2: ModeBench comprises questions with many acceptable solutions across domains. (1) and (2) identify different solution modes across the five domains, which include Graph coloring, a normalized computation tree, executable Python code, algebraic problems and constraint optimization.

Figure 3: Higher performance is not associated with increased solution mode diversity. We show the relationship between a performance-related metric ( pass@ 8; $x$ -axis) and our proposed solution mode diversity metric ( PCMD ; $y$ -axis) for GPT-5.6 Sol (in black), and Qwen2.5-Instruct models of various sizes (where parameter count is indicated by color). Accuracy and model size do not reliably predict solution-mode diversity.

Figure 4: Correctness and solution-mode diversity vary across benchmark levels. Across increasingly difficult ModeBench levels, pass@8 generally declines, while PCMD follows different patterns across models.

Figure 5: Asking the model more times may not recover all certified or enumerated modes. We test directly whether a larger sampling budget recovers the known modes. GPT-5.6 Sol, with medium reasoning on, answers 32 problems per domain and level with 512 responses each. Curves are distinct verified modes in random $k$ -subsets as a share of the enumerated or certified modes available for those problems, and the shaded band is what no level ever produced. Shading around lines is the pointwise 95% CIs using a bootstrap over problems.

Figure 6: RLVR early “winner” (A) becomes the predominant way to solve. The base model supports several modes. RLVR-only training concentrates on one mode. Our replay approach keeps every buffered mode & leaves room for more.

Figure 7: Replay preserves successes that on-policy learning may stop seeing. MaxRL learns from the policy’s current samples, while replay stores one verified exemplar per discovered mode and revisits those exemplars with equal weight. Together, these two learning paths allow Re:Max to preserve previously discovered solution modes while continuing to explore for new ones.

Figure 8: Training reduces solution mode diversity. We compare PCMD before and after RLVR on Qwen2.5-3B, over ModeBench Level 1 test set, under three objectives, five domains, five seeds. The figure shows the change in PCMD from before to after. Marks are means over training seeds with 95% intervals. In 14 of 15 comparisons, we find fewer solution modes after training. See Fig. 20 for other model scales.

	$\Delta$ pass@8						$\Delta$ PCMD
	Graph	Count down	Python	MathIR	Pantry Plan	Avg	Graph	Count down	Python	MathIR	Pantry Plan	Avg
Re:Dr (ours)	$\mathbf{+.646}$	$\mathbf{+.191}$	$+.356$	$\mathbf{+.285}$	$+.207$	$+.337$	$\mathbf{+.559}$	$+.480$	$+.082$	$\mathbf{+.017}$	$+.334$	$+.294$
Re:Max (ours)	$+.621$	$+.185$	$\mathbf{+.509}$	$+.277$	$+.199$	$\mathbf{+.358}$	$+.525$	$\mathbf{+.504}$	$\mathbf{+.249}$	$+.005$	$+.317$	$\mathbf{+.320}$
GAPO	$+.175$	$+.011$	$+.009$	$-.054$	$\mathbf{+.279}$	$+.084$	$+.131$	$+.340$	$.000$	$.000$	$\mathbf{+.653}$	$+.225$
SetPO	$+.064$	$-.050$	$+.497$	$+.051$	$+.154$	$+.143$	$+.039$	$+.007$	$+.013$	$+.003$	$+.266$	$+.066$
MaxRL (no replay)	$+.213$	$+.101$	$.000$	$-.066$	$+.009$	$+.051$	$+.086$	$+.015$	$.000$	$.000$	$.000$	$+.020$

Table 1: Compared objectives under Qwen2.5-0.5B, ModeBench Level 1. All differences are relative to Dr.GRPO; avg weights the five domains equally within seed (5 seeds). Bold marks a column’s best. An em-dash means that there are few more-than-one solution pairs to evaluate, and a superscript gives the paired seed count where it falls below five. Red-to-green shading scales to each column’s largest magnitude.

Figure 9: Replay leads to higher accuracy and solution mode diversity. Diamonds are initial LMs; open/filled markers are training without/with replay. Averages all five domains within seed. (App. Fig. 14 provides per-task results).

Figure 10: Replay gains persist across difficulty levels. Bars show the gain from adding replay to Dr.GRPO (Re:Dr) and MaxRL (Re:Max) at each ModeBench level, averaged across domains for Qwen2.5-0.5B. Replay improves both PCMD (top) and pass@8 (bottom) at all three levels.

Figure 11: Replay and KL anchoring preserve different diversity. Means over the 3 domains with reportable initial PCMD , for Qwen2.5-0.5B. The KL anchor forms a curve because we sweep

\beta

-KL coefficients. Increasing

\beta

preserves more diversity, but at a lower pass@8 . Green indicates higher pass@8 than the anchor at the same diversity (App. E.7 ).

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Generated answer / check	Solution mode	Equivalent responses	Certified modes
Graph coloring	Complete assignment; check fixed colors and edges	Color vector	Spacing only	6.27 4–12
Countdown	Parse operands and execute exactly	Normalized AST	Commutative order	4.52 2–8
Python factors	Execute restricted lambda externally	Return vector	Programs with the same return vector	229.44 16–1,512
MathIR	Execute bounded action program	State trajectory	Programs with the same states	5.00 5
PantryPlan	Check ingredient allocation and all constraints	Ingredient support	Quantity and ordering aliases	18.19 8–40

Appendix

Table 2: Executable solution modes in the five ModeBench domains. The verifier determines both correctness and solution-mode identity. Certified modes gives the mean and range of the Level-1 evaluation catalogue; these catalogues need not enumerate every accepted solution.

Domain	Level 1	Level 2
Graph coloring	Three hidden vertices	Four hidden vertices
Countdown	Three operands	Four operands, with values up to 14
Python factors	Four executed cases, all at most 96	Four cases, including a value above 96
MathIR	One-sided linear families	Variable-on-both-sides families
PantryPlan	Base dietary and composition constraints	More active exclusions and tighter composition constraints

Appendix

Table 3: Level 2 increases task complexity while preserving solution-mode identity. The verifier and definition of a solution mode are unchanged from Level 1.

Figure 12: Correctness and conditional solution diversity vary separately across benchmark levels. Each point is an untrained model evaluated on one domain and level, with pass@8 on the horizontal axis and PCMD on the vertical axis. Filled markers indicate sufficient verified responses for the primary diversity estimate, while hollow markers indicate limited support.

Component	Setting
Data	384 training and 128 disjoint evaluation prompts per domain and level; evaluation prompts are never used for training or replay.
Fresh rollouts	$G=16$ , temperature $1$ , top- $p=1$ , with one prompt group per optimizer update.
Training horizon	Eight passes over the training set, totaling 3,072 optimizer updates, with one PPO epoch per batch.
Evaluation	Four groups of eight responses per prompt at temperature $1$ and top- $p=1$ at passes $0,.5,\ldots,8$ ; final comparisons use the last checkpoint.
Replay	At most $M=16$ solution modes per prompt, one stored exemplar per mode, deterministic round-robin buffer selection, and $\lambda_{\mathrm{replay}}=.10$ .
Training seeds	Five seeds per model–domain condition, with paired comparisons using common seeds whenever available.

Appendix

Table 4: Training and evaluation settings for the matched replay comparisons. Replay and control use the same initialization, fresh-rollout budget, optimizer updates, prompts, and evaluation procedure.

Figure 13: Replay gains persist across model scales. Points show terminal Re:Dr minus Dr.GRPO and Re:Max minus MaxRL effects on pass@8 and PCMD across three model scales and five domains. Filled markers use five paired seeds, hollow markers use fewer, and bars show nominal 95% paired intervals where available.

Model	Comparison	$\Delta$ pass@8	$\Delta$ PCMD
Qwen2.5-0.5B	Re:Dr $-$ Dr.GRPO	$+.337$	$+.294$
Falcon3-1B	Re:Dr $-$ Dr.GRPO	$+.442$	$+.271$
Qwen2.5-3B	Re:Dr $-$ Dr.GRPO	$+.279$	$+.134$
Qwen2.5-0.5B	Re:Max $-$ MaxRL	$+.307$	$+.300$
Falcon3-1B	Re:Max $-$ MaxRL	$+.133$	$+.049$
Qwen2.5-3B	Re:Max $-$ MaxRL	$+.154$	$+.088$

Appendix

Table 5: Replay improves mean correctness and conditional diversity across model scales. Entries are replay minus the matched non-replay objective after eight training passes. pass@8 averages all five domains within matched seeds. PCMD averages a fixed set of domains with defined paired estimates for all contributing seeds. Falcon3-1B Re:Dr pass@8 uses four common seeds; every other entry uses five.

Figure 14: Replay gains vary by domain and model scale. Final pass@8 and PCMD across three model scales for Dr.GRPO, MaxRL, and their replay variants. Markers show seed means; estimates with insufficient verified responses are omitted.

	pass@8		PCMD
Domain	MaxRL	Re:Max	MaxRL	Re:Max
Countdown	.631	.805	.001 (81)	.138 (104)
Graph	.396	.818	.055 (51)	.353 (104)
MathIR	.000	.744	– (0)	.017 (96)
PantryPlan	.760	.779	.384 (97)	.348 (101)

Appendix

Table 6: The Qwen2.5-7B scale extension separates three large replay effects from one near-null comparison. Values are terminal measurements after 3,072 updates from one matched seed per domain. Parentheses give the number of evaluation prompts for which PCMD is defined; an undefined value indicates that too few verified responses are available to estimate conditional diversity.

Figure 15: Replay improves Level-2 Graph accuracy and conditional solution diversity. Points show terminal Re:Dr minus Dr.GRPO and Re:Max minus MaxRL effects on pass@8 and PCMD . Bars show nominal 95% paired intervals where at least two eligible seed pairs are available.

Figure 16: Replay preserves conditional solution diversity throughout training. PCMD across eight training passes for Dr.GRPO, Re:Dr, MaxRL, and Re:Max at three model scales. Lines show means over eligible seeds and bands show seed ranges; gaps indicate insufficient verified responses.

Figure 17: Replay also preserves diversity on harder Level-2 tasks. Qwen2.5-0.5B PCMD across training for the four primary methods. Replay produces the clearest separation on Graph , while MathIR remains concentrated under all four methods.

Domain	Control PCMD	Online RLEP PCMD	Re:Dr PCMD	$\Delta$ PCMD	Replay share
Graph	.003	.344	.711	+.340	.71
Countdown	.017	.054	.538	+.037	.66
Python	.000	.000	.111	.000	.20
MathIR	.000	.013	.018	+.013	.58
PantryPlan	.000	.254	.484	+.254	.93
Mean	.004	.133	.372	+.129	.62

Appendix

Table 7: An online experience pool recovers part, but not all, of Re:Dr’s conditional-diversity gain. Results use five paired seeds per domain against the matched Dr.GRPO control. The effect column is online-pool RLEP-Dr minus control, and replay share is the fraction of updates receiving replay from pass 2 onward.

Figure 18: Re:Dr has the highest terminal mean pass@8 in every displayed comparison. Learning curves compare Dr.GRPO, Re:Dr, UCPO, and sparse RLEP-Dr across two model scales and five domains. Lines show seed means and bands show seed ranges.

Figure 19: Uniform weighting preserves more verified alternatives. Uniform minus frequency-weighted replay at Qwen2.5-0.5B, Level 1. Extra modes are

B_{8}=\texttt{distinct@8}-\texttt{pass@8}

. Bars show nominal 95% paired-seed bootstrap intervals.

Estimate	Replay lower	No change	Replay higher
Pooled estimate	27/29	1/29	1/29
Disjoint split 0	25/29	3/29	1/29
Disjoint split 1	24/29	3/29	2/29

Appendix

Table 8: Replay generally lowers conditional concentration under disjoint sampling splits. Counts summarize Level-1 replay–control comparisons across three model scales, five domains, and two objectives. One Falcon3-1B Python comparison is undefined because too few verified responses are available.

Panel of Figure 20 — Figure 20: Graph and PantryPlan lose conditional solution diversity across scales and evaluation levels. (A) Initial-to-final changes in PCMD at Level 1 across three model scales and no-replay objectives. (B) Level-1-trained policies evaluated on Levels 1–5, with higher levels measuring transfer. Filled marks show well-supported comparisons, open marks indicate limited support, and bars show nominal 95% intervals where available.

Figure 21: RLVR-only training reduces additional verified modes across scales. Bars decompose distinct@8 into pass@8 and additional verified modes before training and after eight passes for Dr.GRPO and GRPO across three model scales and five domains. Whiskers show ranges across five training seeds.

Domain	Mixed groups
Graph	.144
Countdown	.140
Python	.015
MathIR	.364
PantryPlan	.123

Appendix

Table 9: Fresh reward information is sparse in several control runs. Entries are the fraction of Dr.GRPO rollout groups containing both correct and incorrect responses, averaged over five seeds.

Control subset	$n$	$\Delta$ pass@8	$\Delta$ PCMD
All comparisons	15	+.352	+.189
Increasing control pass@8	9	+.379	+.118
Nonzero initial pass@8 , $\geq 10\%$ mixed groups	6	+.247	+.136

Appendix

Table 10: Replay gains persist when controls are learning. Mean Re:Dr minus Dr.GRPO effects across increasingly restrictive subsets of the 15 model–domain comparisons.

Domain	Selected setting	Selected pass@8	Baseline pass@8
Graph	LR $5\times 10^{-8}$	.429	.325
Countdown	entropy $10^{-3}$	.635	.493
Python	baseline	.586	.586
MathIR	entropy $10^{-3}$	.490	.460
PantryPlan	LR $5\times 10^{-8}$	.600	.531

Appendix

Table 11: Correctness-based tuning strengthens the no-replay control in four of five domains. Settings are selected separately within each domain using terminal pass@8 , with ties within

.005

broken by mean@8 ; PCMD is not used for selection.

Domain	Control PCMD	Re:Dr PCMD	$\Delta$ PCMD	$\Delta$ pass@8
Graph	.054	.764	$+.709\;[+.677,+.742]$	$+.515\;[+.439,+.591]$
PantryPlan	.091	.563	$+.473\;[+.371,+.575]$	$+.157\;[+.104,+.210]$

Appendix

Table 12: Replay remains substantially above the selected controls in paired tuned-control comparisons. Effects are Re:Dr minus the correctness-selected no-replay control on matched seeds; brackets give nominal 95% paired intervals.

Benchmark	Base pass@32	Dr.GRPO	Re:Dr	Dr.GRPO avg@32	Re:Dr avg@32
MATH-500	.894	.889	.909	.470	.537
AMC23	.875	.767	.850	.261	.339
OlympiadBench	.628	.546	.620	.196	.234
Minerva	.474	.491	.540	.135	.167
AIME24	.367	.333	.383	.057	.081
AIME25	.200	.211	.217	.021	.025

Appendix

Table 13: Re:Dr restores sampled mathematical coverage lost under Dr.GRPO. pass@32 is the probability that at least one of 32 sampled responses is correct, while avg@32 is mean per-response correctness across the same samples.

Benchmark	Base	Dr.GRPO	MaxRL	Re:Max
MATH-500	.894	.889	.902	.904
AMC23	.875	.767	.850	.850
AIME24	.367	.333	.333	.300
AIME25	.200	.211	.167	.200

Appendix

Table 14: MaxRL largely preserves sampled mathematical coverage before replay is added. Results use one matched seed per MaxRL–Re:Max arm and the same 32-sample evaluation.

Model	Cell	pass@8 : benchmark $\to$ neutral	PCMD : benchmark $\to$ neutral
Dr.GRPO	Python L2	$.812\to.000$	–
Dr.GRPO	Python L3	$1.000\to.000$	–
Re:Dr	Python L2	$.887\to.556$	$.231\to.173$
Re:Dr	Python L3	$1.000\to.494$	$.256\to.323$
Dr.GRPO	MathIR L2	$.531\to.506$	$.000\to.000$
Re:Dr	MathIR L2	$.994\to.975$	$.000\to.000$

Appendix

Table 15: Removing guidance leaves MathIR concentrated but sharply reduces Python correctness. Results use matched Qwen2.5-0.5B checkpoints and strict verification on 32 paired problems per level. PCMD is reported only where sufficient jointly eligible problems remain.

	One withdrawal		Two withdrawals
Domain	Re:Dr	Re:Max	Re:Dr	Re:Max
Graph	19.65	10.99	23.65	12.51
Countdown	9.58	0.10	6.11	-4.87
Python	7.67	5.68	10.10	8.16
MathIR	0.14	0.09	0.20	0.12
PantryPlan	9.78	6.61	7.23	4.94

Appendix

Table 16: Replay generally improves survival after one or two withdrawn options. Entries are replay minus matched control survival in percentage points at four verified draws.

Domain	$\Delta$ saved	$\Delta$ by 8	$\Delta$ calls
Graph	67.58	68.36	-5.37
Countdown	6.76	8.11	-0.55
Python	55.86	63.67	-4.67
MathIR	20.54	20.98	-1.68
PantryPlan	16.41	20.70	-1.44

Appendix

Table 17: Replay improves ordinary-sampling recovery after a withdrawal. Entries are Re:Dr minus Dr.GRPO on the same withdrawn tasks. Saved and resolved-by-eight differences are percentage points; calls are additional recovery responses.

Level	$P$ (%)	$D$	$C$ (%)
1	95.8	1.71	74.0 [71.9, 76.1]
2	77.5	1.16	82.3 [80.3, 84.4]
3	80.3	1.18	84.5 [82.8, 86.2]

Appendix

Table 18: GPT-5.6 Sol produces few verified solution modes at each benchmark level. Values average the five domains equally under strict grading.

P

is pass@8 ,

D

is mean distinct@8 , and

C

is correct-pair collision.

Deployment	Macro PCMD	$N^{(2)}_{\mathrm{eff}}$	L1	L2	L3
			Macro PCMD by level
DeepSeek V4 Pro	.320	1.47	.289	.319	.232
Kimi K3	.302	1.43	.374	.307	.253
GPT-5.4	.264	1.36	.338	.247	.224
Grok 4.3	.239	1.31	.275	.179	.171
GPT-5.6 Sol	.226	1.29	.256	.229	.183

Appendix

Table 19: Hosted deployments remain concentrated among verified solutions. Macro PCMD averages the thirteen domain–level cells reportable for every deployment. Cells require at least thirty prompts with two verified responses.

	Level 1		Level 2		Level 3
Deployment	pass@8	# modes	pass@8	# modes	pass@8	# modes
GPT-5.6 Sol	97.5	1.90	98.2	1.59	99.4	1.54
Claude Opus 5	97.9	1.42	99.6	1.45	99.8	1.35
GPT-5.4	97.7	2.00	98.0	1.68	99.4	1.62
Grok 4.3	99.4	2.12	100.0	1.70	100.0	1.68
Kimi K3	98.0	2.26	100.0	1.95	100.0	1.79

Appendix

Table 20: Hosted deployments produce few verified modes on common prompts. Equal-domain means over Graph , Countdown , MathIR , and PantryPlan , with 128 prompts per domain and level and eight responses per prompt. pass@8 is prompt success and # modes is mean distinct@8 , including unsolved prompts.

Deployment	Level	Refusals	Accuracy (%)	$D$
Claude Opus 5	1	$409\to 0$	$60.0\to 100.0$	$1.12\to 1.49$
Claude Opus 5	2	$1021\to 0$	$.3\to 100.0$	$.02\to 1.38$
Claude Opus 5	3	$1024\to 3$	$.0\to 99.7$	$.00\to 1.41$
GPT-5.6 Sol	1	$0\to 0$	$89.8\to 20.9$	$1.36\to.41$
GPT-5.6 Sol	2	$0\to 0$	$58.7\to 16.7$	$1.15\to.45$
GPT-5.6 Sol	3	$0\to 0$	$56.3\to 19.3$	$1.13\to.48$

Appendix

Table 21: Python prompt-interface effects differ across deployments. Entries show benchmark

\to

direct-expression results under formatting-normalized grading. Accuracy is per-response verified accuracy, and

D

is mean distinct@8 .

Deployment	pass@8 (%)	# modes	PCMD
GPT-5.6 Sol	$98.1\to 75.2$	$1.62\to.99$	$.226\to.145$
GPT-5.4	$98.5\to 74.6$	$1.85\to 1.18$	$.255\to.224$
Grok 4.3	$100.0\to 86.5$	$1.70\to 1.50$	$.154\to.364$
Kimi K3	$99.0\to 93.5$	$2.06\to 1.62$	$.293\to.426$
Claude Opus 4.8	$99.4\to 97.1$	$1.62\to 1.53$	$.202\to.209$

Appendix

Table 22: Disabling explicit reasoning lowers sampled coverage for all five tested deployments. Entries show medium reasoning

\to

disabled reasoning. PCMD uses jointly eligible prompts and therefore need not move with raw mode counts.

$T$	pass@8 (%)	distinct@8	$A$ (%)
0.0	65.6	.673	62.1
0.5	72.3	.840	61.2
1.0	75.2	.963	59.1
1.5	79.8	1.125	57.7
2.0	79.4	1.146	55.9

Appendix

Table 23: GPT-5.6 Sol exposes more sampled modes at higher requested temperature in this sweep. Each temperature uses the same 480 prompts and eight responses per prompt.

A

is per-response verified accuracy.

Deployment	Accuracy (%)	distinct@8
Grok 4.3	$98.65\to 96.35$	$1.608\to 1.658$
Kimi K3	$96.56\to 21.25$	$1.933\to.983$

Appendix

Table 24: Temperature effects are deployment-dependent. Entries show

T=1.0\to 1.5

under strict grading.

Explore similar work

Understanding Diversity Collapse in RLVR via the Lens of Overtraining
Jun 13, 2026Suqin Yuan, Jinkun Chen, Jiyang Zheng +6Reinforcement Learning Reasoning Diversity
Sydney AI Centre, The University of Sydney · Southeast University · Microsoft +3
Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR
Jun 2, 2026Chuanyu Qin, Chenxu Yang, Qingyi Si +3Continual Learning for LLMs Reinforcement Learning
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Improving Generalization Robustness of Multimodal RLVR
Aug 9, 2026Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng +11Multimodal Robustness Multimodal Large Language Models
1National University of Singapore, HPC-AI Lab · 2DAMO Academy, Alibaba Group · 3Hupan Lab +7

Jun 13, 2026, cs.LGJ/K move · Enter open · S save

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Suqin Yuan, Jinkun Chen, Jiyang Zheng, Muyang Li+5

Sydney AI Centre, The University of Sydney · Southeast University · Microsoft+3

Reinforcement Learning Reasoning Diversity RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from \emph{diversity collapse}: Pass@

1

improves while high-

k

Pass@

k

degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of \emph{overtraining}: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-

k

Pass@

k

, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-

k

Pass@

k

, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@

256

above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose \emph{Bayesian Boundary Gating} (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@

k

across a wide range of

k

Abstract

Explore similar work

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR

Improving Generalization Robustness of Multimodal RLVR