The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Fang Wu; Weihao Xuan; Danny Xing; Ximing Lu; Mingjie Liu; Yi Dong; Zaid Harchaoui; Yejin Choi

cs.LGJul 20, 2025

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Authors: Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi

Organizations: Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan · University of Washington, Seattle, WA, USA ·

Abstract

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, the shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.

Figures & tables

Figure 1: Empirical support dynamics compares observed solution access question by question. Both models are evaluated on the same questions under a fixed response budget, decoding protocol, and verifier. Pairing whether each model produces at least one verified success yields preservation, shrinkage, expansion, and out of support. Their counts quantify retention, discovery, and net change across models, tasks, and training checkpoints.

Model	Domain	SRR	NDR	SDS	NSCR	P	E	S	O
ProRL-1.5B-v1	Math	0.96	0.00	0.01	-0.04	1355	5	56	131
	Non-Math	0.91	0.03	0.06	-0.06	1045	31	107	674
	Overall	0.94	0.01	0.03	-0.05	2400	36	163	805
ProRL-1.5B-v2	Math	0.96	0.01	0.01	-0.04	1349	9	62	127
	Non-Math	0.90	0.04	0.07	-0.06	1039	39	113	666
	Overall	0.93	0.02	0.04	-0.05	2388	48	175	793

Table 1: Aggregate observed support dynamics. Each question is classified by whether at least one response passes verification for both checkpoints (

P

), only the trained checkpoint (

E

), only the reference checkpoint (

S

), or neither (

O

), at the benchmark-specific sampling budgets. SRR summarizes retention, NDR the fraction of trained-model successes newly observed relative to the reference, and SDS their harmonic mean. Positive NSCR denotes net expansion; negative NSCR denotes net shrinkage. Read the rates with the four counts. Kangheng-OVR-7B is a vision-language model (VLM). Per-benchmark counts and budgets appear in Appx. C .

Figure 2: Empirical support preservation across nine Reasoning Gym tasks. ProRL reaches high coverage with fewer samples, while the base model catches up as

k

grows. Both models reach full coverage at the largest displayed budgets. Values below each panel report pass@

k

at that budget; values for all budgets appear in Appx. G .

Figure 3: Instances of empirical support expansion on arc_1d , boxnet , and graph_color_vertex20 . ProRL retains a coverage advantage through the largest sampling budget shown in each panel. Values below each panel report pass@

k

at the largest displayed budget. Values for all budgets appear in Appx. G .

Figure 4: Empirical support shrinkage on leg_counting , family_relationships , and power_function . ProRL leads with one attempt, but the base model reaches greater coverage at larger sampling budgets. Values below each panel report pass@

k

at the largest displayed budget; values for all budgets appear in Appx. G .

Category	Correctness	Reference	Target	AIME24	AIME25	Olympiad
Shrinkage	✓ Base, ✗ ProRL	DeepSeek-R1	Base	1.24	1.39	1.17
		DeepSeek-R1	ProRL	1.39	1.70	1.25
		Claude Sonnet 4	Base	1.70	1.54	1.51
		Claude Sonnet 4	ProRL	2.12	1.98	1.83
Expansion	✗ Base, ✓ ProRL	DeepSeek-R1	Base	-	-	1.41
		DeepSeek-R1	ProRL	-	-	1.28

Table 2: Reasoning-token perplexity of traces from DeepSeek-R1 and Sonnet 4, evaluated under the base model and ProRL. Questions are grouped by whether each model produces a verified success: only the base model ( shrinkage ), only ProRL ( expansion ), or neither. Higher perplexity indicates lower likelihood of the supplied reasoning traces.

Dataset	Step	Pass@256	Support Dynamics Metrics				Support Counts				Dataset	Step	Pass@256	Support Dynamics Metrics				Support Counts
Dataset	Step	RLVR	SRR	NDR	SDS	NSCR	P	E	S	O	Dataset	Step	RLVR	SRR	NDR	SDS	NSCR	P	E	S	O
AIME24	0	83.33	-	-	-	-	-	-	-	-	AIME25	0	70.00	-	-	-	-	-	-	-	-
	30	80.00	0.960	0.000	0.000	-0.040	24	0	1	5		30	70.00	0.905	0.095	0.172	0.000	19	2	2	7
	60	80.00	0.960	0.000	0.000	-0.040	24	0	1	5		60	73.33	0.905	0.136	0.237	0.042	19	3	2	6
	90	83.33	1.000	0.000	0.000	0.000	25	0	0	5		90	70.00	0.905	0.095	0.172	0.000	19	2	2	7
	120	80.00	0.960	0.000	0.000	-0.040	24	0	1	5		120	73.33	0.905	0.136	0.237	0.042	19	3	2	6

Table 4: Effect of RLVR (DAPO) training on DeepSeek 1.5B across math benchmarks. Pass@256 values are percentages. Step = 0 corresponds to the non-RLVR baseline.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

AIME2024	@1	@2	@4	@8	@16	@32	@64	@128	@256
SFT	20.00	30.00	30.00	33.33	43.33	50.00	63.33	63.33	63.33
DAPO	20.00	26.67	26.67	40.00	46.67	56.67	60.00	63.33	66.67
AIME2025	@1	@2	@4	@8	@16	@32	@64	@128	@256
SFT	16.67	16.67	26.67	30.00	33.33	40.00	43.33	50.00	63.33
DAPO	16.67	20.00	30.00	43.33	46.67	50.00	50.00	50.00	56.67
AMC23	@1	@2	@4	@8	@16	@32	@64	@128	@256

Appendix

Table 5: Detailed pass@k comparison of Qwen2.5-Math-7B trained with SFT or DAPO across five math benchmarks. Values are percentages; column headings give the number of responses

k

. Higher values indicate greater observed verified coverage.

Metric	Model	AIME24	AMC23	MATH500	Minerva	Olympiad	Avg.
	DeepSeek-1.5B	31.15	72.81	85.01	32.18	51.55	54.54
	ProRL-1.5B	45.62	85.70	92.01	39.27	64.56	65.43
	DeepSeek-7B	53.23	89.30	93.95	43.07	66.67	69.24
avg@32	AceReason-7B	65.83	95.08	95.81	45.35	73.92	75.20
Acc. (%)	Skywork-OR1-7B	67.40	93.59	95.73	43.81	73.05	74.71
	DeepSeek-14B	67.81	95.39	95.28	46.43	72.06	75.39

Appendix

Table 13: Accuracy ( avg@32 , %), response length (tokens), and entropy across five math benchmarks. The Avg. column is the unweighted mean over benchmarks. Row colors: xx reference models, xx RLVR models. Relative to the corresponding reference checkpoint, RLVR increases accuracy and reduces answer entropy in every model–benchmark comparison; token entropy increases for ProRL and DAPO-32B and decreases for AceReason and Skywork.

Name	Full Model Name	Type	Parameters
DeepSeek-1.5B	DeepSeek-R1-Distill-Qwen-1.5B	Base	1.5B
ProRL-1.5B	Nemotron-Research-Reasoning-Qwen-1.5B	RLVR	1.5B
DeepSeek-7B	DeepSeek-R1-Distill-Qwen-7B	Base	7B
AceReason-7B	AceReason-Nemotron-7B	RLVR	7B
Skywork-OR1-7B	Skywork-OR1-7B	RLVR	7B
DeepSeek-14B	DeepSeek-R1-Distill-Qwen-14B	Base	14B

Appendix

Table 14: Models evaluated in the entropy analysis.

$k$	dice		prime_factorization_hard		palindrome_generation
	Base	ProRL	Base	ProRL	Base	ProRL
1	0.000	0.180	0.832	1.000	0.154	0.923
2	0.000	0.460	0.941	1.000	0.235	0.980
4	0.000	0.820	0.970	1.000	0.413	0.980
8	0.020	0.920	0.980	1.000	0.688	0.980
16	0.060	0.980	0.980	1.000	0.825	0.980

Appendix

Table 15: Preservation, top row: pass@

k

values for Figure 2 . Each pair of columns compares the Base model with ProRL at the same sampling budget

k

. A dash marks a sampling budget not shown in the corresponding panel.

$k$	advanced_geometry		advanced_geometry_hard		prime_factorization
	Base	ProRL	Base	ProRL	Base	ProRL
1	0.611	0.865	0.157	0.551	0.980	1.000
2	0.690	0.904	0.275	0.728	0.980	1.000
4	0.828	0.923	0.414	0.845	0.980	1.000
8	0.886	1.000	0.511	0.922	1.000	1.000
16	0.905	1.000	0.686	0.961	1.000	1.000

Appendix

Table 16: Preservation, middle row: pass@

k

values for Figure 2 . Each pair of columns compares the Base model with ProRL at the same sampling budget

k

. A dash marks a sampling budget not shown in the corresponding panel.

$k$	graph_color		graph_color_hard		graph_color_vertex15
	Base	ProRL	Base	ProRL	Base	ProRL
1	0.140	1.000	0.120	0.960	0.100	1.000
2	0.241	1.000	0.121	1.000	0.161	1.000
4	0.421	1.000	0.221	1.000	0.222	1.000
8	0.742	1.000	0.462	1.000	0.382	1.000
16	0.960	1.000	0.582	1.000	0.602	1.000

Appendix

Table 17: Preservation, bottom row: pass@

k

values for Figure 2 . Each pair of columns compares the Base model with ProRL at the same sampling budget

k

. A dash marks a sampling budget not shown in the corresponding panel.

$k$	arc_1d		boxnet		graph_color_vertex20
	Base	ProRL	Base	ProRL	Base	ProRL
1	0.010	0.040	0.024	0.097	0.000	0.881
2	0.037	0.109	0.048	0.151	0.000	0.980
4	0.064	0.158	0.063	0.179	0.001	0.980
8	0.097	0.297	0.083	0.210	0.023	1.000
16	0.128	0.375	0.102	0.239	0.044	1.000

Appendix

Table 18: Expansion: pass@

k

values for Figure 3 . Each pair of columns compares the Base model with ProRL at the same sampling budget

k

. A dash marks a sampling budget not shown in the corresponding panel.

$k$	leg_counting		family_relationships		power_function
	Base	ProRL	Base	ProRL	Base	ProRL
1	0.040	0.280	0.140	0.840	0.465	0.624
2	0.040	0.300	0.220	0.860	0.604	0.802
4	0.120	0.300	0.380	0.860	0.723	0.842
8	0.140	0.300	0.560	0.860	0.842	0.901
16	0.160	0.300	0.680	0.860	0.941	0.901

Appendix

Table 19: Shrinkage: pass@

k

values for Figure 4 . Each pair of columns compares the Base model with ProRL at the same sampling budget

k

. A dash marks a sampling budget not shown in the corresponding panel.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
Apr 6, 2026Kazuki Egashira, Mark Vero, Jasper Dekoninck +3Reinforcement Learning with Verifiable Rewards RL for Language Model Reasoning
ETH Zurich · Max Planck Institute for Intelligent Systems, Tübingen
Understanding Diversity Collapse in RLVR via the Lens of Overtraining
Jun 13, 2026Suqin Yuan, Jinkun Chen, Jiyang Zheng +6Reinforcement Learning Reasoning Diversity
Sydney AI Centre, The University of Sydney · Southeast University · Microsoft +3
On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
Mar 23, 2026Kexin Huang, Haoming Meng, Junkang Wu +10RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Alibaba Group

Apr 6, 2026J/K move · Enter open · S save

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

Kazuki Egashira, Mark Vero, Jasper Dekoninck, Florian E. Dorner+2

ETH Zurich · Max Planck Institute for Intelligent Systems, Tübingen

Reinforcement Learning with Verifiable Rewards RL for Language Model Reasoning

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.

Abstract

RL for Language Model Reasoning

Explore similar work

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation