How Far Can Unsupervised RLVR Scale LLM Training?

Bingxiang He; Yuxin Zuo; Zeyuan Liu; Shangziqi Zhao; Zixuan Fu; Junlin Yang; Cheng Qian; Kaiyan Zhang

cs.LGMar 9, 2026

How Far Can Unsupervised RLVR Scale LLM Training?

Authors: Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang,

Organizations: Tsinghua University · Shanghai AI Lab · Xi’an Jiaotong University · University of Illinois Urbana-Champaign ·

Abstract

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.

Figures & tables

Figure 1: Overview of our paper’s framework. At the center is the taxonomy of Unsupervised RLVR methods, categorized into intrinsic rewards and external rewards. The four surrounding panels illustrate the key findings of our empirical investigation.

Method	Estimator	Formula
RLIF	Self-Certainty	$r(x,y)=\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}D_{\mathrm{KL}}(U\\|\pi_{\theta}(\cdot\|x,y_{<t}))$
EM-RL	Trajectory-Level Entropy	$r(x,y)=\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\log\pi_{\theta}(y_{t}\|x,y_{<t})$
EM-RL, RENT	Token-Level Entropy	$r(x,y)=-\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}H(\pi_{\theta}(\cdot\|x,y_{<t}))$
RLSC	Probability	$r(x,y)=\prod_{t=1}^{\|y\|}\pi_{\theta}(y_{t}\|x,y_{<t})$
RLSF	Probability Disparity	$r(x,y)=\frac{1}{M}\sum_{t=1}^{\|a\|}\big[\max_{a_{t}}\pi_{\theta}(a_{t}\|x,c,a_{<t})-\max_{a_{t}\neq\arg\max\pi_{\theta}}\pi_{\theta}(a_{t}\|x,c,a_{<t})\big]$

Table 1: Overview of certainty-based rewards, estimators and their formulas. All variants reward high-confidence predictions through different formalizations of model certainty.

Method	Estimator	Formula
TTRL, SRT, ETTRL SeRL, SQLM, R-Zero	Majority Voting	$r(x,y)=\mathbb{1}\left[y=\arg\max_{y^{\prime}}\sum_{i=1}^{N}\mathbb{1}[y_{i}=y^{\prime}]\right],\hskip 8.19447pt\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta}(\cdot\|x)$
Co-Reward	Majority Voting across Rephrased Question	$\begin{aligned} r(x,y)&=\mathbb{1}[y=\arg\max_{y^{}}\textstyle\sum_{i=1}^{N}\mathbb{1}[y_{i}=y^{}]],\hskip 8.19447pt\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta}(\cdot\|x)\\ &+\mathbb{1}[y=\arg\max_{y^{}}\textstyle\sum_{j=1}^{N}\mathbb{1}[y_{j}^{\prime}=y^{}]],\hskip 8.19447pt\{y_{j}^{\prime}\}_{j=1}^{N}\sim\pi_{\theta}(\cdot\|\text{rephrase}(x))\end{aligned}$
RLCCF	Self-consistency Weighted Voting	$\begin{aligned} r(x,y)=\mathbb{1}\bigg[y=\arg\max_{a}\textstyle\sum_{n=1}^{N}&\left(\max_{a^{\prime}}\textstyle\sum_{k=1}^{K}\mathbb{1}[o_{n,k}=a^{\prime}]\right)\cdot\textstyle\sum_{k=1}^{K}\mathbb{1}[a=o_{n,k}]\bigg],\\ &\{o_{n,k}\}_{k=1}^{K}\sim\pi_{\theta_{n}}(\cdot\|x),\hskip 8.19447ptn=1,\ldots,N\end{aligned}$
EMPO	Semantic Similarity	$r(x,y)=\frac{\|\mathcal{C}(y)\|}{G},\hskip 8.19447pt\mathcal{C}(y)\in\texttt{SemanticCluster}(\{o_{i}\}_{i=1}^{G}),\hskip 8.19447pt\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot\|x)$
CoVo	Trajectory Consistency and Volatility	$\begin{aligned} r(x,y)=&\frac{1}{G}\left\\|\textstyle\sum_{i=1}^{G}\texttt{Con}(y_{i})\cdot[\cos(\texttt{Vol}(y_{i})),\sin(\texttt{Vol}(y_{i}))]\right\\|+r_{\text{cur}},\\ &\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta}(\cdot\|x),\hskip 8.19447ptG=\|\{i:\text{ans}(y_{i})=\text{ans}(y)\}\|\end{aligned}$

Table 2: Overview of ensemble-based rewards, estimators and their formulas. All variants operationalize the assumption that consistency across independent samples correlates with correctness.

Figure 2: Training dynamics comparing majority-voting training and ground-truth training. Intrinsic rewards initially match supervised performance but eventually collapse: the proxy reward rises while validation accuracy falls, revealing divergence between optimizing confidence and optimizing correctness.

Figure 3: Five intrinsic reward methods exhibit distinct failure patterns. Self-Certainty and Majority Voting degrade gradually while maintaining label accuracy. Probability collapses toward brevity and entropy-based methods drive entropy down through repetition rather than correctness.

Figure 4: Training dynamics on individual representative problems. For each problem, the heatmap shows greedy decoding correctness across epochs (blue = correct, red = wrong; darker = higher confidence), and the green wave indicates whether the highest-reward rollout is correct.

Figure 5: Training Label Accuracy ( blue ) on six MATH500 problems and Test Label Accuracy on two OOD problems: ID 76 ( orange ) and ID 131 ( green ).

Figure 6: Effect of training dataset size from

\{32,128,512,2048,8192,16384\}

. Training with

32

128

samples maintains stable performance without model collapse.

Figure 7: KL divergence at each training step.

Figure 8: Comparison between training and test-time.

Figure 9: Training dynamics of extreme DAPO-32 setting, where almost all initial majority votes are incorrect.

Figure 10: RL Training dynamics after different training stages in Qwen (left) and LLaMA (right) family. In Qwen, SFT enables stable training while base collapses by step 200. In LLaMA, both eventually collapse but SFT delays failure.

Figure 11: Comparison between Model Collapse Step (left) and

\mathbf{Pass@k}

Gain (right) as predictors of RL trainability, with GT Gain (middle) as reference. Later collapse strongly predicts better RL performance.

Figure 12: Consistency across different rollout counts (left) and mini-batch sizes (right). Aggressive hyperparameters accelerate collapse but preserve relative model rankings and predictive power.

Indicator	Computation Cost	Total Tokens	Requires GT
GT Gain	$7k×8×17k×7$	6.66B	Yes
	(response × rollouts × problems × models)	(baseline)
Model Collapse Step	$7k×8×662×32$	1.19B	No
	(response × rollouts × total steps × batch)	(5.6× faster)

Table 3: Computation cost comparison between Model Collapse Step and the gold standard (GT Gain) for assessing RL trainability across 7 models.

Figure 13: Top: Validation accuracy comparing Self-Verification with Trajectory-Level Entropy and Oracle Supervision. Bottom: Training dynamics for Self-Verification.

Figure 14: Prompt sensitivity across base and instruction-aligned models.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Step	1	2	3	4	5	6	7	8	9	10
level3_id146	12.70	15.53	15.92	16.21	18.46	22.07	22.56	24.80	31.35	39.36
level1_id187	6.64	6.69	6.84	7.42	10.45	11.04	11.43	11.82	15.82	18.46
level1_id262	15.14	17.19	17.48	18.85	20.02	22.07	24.12	25.20	34.67	39.84
level3_id122	11.33	12.01	12.40	14.06	17.87	18.46	20.31	21.29	33.59	33.89

Appendix

Table 4: Monotonic increase of

p_{\text{maj}}

(%) in early training steps.

Step	5	10	15	20	25	30	35	40	45	50
level3_id146	18.46	39.36	48.93	91.11	95.41	98.14	98.54	99.02	99.61	99.80
level1_id187	11.04	18.46	26.37	79.88	89.84	93.07	96.09	97.66	98.54	99.02
level1_id262	22.07	39.84	51.37	90.14	95.90	96.80	97.46	98.05	98.63	99.41
level3_id122	17.87	33.59	43.55	84.28	92.19	93.26	95.80	96.09	98.34	98.54

Appendix

Table 5: Geometric convergence of

p_{\text{maj}}

(%) to 1.0 over 50 training steps.

Method	Estimator	Formula	Monotonic transformation $\psi$	Anchor distribution $q$	Model distribution $\pi_{\theta}$
RLIF	Self-Certainty	$r_{\text{SC}}=\displaystyle\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\mathbb{H}(U_{V},\pi_{\theta}^{t})+\log\|V\|$	$z+\log\|V\|$	$\{U_{V}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
EM-RL, RENT	Token-Level Entropy	$r_{\text{H}}=-\displaystyle\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\mathbb{H}(\pi_{\theta}^{t})$	$z$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
EM-RL	Trajectory-Level Entropy	$r_{\text{Traj}}=\displaystyle-\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\mathbb{H}(\delta^{t},\pi_{\theta}^{t})$	$z$	$\{\delta^{t}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
RLSC	Probability	$r_{\text{Prob}}=\displaystyle\exp(-\sum_{t=1}^{\|y\|}\mathbb{H}(\delta^{t},\pi_{\theta}^{t}))$	$\exp\!\bigl(\|\mathcal{I}\|\cdot z\bigr)$	$\{\delta^{t}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
EMPO	Semantic Entropy	$r_{\text{SE}}=\displaystyle\exp(-\mathbb{H}(\delta^{A},\pi_{\theta}^{A}))$	$\exp(z)$	$\delta^{A}$	$\pi_{\theta}^{A}$
TTRL, SRT, ETTRL	Majority Voting	$r_{\text{MV}}=\displaystyle\lim_{\tau\to 0^{+}}\exp(-\mathbb{H}(\delta^{A},\tilde{\pi}_{\theta}^{A}))$	$\exp(z)$	$\delta^{A}$	$\tilde{\pi}_{\theta}^{A}$

Appendix

Table 6: Instantiations for the unified reward framework of representative intrinsic rewards. Each method is specified by its estimator, anchor and model distributions, and a monotonic transformation of the cross-entropy between them. Token-level (

t

) and answer-level (

A

) variants capture different granularities of aggregation.

Figure 15: Examples of per-problem training dynamics from MATH-500.

Advantage Estimator	Training Temperature	Global Batch Size	Mini Batch Size	Rollout Number	Regularization	Max Prompt Length	Max Response Length	Learning Rate	Epoch
GRPO	1.0	64	64	8	w/o KL/Entropy	1024	7168	1e-6	1

Appendix

Table 7: Default hyperparameters for training.

Figure 16: Effect of training temperature for Majority Voting method.

Figure 17: Effect of mini-batch size for Majority Voting method.

Figure 18: Effect of KL divergence regularization for Majority Voting method.

Figure 19: Effect of rollout number for Majority Voting method.

Figure 20: Effect of training temperature on Self-Certainty performance. Note that Point-Biserial Correlation is replaced with Mean Response Length due to Self-Certainty’s scoring characteristics.

Figure 21: Effect of training temperature on Token-Level Entropy performance.

Figure 22: Effect of training temperature on Trajectory-Level Entropy performance.

Figure 23: Effect of training temperature on Probability-based certainty performance.

Figure 24: Effect of mini-batch size on Self-Certainty performance.

Figure 25: Effect of mini-batch size on Token-Level Entropy performance.

Figure 26: Effect of mini-batch size on Trajectory-Level Entropy performance.

Figure 27: Effect of mini-batch size on Probability performance.

Figure 28: Effect of KL divergence regularization on Self-Certainty performance.

Figure 29: Effect of KL divergence regularization on Token-Level Entropy performance.

Figure 30: Effect of KL divergence regularization on Trajectory-Level Entropy performance.

Figure 31: Effect of KL divergence regularization on Probability performance.

Figure 32: Effect of rollout number on Self-Certainty performance.

Figure 33: Effect of rollout number on Token-Level Entropy performance.

Figure 34: Effect of rollout number on Trajectory-Level Entropy performance.

Figure 35: Effect of rollout number on Probability performance.

Figure 36:

Pass@k

performance across different validation benchmarks and models, showing how trends vary as

k

increases from 1 to 256.

Family	Model	Abbrev.	Stage	Size
Qwen	Qwen2.5-1.5B	Q2.5-1.5B	Base	1.5B
	Qwen2.5-Math-1.5B	Q2.5-Math-1.5B	Math Base	1.5B
	DeepSeek-R1-Distill-Qwen-1.5B	DS-R1-1.5B	SFT	1.5B
	Qwen2.5-1.5B-Instruct	Q2.5-1.5B-Inst	Instruct	1.5B
	Qwen3-1.7B-Base	Q3-1.7B	Base	1.7B
	Qwen3-4B-Base	Q3-4B	Base	4B

Appendix

Table 8: Model configurations for backbone experiments. Models are categorized by family, training stage, and size.

Figure 37: Training dynamics across different training stages in Qwen family models.

Figure 38: Training dynamics across different training stages in Llama family models.

Figure 39: Effect of model size on stability across both Qwen and Llama families.

Figure 40: Comparison of Qwen 2.5 and Qwen 3 generations across comprehensive training metrics. Results reveal improved stability in the newer generation, with Qwen3 models demonstrating more gradual and controlled training dynamics compared to Qwen2.5 counterparts.

Figure 41: Comparison of different training data sources.

Metric	Qwen2.5-Math-1.5B	Qwen2.5-Math-7B
maj@2	$28.09$	$33.23$
maj@4	$33.68$	$41.20$
maj@8	$37.23$	$45.73$
maj@16	$38.10$	$47.98$
maj@32	$38.43$	$49.19$
maj@64	$38.17$	$49.87$

Appendix

Table 9: Comparisons before and after TTRL. The results show that TTRL-trained models significantly surpass the base models’ majority-vote performance in accuracy.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

The Invisible Leash: Why RLVR May or May Not Escape Its Origin
Jul 20, 2025Fang Wu, Weihao Xuan, Danny Xing +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan +2
You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories
May 20, 2026Zhepei Wei, Xinyu Zhu, Wei-Lin Chen +3RL for Language Model Reasoning Low-Rank Approximation
University of Virginia · Washington University in St. Louis
Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework
May 21, 2026Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash +2KL-Regularized RL Intrinsic Reward Methods
Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK +2

Jul 20, 2025J/K move · Enter open · S save

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu+4

Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan+2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, the shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.

Abstract

RL for Language Models

Explore similar work

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework