How Far Can Unsupervised RLVR Scale LLM Training?

Bingxiang He; Yuxin Zuo; Zeyuan Liu; Shangziqi Zhao; Zixuan Fu; Junlin Yang; Cheng Qian; Kaiyan Zhang

cs.LGMar 9, 2026

How Far Can Unsupervised RLVR Scale LLM Training?

Authors: Bingxiang He, Yuxin Zuo, Zeyuan Liu, Shangziqi Zhao, Zixuan Fu, Junlin Yang, Cheng Qian, Kaiyan Zhang,

Organizations: Tsinghua University · Shanghai AI Lab · Xi’an Jiaotong University · University of Illinois Urbana-Champaign ·

Abstract

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.

Figures & tables

Figure 1: Overview of our paper’s framework. At the center is the taxonomy of Unsupervised RLVR methods, categorized into intrinsic rewards and external rewards. The four surrounding panels illustrate the key findings of our empirical investigation.

Method	Estimator	Formula
RLIF	Self-Certainty	$r(x,y)=\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}D_{\mathrm{KL}}(U\\|\pi_{\theta}(\cdot\|x,y_{<t}))$
EM-RL	Trajectory-Level Entropy	$r(x,y)=\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\log\pi_{\theta}(y_{t}\|x,y_{<t})$
EM-RL, RENT	Token-Level Entropy	$r(x,y)=-\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}H(\pi_{\theta}(\cdot\|x,y_{<t}))$
RLSC	Probability	$r(x,y)=\prod_{t=1}^{\|y\|}\pi_{\theta}(y_{t}\|x,y_{<t})$
RLSF	Probability Disparity	$r(x,y)=\frac{1}{M}\sum_{t=1}^{\|a\|}\big[\max_{a_{t}}\pi_{\theta}(a_{t}\|x,c,a_{<t})-\max_{a_{t}\neq\arg\max\pi_{\theta}}\pi_{\theta}(a_{t}\|x,c,a_{<t})\big]$

Table 1: Overview of certainty-based rewards, estimators and their formulas. All variants reward high-confidence predictions through different formalizations of model certainty.

Method	Estimator	Formula
TTRL, SRT, ETTRL SeRL, SQLM, R-Zero	Majority Voting	$r(x,y)=\mathbb{1}\left[y=\arg\max_{y^{\prime}}\sum_{i=1}^{N}\mathbb{1}[y_{i}=y^{\prime}]\right],\hskip 8.19447pt\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta}(\cdot\|x)$
Co-Reward	Majority Voting across Rephrased Question	$\begin{aligned} r(x,y)&=\mathbb{1}[y=\arg\max_{y^{}}\textstyle\sum_{i=1}^{N}\mathbb{1}[y_{i}=y^{}]],\hskip 8.19447pt\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta}(\cdot\|x)\\ &+\mathbb{1}[y=\arg\max_{y^{}}\textstyle\sum_{j=1}^{N}\mathbb{1}[y_{j}^{\prime}=y^{}]],\hskip 8.19447pt\{y_{j}^{\prime}\}_{j=1}^{N}\sim\pi_{\theta}(\cdot\|\text{rephrase}(x))\end{aligned}$
RLCCF	Self-consistency Weighted Voting	$\begin{aligned} r(x,y)=\mathbb{1}\bigg[y=\arg\max_{a}\textstyle\sum_{n=1}^{N}&\left(\max_{a^{\prime}}\textstyle\sum_{k=1}^{K}\mathbb{1}[o_{n,k}=a^{\prime}]\right)\cdot\textstyle\sum_{k=1}^{K}\mathbb{1}[a=o_{n,k}]\bigg],\\ &\{o_{n,k}\}_{k=1}^{K}\sim\pi_{\theta_{n}}(\cdot\|x),\hskip 8.19447ptn=1,\ldots,N\end{aligned}$
EMPO	Semantic Similarity	$r(x,y)=\frac{\|\mathcal{C}(y)\|}{G},\hskip 8.19447pt\mathcal{C}(y)\in\texttt{SemanticCluster}(\{o_{i}\}_{i=1}^{G}),\hskip 8.19447pt\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta}(\cdot\|x)$
CoVo	Trajectory Consistency and Volatility	$\begin{aligned} r(x,y)=&\frac{1}{G}\left\\|\textstyle\sum_{i=1}^{G}\texttt{Con}(y_{i})\cdot[\cos(\texttt{Vol}(y_{i})),\sin(\texttt{Vol}(y_{i}))]\right\\|+r_{\text{cur}},\\ &\{y_{i}\}_{i=1}^{N}\sim\pi_{\theta}(\cdot\|x),\hskip 8.19447ptG=\|\{i:\text{ans}(y_{i})=\text{ans}(y)\}\|\end{aligned}$

Table 2: Overview of ensemble-based rewards, estimators and their formulas. All variants operationalize the assumption that consistency across independent samples correlates with correctness.

Figure 2: Training dynamics comparing majority-voting training and ground-truth training. Intrinsic rewards initially match supervised performance but eventually collapse: the proxy reward rises while validation accuracy falls, revealing divergence between optimizing confidence and optimizing correctness.

Figure 3: Five intrinsic reward methods exhibit distinct failure patterns. Self-Certainty and Majority Voting degrade gradually while maintaining label accuracy. Probability collapses toward brevity and entropy-based methods drive entropy down through repetition rather than correctness.

Figure 4: Training dynamics on individual representative problems. For each problem, the heatmap shows greedy decoding correctness across epochs (blue = correct, red = wrong; darker = higher confidence), and the green wave indicates whether the highest-reward rollout is correct.

Figure 5: Training Label Accuracy ( blue ) on six MATH500 problems and Test Label Accuracy on two OOD problems: ID 76 ( orange ) and ID 131 ( green ).

Figure 6: Effect of training dataset size from

\{32,128,512,2048,8192,16384\}

. Training with

32

128

samples maintains stable performance without model collapse.

Figure 7: KL divergence at each training step.

Figure 8: Comparison between training and test-time.

Figure 9: Training dynamics of extreme DAPO-32 setting, where almost all initial majority votes are incorrect.

Figure 10: RL Training dynamics after different training stages in Qwen (left) and LLaMA (right) family. In Qwen, SFT enables stable training while base collapses by step 200. In LLaMA, both eventually collapse but SFT delays failure.

Figure 11: Comparison between Model Collapse Step (left) and

\mathbf{Pass@k}

Gain (right) as predictors of RL trainability, with GT Gain (middle) as reference. Later collapse strongly predicts better RL performance.

Figure 12: Consistency across different rollout counts (left) and mini-batch sizes (right). Aggressive hyperparameters accelerate collapse but preserve relative model rankings and predictive power.

Indicator	Computation Cost	Total Tokens	Requires GT
GT Gain	$7k×8×17k×7$	6.66B	Yes
	(response × rollouts × problems × models)	(baseline)
Model Collapse Step	$7k×8×662×32$	1.19B	No
	(response × rollouts × total steps × batch)	(5.6× faster)

Table 3: Computation cost comparison between Model Collapse Step and the gold standard (GT Gain) for assessing RL trainability across 7 models.

Figure 13: Top: Validation accuracy comparing Self-Verification with Trajectory-Level Entropy and Oracle Supervision. Bottom: Training dynamics for Self-Verification.

Figure 14: Prompt sensitivity across base and instruction-aligned models.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Step	1	2	3	4	5	6	7	8	9	10
level3_id146	12.70	15.53	15.92	16.21	18.46	22.07	22.56	24.80	31.35	39.36
level1_id187	6.64	6.69	6.84	7.42	10.45	11.04	11.43	11.82	15.82	18.46
level1_id262	15.14	17.19	17.48	18.85	20.02	22.07	24.12	25.20	34.67	39.84
level3_id122	11.33	12.01	12.40	14.06	17.87	18.46	20.31	21.29	33.59	33.89

Appendix

Table 4: Monotonic increase of

p_{\text{maj}}

(%) in early training steps.

Step	5	10	15	20	25	30	35	40	45	50
level3_id146	18.46	39.36	48.93	91.11	95.41	98.14	98.54	99.02	99.61	99.80
level1_id187	11.04	18.46	26.37	79.88	89.84	93.07	96.09	97.66	98.54	99.02
level1_id262	22.07	39.84	51.37	90.14	95.90	96.80	97.46	98.05	98.63	99.41
level3_id122	17.87	33.59	43.55	84.28	92.19	93.26	95.80	96.09	98.34	98.54

Appendix

Table 5: Geometric convergence of

p_{\text{maj}}

(%) to 1.0 over 50 training steps.

Method	Estimator	Formula	Monotonic transformation $\psi$	Anchor distribution $q$	Model distribution $\pi_{\theta}$
RLIF	Self-Certainty	$r_{\text{SC}}=\displaystyle\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\mathbb{H}(U_{V},\pi_{\theta}^{t})+\log\|V\|$	$z+\log\|V\|$	$\{U_{V}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
EM-RL, RENT	Token-Level Entropy	$r_{\text{H}}=-\displaystyle\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\mathbb{H}(\pi_{\theta}^{t})$	$z$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
EM-RL	Trajectory-Level Entropy	$r_{\text{Traj}}=\displaystyle-\frac{1}{\|y\|}\sum_{t=1}^{\|y\|}\mathbb{H}(\delta^{t},\pi_{\theta}^{t})$	$z$	$\{\delta^{t}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
RLSC	Probability	$r_{\text{Prob}}=\displaystyle\exp(-\sum_{t=1}^{\|y\|}\mathbb{H}(\delta^{t},\pi_{\theta}^{t}))$	$\exp\!\bigl(\|\mathcal{I}\|\cdot z\bigr)$	$\{\delta^{t}\}_{t=1}^{\|y\|}$	$\{\pi_{\theta}^{t}\}_{t=1}^{\|y\|}$
EMPO	Semantic Entropy	$r_{\text{SE}}=\displaystyle\exp(-\mathbb{H}(\delta^{A},\pi_{\theta}^{A}))$	$\exp(z)$	$\delta^{A}$	$\pi_{\theta}^{A}$
TTRL, SRT, ETTRL	Majority Voting	$r_{\text{MV}}=\displaystyle\lim_{\tau\to 0^{+}}\exp(-\mathbb{H}(\delta^{A},\tilde{\pi}_{\theta}^{A}))$	$\exp(z)$	$\delta^{A}$	$\tilde{\pi}_{\theta}^{A}$

Appendix

Table 6: Instantiations for the unified reward framework of representative intrinsic rewards. Each method is specified by its estimator, anchor and model distributions, and a monotonic transformation of the cross-entropy between them. Token-level (

t

) and answer-level (

A

) variants capture different granularities of aggregation.

Figure 15: Examples of per-problem training dynamics from MATH-500.

Advantage Estimator	Training Temperature	Global Batch Size	Mini Batch Size	Rollout Number	Regularization	Max Prompt Length	Max Response Length	Learning Rate	Epoch
GRPO	1.0	64	64	8	w/o KL/Entropy	1024	7168	1e-6	1

Appendix

Table 7: Default hyperparameters for training.

Figure 16: Effect of training temperature for Majority Voting method.

Figure 17: Effect of mini-batch size for Majority Voting method.

Figure 18: Effect of KL divergence regularization for Majority Voting method.

Figure 19: Effect of rollout number for Majority Voting method.

Figure 20: Effect of training temperature on Self-Certainty performance. Note that Point-Biserial Correlation is replaced with Mean Response Length due to Self-Certainty’s scoring characteristics.

Figure 21: Effect of training temperature on Token-Level Entropy performance.

Figure 22: Effect of training temperature on Trajectory-Level Entropy performance.

Figure 23: Effect of training temperature on Probability-based certainty performance.

Figure 24: Effect of mini-batch size on Self-Certainty performance.

Figure 25: Effect of mini-batch size on Token-Level Entropy performance.

Figure 26: Effect of mini-batch size on Trajectory-Level Entropy performance.

Figure 27: Effect of mini-batch size on Probability performance.

Figure 28: Effect of KL divergence regularization on Self-Certainty performance.

Figure 29: Effect of KL divergence regularization on Token-Level Entropy performance.

Figure 30: Effect of KL divergence regularization on Trajectory-Level Entropy performance.

Figure 31: Effect of KL divergence regularization on Probability performance.

Figure 32: Effect of rollout number on Self-Certainty performance.

Figure 33: Effect of rollout number on Token-Level Entropy performance.

Figure 34: Effect of rollout number on Trajectory-Level Entropy performance.

Figure 35: Effect of rollout number on Probability performance.

Figure 36:

Pass@k

performance across different validation benchmarks and models, showing how trends vary as

k

increases from 1 to 256.

Family	Model	Abbrev.	Stage	Size
Qwen	Qwen2.5-1.5B	Q2.5-1.5B	Base	1.5B
	Qwen2.5-Math-1.5B	Q2.5-Math-1.5B	Math Base	1.5B
	DeepSeek-R1-Distill-Qwen-1.5B	DS-R1-1.5B	SFT	1.5B
	Qwen2.5-1.5B-Instruct	Q2.5-1.5B-Inst	Instruct	1.5B
	Qwen3-1.7B-Base	Q3-1.7B	Base	1.7B
	Qwen3-4B-Base	Q3-4B	Base	4B

Appendix

Table 8: Model configurations for backbone experiments. Models are categorized by family, training stage, and size.

Figure 37: Training dynamics across different training stages in Qwen family models.

Figure 38: Training dynamics across different training stages in Llama family models.

Figure 39: Effect of model size on stability across both Qwen and Llama families.

Figure 40: Comparison of Qwen 2.5 and Qwen 3 generations across comprehensive training metrics. Results reveal improved stability in the newer generation, with Qwen3 models demonstrating more gradual and controlled training dynamics compared to Qwen2.5 counterparts.

Figure 41: Comparison of different training data sources.

Metric	Qwen2.5-Math-1.5B	Qwen2.5-Math-7B
maj@2	$28.09$	$33.23$
maj@4	$33.68$	$41.20$
maj@8	$37.23$	$45.73$
maj@16	$38.10$	$47.98$
maj@32	$38.43$	$49.19$
maj@64	$38.17$	$49.87$

Appendix

Table 9: Comparisons before and after TTRL. The results show that TTRL-trained models significantly surpass the base models’ majority-vote performance in accuracy.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 433 papers

Jul 20, 2025

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, the shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.

Fang Wu, Weihao Xuan, Danny Xing +5

Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan +2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

May 20, 2026

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving reasoning in large language models (LLMs), yet the underlying geometry of the resulting parameter trajectories remains underexplored. In this work, we demonstrate that RLVR weight trajectories are extremely low-rank and highly predictable. Specifically, we find that the majority of downstream performance gains are captured by a rank-1 approximation of the parameter deltas, where the magnitude of this projection evolves near-linearly with training steps. Motivated by this, we propose a simple and compute-efficient method RELEX (REinforcement Learning EXtrapolation), which estimates the rank-1 subspace from a short observation window and extrapolates future checkpoints via linear regression, with no learned model required. Across three models (i.e., Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base), RELEX produces checkpoints that match or exceed RLVR performance on both in-domain and out-of-domain benchmarks, requiring as few as 15% steps of full RLVR training. Remarkably, RELEX is able to extrapolate far beyond the observation window at no training cost, predicting checkpoints up to 10-20

\times

beyond the observed prefix with continued improvement (e.g., observe only the first 50 steps and extrapolate to 1000 steps). Our ablation analysis confirms the minimalist sufficiency of RELEX: neither increasing the subspace rank nor employing non-linear modeling yields further gains in extrapolation. Finally, we show that RELEX's success stems from a "denoising" effect: by projecting updates onto the rank-1 subspace, the model discards stochastic optimization noise that would otherwise degrade performance during extrapolation. Our code is available at https://github.com/weizhepei/RELEX.

Zhepei Wei, Xinyu Zhu, Wei-Lin Chen +3

University of Virginia · Washington University in St. Louis

RL for Language Model Reasoning Low-Rank Approximation Reinforcement Learning with Verifiable Rewards RL Post-Training

Ask PDF

May 21, 2026

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of LLMs, but often depends on external supervision from human annotations or gold-standard solutions. Reinforcement learning from internal feedback (RLIF) has recently emerged as a scalable unsupervised alternative, using signals extracted from the model itself. However, existing RLIF methods typically rely on a single internal reward, which can lead to reward hacking, entropy collapse, and degraded reasoning structure. We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components: an answer-level reward based on cluster voting and a completion-level reward based on token-wise self-certainty. To combine these signals robustly, we apply GDPO-based normalization to reduce reward-scale imbalance. We further introduce KL-Cov regularization, which targets low-entropy token distributions responsible for disproportionate entropy reduction, preserving exploration and preventing late-stage collapse. Across mathematical reasoning and code-generation benchmarks, our method improves stability and robustness over prior unsupervised RL approaches, while achieving performance close to supervised RLVR methods. These results show that complementary internal rewards, combined with targeted regularization, can support stable long-horizon reasoning without relying on external ground-truth supervision. Code will be released soon.

Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash +2

Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK +2

KL-Regularized RL Intrinsic Reward Methods Reinforcement Learning Multi-Objective Reinforcement Learning RL for Language Model Reasoning Reward Design for RL+1

Ask PDF

Abstract

RL for Language Models

Explore similar work

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework