Efficient RLVR Training via Weighted Mutual Information Data Selection

Xinyu Zhou; Boyu Zhu; Haotian Zhang; Huiming Wang; Zhijiang Guo

cs.LGMar 2, 2026

Efficient RLVR Training via Weighted Mutual Information Data Selection

Authors: Xinyu Zhou, Boyu Zhu, Haotian Zhang, Huiming Wang, Zhijiang Guo

Organizations: HKUST(GZ) · UCL · Kuaishou Technology · HKUST

Abstract

Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.

Figures & tables

Figure 1: Overview of the InSight pipeline. InSight maintains a Bayesian belief over data success rates, scores candidate datapoints using Weighted Mutual Information (WMI), and selects the top- $M$ datapoints for RL training. Observed rewards update the posterior beliefs, enabling adaptive data selection that jointly accounts for data difficulty and accumulated evidence, unlike difficulty-only heuristics.

Figure 2: Expected variance reduction as a function of prior mean $\bar{\phi}_{\tau}$ and accumulated evidence $n$ . While difficulty-based heuristics focus solely on $\bar{\phi}_{\tau}\approx 0.5$ , the expected uncertainty reduction decays rapidly with evidence, revealing a fundamental limitation of difficulty-only selection.

Figure 3: Weighted functions $w(\bar{\phi})$ under different $\eta,\mu$ .

Models	Methods	CountDown	AIME24	AMC23	MATH500	Minerva.	Olympiad.	Avg.
	Random	71.57	10.41	43.90	71.30	21.60	34.20	42.16
Qwen3-0.6B	MoPPS	76.28	10.83	44.16	70.60	21.60	34.25	42.95
	Inverse-Evidence	72.13	10.60	42.50	71.80	21.70	34.80	42.25
	InSight (Ours)	76.70 (5.13 $\uparrow$ )	11.90 (1.49 $\uparrow$ )	44.30 (0.40 $\uparrow$ )	71.75 (0.45 $\uparrow$ )	21.94 (0.34 $\uparrow$ )	34.80 (0.60 $\uparrow$ )	43.56 (1.40 $\uparrow$ )
	Random	84.90	51.45	78.01	91.22	41.13	56.24	67.16
Qwen3-4B	MoPPS	87.80	50.00	77.41	90.82	41.59	56.45	67.35

Table 1: Evaluation results across planning and mathematics benchmarks. Accuracy is computed as the average pass@1 over 16 independent generations per problem. Bold indicates the best results, and gray denotes the oracle baseline.

\uparrow(\downarrow)

indicates the improvement(degradation) compared to the Random baseline.

Models	Methods	MMLU				GPQA	Avg.
Models	Methods	Humanities	Other	Social-Sci.	STEM	GPQA	Avg.
	Random	38.83	47.22	52.32	40.95	26.34	41.13
Qwen3-0.6B	MoPPS	39.17	47.12	51.12	38.69	27.90	40.80
	Inverse-Evidence	38.55	45.19	49.76	36.95	29.46	39.98
	InSight (Ours)	39.21 (0.38 $\uparrow$ )	47.34 (0.12 $\uparrow$ )	52.84 (0.52 $\uparrow$ )	41.80 (0.85 $\uparrow$ )	29.50 (3.16 $\uparrow$ )	42.14 (1.01 $\uparrow$ )
	Random	48.63	60.41	64.09	53.47	30.20	51.36

Table 2: Evaluation results across general-reasoning benchmarks, via LM-Evaluation-Harness framework. Bold indicates the best results.

\uparrow

indicates the improvement compared to the Random baseline.

Figure 4: Performance comparisons among different methods on the Countdown task. Our proposed InSight outperforms the existing SOTA ( MoPPS ) and other baselines in both training efficiency and performance.

Models	Component	AIME24	AMC23	Olympiad.
	$I(R_{1:K},\Phi_{\tau})$	9.37	43.52	34.01
Qwen3-0.6B	${\color[rgb]{0.06,0.46,1}w(\bar{\phi}_{\tau})}\cdot{\color[rgb]{0.72,0,0}I(R_{1:K},\Phi_{\tau})}$	11.90	44.30	34.80
	$I(R_{1:K},\Phi_{\tau})$	51.66	76.43	56.24
Qwen3-4B	${\color[rgb]{0.06,0.46,1}w(\bar{\phi}_{\tau})}\cdot{\color[rgb]{0.72,0,0}I(R_{1:K},\Phi_{\tau})}$	53.75	79.00	57.00
	${\color[rgb]{0.06,0.46,1}w(\bar{\phi}_{\tau})}$	46.67	77.33	52.25
R1-Distill-7B	$I(R_{1:K},\Phi_{\tau})$	46.67	77.90	52.35

Table 3: Evaluation comparisons of WMI components across different models. Bold indicates the best results.

Models	$\bar{\phi}_{\tau}$ / $\hat{\phi}_{\tau}$	AIME24	AMC23	Olympiad.
	$\hat{\phi}_{\tau}$	10.41	43.22	34.25
Qwen3-0.6B	$\bar{\phi}_{\tau}$	11.90	44.30	34.80
	$\hat{\phi}_{\tau}$	51.45	77.48	56.33
Qwen3-4B	$\bar{\phi}_{\tau}$	53.75	79.00	57.00
	$\hat{\phi}_{\tau}$	46.25	77.93	52.37
R1-Distill-7B	$\bar{\phi}_{\tau}$	47.71	78.40	52.50

Table 4: Evaluation comparisons between

\bar{\phi}_{\tau}

and

\hat{\phi}_{\tau}

variants in WMI across different models. Bold indicates the best results.

Models	$\mu$	AIME24	AMC23	Olympiad.
	$0.1$	10.41	44.80	34.75
Qwen3-0.6B	$0.3$	11.90	44.30	34.80
	$0.7$	10.41	42.62	33.09
	$0.1$	50.83	77.03	56.32
Qwen3-4B	$0.3$	53.75	79.00	57.00
	$0.7$	52.91	77.41	56.58

Table 5: Evaluation comparisons of different

\mu

in weighted function across different models. Bold indicates the best results.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Models	Methods	AIME24	AMC23	MATH500	Minerva.	Olympiad.	Training Hours
	DS	50.62	80.79	90.82	38.39	53.38	$\sim$ 30.5 Hours
R1-Distill-Qwen-7B	Random	46.25	77.90	90.50	37.98	51.80	$\sim$ 12.5 Hours
	InSight (Ours)	47.71	78.40	90.45	39.06	52.50	$\sim$ 12.5 Hours

Appendix

Table 6: Evaluation results across mathematics benchmarks. Accuracy is computed as the average pass@1 over 16 independent generations per problem.

Models	Methods	$\hat{M}$	AIME24	AMC23	MATH500	Minerva.	Olympiad.
	MoPPS	8x	10.20	44.42	71.46	21.64	34.07
	InSight	8x	10.20	44.50	72.53	21.69	35.07
Qwen3-0.6B	InSight	12x	10.83	43.30	70.85	22.20	34.33
	InSight	16x	11.90	44.30	71.75	21.94	34.80
	MoPPS	8x	52.29	77.71	91.08	41.40	56.70
	InSight	8x	52.29	78.54	91.09	41.73	56.91

Appendix

Table 7: Evaluation comparisons of different

\hat{M}

across different models, on mathematics benchmarks. Bold indicates the best results.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Smart Picks in the Dark: Towards Efficient RLVR for Reasoning via Tracing Metacognitive Pivots
Jun 3, 2026Guangcheng Zhu, Shenzhi Yang, Haobo Wang +7Uncertainty Quantification RL for Language Model Reasoning
Zhejiang University · Ant Group
Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection
May 27, 2026Jianghao Wu, Jianfei Cai, Weiqiang Wang +3Training Data Selection Reinforcement Learning with Verifiable Rewards
Faculty of Information Technology, Monash University, Melbourne, Australia.
Training-Free Prompt Selection for Efficient Reinforcement Learning With Partial Rollouts
Oct 16, 2026Lei Jiang, Honghui Bao, Yue Zhou +3Prompt Selection for RL RL for Language Model Reasoning
University of Illinois Chicago · LLM Core and Agentic AI, CapitalOne

Jun 3, 2026J/K move · Enter open · S save

Smart Picks in the Dark: Towards Efficient RLVR for Reasoning via Tracing Metacognitive Pivots

Guangcheng Zhu, Shenzhi Yang, Haobo Wang, Xing Zheng+6

Zhejiang University · Ant Group

Uncertainty Quantification RL for Language Model Reasoning Active Learning Reinforcement Learning with Verifiable Rewards Data-Efficient Learning

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) has greatly advanced large reasoning models (LRMs), but it requires timely training on a huge fully-annotated dataset. To this end, data-efficient RLVR methods have been widely studied from two perspectives: (i) data selection methods identify a small subset of "golden" samples that yield near-full-data performance, but they rely on a pre-existing pool of labeled data. (ii) unsupervised RLVR methods train the model using its own internal supervision signals on large-scale unlabeled data, yet they exhibit suboptimal performance. Accordingly, we investigate the "pick in the dark" setup for RLVR, which aims to select, without prior supervision, unlabeled samples that are most beneficial for training and worthy of annotation. Through systematic analysis, we demonstrate that smart picks hinge on a well-calibrated uncertainty estimator to enable strategic partitioning of data for adaptive training regimes. Building on this insight, we propose PivotTrace, a three-way data triage framework that leverages attention dynamics to trace metacognitive pivots during reasoning. By precisely quantifying uncertainty through pivot density, PivotTrace achieves automated data routing to synergistically maximize both annotation and training efficiency. Empirically, PivotTrace surpasses the fully supervised LRM with only 29.3% annotated samples and 2.75 faster convergence.

Abstract

Reinforcement Learning

Explore similar work

Smart Picks in the Dark: Towards Efficient RLVR for Reasoning via Tracing Metacognitive Pivots

Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection

Training-Free Prompt Selection for Efficient Reinforcement Learning With Partial Rollouts