The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

Jin Cui; Xinyue Long; Boran Zhao; Pengju Ren; Hao Dong

cs.AISep 27, 2026

The Model Knows Another Way: Strategy Switching for Effective RLVR Exploration

Authors: Jin Cui, Xinyue Long, Boran Zhao, Pengju Ren, Hao Dong

Organizations: State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, and Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Software Engineering, Xi’an Jiaotong University · ELLIS Institute Finland and Tampere University

Abstract

Reinforcement learning with verifiable rewards (RLVR) is often limited by insufficient exploration: difficult problems can yield uniformly incorrect rollout groups and therefore little learning signal. We show that such failures need not reflect missing capability. Instead, finite sampling often concentrates on a problem-specific dominant reasoning strategy while leaving alternative strategies already supported by the model unexplored. Moreover, the accessibility of these strategies evolves during RL: some are internalized into autonomous behavior, while others become difficult to elicit before being absorbed. Motivated by these observations, we introduce Problem--Strategy Rollout Allocation (PSRA), which treats unguided and strategy-conditioned prompts as competing exploration arms and uses Bayesian sequential allocation to direct a fixed rollout budget toward arms most likely to yield informative, non-saturated groups. A preservation objective keeps useful strategy-conditioned routes accessible while successful guided behaviors are transferred to the unguided policy. Across Qwen2.5 models from 1.5B to 7B and two RL training corpora, PSRA consistently improves reasoning performance, reduces dead saturation, strengthens out-of-distribution transfer, and maintains larger gains under increased inference budgets.

Figures & tables

Figure 1: Diagnosing strategy-level exploration in the base policy. (a) Representation-space clustering reveals a compact set of reasoning strategies. (b) Autonomous rollouts are strongly concentrated. (c) Strategy conditioning reliably redirects generation toward distinct reasoning regions. (d) Switching to the best alternative strategy improves rescue rate over continued dominant-strategy or unguided sampling. (e) Pre-generation states predict reasoning directions missed by finite rollouts. (f) A small set of certified strategies covers most of the predicted exploration gaps. (g) During RL, strategy accessibility exhibits both internalization and displacement.

Figure 2: Overview of PSRA Structure. PSRA allocates rollout budget over unguided and strategy-conditioned arms, then transfers successful guided behaviors while preserving strategy accessibility.

Qwen2.5-7B/3B/1.5B-Instruct on DAPO-Math-17K
Model	In-Distribution Performance						Out-of-Distribution Performance
Model	AIME 24/25	AMC	MATH-500	Minerva	OmniMath	Avg@8	StrategyQA	GPQA	MMLU-Pro	Avg@8
Qwen2.5-7B-Instruct	10.2/6.9	51.8	74.4	35.5	24.6	33.9	63.0	27.3	23.8	38.0
+Vanilla GRPO	10.8/6.9	53.3	75.2	36.7	25.1	34.6	63.0	27.0	22.1	37.4
LUFFY	12.2/6.6	54.2	75.1	36.3	25.3	35.0	63.4	29.6	25.4	39.5
BREAD	12.6/7.7	54.2	74.4	35.6	24.6	34.9	63.0	30.0	24.2	39.1

Table 1: Main results across model scales, training dataset, and evaluation distributions.

Figure 3: Strategy-level effects of PSRA. (a) PSRA successfully reduces strategy concentration. (b) Its unguided policy shifts probability mass from the base-dominant strategy toward alternative strategies. (c) This produces higher rescue rates on initially failed problems across rollout budgets. (d) PSRA maintains a growing advantage over vanilla GRPO at larger inference budgets.

Figure 4: Training dynamics on Qwen2.5-7B-Instruct and Qwen2.5-1.5B-Instruct. Under the same rollout budget, PSRA achieves faster and more stable reward improvement, maintains higher policy entropy, and reduces dead-saturated groups. The dashed curve denotes the best strategy arm, illustrating the remaining strategy-conditioned performance frontier.

Figure 5: Strategy preservation ablation.

	Number of strategies				Bayes allocation budget
Variant	#4	#3	#2	#1	Rand	$B0$	$B5$	$B15$	$B20$
IDP Avg.	$36.3_{\pm.4}$	$36.1_{\pm.3}$	$34.6_{\pm.6}$	$35.3_{\pm.5}$	$35.6_{\pm.5}$	$35.9_{\pm.7}$	$36.3_{\pm.7}$	$36.9_{\pm.5}$	$\mathbf{37.2}_{\pm.6}$
ODP Avg.	$40.3_{\pm.3}$	$40.3_{\pm.4}$	$38.0_{\pm.8}$	$38.1_{\pm.6}$	$38.2_{\pm.6}$	$38.0_{\pm.5}$	$38.8_{\pm.5}$	$40.3_{\pm.9}$	$\mathbf{40.7}_{\pm.6}$

Table 2: Ablation of strategy-set size and Bayesian rollout allocation.

B

denotes reasoning budget.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	7B / DeepScaleR	7B / DAPO	3B / DAPO	1.5B / DAPO
Backbone	Qwen2.5-7B	Qwen2.5-7B	Qwen2.5-3B	Qwen2.5-1.5B
Training data	DeepScaleR-40K	DAPO-Math-17K	DAPO-Math-17K	DAPO-Math-17K
Training split	7B pass@8 mixed	7B pass@8 mixed	3B pass@8 mixed	1.5B pass@8 mixed
GPUs	$4\times$ GH200	$4\times$ GH200	$4\times$ GH200	$4\times$ GH200
Prompts / rank / step	32	8	8	8
Global prompts / step	128	32	32	32

Appendix

Table 3: Training configurations across model scales and training corpora. All PSRA-specific hyperparameters and per-prompt rollout budgets are shared across settings.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,103 papers

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
Jul 28, 2026Pixel Nomand, Elena Voss, Marcus Hale +1Reinforcement Learning RL for Language Model Reasoning
University of Wisconsin–Madison · University of Washington
Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR
May 15, 2026Chanuk Lee, Sangwoo Park, Minki Kang +1RL for Language Model Reasoning RL Exploration
KAIST
Decoupling Exploration from Optimization in RLVR
Oct 7, 2026Saif Punjwani, Micah GoldblumRL for Language Model Reasoning RL Exploration
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University

Jul 28, 2026J/K move · Enter open · S save

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes

University of Wisconsin–Madison · University of Washington

Reinforcement Learning RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).

Abstract

RL for Language Model Reasoning

Explore similar work

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVR

Decoupling Exploration from Optimization in RLVR