Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR

Yukang Zhu; Zhen Han

cs.CLSep 12, 2026

Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR

Authors: Yukang Zhu, Zhen Han

Organizations: Independent Researcher · Amazon

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most informative training frontier. We show that these otherwise-inert problems can be unlocked via teacher-guided curriculum learning: partial reasoning traces from a stronger model create a graded difficulty landscape, and a backward-chaining curriculum progressively withdraws guidance until the model solves problems unaided. Training on only 128 unsolvable problems matches or exceeds GRPO trained on a full 2,000-problem corpus (~16x data efficiency) on the nine-benchmark average for both base models, while substantially expanding the reasoning boundary measured by pass@k at large k. Furthermore, we identify a distribution-shift cost that is particularly acute in the unsolvable-only regime and propose Monotone Frontier Curriculum (MFC), a method that monotonically drives training toward unguided solving, consistently outperforming existing curriculum methods.

Figures & tables

Figure 1: Overview of our unsolvable-curriculum pipeline. Top path: standard GRPO trained on the full $2{,}000$ -problem mixed-difficulty corpus. Bottom path: we filter the same corpus to its $147$ unsolvable problems ( $\mathrm{pass}@64=0$ on the student), generate a step-structured trace with a stronger teacher, discard teacher failures ( $136$ remain), and select the $128$ shortest examples as the curriculum-training set. We compare three backward-chaining curricula: Mixture (R 3 ), AdaBack, and our MFC . MFC is designed for the unsolvable-only regime. With $\sim$ 16 $\times$ less source data, the curriculum methods match or exceed the GRPO baseline on the cross-benchmark average (right; Qwen3-1.7B, $9$ benchmarks).

Figure 2: Average

\mathrm{pass}@8

on the

147

initially unsolvable problems (

\mathrm{pass}@64=0

) across GRPO training checkpoints.

Guidance $\rho$	0	0.25	0.50	0.75	1.00
$\mathrm{pass}@8$	2.2%	34.0%	53.6%	72.3%	85.4%

Table 1:

\mathrm{pass}@8

of Qwen3-1.7B on the

147

unsolvable problems at varying guidance levels

\rho

(averaged over

4

independent evaluations).

Figure 3: Unguided training mass

M_{0}^{(t)}

of AdaBack on the

128

-problem unsolvable corpus (Qwen3-1.7B, the same run in § 4 ).

M_{0}^{(t)}

rises initially but plateaus once per-sample intervals settle in the learnable band.

		In-domain		Math OOD					Non-math OOD
Base Model	Method	OR1-200	Uns-22	MATH-500	AMC23	AIME24	AIME25	AIME26	SciBench	GPQA-D	Avg.
Qwen3-1.7B	Base	47.5	0.6	83.5	62.8	23.2	22.6	22.1	34.1	34.9	36.8
	GRPO	68.6	23.9	89.5	73.8	26.7	26.3	21.3	34.5	33.8	44.3
	Mixture	66.9	25.6	88.2	72.2	29.8	25.8	27.1	36.3	35.0	45.2
	AdaBack	67.2	29.5	89.1	72.5	30.0	29.2	26.3	34.1	34.1	45.8
	MFC	68.2	31.8	89.1	75.0	32.5	28.8	27.2	37.1	36.1	47.3

Table 2: Main result.

\mathrm{pass}@1

(%, average of

8

completions at temperature

1

) on nine benchmarks for two base models. Bold marks the column-wise best and underline the column-wise second-best, both computed within each base-model block.

Figure 4: Reasoning boundary measured by

\mathrm{pass}@k

on the union of AIME24/25/26 (

90

problems),

k\in\{1,2,4,8,16,32,64,128,256\}

on a

\log_{2}

axis. Left: Qwen3-1.7B; right: Qwen3-0.6B. The unsolvable-curriculum methods (Mixture, AdaBack, MFC) widen the gap over both Base and GRPO at large

k

, with MFC uniformly best.

Figure 5: Curriculum dynamics of MFC vs. AdaBack on Uns-128 (Qwen3-1.7B). Top: unguided training mass

M_{0}^{(t)}

; bottom: average guidance level used per step. MFC continues to drive training toward the unguided regime throughout the

1{,}200

-step run, whereas AdaBack largely plateaus after step

\sim

800.

Method	Avg. (9 benchmarks)
Base	36.8
SFT (full trace)	39.8
AdaBack	45.8
MFC	47.3

Table 3: Cross-benchmark average

\mathrm{pass}@1

(%) on Qwen3-1.7B. SFT uses the same Uns-128 corpus and full teacher traces; curriculum RL methods use the same data under adaptive guidance. Per-benchmark breakdown in Appendix E .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Rollouts per prompt ( $n$ )	$8$
Training batch size	$128$
PPO mini-batch size	$128$ (full update)
Rollout temperature	$1.0$ (top- $p$ $1.0$ )
Max prompt length	$800$ (GRPO 2k) / $2{,}500$ (hint)
Max response length	$8{,}192$

Appendix

Table 4: Shared GRPO training core, identical across all four trained algorithms and both base models in § 4 .

Method	Parameter	Value
GRPO	training corpus	OR1-2k ( $2{,}000$ problems)
	teacher guidance	none ( $\rho\equiv 0$ )
Mixture	training corpus	Uns-128 expansion ( $640$ rows)
	$\rho$ -level grid	$\{0,0.25,0.5,$ $0.75,1\}$
AdaBack	training corpus	Uns-128
	success threshold $\tau$	$0.5$

Appendix

Table 5: Method-specific curriculum hyperparameters. The three teacher-guided methods all use hint-mode guidance with the discretization of § 3.1 .

Figure 6: Curriculum dynamics of MFC vs. AdaBack on Uns-128 (Qwen3-0.6B). Left: unguided training mass

M_{0}^{(t)}

; right: average guidance level. The qualitative separation is identical to the 1.7B results (Figure 5 ).

Figure 7: Training reward (mean per batch) across

1{,}200

steps for all four methods on both base models. Note that training reward is not directly comparable across methods because curriculum mechanisms present samples at different effective difficulty levels; see text for interpretation.

Figure 8: Validation reward on OR1-200 (no guidance,

\rho=0

) evaluated every

10

steps. This is the fair comparison of unguided capability across methods. All methods converge by step

1{,}200

; the curriculum methods (especially MFC) rise faster in the first

\sim

400 steps.

Figure 9: SFT training NLL loss over

50

epochs on Uns-128 (Qwen3-1.7B). The dashed line marks the selected checkpoint at step

40

where loss has effectively converged.

Method	OR1-200	Uns-22	MATH-500	AMC23	AIME24	AIME25	AIME26	SciBench	GPQA-D	Avg.
SFT	54.5	13.6	83.4	61.9	26.7	26.7	19.2	36.0	36.5	39.8

Appendix

Table 6: Per-benchmark

\mathrm{pass}@1

(%) for the SFT baseline on Qwen3-1.7B. Compare with Base (

36.8

avg) and MFC (

47.3

avg) in Table 2 .

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards
May 3, 2026Rudray Dave, Vedang Dubey, Smit Deoghare +1Reinforcement Learning RL for Language Model Reasoning
Department of Artificial Intelligence Sardar Vallabhbhai National Institute of Technology Surat, India
The Unlearnability Phenomenon in RLVR for Language Models
May 16, 2026Yulin Chen, He He, Chen ZhaoReinforcement Learning RL for Language Model Reasoning
1New York University · 2NYU Shanghai
On the Emergence of Implicit Curriculum in RLVR Learning Dynamics
Feb 16, 2026Yu Huang, Zixin Wen, Yuejie Chi +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Department of Statistics and Data Science, Wharton School, University of Pennsylvania · Machine Learning Department, Carnegie Mellon University · Department of Statistics and Data Science, Yale University +1

May 3, 2026J/K move · Enter open · S save

Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards

Rudray Dave, Vedang Dubey, Smit Deoghare, Sudhakar Mishra

Department of Artificial Intelligence Sardar Vallabhbhai National Institute of Technology Surat, India

Reinforcement Learning RL for Language Model Reasoning Curriculum RL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Recently, Reinforcement Learning from Verifiable Rewards (RLVR) has been established as a highly effective technique for augmenting the math reasoning skills of Large Language Models (LLMs) based on a single instance. Current state-of-the-art 1-shot RLVR models adopt heuristics for selecting instances, mostly based on historical variance in rewards, which we find to be inherently misleading as a measure of transferability value. In this paper, we propose a Selector-Guided Autonomous Curriculum (SGAC) approach, which employs a learnable selector model on a multi-dimensional feature space consisting of success probability, reward variance, output disagreement (entropy), and semantic difficulty level, instead of the static reward variance heuristic. In our empirical evaluation on pools of candidate problems, we observed that output disagreement, rather than reward variance, is the strongest predictor of reasoning gains in subsequent iterations. Leveraging this finding, we develop an autonomous curriculum algorithm for dynamically siphoning candidate problems from a large pool, ranking them by the learned selector, and running micro-bursts of 1-shot GRPO. Our framework is evaluated using the Hendrycks MATH benchmark, with the Qwen2.5-Math-1.5B model serving as the baseline. Our framework obtains an accuracy of 68.0% on the hold-out dataset, which is better than the accuracy obtained from the state-of-the-art model, 64.0%, as well as the 1-shot RLVR checkpoint proposed by Wang et al., which achieved an accuracy of 66.0%. The results confirm that entropy-based intelligent data curation leads to strict reasoning improvement over static training methods, particularly in severely limited data conditions.

Abstract

RL for Language Model Reasoning

Explore similar work

Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards

The Unlearnability Phenomenon in RLVR for Language Models

On the Emergence of Implicit Curriculum in RLVR Learning Dynamics