Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of LLMs, yet a fundamental limitation remains: models cannot learn from problems that are too difficult to solve under their current policy, as these yield no meaningful reward signal. We propose a simple yet effective solution based on task reformulation. We transform challenging open-ended problems into cognitively simpler variants -- such as multiple-choice and cloze formats -- that preserve the original answer while reducing the effective search space and providing denser learning signals. These reformulations span a spectrum from discriminative to generative tasks, which we exploit to bootstrap learning: models first learn from structured, easier formats, and this knowledge transfers back to improve performance on the original open-ended problems. Building on this insight, we introduce Cog-DRIFT, a framework that constructs reformulated variants and organizes them into an adaptive curriculum based on difficulty. Training progresses from easier to harder formats, enabling the model to learn from problems that previously yielded zero signal under standard RL post-training. Cog-DRIFT not only improves on the originally unsolvable hard problems (absolute +10.11% for Qwen and +8.64% for Llama) but also generalizes well to other held-out datasets. Across 2 models and 6 reasoning benchmarks, our method consistently outperforms standard GRPO and strong guided-exploration baselines. On average, Cog-DRIFT shows +4.72% (Qwen) and +3.23% (Llama) improvements over the second-best baseline. We further show that Cog-DRIFT improves pass@k at test time, and the curriculum improves sample efficiency. Overall, our results highlight task reformulation and curriculum learning as an effective paradigm for overcoming the exploration barrier in LLM post-training.
Figures & tables
Figure 1: (A) If a problem is too hard (e.g., pass@64=0), the model cannot learn from it. Reformulation into an MCQ or a cloze task can effectively reduce cognitive load, as well as difficulty for the model. (B) We find that learning from cognitive-load-reduced tasks transfers back to the original hard questions (i.e., learning from MCQ/cloze also improves open-ended questions). (C) These improvements on the original hard questions also generalize to held-out datasets. Results in (B) and (C) are based on Llama3.2-3B-Instruct.
Figure 2: (a) Reformulating open-ended math problems into alternative formats consistently increases accuracy by easing structural constraints. (b) After Rejection Fine-Tuning (RFT), these performance gains successfully transfer back to the original open-ended problems used for training. (c) On the unseen MATH 500 benchmark, the model trained with an easier format incurs only a modest drop.
BMH
Omni-Hard
AIME24
AIME25
GPQA
Date
Avg.
Qwen3-4B-Instruct-2507
Zero-shot
0.0
15.67
46.92
41.71
23.57
87.71
35.93
Few-shot
0.0
16.31
44.78
37.98
26.12
88.37
35.59
RFT
4.32
15.13
40.06
34.88
23.04
85.11
33.76
GRPO
6.37
16.37
31.87
27.29
31.81
89.53
33.87
NuRL (Abstract)
7.73
16.03
44.81
36.44
24.72
87.48
36.20
Table 1: Performance comparison on six benchmarks. We bold the best and underline the second-best results. Cog-DRIFT achieves the strongest overall results, improving avg. accuracy by +4.72% (Qwen) and +3.23% (Llama). Crucially, it unlocks gains on BMH (0 → 10.11% on Qwen / 8.64% on Llama), where all the problems are collected with pass@64=0.
Figure 3: When training on hard open-ended problems and evaluating on AIME24, AIME25, and GPQA using Qwen, Cog-DRIFT generally achieves higher pass@k compared to both the base Qwen model and the GRPO-trained model, particularly as k increases.
Data Source
Omni
MATH
-
15.67
78.80
OEQ
14.16
76.20
4-MCQ
14.82
79.80
Cloze
16.15
80.20
4-MCQ + Cloze
18.36
82.60
OEQ + 4-MCQ
12.39
75.40
Table 2: Combining multiple reformulations (e.g., 4 Choice + Cloze, or Original + 4 Choice + Cloze) leads to the largest gains, suggesting that diversity of reformulations is critical for learning from hard problems .
Figure 4: Left: Instance-level curriculum adaptively reallocates samples from easier (MCQ) to harder (OEQ) reformulations based on per-instance accuracy, leading to improved sample efficiency and continued performance gains. Right: A static uniform mixture (always 25% for each format) shows stagnated performance improvement. Test accuracy is reported on open-ended questions from OmniMATH-Hard.
# Data
Qwen-MATH
Qwen-GPQA
Llama-MATH
Llama-GPQA
Original Dhard
8922
82.2
33.1
24.2
20.7
Binary Filter
5922
81.8
34.7
28.0
22.8
Silver Ans Filter
958
81.6
36.0
32.2
23.3
Table 3: Impact of data quality filtering on model performance. While the Silver Answer Filter is more selective and substantially reduces dataset size, it consistently improves performance for Llama and slightly for Qwen on GPQA. Overall, Llama is more sensitive to data quality, whereas Qwen is less affected by variations in data quality.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Llama
Qwen
num_train_samples
958
958
max_prompt_length
1024
1024
max_response_length
8192
8192
lr
1e-6
1e-6
clip_ratio_low
0.2
0.2
clip_ratio_high
0.28
0.28
Appendix
Table 4: Configurations and hyperparameters we use for training.
Sample Size
License
BigMath (Train) ( Albalak et al., 2025 )
958
Apache License
BigMath (Test) ( Albalak et al., 2025 )
192
Apache License
OmniMATH ( Gao et al., 2024 )
452
Apache License
AIME 2024 ( AIME, 2025 )
30
CC0
AIME 2025 ( AIME, 2025 )
30
CC0
MATH500 ( Lightman et al., 2023 ; Hendrycks et al., 2021 )
500
MIT License
Appendix
Table 5: The statistics and licenses of the datasets used in this study. Unless otherwise specified, like BigMath has a train split, all the datasets are used for evaluation.
Reinforcement learning from verifiable rewards (RLVR) has shown strong promise for LLM reasoning, but outcome-based RLVR remains inefficient on hard problems because correct final-answer rollouts are rare and sample-level credit assignment cannot use partial progress in failed attempts. We introduce SCRL (Subproblem Curriculum Reinforcement Learning), a curriculum RL framework that derives verifiable subproblems from reference reasoning chains and fixes the final subproblem as the original problem. This turns partial progress on hard problems into verifiable learning signals. Algorithmically, SCRL uses subproblem-level normalization, which normalizes rewards independently at each subproblem position and assigns the resulting advantages to the corresponding answer spans, enabling finer-grained credit assignment without external rubrics or reward models. Our analysis shows that subproblem curricula lift hard problems out of gradient dead zones, with larger relative gains as the original problem becomes harder. Across seven mathematical reasoning benchmarks, SCRL outperforms strong curriculum-learning baselines, improving average accuracy over GRPO by +4.1 points on Qwen3-4B-Base and +1.9 points on Qwen3-14B-Base. On AIME24, AIME25, and IMO-Bench, SCRL further improves pass@1 by +3.7 points and pass@64 by +4.6 points on Qwen3-4B-Base, indicating better exploration on hard reasoning problems.
Xitai Jiang, Zihan Tang, Wenze Lin +3
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · B Gao Huang
Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.
Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic +1
University College London London, UK · University of Basel Basel, Switzerland
RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model's current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.
Zhanke Zhou, Xiangyu Lu, Chentao Cao +4
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney