Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning

Ilia Mahrooghi; Aryo Lotfi; Emmanuel Abbe

cs.LGFeb 16, 2026

Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning

Authors: Ilia Mahrooghi, Aryo Lotfi, Emmanuel Abbe

Organizations: EPFL · Apple

Abstract

Reinforcement learning has emerged as a powerful paradigm for unlocking reasoning capabilities in language models. However, relying on sparse rewards makes this process highly sample-inefficient, as models must navigate vast search spaces with minimal feedback. While classic curriculum learning aims to mitigate this by ordering data based on complexity, prior works have primarily targeted small datasets and do not directly transfer to the large-scale settings typical of modern language model training. Furthermore, the right ordering for a specific model is often unclear. To address this, we propose Goldilocks, an adaptive data-selection strategy that uses a Selector network to predict the standard deviation of rewards across the model's rollouts for each candidate question. The Selector prioritizes questions with high predicted reward variability, corresponding to questions that are neither too easy nor too hard for the model's current capabilities (Goldilocks principle), while training the model with GRPO. By leveraging the model's performance on seen samples, the Selector continuously adapts to the model's evolving abilities. Across the OpenMathReasoning and Polaris datasets, Goldilocks consistently improves over standard GRPO, requiring up to 78% fewer optimization steps to reach the corresponding GRPO performance.

Figures & tables

Figure 1: Overview of the Goldilocks Framework. The continuous training loop.

Model	Eval. Step	GRPO	PCL	Goldilocks	PCL	Goldilocks
		Accuracy (%)			Step Reduction
OLMo	40,000	16.02	14.47	20.01	$\times$	64.0%
Qwen2.5-1.5B	13,400	40.99	37.46	44.56	$\times$	28.4%
Qwen3-4B	21,000	54.00	53.86	56.67	$\times$	41.9%
Phi (4B)	16,000	40.12	38.39	44.27	$\times$	48.8%
Qwen2.5-7B	17,000	48.27	50.63	52.61	32.9%	49.4%

Table 1: OpenMathReasoning Results. Validation accuracy (%) is averaged over the final five evaluation points at each model’s respective training horizon. Step Reduction reports the relative reduction in optimization steps required for each method to reach GRPO’s final accuracy, using five-point rolling averages. A

\times

indicates that the method does not reach GRPO’s final accuracy within the available training horizon.

Figure 2: Long-run reasoning training. Held-out accuracy on OpenMathReasoning compares curriculum benefits across a large question pool.

Model	End Step	GRPO	PCL	MoPPS	Goldilocks	PCL	MoPPS	Goldilocks
		Accuracy (%)				Step Reduction
OLMo	29,000	21.15	18.65	20.04	23.12	$\times$	$\times$	51.7%
Qwen2.5-1.5B	6,000	21.52	19.85	21.56	22.26	$\times$	23.3%	40.0%
Qwen3-4B	10,000	31.32	30.08	30.95	31.60	$\times$	$\times$	26.0%
Phi (4B)	10,000	21.35	23.39	22.65	25.87	48.0%	38.0%	78.0%
Qwen2.5-7B	17,400	33.63	33.94	38.89	35.95	16.1%	58.6%	42.5%

Table 2: Training on the smaller Polaris dataset. Validation accuracy (%) averaged over the final five evaluation points at a common end step for each model. All methods are truncated at the shortest run’s end step. The smaller training pool leads to more frequent question revisits, providing a favorable setting for history-based selection methods such as MoPPS. Step Reduction reports the relative reduction in optimization steps required for each method to reach GRPO’s final accuracy, using five-point rolling averages. A

\times

indicates that the method does not reach GRPO’s final accuracy within the available training horizon.

Panel of Figure 3 — Figure 3: Training dynamics on Qwen2.5-7B.

Figure 4: Seed robustness on Qwen2.5-1.5B. GRPO and Goldilocks across four model seeds. Solid lines represent the mean performance, while the shaded regions show one standard deviation of the raw validation accuracies across seeds at each evaluation step.

			Accuracy (%)		Efficiency
Group	Setting	Steps	GRPO	Goldilocks	Step Red.
Main Results	Standard	13,400	40.99	44.56	28.4%
Objective Robustness	DAPO Loss	13,400	39.50	43.66	38.8%
Objective Robustness	Entropy Reg. $\beta=3\times 10^{-4}$	13,400	40.28	44.24	29.9%
Seeds	Four seeds	13,400	$40.33\pm 0.58$	$\mathbf{44.46\pm 0.30}$	32.8%
Selection Ablations	$\epsilon=0$	13,400	40.99	44.26	31.3%

Table 3: Ablations on OpenMathReasoning. Validation accuracy (%) with Qwen2.5-1.5B, averaged over the final five evaluations at the listed training horizon, shared by all runs in each comparison. The reference configuration uses

G=16

; Goldilocks additionally uses

K=8

and

\epsilon=0.2

. Seed results are mean

\pm

sample standard deviation across four seeds; their step reduction is computed from the mean curves. Step Red. reports the reduction in optimization steps required for Goldilocks to reach GRPO’s final rolling-average accuracy at the same horizon.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Batch-size search space	256, 512, 768, 1,024, 2,048
Learning-rate search space	$10^{-5}$ , $10^{-6}$ , $3.33\times 10^{-6}$ , $6.66\times 10^{-6}$
Selection criterion	Validation accuracy
Generations per question $G$	16

Appendix

Table 4: GRPO hyperparameter search and shared Model configuration. Batch size and learning rate are tuned separately for each Model–dataset pair. Batch sizes count generated completions per optimization step.

	OpenMathReasoning		Polaris
Model	Batch size	Learning rate	Batch size	Learning rate
OLMo2-1B	768	$10^{-5}$	1,024	$10^{-5}$
Qwen2.5-1.5B	1,024	$10^{-5}$	1,024	$10^{-5}$
Qwen3-4B-Base	256	$3.33\times 10^{-6}$	256	$3.33\times 10^{-6}$
Phi-4-mini-Instruct	256	$10^{-5}$	256	$10^{-5}$
Qwen2.5-7B	256	$3.33\times 10^{-6}$	256	$3.33\times 10^{-6}$

Appendix

Table 5: Selected GRPO Model hyperparameters. Batch size is the total number of completions per optimization step across all eight Model GPUs. All compared methods use the listed Model learning rate; Goldilocks and PCL use

7/8

of the listed batch size when one GPU is reserved for the Selector.

Setting	Value
Backbone	Qwen3-0.6B
Representation	Masked mean of final hidden states
Output	Scaled sigmoid in $[0,1/2]$
Adaptation	Full-parameter fine-tuning
Optimizer / learning rate	AdamW / $10^{-5}$
Selector minibatch size	32

Appendix

Table 6: Goldilocks Selector configuration.

Figure 5: Accuracy versus wall-clock time. Held-out accuracy of Qwen2.5-7B on OpenMathReasoning throughout training for GRPO, PCL, and Goldilocks.

Figure 6: Accuracy versus generated-token budget. Held-out accuracy of Qwen2.5-7B on OpenMathReasoning as a function of cumulative Model-generated tokens for GRPO, PCL, and Goldilocks.

Figure 7: Selector prediction error on OpenMathReasoning. Absolute error between predicted and realized reward-variability targets throughout training. The x-axis reports Selector optimization steps; each Model optimization step corresponds to four Selector steps.

Figure 8: Distribution of Goldilocks Selector scores. Mean predicted utility with

\pm

one standard deviation over Selector optimization steps.

Panel of Figure 9 — Figure 9: Optimization-objective ablation. GRPO and Goldilocks under standard GRPO, DAPO, and entropy-regularized objectives.

Panel of Figure 10 — Figure 10: Exploration-probability ablation. Training dynamics for $\epsilon\in\{0,0.2,0.5\}$ .

Panel of Figure 11 — Figure 11: Candidate-pool size ablation. Training dynamics for $K\in\{8,16,32\}$ .

Panel of Figure 12 — Figure 12: Rollout-group-size ablation. GRPO and Goldilocks with $G\in\{8,16,32\}$ at a fixed global batch size.

Figure 13: Seed robustness. Validation accuracy for GRPO and Goldilocks across four Model seeds.

Panel of Figure 14 — Figure 14: OLMo2-1B on OpenMathReasoning. Validation performance and training dynamics throughout optimization.

Panel of Figure 15 — Figure 15: Qwen2.5-1.5B on OpenMathReasoning. Validation performance and training dynamics throughout optimization.

Panel of Figure 16 — Figure 16: Qwen3-4B on OpenMathReasoning. Validation performance and training dynamics throughout optimization.

Panel of Figure 17 — Figure 17: Phi-4-mini-Instruct on OpenMathReasoning. Validation performance and training dynamics throughout optimization.

Panel of Figure 18 — Figure 18: Qwen2.5-7B on OpenMathReasoning. Validation performance and training dynamics throughout optimization.

Panel of Figure 19 — Figure 19: OLMo2-1B on Polaris. Validation performance and training dynamics throughout optimization.

Panel of Figure 20 — Figure 20: Qwen2.5-1.5B on Polaris. Validation performance and training dynamics throughout optimization.

Panel of Figure 21 — Figure 21: Qwen3-4B on Polaris. Validation performance and training dynamics throughout optimization.

Panel of Figure 22 — Figure 22: Phi-4-mini-Instruct on Polaris. Validation performance and training dynamics throughout optimization.

Panel of Figure 23 — Figure 23: Qwen2.5-7B on Polaris. Validation performance and training dynamics throughout optimization.

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
Oct 5, 2026Hoang Phan, Minh Pham, Chau Pham +3RL for Language Model Reasoning LLM-Guided RL
New York University · University at Buffalo · Monash University
Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1RL for Language Model Reasoning Efficient Language Model Reasoning
University of Southern California · 2DEVCOM ARL
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model Reasoning Policy Optimization
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney

Oct 5, 2026J/K move · Enter open · S save

Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

Hoang Phan, Minh Pham, Chau Pham, Chinmay Hegde+2

New York University · University at Buffalo · Monash University

RL for Language Model Reasoning LLM-Guided RL Sparse-Reward RL RL for VLMs

Open paper AskPDF arXiv

On-policy reinforcement learning has become a central paradigm for improving the reasoning abilities of large language models. However, its effectiveness is often limited by reward sparsity: when a model fails to discover correct trajectories for difficult problems, the optimization process receives little useful signal and may stagnate. Existing approaches mitigate this issue by incorporating off-policy demonstrations, expert traces, or model-generated solutions, but they typically require the auxiliary data to match the format of the reinforcement-learning task, often relying on rejection sampling from stronger models to obtain suitable training trajectories. We introduce Rationale-Guided Policy Optimization (RGPO), a framework that adaptively leverages ground-truth rationale information according to the model's current capability while preserving its freedom to explore. Rather than treating reference solutions as fixed imitation targets, RGPO uses them as temporary scaffolds: rationales help the model generate improved responses, after which only higher-reward, model-generated solutions are transferred back to the original unguided setting. This design allows training to exploit available ground-truth information without requiring off-policy data to follow the same format as the RL task. Across both language-only and vision-language reasoning settings, RGPO consistently improves performance over RLVR baselines, and ablation studies show that adaptive rationale guidance is a key contributor to these gains. These results suggest that RGPO offers a practical and general approach for reducing reward sparsity, stabilizing reinforcement learning, and improving reasoning performance in both text-only and multimodal models.

Abstract

RL for Language Model Reasoning

Explore similar work

Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning