Frontier Learning: Training LLM Reasoners at the Edge of Capability

Robin Faro; Shyam Sundhar Ramesh; Ilija Bogunovic; Aurelien Lucchi

cs.LGSep 28, 2026

Frontier Learning: Training LLM Reasoners at the Edge of Capability

Authors: Robin Faro, Shyam Sundhar Ramesh, Ilija Bogunovic, Aurelien Lucchi

Organizations: University College London London, UK · University of Basel Basel, Switzerland

Abstract

Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

Figures & tables

Figure 1: Frontier learning with procedural data generation. Levels vary along task-specific difficulty dimensions, here abstracted as structural complexity and reasoning depth. As the model improves, its frontier of capabilities shifts outward; frontier learning uses procedural generation to continually construct training data near this moving region.

Figure 2: One iteration of frontier learning with GRPO. At each step, the algorithm initially constructs the level batch with a mix of exploratory levels and high-regret frontier levels sampled from the buffer. A level sampled from the buffer is locally mutated with probability $p_{\mathrm{mut}}(\ell)$ , by perturbing its attributes in order to evaluate and potentially extend the policy’s capability frontier. The resulting level batch is then passed through the generator to construct the training batch of problems for the GRPO update. The resulting rewards and regret estimates are then used for future level selection.

	Puzzle		Math
Task	Countdown	Sokoban	Dec. Arith.
Model	Qwen3-4B-Base	Qwen3-4B	Qwen3-4B-Base
Method
Domain Rand.	44.7 ±1.7	37.6 ±2.6	31.0 ±2.8
Uniform	44.2 ±4.0	39.5 ±1.3	28.0 ±2.6
SEC	40.1 ±8.1	34.3 ±6.6	29.1 ±4.1

Table 1: Accuracy (%) on the fixed anchor evaluation set after 200 training steps. Best result per task in bold ; second-best underlined .

Method	Accuracy
Domain Rand.	23.4 ±4.7
Uniform	31.9 ±1.5
SEC	33.3 ±1.2
PLR	31.2 ±1.9
ACE-GRPO	33.2 ±2.1
DAPO	25.4 ±5.9

Table 2: Accuracy (%) on Dice for the baselines, Frontier Learning and its components ablation. Best result in bold ; second-best within each block underlined .

Task	Countdown	Largest Island
Model	Llama-3.2-3B-Instruct	Olmo3-7B-Instruct
Method
Domain Rand.	33.8 ±1.9	45.2 ±1.8
Uniform	25.3 ±4.7	45.3 ±3.4
SEC	29.5 ±1.6	48.3 ±4.7
PLR	29.0 ±1.8	52.0 ±5.3

Table 3: Accuracy (%) on Countdown and Largest Island . Best result per task in bold ; second-best underlined .

Figure 3: Per-difficulty accuracy (%) on Largest Island with Olmo3-7B-Instruct over 500 GRPO steps. All methods saturate on Easy levels, but only frontier learning keeps improving on Medium and Hard levels throughout training, while the baselines plateau after roughly 200 steps.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Example Sokoban levels at the two extremes of the anchor ladder. Left: L1 — grid $6{\times}6$ , 1 box, depth $\leq 12$ . Right: L7 — grid $10{\times}10$ , 6 boxes, depth $\leq 80$ . Colours: dark grey = wall, beige = floor, brown = box, yellow = goal, green = box on goal.

Task	Level	Bin	Config
Countdown	1	Easy	num =3	mv =20	mt =80
	2	Easy	3	40	150
	3	Easy	3–4	60	250
	4	Medium	4	80	350
	5	Medium	4–5	100	500
	6	Medium	5	120	800

Appendix

Table 4: Full anchor-level specifications, grouped into Easy/Medium/Hard (and Extra Hard for Dice) difficulty bins used throughout the paper. Countdown: num = num_numbers , mv = max_value , mt = max_target . Sokoban: grid is square; d = max_depth . Decimal: dp = decimal_places , pr = precision , t = terms . Largest Island: grid = rows

\times

cols ; isl = num_islands ; sz = island_size .

Hyperparameter	Symbol	Value
Learning rate		$10^{-6}$
Train batch size		64
Rollouts per problem	$n_{r}$	8
PPO mini-batch size		16
PPO micro-batch size per GPU		8
Max problem length (tokens)		1024

Appendix

Table 5: GRPO training hyperparameters (shared across all methods and tasks).

Buffer and sampling
Hyperparameter	Symbol	Value
Levels per step	$n_{\ell}$	4
Problems per level	$n_{p}$	16
Buffer capacity	$B_{\max}$	100
Seed levels		8
Seed sampling mode		grid

Appendix

Table 6: Frontier Learning curriculum hyperparameters.

Figure 5: Accuracy (%) on the fixed anchor evaluation sets over 200 GRPO steps for Countdown , Sokoban , and Decimal Arithmetic .

Figure 6: Validation accuracy across the three difficulty bins throughout the full training horizon for Countdown , Sokoban , and Decimal Arithmetic .

Figure 7: Final validation accuracy per difficulty bin for Countdown , Sokoban , and Decimal Arithmetic .

Figure 8: Long-horizon Dice learning curves over 500 GRPO steps. DAPO is omitted from the step-based curves because it is matched by total rollout budget rather than optimizer updates.

Figure 9: Validation accuracy across the four difficulty bins throughout the full training horizon for Dice .

Figure 10: Final validation accuracy per difficulty bin for Dice

Variant	Accuracy (%)
Default frontier learning	$39.2\pm 1.3$
$\lambda_{s}=0.02$	$40.7\pm 1.1$
Informative interval $=[0.10,0.90]$	$40.5\pm 0.8$
Informative interval $=[0.02,0.98]$	$39.8\pm 1.0$
$\lambda_{s}=0$	$38.8\pm 1.4$

Appendix

Table 7: Hyperparameter sensitivity on Countdown using Llama-3.2-3B-Instruct.

Figure 11: Learning-rate sweep on Dice (seed 42) for Uniform and SEC, the non-regret-based baseline and the best-performing baseline in Table 2 , respectively. † Denotes the learning rate used for the main paper’s results ( $1\times 10^{-6}$ ). No swept value raises either baseline’s final accuracy above $\approx$ 35%, indicating that an under-tuned learning rate does not explain the gap to frontier learning.

Method	Runtime (h)	Response length (tokens)	Accuracy (%)
Domain Rand.	15.4 $\pm$ 2.0	3838 $\pm$ 33	37.6 $\pm$ 2.6
PLR	14.8 $\pm$ 1.8	3698 $\pm$ 103	41.1 $\pm$ 3.9
Uniform	15.1 $\pm$ 1.8	3761 $\pm$ 51	39.5 $\pm$ 1.3
SEC	14.8 $\pm$ 1.8	3709 $\pm$ 81	34.3 $\pm$ 6.6
Frontier Learning (ours)	14.2 $\pm$ 1.7	3529 $\pm$ 49	45.2 $\pm$ 1.9

Appendix

Table 8: End-to-end compute accounting on Sokoban (200 steps).

Method	Runtime (h)	Response length (tokens)	Accuracy (%)
Domain Rand.	17.4 $\pm$ 0.1	1538 $\pm$ 14	44.7 $\pm$ 1.7
PLR	11.3 $\pm$ 0.3	1521 $\pm$ 44	44.3 $\pm$ 2.9
Uniform	11.8 $\pm$ 0.3	1546 $\pm$ 50	44.2 $\pm$ 4.0
SEC	11.2 $\pm$ 0.4	1446 $\pm$ 65	40.1 $\pm$ 8.1
Frontier Learning (ours)	11.7 $\pm$ 0.5	1269 $\pm$ 28	50.9 $\pm$ 0.5

Appendix

Table 9: End-to-end compute accounting on Countdown (200 steps).

Method	Runtime (h)	Response length (tokens)	Accuracy (%)
Domain Rand.	4.3 $\pm$ 1.1	837 $\pm$ 31	31.0 $\pm$ 2.8
PLR	6.0 $\pm$ 1.1	677 $\pm$ 30	27.5 $\pm$ 2.9
Uniform	5.7 $\pm$ 1.0	642 $\pm$ 33	28.0 $\pm$ 2.6
SEC	6.1 $\pm$ 1.2	665 $\pm$ 54	29.1 $\pm$ 4.1
Frontier Learning (ours)	4.2 $\pm$ 1.1	758 $\pm$ 50	34.7 $\pm$ 3.6

Appendix

Table 10: End-to-end compute accounting on Decimal Arithmetic (200 steps).

Baselines
Method	Runtime (h)	Response length (tokens)	Accuracy (%)
Domain Rand.	19.4 $\pm$ 1.9	907 $\pm$ 104	23.4 $\pm$ 4.7
PLR	18.7 $\pm$ 1.1	877 $\pm$ 65	31.2 $\pm$ 1.9
Uniform	20.8 $\pm$ 1.9	994 $\pm$ 108	31.9 $\pm$ 1.5
SEC	18.9 $\pm$ 2.3	866 $\pm$ 129	33.3 $\pm$ 1.2
ACE-GRPO	22.0 $\pm$ 0.8	1063 $\pm$ 45	33.2 $\pm$ 2.1

Appendix

Table 11: End-to-end compute accounting on Dice (500 steps).

Method	Accuracy (%)
DR	25.4 $\pm$ 3.7
PLR	31.2 $\pm$ 1.9
Uniform	32.2 $\pm$ 1.7
SEC	33.5 $\pm$ 1.6
AceGRPO	31.8 $\pm$ 3.3
Frontier Learning (ours)	68.0 $\pm$ 5.5

Appendix

Table 12: Dice performance at matched PLR compute time (18.7 hours).

Method	Runtime (h)	Response length (tokens)	Accuracy (%)
Domain Rand.	63.2 $\pm$ 0.3	1578 $\pm$ 29	45.2 $\pm$ 1.8
PLR	44.3 $\pm$ 6.0	1276 $\pm$ 96	52.0 $\pm$ 5.3
Uniform	53.1 $\pm$ 2.5	1132 $\pm$ 125	45.3 $\pm$ 3.4
SEC	44.4 $\pm$ 7.1	1218 $\pm$ 102	48.3 $\pm$ 4.7
ACE-GRPO	43.8 $\pm$ 4.7	1243 $\pm$ 106	45.3 $\pm$ 5.5
frontier learning (ours)	52.4 $\pm$ 0.7	1498 $\pm$ 6	73.7 $\pm$ 3.3

Appendix

Table 13: End-to-end compute accounting on Largest Island (500 steps, Olmo-3-7B-Instruct).

Explore similar work

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
May 7, 2026Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger +1LLM Reasoning Strategies Reasoning Skills
University of Southern California · 2DEVCOM ARL
Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
Jul 29, 2025Carel van Niekerk, Renato Vukovic, Benjamin Ruppik +3Large Language Model Reinforcement Learning LLM Reasoning Strategies
Heinrich Heine University Düsseldorf, Germany
Understanding Reasoning from Pretraining to Post-Training
Jul 17, 2026Jingyan Shen, Ang Li, Salman Rahman +4LLM Reasoning Strategies Large Language Model Pretraining
New York University · Columbia University · University of California, Los Angeles +2

May 7, 2026, cs.CLJ/K move · Enter open · S save

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna

University of Southern California · 2DEVCOM ARL

LLM Reasoning Strategies Reasoning Skills

Open paper AskPDF arXiv

Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.

Abstract

Explore similar work

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Post-Training Large Language Models via Reinforcement Learning from Self-Feedback

Understanding Reasoning from Pretraining to Post-Training