Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. Training-time scaffolds, such as strategies or hints added to the prompt during training, offer a natural way to turn this information into guidance, but their design and adaptation are largely left to humans and tailored to specific domains. We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. A frozen teacher periodically examines this experience, diagnoses bottlenecks, and designs scaffolds in two forms: (i) strategies shared across a class of tasks and (ii) hints for individual hard tasks, adjusting the content and amount of support as the student learns. The scaffolds are removed at evaluation. We apply the same framework to ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation. When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match. On GPU kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to 5.0 points. On math, our student surpasses QuestA's reported accuracy using approximately one-third as many student training steps. Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves, and that its revised scaffolds help the student learn more than keeping the previous scaffold or using no scaffold.
Figures & tables
Figure 1: A teacher LLM helps a student LLM learn in AutoScaffold. During RL training, the frozen teacher reads the student’s training experience and writes scaffolds into its training prompts. Scaffolds are removed at evaluation.
Figure 2: AutoScaffold: a teacher LLM designs and adapts scaffolds during RL training. (a) Unlike hand-designed scaffolding, AutoScaffold is automatic, adaptive, and unified. (b) Evaluation curves during training (success rate on ALFWorld, exact match on search-based QA), compared with GiGPO. (c) Gains over the RL baseline of each experiment, evaluated without scaffolds: success rate on ALFWorld, average exact match on search-based QA, average accuracy on math, and KernelBench correctness pass@3 averaged over Levels 1 to 3. Gray dots are prior scaffolding methods. All methods use the same student size as AutoScaffold, except Skill0, SkillRL and SkillC in ALFWorld using 7B models. (d) The teacher sets each problem’s solution-prefix ratio as instance-level scaffold: it first reveals more of the reference solutions and then withdraws support, whereas QuestA switches from 50% to 25% at a fixed step.
Method
Stage
What it changes
Granularity
Automatic
Adapts to student progress
No extra runs
Scaffold-free eval.
Domains
AI for automating model development
AHE ( Lin et al., 2026b )
inference
agent harness
–
✓
–
–
–
coding
autoresearch ( Karpathy, 2026 )
training
training code
–
✓
✗
✗
–
pretraining
Eureka ( Ma et al., 2024 )
training
reward function
–
✓
✗
✗
–
robotics
Trainee-to-Trainer ( Chen et al., 2026a )
training
environment
–
✓
✓
✓
–
multi-agent
LLMZero ( Fang et al., 2026 )
training
hyperparameters
–
✓
✓
∘
–
sci. QA, chem., music
Table 1: Positioning of AutoScaffold. Stage : whether the method changes how a model behaves at inference time or how it is trained. What it changes : the object that is designed or edited. Granularity : whether guidance is shared by a class of tasks, specific to one task, or both. Automatic : an AI system decides the changes, without hand-designed content or schedules ( ∘ : content is generated by a model, but its use follows hand-designed rules). Adapts to student progress : whether guidance or training settings are revised in response to the student’s current performance and observed failures ( ∘ : changes follow predefined rules or schedules). No extra runs : changes are decided and tested within a single training run, without training separate runs or branches ( ∘ : branches are trained from checkpoints). Scaffold-free eval. : the learner is evaluated without the guidance it received during training ( ∘ : some guidance is kept). Domains : the task types on which the method is evaluated (A = agentic, S = search QA, M = math, K = GPU kernels). –: not applicable.
Figure 3: Overview of AutoScaffold . (a) A class-level scaffold, injected with probability q[C] , and an instance-level scaffold are added to the student’s training prompt. On a hard task this turns all-failure groups into mixed groups that produce training signal; the RL algorithm is unchanged, and all scaffolds are removed at evaluation. (b) At the end of each block of K steps, a frozen teacher gathers evidence from the training log, proposes edits, validates class-level scaffolds on held-out tasks, and deploys the updated scaffold for the next block. Bottom: an actual update on ALFWorld.
Table 2: Task classes used for class-level scaffolds in each domain. Classes follow the categories provided by each training dataset; math has no such categories and forms a single class.
Method
Pick
Look
Clean
Heat
Cool
Pick2
Overall
Qwen2.5-1.5B-Instruct
5.9
5.5
3.3
9.7
4.2
0.0
4.1
Training-free methods
ReAct
17.4
20.5
15.7
6.2
7.7
2.0
12.8
Reflexion
35.3
22.2
21.7
13.6
19.4
3.7
21.8
RL baselines without scaffold
PPO
64.8
40.5
57.1
60.6
46.4
47.4
54.4
Table 3: Success rates (%) on ALFWorld with Qwen2.5-1.5B-Instruct. Overall for AutoScaffold is the unweighted mean over the six task types. Prompting and RL baseline results are from Feng et al. (2025) ; ours are mean ± standard deviation over 3 seeds. Gray rows are from Lin et al. (2026a) and use the larger Qwen2.5-7B-Instruct (reference only). Bold : best among 1.5B models.
Method
Single-Hop QA
Multi-Hop QA
Avg.
NQ †
TriviaQA ⋆
PopQA ⋆
HotpotQA †
2Wiki ⋆
MuSiQue ⋆
Bamboogle ⋆
RL baselines without scaffold
R1-Instruct
27.0
53.7
19.9
23.7
29.2
7.2
29.3
27.1
Search-R1
34.1
54.5
37.8
32.4
31.9
10.3
26.4
32.5
ZeroSearch
41.4
57.4
44.8
27.4
30.0
9.8
11.1
31.7
StepSearch
–
–
–
34.5
32.0
17.4
34.4
–
Table 4: Exact match (%) on search-based QA with Qwen2.5-3B-Instruct, trained on NQ and HotpotQA ( † : in-domain; ⋆ : out-of-domain). RL baseline results are from Feng et al. (2025) ; SkillRL and Skill0 are from their original papers ( Xia et al., 2026 ; Lu et al., 2026 ) ; ours are mean ± standard deviation over 3 seeds. Gray rows use the larger Qwen2.5-7B-Instruct (reference only). Bold : best among 3B models.
Method
AIME24
AIME25
AMC23
MATH-500
Minerva
Olympiad
Avg.
Qwen3-4B-Instruct
52.1
43.1
92.2
93.6
46.1
67.7
65.8
RL baselines without scaffold
SFT
14.4
22.1
55.2
78.9
38.1
40.2
41.5
GRPO
55.8
45.0
95.0
96.0
50.1
70.4
68.7
RL baselines with scaffold
LUFFY
42.3
36.0
86.4
91.1
48.2
59.4
60.6
Table 5: Accuracy (%) on math benchmarks with Qwen3-4B-Instruct. RL baseline results are from Liao et al. (2026) ; ours are mean ± standard deviation over 3 seeds. Avg.: mean over the six benchmarks. Bold / underline : best / second best.
Method
AIME24
AIME25
HMMT Feb 25
OlympiadBench
BRUMO25
Avg.
Training-free base models
DeepSeek-R1-Distill-1.5B
28.7
22.3
12.0
52.4
31.8
29.4
Qwen3-1.7B
48.3
36.8
22.2
56.1
44.1
41.5
Nemotron-1.5B
61.8
49.5
31.6
64.6
58.2
53.1
Larger models (reference only)
DeepSeek-R1-Distill-32B
72.6
51.8
33.0
65.0
68.0
58.1
Table 6: Accuracy (%) on math benchmarks with Nemotron-1.5B. DAPO and QuestA results are from Li et al. (2026a) . Gray: larger models (reference only). Bold : best among small models.
Method
Level 1
Level 2
Level 3
Correctness
Fast 1
Correctness
Fast 1
Correctness
Fast 1
Qwen3-8B-Cold-Start
51
14
43
16
32
8
StitchCUDA
59
17
50
20
32
13
AutoScaffold (ours)
63
19
54
25
33
14
Table 7: Pass@3 of Correctness and Fast 1 on the KernelBench test set. Fast 1 : the generated kernel is correct and faster than the PyTorch reference ( Ouyang et al., 2025 ) . Bold / underline : best / second best.
AutoScaffold
GiGPO
Teacher decision
Scaffold adaptation & injection
w/o tools & memory
w/o validation
Static
K=5
K=20
qmax=1
Success
93.9
86.7
85.3
79.8
78.6
90.1
90.4
86.0
Δ
–
− 7.2
− 8.6
− 14.1
− 15.3
− 3.8
− 3.5
− 7.9
Table 8: Ablations on ALFWorld (Qwen2.5-1.5B-Instruct), success rate (%) evaluated without scaffold. Static: Skill0’s published ALFWorld skill bank used as a fixed class-level scaffold. Δ : gap to full AutoScaffold .
Figure 4: One teacher update on ALFWorld (step 40). Top: the teacher queries tools and memory, then gathers evidence from class statistics, failed trajectories, and its own past decisions. It reads them and finds meaningful information for further diagnosis and edits. Bottom: it diagnoses a different failure in each class and responds with a matching edit: adding, rewriting, reducing, or omitting a scaffold.
Figure 5: Scaffold changes over training on ALFWorld. Left: scaffold-free success (top) and mean class-level injection probability (bottom), with training divided into three stages. Right: what the teacher changed in each stage (+ add, ∼ revise, − remove, ↑↓ raise or lower injection probability).
Figure 6: From each branching point, we train 20 more steps with the updated scaffold ( AutoScaffold , dark blue), with the previous scaffold kept fixed (orange), or with vanilla RL (gray). AutoScaffold gives the highest success at every branching point.
Figure 7: Solution-prefix ratios during math training (Nemotron-1.5B). Mean ratio across all training problems. AutoScaffold (red) adjusts each problem’s ratio using training evidence. QuestA (gray) switches from 50% to 25% at step 100.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Class
No scaffold
Current scaffold
Proposed scaffold
Outcome
Heat
6.7
30.0
40.0
adopted
Look
26.7
30.0
30.0
adopted
Pick2
13.3
–
10.0
dropped
Appendix
Table 9: Validation of the scaffolds proposed in this update: success rate (%) on 30 held-out games per class. Pick2 has no current scaffold.
Outcome in the current block
Selection
Change in rx
All-fail
0≤rx≤25
+20
All-fail
26≤rx≤49
+10
Mixed
rx≥1 , success rate >50%
−5
All-pass
rx≥1
−10
Appendix
Table 10: Prefix-ratio changes in this update. Ratios are capped at 50.
ALFWorld
Search QA
Math
KernelBench
Teacher wall-clock per update (min)
4
3
5
4
Validation wall-clock per update (min)
11
13
21
16
Total teacher API cost per run (USD)
2.35
3.13
5.57
2.76
Appendix
Table 11: Teacher cost per domain. Per-update values are means over all scaffold updates in a run.
Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the Exploration Potential Score(EPS), a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7% relative improvement in-domain and gains of 11.5% on MathVision and 11.1% on MMMU-Pro.
Yuanhao Yue, Qianli Ma, Chengyu Wang +3
Alibaba Cloud Computing · Shanghai Jiao Tong University · Fudan University +1
Reinforcement learning is critical to improving large reasoning models, but its success relies heavily on verifiable rewards (RLVR), making it hard to use in open-ended domains where correctness is ambiguous and cannot be verified. Moreover, reasoning trajectories remain largely unconstrained, and optimizing solely toward the final answer can favor early exploitation over generalization. In this work, we ask whether general reasoning ability can be improved by teaching models how to think (the structure of reasoning) rather than what to produce (the outcome of reasoning), and we extend traditional RLVR to open-ended settings. We introduce Structure-Aware Reinforcement Learning (SARL), a label-free framework that constructs per-response reasoning maps from intermediate thinking steps and rewards their reasoning topology. SARL shifts supervision from destination to path, encouraging reasoning trajectories that are both locally coherent and globally efficient. On verifiable math tasks, SARL outperforms prior label-free RL baselines and even exceeds RL methods with ground truth supervision, with average gains of +9.1% under PPO and +11.6% under GRPO across four math benchmarks, with particularly large improvements on AIME25 (+35.5% with PPO and +44.7% with GRPO). On non-verifiable open-ended tasks, SARL achieves average gains of +34.6% under PPO and +30.4% under GRPO on WildBench across five task categories, outperforming prior label-free RL methods and DPO, which relies on additional preference labels. Beyond strong performance, SARL exhibits substantially lower KL divergence and higher policy entropy, indicating more stable and exploratory training dynamics. Code and data are available at Code Link.
Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning, but scalar trajectory rewards offer limited guidance for token-level credit assignment. Existing self-distillation methods incorporate privileged teacher information through fixed strategies: direct distribution matching may destabilize successful behavior, while magnitude modulation provides limited corrective guidance after failure. We propose Hybrid Hindsight Self-Distillation (H2SD), which adapts both teacher context and update strategy to trajectory correctness. For successful trajectories, the teacher conditions on the verified student response and a rephrasing instruction to rescore the original tokens. These scores modulate reward-derived updates while preserving advantage signs. For failed trajectories, a verified reference hint guides corrective supervision through reverse KL distillation. Across Sudoku, Calcudoku, Arrow Maze, and mathematical reasoning with two Qwen3 models, H2SD achieves an overall average accuracy of 43.33%, outperforming the strongest baseline by 11.43 percentage points while maintaining stable optimization.
Qiye Cai, Yichuan Ma, Weitao Ma +6
Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +1