AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models

Zijian Zhang; Rizhen Hu; Jiangweizhi Peng; Shiyang Li; Sujay Bhatt; Sihan Zeng; Mingyi Hong

cs.LGNov 10, 2026

AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models

Authors: Zijian Zhang, Rizhen Hu, Jiangweizhi Peng, Shiyang Li, Sujay Bhatt, Sihan Zeng, Mingyi Hong

Organizations: University of Minnesota, Twin Cities · JPMorgan AI Research

Abstract

Reinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused. Training-time scaffolds, such as strategies or hints added to the prompt during training, offer a natural way to turn this information into guidance, but their design and adaptation are largely left to humans and tailored to specific domains. We introduce AutoScaffold, a unified AI-for-AI framework that turns a student's training experience into adaptive scaffolds, with minimal domain-specific information and no human intervention during training. A frozen teacher periodically examines this experience, diagnoses bottlenecks, and designs scaffolds in two forms: (i) strategies shared across a class of tasks and (ii) hints for individual hard tasks, adjusting the content and amount of support as the student learns. The scaffolds are removed at evaluation. We apply the same framework to ALFWorld, search-based QA, mathematical reasoning, and GPU kernel generation. When evaluated without scaffolds, students trained with AutoScaffold outperform those trained with the corresponding RL baselines in all four domains, with gains of 7.2 points in ALFWorld success and 4.0 points in average QA exact match. On GPU kernel generation, AutoScaffold increases the fraction of tasks with a correct kernel that runs faster than the PyTorch reference by up to 5.0 points. On math, our student surpasses QuestA's reported accuracy using approximately one-third as many student training steps. Analyses show that the teacher identifies a different failure in each task class and shifts its scaffolds from step-by-step workflows to targeted fixes as the student improves, and that its revised scaffolds help the student learn more than keeping the previous scaffold or using no scaffold.

Figures & tables

Figure 1: A teacher LLM helps a student LLM learn in AutoScaffold. During RL training, the frozen teacher reads the student’s training experience and writes scaffolds into its training prompts. Scaffolds are removed at evaluation.

Figure 2: AutoScaffold: a teacher LLM designs and adapts scaffolds during RL training. (a) Unlike hand-designed scaffolding, AutoScaffold is automatic, adaptive, and unified. (b) Evaluation curves during training (success rate on ALFWorld, exact match on search-based QA), compared with GiGPO. (c) Gains over the RL baseline of each experiment, evaluated without scaffolds: success rate on ALFWorld, average exact match on search-based QA, average accuracy on math, and KernelBench correctness pass@3 averaged over Levels 1 to 3. Gray dots are prior scaffolding methods. All methods use the same student size as AutoScaffold, except Skill0, SkillRL and SkillC in ALFWorld using 7B models. (d) The teacher sets each problem’s solution-prefix ratio as instance-level scaffold: it first reveals more of the reference solutions and then withdraws support, whereas QuestA switches from 50% to 25% at a fixed step.

Method	Stage	What it changes	Granularity	Automatic	Adapts to student progress	No extra runs	Scaffold-free eval.	Domains
AI for automating model development
AHE ( Lin et al., 2026b )	inference	agent harness	–	✓	–	–	–	coding
autoresearch ( Karpathy, 2026 )	training	training code	–	✓	✗	✗	–	pretraining
Eureka ( Ma et al., 2024 )	training	reward function	–	✓	✗	✗	–	robotics
Trainee-to-Trainer ( Chen et al., 2026a )	training	environment	–	✓	✓	✓	–	multi-agent
LLMZero ( Fang et al., 2026 )	training	hyperparameters	–	✓	✓	$\circ$	–	sci. QA, chem., music

Table 1: Positioning of AutoScaffold. Stage : whether the method changes how a model behaves at inference time or how it is trained. What it changes : the object that is designed or edited. Granularity : whether guidance is shared by a class of tasks, specific to one task, or both. Automatic : an AI system decides the changes, without hand-designed content or schedules (

\circ

: content is generated by a model, but its use follows hand-designed rules). Adapts to student progress : whether guidance or training settings are revised in response to the student’s current performance and observed failures (

\circ

: changes follow predefined rules or schedules). No extra runs : changes are decided and tested within a single training run, without training separate runs or branches (

\circ

: branches are trained from checkpoints). Scaffold-free eval. : the learner is evaluated without the guidance it received during training (

\circ

: some guidance is kept). Domains : the task types on which the method is evaluated (A = agentic, S = search QA, M = math, K = GPU kernels). –: not applicable.

Figure 3: Overview of AutoScaffold . (a) A class-level scaffold, injected with probability

q[C]

, and an instance-level scaffold are added to the student’s training prompt. On a hard task this turns all-failure groups into mixed groups that produce training signal; the RL algorithm is unchanged, and all scaffolds are removed at evaluation. (b) At the end of each block of

K

steps, a frozen teacher gathers evidence from the training log, proposes edits, validates class-level scaffolds on held-out tasks, and deploys the updated scaffold for the next block. Bottom: an actual update on ALFWorld.

Domain	Source of classes	Classes
ALFWorld	task type	Pick, Look, Clean, Heat, Cool, Pick2
Search-based QA	training source	NQ, HotpotQA
KernelBench	kernel type	Conv, Elementwise, Loss, Matmul, Norm/Softmax, Reduce
Math reasoning	none	a single class

Table 2: Task classes used for class-level scaffolds in each domain. Classes follow the categories provided by each training dataset; math has no such categories and forms a single class.

Method	Pick	Look	Clean	Heat	Cool	Pick2	Overall
Qwen2.5-1.5B-Instruct	5.9	5.5	3.3	9.7	4.2	0.0	4.1
Training-free methods
ReAct	17.4	20.5	15.7	6.2	7.7	2.0	12.8
Reflexion	35.3	22.2	21.7	13.6	19.4	3.7	21.8
RL baselines without scaffold
PPO	64.8	40.5	57.1	60.6	46.4	47.4	54.4

Table 3: Success rates (%) on ALFWorld with Qwen2.5-1.5B-Instruct. Overall for AutoScaffold is the unweighted mean over the six task types. Prompting and RL baseline results are from Feng et al. (2025) ; ours are mean

\pm

standard deviation over 3 seeds. Gray rows are from Lin et al. (2026a) and use the larger Qwen2.5-7B-Instruct (reference only). Bold : best among 1.5B models.

Method	Single-Hop QA			Multi-Hop QA				Avg.
Method	NQ †	TriviaQA ⋆	PopQA ⋆	HotpotQA †	2Wiki ⋆	MuSiQue ⋆	Bamboogle ⋆	Avg.
RL baselines without scaffold
R1-Instruct	27.0	53.7	19.9	23.7	29.2	7.2	29.3	27.1
Search-R1	34.1	54.5	37.8	32.4	31.9	10.3	26.4	32.5
ZeroSearch	41.4	57.4	44.8	27.4	30.0	9.8	11.1	31.7
StepSearch	–	–	–	34.5	32.0	17.4	34.4	–

Table 4: Exact match (%) on search-based QA with Qwen2.5-3B-Instruct, trained on NQ and HotpotQA (

\dagger

: in-domain;

\star

: out-of-domain). RL baseline results are from Feng et al. (2025) ; SkillRL and Skill0 are from their original papers ( Xia et al., 2026 ; Lu et al., 2026 ) ; ours are mean

\pm

standard deviation over 3 seeds. Gray rows use the larger Qwen2.5-7B-Instruct (reference only). Bold : best among 3B models.

Method	AIME24	AIME25	AMC23	MATH-500	Minerva	Olympiad	Avg.
Qwen3-4B-Instruct	52.1	43.1	92.2	93.6	46.1	67.7	65.8
RL baselines without scaffold
SFT	14.4	22.1	55.2	78.9	38.1	40.2	41.5
GRPO	55.8	45.0	95.0	96.0	50.1	70.4	68.7
RL baselines with scaffold
LUFFY	42.3	36.0	86.4	91.1	48.2	59.4	60.6

Table 5: Accuracy (%) on math benchmarks with Qwen3-4B-Instruct. RL baseline results are from Liao et al. (2026) ; ours are mean

\pm

standard deviation over 3 seeds. Avg.: mean over the six benchmarks. Bold / underline : best / second best.

Method	AIME24	AIME25	HMMT Feb 25	OlympiadBench	BRUMO25	Avg.
Training-free base models
DeepSeek-R1-Distill-1.5B	28.7	22.3	12.0	52.4	31.8	29.4
Qwen3-1.7B	48.3	36.8	22.2	56.1	44.1	41.5
Nemotron-1.5B	61.8	49.5	31.6	64.6	58.2	53.1
Larger models (reference only)
DeepSeek-R1-Distill-32B	72.6	51.8	33.0	65.0	68.0	58.1

Table 6: Accuracy (%) on math benchmarks with Nemotron-1.5B. DAPO and QuestA results are from Li et al. (2026a) . Gray: larger models (reference only). Bold : best among small models.

Method	Level 1		Level 2		Level 3
Method	Correctness	Fast 1	Correctness	Fast 1	Correctness	Fast 1
Qwen3-8B-Cold-Start	51	14	43	16	32	8
StitchCUDA	59	17	50	20	32	13
AutoScaffold (ours)	63	19	54	25	33	14

Table 7: Pass@3 of Correctness and Fast 1 on the KernelBench test set. Fast 1 : the generated kernel is correct and faster than the PyTorch reference ( Ouyang et al., 2025 ) . Bold / underline : best / second best.

	AutoScaffold	GiGPO	Teacher decision		Scaffold adaptation & injection
	AutoScaffold	GiGPO	w/o tools & memory	w/o validation	Static	$K{=}5$	$K{=}20$	$q_{\max}{=}1$
Success	93.9	86.7	85.3	79.8	78.6	90.1	90.4	86.0
$\Delta$	–	$-$ 7.2	$-$ 8.6	$-$ 14.1	$-$ 15.3	$-$ 3.8	$-$ 3.5	$-$ 7.9

Table 8: Ablations on ALFWorld (Qwen2.5-1.5B-Instruct), success rate (%) evaluated without scaffold. Static: Skill0’s published ALFWorld skill bank used as a fixed class-level scaffold.

\Delta

: gap to full AutoScaffold .

Figure 4: One teacher update on ALFWorld (step 40). Top: the teacher queries tools and memory, then gathers evidence from class statistics, failed trajectories, and its own past decisions. It reads them and finds meaningful information for further diagnosis and edits. Bottom: it diagnoses a different failure in each class and responds with a matching edit: adding, rewriting, reducing, or omitting a scaffold.

Figure 5: Scaffold changes over training on ALFWorld. Left: scaffold-free success (top) and mean class-level injection probability (bottom), with training divided into three stages. Right: what the teacher changed in each stage (+ add,

\sim

revise,

-

remove,

\uparrow\downarrow

raise or lower injection probability).

Figure 6: From each branching point, we train 20 more steps with the updated scaffold ( AutoScaffold , dark blue), with the previous scaffold kept fixed (orange), or with vanilla RL (gray). AutoScaffold gives the highest success at every branching point.

Figure 7: Solution-prefix ratios during math training (Nemotron-1.5B). Mean ratio across all training problems. AutoScaffold (red) adjusts each problem’s ratio using training evidence. QuestA (gray) switches from 50% to 25% at step 100.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Class	No scaffold	Current scaffold	Proposed scaffold	Outcome
Heat	6.7	30.0	40.0	adopted
Look	26.7	30.0	30.0	adopted
Pick2	13.3	–	10.0	dropped

Appendix

Table 9: Validation of the scaffolds proposed in this update: success rate (%) on 30 held-out games per class. Pick2 has no current scaffold.

Outcome in the current block	Selection	Change in $r_{x}$
All-fail	$0\leq r_{x}\leq 25$	$+20$
All-fail	$26\leq r_{x}\leq 49$	$+10$
Mixed	$r_{x}\geq 1$ , success rate $>50\%$	$-5$
All-pass	$r_{x}\geq 1$	$-10$

Appendix

Table 10: Prefix-ratio changes in this update. Ratios are capped at 50.

	ALFWorld	Search QA	Math	KernelBench
Teacher wall-clock per update (min)	4	3	5	4
Validation wall-clock per update (min)	11	13	21	16
Total teacher API cost per run (USD)	2.35	3.13	5.57	2.76

Appendix

Table 11: Teacher cost per domain. Per-update values are means over all scaffold updates in a run.

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 456 papers

Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training
Sep 14, 2026Yuanhao Yue, Qianli Ma, Chengyu Wang +3Multimodal Large Language Models Curriculum RL
Alibaba Cloud Computing · Shanghai Jiao Tong University · Fudan University +1
SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
Mar 30, 2026Yifan Wang, Bolian Li, David Cho +3RL for Language Model Reasoning Reinforcement Learning
Purdue University · Texas Instruments
H $^2$ SD: Hybrid Hindsight Self-Distillation
Jul 21, 2026Qiye Cai, Yichuan Ma, Weitao Ma +6RL for Language Model Reasoning Language Model Distillation
Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +1

Sep 14, 2026J/K move · Enter open · S save

Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

Yuanhao Yue, Qianli Ma, Chengyu Wang, Haoting Wang+2

Alibaba Cloud Computing · Shanghai Jiao Tong University · Fudan University+1

Multimodal Large Language Models Curriculum RL Multimodal Reasoning RL Post-Training

Open paper AskPDF arXiv

Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the

\textit{Exploration Potential Score} (EPS)

, a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7% relative improvement in-domain and gains of 11.5% on MathVision and 11.1% on MMMU-Pro.

Abstract

RL for Language Models

Explore similar work

Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology

H2^22SD: Hybrid Hindsight Self-Distillation

H $^2$ SD: Hybrid Hindsight Self-Distillation