Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL

Youling Huang; Tiankuo Xu; Jiaji Liu; Tong Zheng; Shuo Zhou; Shaotong Qi; Junchi Yao; Shiyang Liu

cs.AISep 29, 2026

Guide, Then Let Go: Gap-Adaptive Teacher Scheduling for Sparse-Reward Agentic RL

Authors: Youling Huang, Tiankuo Xu, Jiaji Liu, Tong Zheng, Shuo Zhou, Shaotong Qi, Junchi Yao, Shiyang Liu, +5 more

Organizations: DUT · Kuaishou · XJTU · THU · UCAS · BFSU · SEU · MBZUAI

Abstract

Reinforcement learning for long-horizon agents typically relies on sparse outcome-based rewards. This leads to a severe cold-start problem, as early-stage policies often fail to solve sampled tasks, leaving little useful reward signal for learning. To mitigate this problem, we use on-policy distillation (OPD) to provide token-level guidance on the student's own rollouts. We find that the benefit of this guidance depends on the performance gap between the teacher and the student. When the teacher substantially outperforms the student, distillation helps guide the student through the early training stage where outcome rewards provide little learning signal. As the gap narrows and eventually reverses, however, continued distillation becomes less beneficial and may hinder further improvement. Motivated by this observation, we propose Gap-Adaptive Teacher Scheduling (GATS), which augments the student's RL objective with an OPD term whose weight adapts to the teacher-student performance gap. Specifically, GATS gradually reduces teacher guidance as the student approaches the teacher's reference performance and withdraws it once that reference is reached. This enables GATS to leverage task-trained teachers smaller than the student, since teacher guidance is primarily needed during early training. Across ALFWorld, WebShop, and ScienceWorld with three Qwen2.5 teacher-student configurations, GATS achieves the highest average success rate among the compared methods in all three configurations, improving over reward-only GRPO by 4.37%-11.87% under matched student rollout budgets. Code is available at https://github.com/Ricardo-H/guide-then-let-go.

Figures & tables

Figure 1: Guide early, then let go. Representative training dynamics on WebShop for the Qwen2.5-3B

\rightarrow

7B configuration. (a) OPD reduces all-failure rollout groups during the sparse-reward cold start. (b) After teacher withdrawal, GATS continues to improve beyond the teacher reference, whereas fixed-weight OPD plateaus near it. Dashed lines indicate the teacher reference

M_{T}

, and dotted lines mark the withdrawal step.

Figure 2: Overview of GATS. The student generates trajectory groups that are used to compute both the GRPO loss and the teacher-guided OPD loss. GATS adaptively weights the OPD loss according to the teacher–student capability gap and combines it with the GRPO loss to update the student. As the gap narrows, the OPD weight decreases to zero, after which the teacher is permanently withdrawn and training continues with GRPO alone.

Figure 3: OPD gain over GRPO versus the teacher–student performance gap. The observed gain diminishes as the gap narrows and becomes negative near parity, motivating gap-adaptive supervision and eventual teacher withdrawal.

Student: Qwen2.5-7B
	ALFWorld		WebShop	ScienceWorld		Avg. SR
Method	ID	OOD	Eval	ID	OOD
Teacher: Qwen2.5-1.5B	53.65	60.42	63.80	12.76	13.80	40.89
Prompt-only	14.84	13.02	0.26	10.94	7.03	9.22
GRPO	63.02	73.70	61.20	38.02	28.91	52.97
GRPO $+$ OPD	55.21	46.09	61.20	18.49	14.58	39.11

Table 1: Final success rates (%) on ALFWorld, WebShop, and ScienceWorld. Each entry is averaged over three evaluation runs. Avg. SR denotes the unweighted mean of the five reported metrics. Within each student block, the best and second-best results in each column are shown in bold and underlined, respectively.

Figure 4: Training dynamics for Qwen2.5-3B

\rightarrow

7B on ALFWorld, WebShop, and ScienceWorld. Top: training success rate. Bottom: all-failure group rate. Horizontal dashed lines denote the teacher reference

M_{T}

; vertical dotted lines mark GATS teacher withdrawal at

t_{w}

. Teacher guidance accelerates early learning, while GATS continues to improve after withdrawal.

Different decay shapes, $N=80$
Schedule	ALF	Web	Sci.	Avg.
Linear	77.86	68.75	44.01	63.54
Cosine	71.35	71.09	44.01	62.15
Step	69.79	65.10	36.72	57.20
Linear decay with varying $N$
Linear, $N=80$	77.86	68.75	44.01	63.54

Table 2: Comparison of OPD-weight decay schedules with a Qwen2.5-1.5B teacher and Qwen2.5-7B student. ALF and Sci. denote the ID splits of ALFWorld and ScienceWorld, respectively, while Web denotes the evaluation split of WebShop. All results are averaged over three evaluation runs.

Figure 5: Training success versus GPU-hours for the Qwen2.5-3B

\rightarrow

7B configuration, including teacher costs. Vertical dotted lines mark the GRPO 150-update budgets; horizontal dashed lines indicate the teacher reference

M_{T}

Environment	Budget	GRPO	GRPO+OPD	GATS	$M_{T}$	$\Delta$ (GATS $-$ GRPO)
	(GPU-h)					(pp)
ALFWorld	76.4	71.9	73.6	84.9	74.4	$+$ 13.1
WebShop	39.5	64.0	58.8	77.1	58.6	$+$ 13.2
ScienceWorld	46.3	27.8	39.7	46.0	32.5	$+$ 18.2

Table 3: Smoothed training success rate (%) at the GRPO 150-update compute budget for each environment.

M_{T}

is the teacher’s training-split success rate used as the withdrawal reference. Bold indicates the best method in each environment.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Environment	Train	ID	OOD	Eval
ALFWorld	3,553	140	134	–
WebShop	6,410	–	–	500
ScienceWorld	3,322	1,661	1,684	–

Appendix

Table 4: Task-pool sizes for training and evaluation. Dashes indicate splits not used in the reported evaluation.

Setting	Value
Training updates	150
GRPO group size	8 rollouts per prompt
Prompts per batch	16
Optimizer	AdamW
Learning rate	$1\times 10^{-6}$
Weight decay	$0.01$

Appendix

Table 5: Core training hyperparameters shared across student runs.

Setting	ALFWorld	WebShop	ScienceWorld
Max prompt length	2,048	4,096	6,000
Max response length	512	1,024	1,024
PPO mini-batch size (configuration)	256	64	256
Discount $\gamma$	0.95	1.0	1.0
Max turns	50	15	30

Appendix

Table 6: Environment-specific training settings.

Teacher–student gap (pp)	OPD gain over GRPO (pp)
$+41.3$	$+14.1$
$+31.7$	$+6.0$
$+19.0$	$+5.2$
$+14.3$	$+3.4$
$+9.6$	$+2.2$
$-0.8$	$-3.0$

Appendix

Table 7: Task-performance-gap diagnostic on ALFWorld. The gap is teacher SR minus student SR before the continuation. OPD gain is the final SR difference between the matched GRPO+OPD and GRPO continuations. Both quantities are reported in percentage points (pp).

Explore similar work

TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL
Sep 28, 2026Yibin Huang, Xinming Xu, Conghui ZhuAgentic Reinforcement Learning Teacher
Faculty of Computing, Harbin Institute of Technology · Tsinghua University
Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents
Sep 28, 2026Tong Zhang, Zhou Liu, Yihao Liu +8Efficient On-Policy Distillation Teacher
Qwen Large Model Application Team, Alibaba · Peking University · University of Chinese Academy of Sciences +1
Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
Aug 1, 2026Zhuowen Han, Jinwei Xiao, Zhengxi Lu +9Teacher Uni-Opd
TJUNLP Lab, School of Computer Science and Technology, Tianjin University · Meituan Longcat Team

Sep 28, 2026, cs.LGJ/K move · Enter open · S save

TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL

Yibin Huang, Xinming Xu, Conghui Zhu

Faculty of Computing, Harbin Institute of Technology · Tsinghua University

Agentic Reinforcement Learning Teacher Exploration

Open paper AskPDF arXiv

Effective multi-turn agents require interaction strategies that coordinate information gathering, actions, and feedback over long horizons. GRPO is a reinforcement learning algorithm used to train these agents, but sparse trajectory-level rewards limit early exploration in small models. Recent methods augment RL with on-policy distillation (OPD) from a stronger teacher. However, a fixed mixture assumes that teacher guidance and reward optimization should retain a constant relative role throughout training and across interaction turns. This assumption can fail at two scales. Globally, as training progresses, maintaining strong distillation pressure can constrain the model from moving beyond the teacher's capabilities. Locally, teacher--student disagreement identifies where the student departs from the teacher, but cannot tell whether that departure is exploration supported by better outcomes or low-quality policy drift. Our methodological insight is that teacher guidance and reward optimization should be dynamically rebalanced over training and jointly allocated across turns. We instantiate this insight in \tide. Globally, \tide uses the measured disagreement trend as a practical schedule signal, advancing an OPD-to-RL handoff when discrepancy reduction becomes slow but remains positive and progressively increasing the relative weight of RL. Locally, \tide jointly modulates teacher-guided and reward-driven updates: relative action value and disagreement prioritize the OPD signal, whereas relative action value supplies the RL advantage and normalized disagreement reweights it across turns. Coupled with the global handoff, \tide allocates stronger teacher guidance early and gives reward-driven updates greater relative weight later in training. Experiments across multiple benchmarks, student scales, and controlled ablations support the effectiveness of TIDE's adaptive OPD--RL coordination.

Abstract

Explore similar work

TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL

Teacher-Student Gaps Are Not Enough: Outcome-Guided On-Policy Distillation for Multi-Turn Autonomous Agents

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance