State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

Zhentao Zhang; Yanzhao Zheng; Yanlin Liu; Yuanqiang Yu; Tianze Xu; Chao Ma; Jihuai Zhu; Lelun Wu

cs.LGOct 9, 2026

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

Authors: Zhentao Zhang, Yanzhao Zheng, Yanlin Liu, Yuanqiang Yu, Tianze Xu, Chao Ma, Jihuai Zhu, Lelun Wu,

Organizations: Alibaba Group · Tsinghua University · Shanghai Jiao Tong University · University of Science and Technology of China

Abstract

On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and executed action-observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning-action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. We introduce State-Grounded On-Policy Distillation, a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning-action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07% with the strongest baseline to 31.74% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.

Figures & tables

Figure 1: Overview of State-OPD. The teacher conditions on the interaction state rather than prior student reasoning; the student retains its full history and samples additional complete responses at states with high response entropy.

Panel of Figure 2 — Figure 2: Removing prior student reasoning improves Qwen3-32B teacher continuation at ALFWorld states visited by Qwen3-0.6B and Qwen3-4B students and changes the reverse-KL depth profile. Continuations run for at most 30 additional turns. Paired intervals appear in Table 3 .

Panel of Figure 3 — Figure 3: Resampling reasoning reveals more actions at the same state. (a) For each state, we draw 16 responses with fixed reasoning and 16 complete responses. Side-by-side bars show the percentage of states in each action-count bin within each condition. (b) Faint points show individual states; solid markers show equal-count-bin means with 95% trajectory-bootstrap intervals. Higher response entropy accompanies more additional actions.

Panel of Figure 4 — Figure 4: Training dynamics on ALFWorld for a Qwen3-4B student distilled from Qwen3-32B. (a,b) Seen/Unseen averages. (c) Success-rate gap at matched supervised-turn budgets (State-OPD minus linearly interpolated OPD). (d) Mean thinking tokens per root-trajectory turn; faint lines show step values and marked lines show centered 11-step moving averages.

Method	Seen SR $\uparrow$	Unseen SR $\uparrow$	Avg. SR $\uparrow$
State-OPD (Ours)	43.39	45.71	44.55
w/o State-Centered Response Allocation	40.36	43.10	41.73
w/o State-Grounded Supervision	33.39	37.31	35.35
Vanilla OPD	30.89	33.96	32.42

Table 2: State-OPD ablation on ALFWorld. A Qwen3-4B student is distilled from Qwen3-32B; Avg. is the mean of Seen and Unseen SR.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Student	Anchor depth	$T_{h}$	$T_{s}$	$\Delta_{T}$ [95% CI]
Qwen3-0.6B	Early	5.47	17.97	$+12.50\;[+4.41,+21.67]$
	Middle	2.34	15.63	$+13.28\;[+6.45,+20.51]$
	Late	0.00	14.06	$+14.06\;[+4.61,+25.78]$
Qwen3-4B	Early	14.84	15.63	$+0.78\;[-4.84,+6.90]$
	Middle	17.97	21.09	$+3.13\;[-9.48,+17.74]$
	Late	10.16	30.47	$+20.31\;[+5.47,+37.07]$

Appendix

Table 3: Teacher continuation from matched ALFWorld anchors. Each row uses 32 anchors from 30 source episodes and four repeats per condition (128 paired continuations).

T_{h}

and

T_{s}

are success rates (%);

\Delta_{T}

is their paired difference in percentage points with a 95% source-episode bootstrap interval.

Optimization
Hyperparameter	Value
Optimizer	AdamW
Training steps	200
Root tasks per step	16
Baseline rollouts per task	4
Learning rate	$1\times 10^{-6}$ (constant)

Appendix

Table 4: Training hyperparameters across environments.

Generation
Hyperparameter	Value
Temperature	$0.4$
Top- $p$	$1.0$
Top- $k$	$-1$
Evaluation protocol
Evaluation interval	20 training steps

Appendix

Table 5: Evaluation hyperparameters across environments.

Environment	Train	Seen evaluation	Unseen evaluation / test
ALFWorld	3,553	140	134
ScienceWorld	2,294 (17)	456 (17)	782 (13)
WebShop	4,096	–	500

Appendix

Table 6: Training and evaluation sets. Parentheses give the number of ScienceWorld task types.

State-OPD (ours)
Hyperparameter	Value
Concentration factor $\beta$	2
TCOD-F2B
Starting horizon	1 turn
Pacing factor $\eta$	2
Guided-OPD

Appendix

Table 7: Method-specific hyperparameters.

Setting	Baseline	Qwen3-1.7B	Qwen3-4B
ALFWorld Seen	Vanilla OPD	$+4.82\;[+0.54,+9.29]$	$+12.50\;[+8.39,+16.79]$
	TCOD-F2B	$+2.14\;[-2.32,+6.61]$	$+14.11\;[+9.64,+18.57]$
	Guided-OPD	$+3.39\;[-0.89,+7.68]$	$+16.25\;[+11.43,+21.07]$
	TurnOPD	$+3.93\;[-0.36,+8.39]$	$+16.61\;[+11.79,+21.61]$
ALFWorld Unseen	Vanilla OPD	$+3.54\;[-1.49,+8.58]$	$+11.75\;[+6.53,+17.35]$
	TCOD-F2B	$+7.46\;[+2.05,+13.06]$	$+10.45\;[+5.04,+15.86]$

Appendix

Table 8: Paired success-rate differences, State-OPD minus the indicated baseline, in percentage points. Brackets give 95% task-bootstrap intervals. Positive differences favor State-OPD. Each student column contains 20 comparisons.

Figure 6: State-OPD reduces persistent failed-action loops throughout ALFWorld training. For a Qwen3-4B student distilled from Qwen3-32B, each panel shows the share of failed-action starts with at least

k

further identical actions, each returning Nothing happens.

Panel of Figure 7 — Figure 7: State selection and allocation sensitivity on ALFWorld. (a) Share of additional distinct actions in a 16-response probe recovered by entropy-ranked or random selection under matched budgets; error bars are 95% root-cluster bootstrap intervals. (b) Seen/Unseen SR across $\beta$ for a Qwen3-4B student distilled from Qwen3-32B; each setting has one training run.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 512 papers

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
Jun 14, 2026Gengsheng Li, Mao Zheng, Mingyang Song +8Language Model Distillation LLM Agent Training
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent +4
Stochastic Teacher Intervention for Agentic On-Policy Distillation
Oct 7, 2026Junnan Liu, Linhao Luo, Zhijun Chen +3On-Policy Distillation Teacher-Student Learning
Department of Data Science and AI, Faculty of Information Technology, Monash University, Australia · The Hong Kong Polytechnic University, Hong Kong SAR, China · Zhongguancun Laboratory, Beijing, P.R. China
Multi-Turn On-Policy Distillation with Prefix Replay
Jul 6, 2026Baohao Liao, Hanze Dong, Christof Monz +3LLM Agent Training On-Policy Distillation
1Microsoft Research · University of Amsterdam

Jun 14, 2026J/K move · Enter open · S save

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu+7

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent+4

Language Model Distillation LLM Agent Training On-Policy Distillation Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice. On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students, but we find that it suffers a characteristic failure mode in this setting: small student errors compound across turns and push the trajectory out of the teacher's familiar state distribution, so the teacher's supervision becomes least reliable precisely where the student needs it most. We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout and schedules the teacher's intervention probability along a curriculum that decays to zero. Strong guidance keeps early trajectories close to the teacher distribution and is then gradually withdrawn to recover the purely on-policy regime used at inference. On ALFWorld, ScienceWorld, and WebShop, distilling Qwen3 students from a Qwen3-30B-A3B teacher, Guided-OPD yields average relative gains of 21.1% in Score and 25.5% in Success Rate over vanilla OPD, with larger gains on smaller students.

Abstract

On-Policy Distillation

Explore similar work

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

Stochastic Teacher Intervention for Agentic On-Policy Distillation

Multi-Turn On-Policy Distillation with Prefix Replay