UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents

Wenbo Zhang; Pengcheng Xu; Weizhi Du; Jing Zhang; Hengrui Cai

cs.LGSep 27, 2026

UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents

Authors: Wenbo Zhang, Pengcheng Xu, Weizhi Du, Jing Zhang, Hengrui Cai

Organizations: University of California, Irvine · University of Michigan, Ann Arbor

Abstract

On-policy distillation (OPD) trains a student on its own rollouts using dense supervision from a teacher. In multi-turn environments, a mistake at a critical decision step can redirect the subsequent rollout toward poor outcomes. We use low teacher confidence on student actions to select high-uncertainty steps for correction. In a controlled ALFWorld study, a single teacher correction at a low-confidence step improves subsequent student behavior and task success, motivating selective intervention during distillation. We propose UOPD, an uncertainty-aware intervention method for on-policy distillation. At low-uncertainty turns, UOPD executes student actions and applies the standard OPD loss. At high-uncertainty turns, it samples and executes teacher actions and trains the student to imitate them through supervised fine-tuning, which minimizes forward Kullback-Leibler divergence in expectation. UOPD utilizes adaptive uncertainty thresholds to target a scheduled intervention rate. Empirically, we evaluate UOPD across a broad range of agentic tasks, including ALFWorld, WebShop, and Search, demonstrating its superior performance over OPD methods and their variants. UOPD improves WebShop score by up to $15.8\%$ relative to standard OPD.

Figures & tables

Figure 1 : OPD versus UOPD. OPD executes student actions ( $a^{S}_{t}$ ) given observations ( $o_{t}$ ) and distills using reverse KL. UOPD uses teacher signals for uncertainty quantification (UQ) and an adaptive threshold to determine when the teacher intervenes. This decision jointly determines the executed action and the learning objective: reverse KL for student actions, or SFT on teacher actions ( $a^{T}_{t}$ ). The latter minimizes forward KL in expectation.

RL-Qwen2.5-7B teacher $\rightarrow$ Qwen2.5-3B student
	ALFWorld				WebShop
	Seen		Unseen		Test
Method	SR $\uparrow$	Turns $\downarrow$	SR $\uparrow$	Turns $\downarrow$	Score $\uparrow$	SR $\uparrow$	Turns $\downarrow$
Student (zero-shot)	21.0 $\pm$ 0.8	43.8 $\pm$ 0.3	15.9 $\pm$ 3.4	46.0 $\pm$ 1.1	12.2 $\pm$ 2.1	1.0 $\pm$ 0.5	13.6 $\pm$ 0.3
Teacher (zero-shot)	97.4 $\pm$ 0.4	9.1 $\pm$ 0.1	92.8 $\pm$ 0.4	12.3 $\pm$ 0.4	85.4 $\pm$ 0.6	77.9 $\pm$ 1.2	6.6 $\pm$ 0.3
OPD	88.1 $\pm$ 1.8	14.2 $\pm$ 0.3	85.3 $\pm$ 1.6	16.3 $\pm$ 0.6	77.5 $\pm$ 2.2	66.9 $\pm$ 3.0	7.4 $\pm$ 0.3

Table 1 : Main results on ALFWorld and WebShop. We report success rate (SR, %) on ALFWorld, and mean task score, SR (%) on WebShop. We also include average number of turns. Results are means

\pm

standard deviations over three evaluation seeds. Bold indicates the best result among distillation methods.

Intervention signal
	ALFWorld		WebShop
Configuration	Seen SR $\uparrow$	Unseen SR $\uparrow$	Score $\uparrow$	SR $\uparrow$	Turns $\downarrow$
UOPD ( $\beta=1$ , $30\%\!\to\!5\%$ )	90.0	88.8	82.4	69.3	6.5
Confidence gap	89.8	87.8	77.3	69.0	7.2
Random intervention	88.3	83.8	77.3	64.6	7.4
SFT weight

Table 2 : Effects of intervention selection, SFT weight, teacher execution, and schedule with a 3B student. The default UOPD configuration uses teacher confidence,

\beta=1

, teacher execution, and a

30\%\!\to\!5\%

intervention schedule. Random intervention samples each turn with the scheduled probability. Values are means over three evaluation seeds, and bold marks the best value in each column.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Task success (%)	Trajectories with state returns (%)
Student	13.98 ( $13/93$ )	93.55 ( $87/93$ )
Random	16.13 ( $15/93$ )	91.40 ( $85/93$ )
Intervention	25.81 ( $24/93$ )	80.65 ( $75/93$ )

Appendix

Table 3: Outcomes on the same 93 ALFWorld tasks used in Fig. 2 (c,d). Random and Intervention each execute one teacher action before returning control to the student.

Optimisation
	ALFWorld	WebShop	Search
Optimizer	AdamW, $\beta{=}(0.9,0.999)$ , weight decay $0.01$ , constant learning rate, no warm-up
Learning rate	$1\times 10^{-6}$	$1\times 10^{-6}$	$1\times 10^{-6}$
Gradient clipping	$1.0$	$1.0$	$1.0$
Training steps	$250$	$150$	$300$
Tasks per rollout batch	$16$	$16$	$64$

Appendix

Table 4 : Training hyperparameters. Settings shared by all methods are listed first; the last block is specific to UOPD.

Role	GPUs	Configuration
Student rollout (vLLM)	$4$	$2$ engines, tensor parallel size $2$ per engine
Teacher inference (vLLM)	$2$	$1$ engine, tensor parallel size $2$
Student updates (verl/FSDP)	$2$	Ulysses sequence parallel size $2$
Total	$8$	Single node

Appendix

Table 5 : Eight-GPU allocation. Student rollout, teacher inference, and student updates run on separate GPU groups.

Explore similar work

DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation
Date pendingYuchen Xia, Qianguo Sun, Chao Song +3Efficient On-Policy Distillation Uni-Opd
The Chinese University of Hong Kong · IDEA Research · Emdoor Research Institute
On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents
Jun 14, 2026Gengsheng Li, Mao Zheng, Mingyang Song +8Teacher Curriculum
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Large Language Model Department, Tencent +4
SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
Jun 17, 2026Yuhang Zhou, Lizhu Zhang, Yifan Wu +5Uni-Opd Selective
Meta AI

Date pending, cs.LGJ/K move · Enter open · S save

DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation

Yuchen Xia, Qianguo Sun, Chao Song, Junlong Wu+2

The Chinese University of Hong Kong · IDEA Research · Emdoor Research Institute

Efficient On-Policy Distillation Uni-Opd Discrepancy Accumulation

Open paper AskPDF arXiv

While on-policy distillation (OPD) reduces exposure bias by training student language models on their own rollouts, early student errors in long-horizon agentic scenarios can lead to contexts unfamiliar to the teacher. To improve trajectory quality, recent work on agentic OPD introduces teacher intervention into training rollouts by switching the executor between the student and the teacher. However, existing methods determine how much teacher intervention is needed---but not when. To address this limitation, we propose DASH-OPD (Discrepancy-Aware Switching with Hysteresis for OPD), the first agentic OPD method to perform adaptive, bidirectional executor switching. At each turn, DASH-OPD measures teacher--student discrepancy using a mean log-probability ratio over action tokens. Student-to-teacher ratios on student turns serve as drift signals, while teacher-to-student ratios on teacher turns serve as recovery signals. These signals are accumulated over multiple turns to form drift and recovery evidence, respectively. DASH-OPD switches executors when either type of evidence exceeds its corresponding switching threshold, introducing hysteresis that prevents rapid switching triggered by transient discrepancy fluctuations. Across three benchmarks and two student model sizes, DASH-OPD outperforms five baselines in all 14 task performance comparisons, while requiring the fewest interaction turns in nine of ten efficiency comparisons. Code, models, and training logs are available at https://github.com/Lucian1115/DASH-OPD

Abstract

Explore similar work

DASH-OPD: Discrepancy-Aware Switching with Hysteresis for On-Policy Distillation

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation