Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents

Md Farhan Ishmam; Kenneth Marino

cs.AIOct 14, 2026

Teaching Agents in their Own Style: Blackbox Teacher Distillation for Web Agents

Authors: Md Farhan Ishmam, Kenneth Marino

Organizations: Kahlert School of Computing, University of Utah

Abstract

Frontier web agents using closed models have high inference and computational costs. Smaller open models can be an alternative, but to be performant, they require training supervision from a human or teacher model. Token-level teacher distillation is not an option for closed teachers as it needs access to the model's logits. The standard approach is to imitate teacher demonstrations offline, but it trains the student on states it has never visited, and in a style it does not write in. We introduce WebKD, where the teacher is the editor of the student's failures. Given a failed episode, the teacher finds the first point of mistake, and rewrites the student's reasoning traces, harness states, and actions from that point onward. The editing reveals that the teacher's corrections can be more surprising to the student than the student's own text. WebKD instead aims to edit minimally. In the data space, the teacher writes each correction in the student's style. In the objective space, the reasoning is weighted by its contribution to the action. A role-asymmetric KL anchor allows the corrected action to move freely while keeping the reasoning near the student's prior. On the TimeWarp benchmark, WebKD students gain about 5.5-10.7 pp over behavior cloning, outcome-reward RL, and teacher editing baselines. The performance transfers to unseen versions and to the live web in the OnlineMind2Web benchmark. WebKD opens a new way of distilling black-box models to improve the capabilities of open-weight web agents.

Figures & tables

Figure 1: Overview of WebKD . (Top) At step $t$ , the harness generates the observation history from the goal, accessibility tree, and previous responses. The web agent emits a five-component response conditioned on the observation history. The T hink, P lan, and S tep components carry their reasoning and running plan, the M emory component reads from and writes to the scratchpad memory, and only the A ction component is executed in the web environment, which returns the next observation. (Bottom) The student is behavior-cloned on the teacher’s rollouts (P1) and then generates rollouts from the training tasks (P2) . The teacher localizes each failure at $t^{\prime}$ (P3) and rewrites the faulty trajectory suffix in the student’s style (P4) . The resulting corpus $\mathcal{D}$ trains the student under action informativeness weighing, and KL anchored objective of § 3.3.1 –§ 3.3.2 .

Panel of Figure 2 — Figure 2: The perplexity premium by response components against the student’s continuations, the teacher’s raw corrections, and the WebKD ’s corpus with in-context edits.

Panel of Figure 3 — Figure 3: Edits for the same student step. The raw teacher edit replaces the student’s action but rewrites the plan in its own style. The style-conditioned edit keeps the student’s nested plan format. Values in brackets give the span weight $v_{s}$ . $\langle\mathrm{\textbf{T}}\rangle$ : think, $\langle\mathrm{\textbf{P}}\rangle$ : plan, $\langle\mathrm{\textbf{M}}\rangle$ : memory, $\langle\mathrm{\textbf{A}}\rangle$ : action.

Model	Baseline	v1	v2	v3	v4	v5	v6 (seen)
–	Teacher (GPT-5.5)	$71.8_{\pm 4.43}$	$71.8_{\pm 4.43}$	$68.0_{\pm 4.60}$	$68.0_{\pm 4.60}$	$69.9_{\pm 4.52}$	$70.9_{\pm 4.58}$
Qwen 3.5 4B	Zero-Shot	$1.0_{\pm 0.57}$	$1.0_{\pm 0.57}$	$0.3_{\pm 0.31}$	$1.6_{\pm 0.71}$	$0.0_{\pm 0.00}$	$1.0_{\pm 0.57}$
	Behavior Cloning	$10.7_{\pm 1.76}$	$9.4_{\pm 1.66}$	$9.7_{\pm 1.68}$	$10.7_{\pm 1.76}$	$5.2_{\pm 1.26}$	$9.7_{\pm 1.68}$
	Behavior Cloning (TW)	$27.5_{\pm 2.54}$	$28.5_{\pm 2.57}$	$27.8_{\pm 2.55}$	$26.2_{\pm 2.50}$	$28.2_{\pm 2.56}$	$29.8_{\pm 2.60}$
	m-GRPO	$30.1_{\pm 2.62}$	$27.2_{\pm 2.54}$	$26.2_{\pm 2.50}$	$28.2_{\pm 2.56}$	$30.1_{\pm 2.61}$	$30.4_{\pm 2.62}$
	Self/Student Edit	$25.2_{\pm 2.47}$	$19.4_{\pm 2.25}$	$27.2_{\pm 2.53}$	$24.3_{\pm 2.56}$	$23.3_{\pm 2.61}$	$23.6_{\pm 2.62}$

Table 1: Success rate (%) on the TimeWarp test split, in domain on v6 and on the five held-out versions v1–v5. Results are averaged over three runs with the standard error as the subscript.

Table 6

Panel of Figure 6 — Figure 6: Change in action distribution before and after teacher distillation using WebKD for the Qwen-3.5 4B baseline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
fine-tuning	full-parameter, from the seed checkpoint
learning rate	$1\times 10^{-5}$ , cosine decay, warmup ratio $0.1$
epochs	$2$
context cutoff	$20{,}480$ tokens
batch	$1$ per device $\times$ $16$ -step gradient accumulation (effective $16$ )
precision	bf16, gradient checkpointing

Appendix

Table 4: Training hyperparameters for the combined objective.

Panel of Figure 7 — Figure 7: Behavior cloning over more epochs doesn’t improve the success rate, as it flattens at $\sim 25\%$ , whereas WebKD continues to improve and peaks at the 3rd total epoch for the Qwen-3.5 9B model. WebKD is trained from the first epoch of the BC seed.

Figure 8: Failure mode analysis of Qwen3.5-4B. Each bar assigns every evaluation trajectory of a method to exactly one outcome for the $103\times 3=309$ tasks.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

Oct 14, 2026

ActionOPD: Distilling Decisions, Not Tokens

Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On

τ^2

-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.

Huining Yuan, Xiangmin Yi, Zelai Xu +5

Tsinghua University

LLM Agent Training Knowledge Distillation On-Policy Distillation

Ask PDF

Oct 2, 2026

Harness-Aware Distillation for Small Language Model Agents

Language model agents are deployed with a harness, the software around the model that manages its context, tools, and feedback. When such an agent is distilled into a smaller one, the harness stays in place, so the student mainly needs the teacher-specific abilities that the harness cannot provide, such as acting correctly on harness information. Standard distillation, however, imitates the teacher's full outputs and treats the harness as part of the input. We propose Harness-Aware Distillation (HAD), which focuses distillation on what the teacher adds beyond the harness. HAD complements on-policy distillation with two components: an action preference that contrasts the same teacher's actions with and without the harness information, scored after the student's own reasoning, and a validity check that drops preference pairs whose preferred action contradicts the harness records. We show that the contrast gives the student information that imitating the teacher alone cannot provide, and HAD needs no task rewards, success labels, or future information. Across multiple long-horizon agent benchmarks and models, HAD outperforms on-policy distillation baselines with the same fixed harness. Our analysis shows that HAD enters fewer unproductive loops and recovers from errors more often than the baselines, and suggests that it adaptively keeps learnable feedback in its weights while reading state information from the harness.

Moonseok Choi, Taehong Moon, Giung Nam +1

KAIST AI · Independent researcher

LLM Agent Harnesses On-Policy Distillation

Ask PDF

Aug 31, 2026

When and What to Teach: Budget-Aware Online Adaptation for Web Agents

Web agents have achieved significant success in automating complex internet tasks but deploying them in real-world environments requires continuous online adaptation. Given that deploying powerful proprietary models remains commercially cost-prohibitive, practitioners must rely on lightweight local models that evolve post-deployment via online teaching from a stronger teacher. However, standard interactive feedback imposes prohibitive costs. We show that conventional trajectory-level preference optimization wastes budget on both unresolvable episodes and redundant execution turns. To resolve these inefficiencies, we propose Score-Guided Online Teaching with Budgeted Trajectory Trimming, a budget-aware framework that systematically orchestrates when and what to teach. Specifically, our framework integrates a solvability-aware teacher gate to dictate when to query the teacher model and a score-guided turn selection mechanism to decide what informative turns to retain. Extensive experiments on MiniWoB and TimeWarp demonstrate that our method achieves comparable first-pass success while reducing teacher calls by 22.6% and student training compute by 52.1% on average. Our code is available at https://github.com/zjw131f1fc/budgeted-online-teaching.

Jianwei Zhang, Sihan Cao, Pengcheng Zheng +7

University of Electronic Science and Technology of China · Institute of Software Chinese Academy of Sciences · Xi’an University of Architecture and Technology

Online Knowledge Distillation LLM Agent Training Web Agents Online Continual Learning Cost-Sensitive Learning Knowledge Distillation

Ask PDF

Abstract

LLM Agent Training

Explore similar work

ActionOPD: Distilling Decisions, Not Tokens

Harness-Aware Distillation for Small Language Model Agents

When and What to Teach: Budget-Aware Online Adaptation for Web Agents