D-Skill: Distilling Teacher Knowledge through Agent Skills

Shuo Yuan; Dacheng Li; Joseph E. Gonzalez

cs.AIOct 20, 2026

D-Skill: Distilling Teacher Knowledge through Agent Skills

Authors: Shuo Yuan, Dacheng Li, Joseph E. Gonzalez

Organizations: University of California, Berkeley

Abstract

Frontier language-model agents excel at complex tasks, but high inference costs make transferring their capabilities to smaller models essential. Traditional knowledge distillation achieves this through parameter updates, requiring access to the student's weights and additional training compute while encoding the transferred knowledge implicitly in the parameters. We introduce D-Skill, a framework that treats the teacher as a black box and distills its behavioral advantage into an interpretable natural-language skill by analyzing paired rollouts and identifying behavioral divergences and agreements. The resulting lessons are consolidated offline, with any compatible skill-update backend, into a skill document that is provided to the student at inference time. D-Skill constructs the skill entirely offline from fixed trajectories, without labels, further environment interaction, or weight updates after rollout collection. Across SpreadsheetBench-Verified, ALFWorld, and chess puzzles spanning multiple teacher-student pairs, D-Skill closes 75%-95% of the teacher-student gap without touching model weights. After a single offline pass over the paired traces, D-Skill outperforms one-epoch SFT baselines and remains competitive with multi-epoch SFT. Under the same model family, the resulting student also incurs 5.7x-8.3x lower inference cost per task than the teacher. Our results demonstrate that natural-language skill distillation offers a performant, interpretable, and efficient alternative to weight-based capability transfer.

Figures & tables

Figure 1: D-Skill closes the gap at a fraction of its cost. D-Skill closes 84% of the teacher-student gap at

8.3\times

lower inference cost on SpreadsheetBench-Verified. Teacher: GPT-5.6 Terra; student: GPT-5.6 Luna.

Figure 2: Ground-truth-free skill distillation from paired rollout traces. (1) Rollout: Paired teacher and student traces are obtained from fresh rollouts or saved executions. (2) Difference analysis: A Difference Analyst Agent compares the fixed traces without answer keys, labels, or further environment interaction, extracting divergence and agreement items. (3) Skill update: A Consolidation Agent converts the lessons into patches, merges them, and applies the update to SKILL.md .

Table 1: Main results. Each block reports the bare student (floor), the student with the distilled D-Skill skill, and the teacher; %GC is the fraction of the teacher–student gap closed.

	Qwen3.6 35B-A3B		GPT-5.6 Luna
Method	Score	%GC	Score	%GC
Student trace only	$47.7$ $\pm$ 6.7	$38.3\%$	$49.8$ $\pm$ 8.8	$45.4\%$
Teacher trace only	$56.3$ $\pm$ 6.1	$55.2\%$	$63.5$ $\pm$ 5.4	$70.7\%$
D-Skill	$\mathbf{71.7}$ $\pm$ 3.6	$\mathbf{85.1\%}$	$\mathbf{73.3}$ $\pm$ 1.0	$\mathbf{89.0\%}$

Table 2: Teacher-student contrast improves performances. Results average 3 independent skills, each evaluated twice.

	Consolidation
Analyst	Sonnet	Qwen
Sonnet	$\mathbf{39.0}$ $\pm$ 3.5 (88%)	$34.3$ $\pm$ 3.4 (59%)
Qwen	$31.1$ $\pm$ 4.5 (39%)	$32.6$ $\pm$ 3.9 (48%)

Table 3: A good analyst is needed. Chess accuracy of the Qwen student with the skill produced by each analyst-consolidation pair. Results average 3 independent skills, each evaluated twice.

Method	Score	%GC
Student (Qwen3.6-35B-A3B)	$24.8$ $\pm$ 0.4	—
SkillOpt	$29.6$ $\pm$ 1.4	$29.7\%$
D-Skill w/ SkillOpt backend	$\mathbf{36.9}$ $\pm$ 5.7	$\mathbf{74.9\%}$
D-Skill	$\mathbf{39.0}$ $\pm$ 3.5	$\mathbf{87.7\%}$

Table 4: D-Skill is compatible with an alternative skill-update backend. Results average three independently distilled skills, each evaluated twice. The SkillOpt-backed variant retains SkillOpt’s held-out validation gate.

	Qwen3.6-35B-A3B		GPT-5.6 Luna
Method	Score	%GC	Score	%GC
Student	$28.0$ $\pm$ 1.4	—	$25.3$ $\pm$ 1.1	—
Teacher (DS-V4-Pro + Skill)	$79.3$ $\pm$ 1.8	100%	$79.3$ $\pm$ 1.8	100%
Trace2Skill (student trace, student-authored)	$33.0$ $\pm$ 4.2	$9.7\%$	$31.1$ $\pm$ 1.6	$10.7\%$
Trace2Skill (teacher trace, teacher-authored)	$34.2$ $\pm$ 2.8	$12.0\%$	$37.0$ $\pm$ 1.6	$21.7\%$
Trace2Skill (student trace, teacher-authored)	$36.0$ $\pm$ 1.1	$15.6\%$	$46.2$ $\pm$ 5.9	$38.6\%$

Table 5: SpreadsheetBench-Verified: D-Skill vs. Trace2Skill baselines. All numbers are means over six evaluations, comprising two passes for each of three independently distilled skills.

Panel of Figure 3 — Figure 3: One-pass D-Skill outperforms one-epoch SFT and remains competitive with its multi-epoch performance. Both methods use Qwen3.6-35B-A3B as the student and the same teacher-trajectory corpus on (a) SpreadsheetBench Verified and (b) ALFWorld. The x-axis reports the number of teacher tasks processed, and the shaded bands show SFT performance over epochs 2-5.

Figure 4: Inference cost per task at model list prices on (a) SpreadsheetBench Verified and (b) ALFWorld. Costs are reported in cents episode.

	Tasks	Total	Per task
SSB-V	$99$	3.28$	$3.3$ ¢
ALFWorld	$57$	1.51$	$2.6$ ¢

Table 6: One-time distillation cost GPT-5.6 Terra

\rightarrow

Luna, billed at list prices. Per task is per training task.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Input and output
<hanging></hanging>	No input. Lists every piece that is attacked and has no defender, for both sides.
<safety>m1, …</safety>	Up to five candidate moves. For each: whether it gives check (checkmate and stalemate named explicitly), the number of legal replies, every forcing reply (checks and captures with the captured piece named), attackers vs. defenders of the destination square, and the agent’s pieces left undefended afterwards.
<threats></threats>	No input. Null-move audit: the opponent’s forcing moves if the agent passed, and any immediate mate threats.
<board>move</board>	One move. The board after the agent’s move only, with material, undefended pieces, and check/checkmate status.
<simulate>move, r1, …</simulate>	One candidate and up to five opponent replies, each simulated independently from the position after the candidate: resulting board, material, undefended pieces, and whether the agent is in check.
<line>m1, r1, m2, …</line>	A complete line of two to six plies supplied by the agent (own moves and opponent replies alternating): the final board, material, undefended pieces, and check/mate status.

Appendix

Table 7: Analysis tools available to the chess agent. Every tool is a static computation on a copy of the board; nothing is searched or evaluated.

		episodes consumed
	Targets	20	40	60	80	99
SSB	actions	$32.5$ $\pm$ 2.8	$17.0$ $\pm$ 1.4	$28.3$ $\pm$ 5.3	$42.0$ $\pm$ 2.1	$49.5$ $\pm$ 0.7
SSB	reasoning + actions	$34.8$ $\pm$ 0.4	$41.0$ $\pm$ 0.7	$43.8$ $\pm$ 4.6	$57.0$ $\pm$ 2.1	$55.5$ $\pm$ 0.7
		10	20	30	40	50	57
ALFWorld	actions	$65.0$ $\pm$ 3.2	$65.3$ $\pm$ 2.7	$73.8$ $\pm$ 1.1	$74.2$ $\pm$ 1.6	$73.1$ $\pm$ 0.0	$75.8$ $\pm$ 0.5
ALFWorld	reasoning + actions	$62.7$ $\pm$ 4.2	$65.7$ $\pm$ 0.0	$53.4$ $\pm$ 1.6	$57.5$ $\pm$ 1.1	$71.6$ $\pm$ 0.0	$78.8$ $\pm$ 2.6

Appendix

Table 8: SFT within epoch 1

	Targets	epoch 2	epoch 3	epoch 4	epoch 5
SSB	actions	$54.3$ $\pm$ 2.5	$58.8$ $\pm$ 5.3	$64.5$ $\pm$ 2.1	$63.8$ $\pm$ 1.8
SSB	reasoning + actions	$61.5$ $\pm$ 0.7	$72.3$ $\pm$ 0.4	$70.8$ $\pm$ 0.4	$71.5$ $\pm$ 1.4
ALFWorld	actions	$79.5$ $\pm$ 1.6	$82.4$ $\pm$ 0.5	$82.0$ $\pm$ 1.1	$83.2$ $\pm$ 0.6
ALFWorld	reasoning + actions	$83.5$ $\pm$ 1.1	$86.2$ $\pm$ 1.6	$85.8$ $\pm$ 1.1	$86.6$ $\pm$ 2.1

Appendix

Table 9: SFT across epochs

Skill Distillation

See all 11 papers

SKILL-KD: Contrastive Skill Distillation for LLM Agents
Jul 30, 2026Qiming Shi, Yibo Dou, Jiawen Zhu +5Skill Distillation LLM Agent Skill Learning
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University +1
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
Apr 12, 2026Hao Wang, Guozhi Wang, Han Xiao +8RL for LLM Agents LLM Agent Skill Learning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong +2
SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization
Jun 4, 2026Qi Zhang, Zhaopeng Feng, Xiaonan Shi +8LLM Agent Self-Improvement LLM Agent Skill Learning
1Zhejiang University · 2Tongyi Lab · 3National University of Singapore

Jul 30, 2026J/K move · Enter open · S save

SKILL-KD: Contrastive Skill Distillation for LLM Agents

Qiming Shi, Yibo Dou, Jiawen Zhu, Yulong Tao+4

State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University+1

Skill Distillation LLM Agent Skill Learning LLM Agent Training Knowledge Distillation

Open paper AskPDF arXiv

Skill-based prompting has become a practical mechanism for improving LLM agents, yet existing methods often treat skills as summaries of the agent's own experience or of successful demonstrations. This creates a mismatch for weaker student agents. A failed trajectory may not reveal the missing knowledge or strategy, while a teacher trajectory may be too implicit to internalize. We propose SKILL-KD, a contrastive skill distillation framework that treats skills as an explicit distillation medium between agents of different capabilities. Given a student failure and the teacher trajectory on the same task, SKILL-KD distills their actionable discrepancy into a textual skill patch, evaluates it by re-running the student, and iteratively refines it when the student still fails. To prevent skill drift from repeated local updates, SKILL-KD maintains trace-linked edit histories and performs Drift-Aware Skill Consolidation to decide whether each patch is added, merged, or skipped. Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines.

Abstract

Skill Distillation

Explore similar work

SKILL-KD: Contrastive Skill Distillation for LLM Agents

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization