Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills

Letian Zhu; Han Li; Yifan Yao; Xinping Lei; Xinyu Che; Jiahao Wang; Dianyi Wang; Yucheng Fu

cs.AIOct 13, 2026

Socratic Skill Learning: Interleaving Teaching and Practice for Transferable Agent Skills

Authors: Letian Zhu, Han Li, Yifan Yao, Xinping Lei, Xinyu Che, Jiahao Wang, Dianyi Wang, Yucheng Fu,

Organizations: Nanjing University · Alibaba Group

Abstract

Agent Skills improve LLM agents by making task knowledge reusable across sessions. Learning transferable Skills requires evidence about a procedure and its conditions of use, which a successful task attempt can leave untested. We introduce Socratic Skill Learning (SSL), which interleaves expert teaching with Student practice. An Educator guides investigation while the Student remains the sole environment operator and Skill author, able to question, test, and adapt advice. To test reuse beyond the learning task, we construct SkillTransferBench, with 35 task families and 408 executable cases, pairing public variation specifications with held-out evaluation in fresh solver sessions. On the main 22-family, 155-case cohort, Fixed-Window SSL reaches 0.801 utility, versus 0.723 without a Skill, 0.761 for adapted Trace2Skill (trajectory distillation), and 0.751 for adapted SKILL-KD (contrastive distillation). Across three model pairs on shared families, interleaved guidance outperforms expert authoring and post-hoc teaching. Process analysis links guidance to Student writing and package revision; a case study shows the Student testing and refining a proposed procedure for later reuse. As demand for new lessons falls, the cost of fixed-window review grows. Issue-Guided SSL tracks unresolved questions across Skill revisions to select and focus review, reaching 0.805 utility with 61.7% fewer Educator tokens and 16.9% fewer total acquisition tokens than Fixed-Window SSL on the main cohort.

Figures & tables

Figure 1: Overview of the study. Panels show (A) task-family construction from six source benchmarks, (B) acquisition-protocol comparisons, external methods, and model pairs, and (C) the path from teaching demand to issue-guided review with utility and cost. Panel C summarizes Qwen experiments on the 22-family main cohort.

Panel of Figure 2 — Figure 2: Task-family catalog and evaluation cohorts. (a) Domain and family composition, (b) family sizes, and (c) Qwen No-Skill Avg@3 scores on the 155 main-cohort cases. (d) Bar length is proportional to case counts in the full catalog, main cohort, and lite subset shared by the additional model pairs.

Figure 3: Socratic Skill Learning. (A) The Student tests guidance and authors the Skill; the Educator reviews recent work and the live package read-only. Only the Skill transfers to fresh solvers. (B) Fixed-Window SSL reviews every

g

tool calls; Issue-Guided SSL reviews unresolved issues when they open, change, or reopen, using focused context.

Skill method	All	Sys.	Serv.	Data	Sci.	Fin.	Office
No Skill score	72.3	78.6	58.2	68.1	66.8	78.0	90.4
Expert Authoring	72.8 $+0.5$	77.9 $-0.6$	58.7 $+0.6$	71.6 $+3.5$	72.4 $+5.7$	72.5 $-5.5$	91.4 $+1.0$
Post-hoc Teaching	74.8 $+2.4$	73.6 $-4.9$	64.4 $+6.3$	73.7 $+5.6$	76.4 $+9.7$	78.1 $+0.0$	91.3 $+0.9$
Trace2Skill	76.1 $+3.8$	85.0 $+6.4$	63.4 $+5.2$	71.0 $+2.9$	73.3 $+6.5$	77.9 $-0.1$	90.0 $-0.4$
SKILL-KD	75.1 $+2.8$	85.5 $+6.9$	60.9 $+2.7$	73.2 $+5.1$	68.2 $+1.5$	75.7 $-2.4$	90.3 $-0.1$
Fixed-Window SSL	80.1 $+7.8$	88.7 $+10.1$	67.3 $+9.1$	75.1 $+6.9$	77.0 $+10.2$	82.4 $+4.4$	94.1 $+3.7$

Table 1: Overall and domain transfer on the Qwen main cohort. Family-macro Avg@3 utilities across 22 families and 155 cases on a 0–100 scale; superscripts show gain over No Skill (pp). Domains: Sys. (systems), Serv. (services), Data (data processing), Sci. (scientific computing), Fin. (finance), and Office (office automation). Bold and underlined scores mark the best and second-best learning methods in each column.

Protocol	Qwen	DeepSeek	Kimi
Fixed-Window SSL	$+4.1$	$+3.1$	$+8.2$
Issue-Guided SSL	$+4.5$	$+2.5$	$+7.4$
Expert Authoring	$+1.1$	$+1.2$	$+5.8$
Post-hoc Teaching	$-0.7$	$+0.5$	$+4.1$

Table 2: Cross-model transfer and selective review. (a) Utility gain over No Skill (pp) on the shared lite subset (10 families, 77 cases). (b) Utility and Educator savings: Qwen main cohort (22 families, 155 cases); DeepSeek and Kimi lite subset. Savings and efficiency ratios compare Issue-Guided with Fixed-Window SSL. Qwen uses three repeats; DeepSeek and Kimi use one. Model pairs are defined in Appendix C.1 .

Panel of Figure 4 — Figure 4: Teaching demand, Student development, and review cost. Qwen Fixed-Window SSL ( $g=10$ ), 22 families and 66 acquisition runs. Panels show (a) anchor scores and teaching rates at regular reviews, (b) teaching–silence differences in writing on the next tool call and Skill changes over ten calls, standardized by round with 95% family-bootstrap intervals, and (c) tokens per review before follow-up, with silent shares.

Verifier	Flagged	Selected line
Generic arithmetic	4	140
Financial relations	1	210

Table 3: Testing teaching advice. Qwen Fixed-Window SSL, HTML-finance family, repeat 1. Recorded anchor checks; the erroneous report entry is line 210.

Panel of Figure 5 — Figure 5: Transfer utility and teaching efficiency across review schedules. Qwen main cohort: 22 families, 155 cases, 3 repeats. Panels compare (a) family-macro utility, (b) Educator tokens, and (c) transfer gain over No Skill per million Educator tokens, with shared schedule rows.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Protocol	Feedback enters through	Skill update mechanism
Trace2Skill	Local trajectory analysis	Extract, merge, and apply lessons
SkillGen	Candidate verification	Generate, verify, and refine packages
SKILL-KD	Teacher–Student trajectory contrast and Student reruns	Consolidation agent proposes and validates patches
SkillLearnBench: Teacher Feedback	Advice after failure, using a reference Skill	Learner revises the Skill before retrying
Fixed-Window SSL	Educator review during practice at fixed tool-call intervals	Student tests advice and writes the live package
Issue-Guided SSL	Review on unresolved issue transitions with local evidence	Student revisions and anchor feedback update persistent issue state

Appendix

Table 4: Feedback and package updates in representative Skill-learning protocols. SSL combines external guidance during ongoing practice with Student-owned tool use and package writing.

Domain	Family	Catalog cases	Main held-out
Systems	Deterministic tarball	12	6
	ACL inheritance	13	6
	SQLite truncation	14	6
	DB WAL recovery	11	7
	Git leak recovery	13	6
	Recover obfuscated files	7	–

Appendix

Table 5: Curated task-family catalog and main evaluation cohort, showing case counts for each family within domain groupings and totals across the catalog.

Family	What changes across cases
Filter JS from HTML	Executable constructs, preservation requirements, malformed syntax, encodings.
API scheduler	Task volume, emergencies, priority rules, dependencies, API capabilities.
SPARQL professors	Rank semantics, affiliation paths, graph noise and duplicate bindings.
Audio STFT peaks	Signal recipe, STFT settings, peak selection and shell output contract.
PyPI server	Package layout, public API, index configuration, publication and installation.
Organization JSON	Entity relationships, field normalization, aggregation distributions.

Appendix

Table 6: Public variation axes for each family in the main evaluation cohort, summarizing the changes across cases declared in the family specifications.

Protocol	Operator / author	Review opportunity	Continuation
Expert Authoring	External model	No separate Educator	Author revises from anchor feedback
Post-hoc Teaching	Student	Once after the authoring trajectory in each round	Student checks and revises
Fixed-Window SSL	Student	Every $g$ tool calls and an unreviewed tail	Student responds; one optional Educator follow-up
Issue-Guided SSL	Student	Unresolved issue transitions	Student investigates and commits revisions before further review

Appendix

Table 7: Authoring and teaching protocols. Each uses 3 acquisition rounds with fresh-solver anchor feedback between rounds.

Condition	Utility $\uparrow$	Gain (pp)	95% CI	Full-pass (%) $\uparrow$	Tokens/run (M)
No Skill	$0.723$	–	–	$32.50$	–
Expert Authoring	$0.728$	$+0.46$	$[-5.46,6.56]$	$40.44$	$4.95$
Post-hoc Teaching	$0.748$	$+2.44$	$[-3.36,7.90]$	$40.37$	$8.71$
Adapted Trace2Skill	$0.761$	$+3.76$	$[-0.50,8.96]$	$39.11$	–
Adapted SKILL-KD	$0.751$	$+2.76$	$[-1.59,8.09]$	$41.65$	–
Fixed-Window SSL ( $g=10$ )	$0.801$	$+7.75$	$[3.53,12.68]$	$49.49$	$6.89$

Appendix

Table 8: Utility, full-pass rate, and acquisition cost on the Qwen 22-family, 155-case main cohort (Avg@3, family-macro). Gain and 95% CI refer to utility differences from No Skill.

Family	No Skill	Expert Authoring	Post-hoc Teaching	Trace2Skill	SKILL-KD	Fixed- Window SSL	Issue- Guided SSL
ACL inheritance	0.933	0.728	0.858	0.885	0.933	0.933	0.881
API scheduler	0.767	0.730	0.745	0.897	0.921	0.791	0.830
Audio STFT peaks	0.563	0.631	0.652	0.608	0.664	0.643	0.656
Bank transaction filter	0.678	0.548	0.550	0.522	0.515	0.593	0.652
Cross-entropy method	0.553	0.556	0.642	0.617	0.556	0.667	0.656
DB WAL recovery	0.473	0.826	0.705	0.905	0.901	0.884	0.878

Appendix

Table 9: Per-family Avg@3 scores on the Qwen main cohort (22 families, 155 cases), including adapted Trace2Skill and SKILL-KD. Only the last row is a macro-average across task families.

All cases ( $n=155{}$ )
Method	Improved	Unchanged	Regressed
Expert Authoring	58	56	41
Post-hoc Teaching	57	53	45
Adapted Trace2Skill	59	65	31
Fixed-Window SSL	72	59	24
Issue-Guided SSL	71	63	21

Appendix

Table 10: Case-level Avg@3 transfer directions relative to No Skill on the Qwen main cohort (22 families, 155 cases), with non-ceiling and ceiling subsets for Fixed-Window SSL and Issue-Guided SSL.

Condition	Repeat 1	Repeat 2	Repeat 3	SD	Mean within-family SD
No Skill	0.719	0.730	0.721	0.006	0.036
Fixed-Window SSL	0.772	0.799	0.831	0.030	0.031
Issue-Guided SSL	0.767	0.810	0.838	0.036	0.037
Adapted SKILL-KD	0.746	0.756	0.750	0.005	0.039

Appendix

Table 11: Repeat variation on the Qwen main cohort (22 families, 155 cases): family-macro utility for each repeat, its standard deviation across repeats, and mean within-family variability.

	DeepSeek		Kimi
Condition	Utility	Gain (pp)	Utility	Gain (pp)
No Skill	.754	–	.679	–
Expert Authoring	.766	+1.21	.738	+5.85
Post-hoc Teaching	.759	+0.54	.721	+4.15
Fixed-Window SSL	.784	+3.08	.762	+8.24
Issue-Guided SSL	.779	+2.50	.753	+7.36

Appendix

Table 12: Utility and gain over No Skill for DeepSeek and Kimi on the lite subset (10 families, 77 cases; single repeat).

Contrast	Qwen (shared)	DeepSeek	Kimi
Fixed-Window SSL $-$ No Skill	[+1.02, +8.03]	[ $-$ 4.63, +13.03]	[+2.48, +15.62]
Issue-Guided SSL $-$ Fixed-Window SSL	[ $-$ 0.84, +1.44]	[ $-$ 5.34, +3.14]	[ $-$ 4.72, +2.89]
Fixed-Window SSL $-$ Expert Authoring	[ $-$ 0.83, +7.14]	[ $-$ 1.93, +7.15]	[ $-$ 2.33, +7.98]
Expert Authoring $-$ Post-hoc Teaching	[ $-$ 1.28, +6.05]	[ $-$ 0.89, +2.23]	[ $-$ 1.13, +4.76]

Appendix

Table 13: Headline contrasts on the shared lite subset (10 families, 77 cases): 95% paired family-bootstrap intervals for Table 2 (a). Qwen averages 3 repeats; DeepSeek and Kimi use one.

Family	No Skill	Expert Authoring	Post-hoc Teaching	Fixed-Window SSL	Issue-Guided SSL
DeepSeek
Audio STFT peaks	.668	.657	.677	.637	.570
Filter JS from HTML	.315	.485	.452	.712	.529
Git leak recovery	.963	.891	.891	.912	.912
HTML finance verification	.833	1.000	1.000	1.000	1.000
JQ processing	.909	.891	.846	.817	.835

Appendix

Table 14: Per-family scores for DeepSeek and Kimi on the lite subset (10 families, 77 cases; single repeat). Qwen scores for these families appear in Table 9 .

Intent	Operational target	Opening	Follow-up
Knowledge transfer	Facts, mechanisms, APIs, formats, boundaries, or expert experience	101 (39.8%)	19 (20.9%)
Reasoning guidance	Representation, decomposition, causal reasoning, comparison, generalization, or decisions	168 (66.1%)	43 (47.3%)
Exploration/verification	Search, experimental design, evidence collection, validation, or tool-use procedure	23 (9.1%)	21 (23.1%)
Error diagnosis/recovery	Attribution of an observed error or stuck state and a recovery direction	99 (39.0%)	46 (50.5%)
Epistemic calibration	Unsupported assumptions, over-generalization, evidence levels, or confidence	39 (15.4%)	11 (12.1%)
Experience consolidation	Retrievable triggers, rules, workflows, scripts, or references from local experience	104 (40.9%)	25 (27.5%)

Appendix

Table 15: Intent definitions and tag frequencies for Qwen Fixed-Window SSL (

g=10

; 22 families, 66 acquisition runs). Opening lessons and follow-up turns use separate denominators of 254 and 91.

Outcome and comparison	$n$	Adjusted difference (pp)
Teaching: observed tool error vs no error	432	$-0.4$ [ $-9.2$ , $+8.1$ ]
Teaching: shell failure vs no failure	413	$-8.5$ [ $-20.8$ , $+2.6$ ]
Teaching: observable vs unobservable package	233	$-31.5$ [ $-41.7$ , $-22.8$ ]
Teaching: preceding package change vs none	130	$-10.2$ [ $-28.7$ , $+12.4$ ]
Teaching: environment-dominant vs other work	432	$+1.5$ [ $-8.2$ , $+10.1$ ]
Teaching: write-dominant vs other work	432	$+1.8$ [ $-21.4$ , $+25.2$ ]

Appendix

Table 16: Round-standardized review-behavior contrasts for Qwen Fixed-Window SSL (

g=10

; 22 families, 66 acquisition runs). Counts give supported observation windows, and brackets give cluster-bootstrap confidence intervals over task families.

Figure 6: Teaching emphasis and development after anchor success. Qwen Fixed-Window SSL (

g=10

), 22 families and 66 acquisition runs. (a) Educator-reported intents in first-round versus later lessons; labels may overlap. (b) Package revisions in rounds following a perfect anchor score, grouped by whether a new lesson arrives.

Teaching schedule	Utility $\uparrow$	Tokens/run (M) $\downarrow$	Reviews	Lessons
Every 5 tool calls	$0.776$	$9.56$	$1126$	$403$
Every 10 tool calls	$0.801$	$6.89$	$630$	$254$
Every 20 tool calls	$0.776$	$6.61$	$375$	$169$
Issue transitions (Issue-Guided SSL)	0.805	5.72	329	135

Appendix

Table 17: Utility, acquisition cost, reviews, and opening lessons for the four Qwen schedules on the main cohort (22 families, 155 cases; 3 repeats and 66 acquisition runs per schedule).

Figure 7: Acquisition-cost allocation across the four teaching schedules. Qwen, 22 families and 66 acquisition runs per schedule. Stacked bars show mean Student-author, Educator, and anchor-evaluation tokens per run, with the total labeled for each schedule.

Pair	Runs	Educator saved	Author added	Anchor added	Net saved
Qwen	66	1.479	0.218	0.097	1.164
DeepSeek	10	4.175	5.276	–	$-$ 1.101
Kimi	10	3.250	$-$ 0.870	–	4.120

Appendix

Table 18: Role-level token changes and net acquisition savings for Issue-Guided SSL relative to Fixed-Window SSL, in millions of tokens per run. Qwen uses the 22 main-cohort families (3 repeats); DeepSeek and Kimi use the 10 shared families (one repeat). DeepSeek and Kimi rows cover the persistent author and Educator sessions.

Pair	Schedule	Student (M/run)	Educator (M/run)	Total (M/run)
DeepSeek	Fixed-Window SSL	13.46	6.55	20.01
	Issue-Guided SSL	18.74	2.37	21.11
Kimi	Fixed-Window SSL	5.41	4.20	9.61
	Issue-Guided SSL	4.54	0.95	5.49

Appendix

Table 19: Per-run acquisition tokens by session role for Fixed-Window SSL and Issue-Guided SSL for DeepSeek and Kimi on the 10-family lite subset (single repeat), covering the persistent author (Student) and Educator sessions.

Persistent issues	Count or summary
Persistent issue nodes	177
Issues/run: mean; median; range	2.68; 3; 1–6
Nodes reopened at least once	23/177 (13.0%)
Reopen events	28
Reopen events followed by a new lesson	28/28
Nodes closed by run end	175

Appendix

Table 20: Qwen Issue-Guided SSL lifecycle accounting across 22 families and 66 acquisition runs, listing persistent issues, reopen events, closure dispositions, and issue status at run end.

LLM Agent Skill Learning

Large language model agent skill learning studies how agents acquire, represent, reuse, and refine procedures for multi-step tasks.

See all 402 papers

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents
Aug 20, 2026Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian +1LLM Agent Skill Learning LLM Agent Skill Evaluation
Computer Science Department, Stony Brook University, Stony Brook, USA
SKILL-KD: Contrastive Skill Distillation for LLM Agents
Jul 30, 2026Qiming Shi, Yibo Dou, Jiawen Zhu +5Skill Distillation LLM Agent Skill Learning
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University +1
Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills
Mar 26, 2026Jingwei Ni, Yihao Liu, Xinpeng Liu +7LLM Agent Skill Learning LLM Agent Self-Improvement
ETH Zürich · University of Zurich · Peking University +2

Aug 20, 2026J/K move · Enter open · S save

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents

Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, Jiawei Zhou

Computer Science Department, Stony Brook University, Stony Brook, USA

LLM Agent Skill Learning LLM Agent Skill Evaluation Memory-Augmented Language Model Agents Transfer Learning Negative Transfer

Open paper AskPDF arXiv

Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience. In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them. When agent-induced skills transfer reliably across tasks remains an open question. We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks. Specifically, we compare task-level with subtask-level skill induction and text with code skill formats, the two axes along which existing methods differ. Task-level skills mostly reduce the agent's performance below its no-memory baseline while subtask-level skills raise it above on average, and text skills transfer better than code skills. To further understand our findings, we examine two complementary properties of the induced skills: specificity, which measures how closely a skill matches real tasks, and abstractness, which measures how evenly its relevance spreads across tasks. Neither property alone predicts task success, but their combined effect does, which we propose as a skill utility score. The score correlates consistently with task success when skills are transferred, and subtask-level and text skills score higher. Computing skill utility only needs the skills and task descriptions, so our score serves as a lightweight diagnostic of a skill memory for practitioners.

Abstract

LLM Agent Skill Learning

Explore similar work

Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents

SKILL-KD: Contrastive Skill Distillation for LLM Agents

Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills