Agent Skills improve LLM agents by making task knowledge reusable across sessions. Learning transferable Skills requires evidence about a procedure and its conditions of use, which a successful task attempt can leave untested. We introduce Socratic Skill Learning (SSL), which interleaves expert teaching with Student practice. An Educator guides investigation while the Student remains the sole environment operator and Skill author, able to question, test, and adapt advice. To test reuse beyond the learning task, we construct SkillTransferBench, with 35 task families and 408 executable cases, pairing public variation specifications with held-out evaluation in fresh solver sessions. On the main 22-family, 155-case cohort, Fixed-Window SSL reaches 0.801 utility, versus 0.723 without a Skill, 0.761 for adapted Trace2Skill (trajectory distillation), and 0.751 for adapted SKILL-KD (contrastive distillation). Across three model pairs on shared families, interleaved guidance outperforms expert authoring and post-hoc teaching. Process analysis links guidance to Student writing and package revision; a case study shows the Student testing and refining a proposed procedure for later reuse. As demand for new lessons falls, the cost of fixed-window review grows. Issue-Guided SSL tracks unresolved questions across Skill revisions to select and focus review, reaching 0.805 utility with 61.7% fewer Educator tokens and 16.9% fewer total acquisition tokens than Fixed-Window SSL on the main cohort.
Figures & tables
Figure 1: Overview of the study. Panels show (A) task-family construction from six source benchmarks, (B) acquisition-protocol comparisons, external methods, and model pairs, and (C) the path from teaching demand to issue-guided review with utility and cost. Panel C summarizes Qwen experiments on the 22-family main cohort.
Figure 2: Task-family catalog and evaluation cohorts. (a) Domain and family composition, (b) family sizes, and (c) Qwen No-Skill Avg@3 scores on the 155 main-cohort cases. (d) Bar length is proportional to case counts in the full catalog, main cohort, and lite subset shared by the additional model pairs.
Figure 3: Socratic Skill Learning. (A) The Student tests guidance and authors the Skill; the Educator reviews recent work and the live package read-only. Only the Skill transfers to fresh solvers. (B) Fixed-Window SSL reviews every g tool calls; Issue-Guided SSL reviews unresolved issues when they open, change, or reopen, using focused context.
Skill method
All
Sys.
Serv.
Data
Sci.
Fin.
Office
No Skill score
72.3
78.6
58.2
68.1
66.8
78.0
90.4
Expert Authoring
72.8 +0.5
77.9 −0.6
58.7 +0.6
71.6 +3.5
72.4 +5.7
72.5 −5.5
91.4 +1.0
Post-hoc Teaching
74.8 +2.4
73.6 −4.9
64.4 +6.3
73.7 +5.6
76.4 +9.7
78.1 +0.0
91.3 +0.9
Trace2Skill
76.1 +3.8
85.0 +6.4
63.4 +5.2
71.0 +2.9
73.3 +6.5
77.9 −0.1
90.0 −0.4
SKILL-KD
75.1 +2.8
85.5 +6.9
60.9 +2.7
73.2 +5.1
68.2 +1.5
75.7 −2.4
90.3 −0.1
Fixed-Window SSL
80.1 +7.8
88.7 +10.1
67.3 +9.1
75.1 +6.9
77.0 +10.2
82.4 +4.4
94.1 +3.7
Table 1: Overall and domain transfer on the Qwen main cohort. Family-macro Avg@3 utilities across 22 families and 155 cases on a 0–100 scale; superscripts show gain over No Skill (pp). Domains: Sys. (systems), Serv. (services), Data (data processing), Sci. (scientific computing), Fin. (finance), and Office (office automation). Bold and underlined scores mark the best and second-best learning methods in each column.
Protocol
Qwen
DeepSeek
Kimi
Fixed-Window SSL
+4.1
+3.1
+8.2
Issue-Guided SSL
+4.5
+2.5
+7.4
Expert Authoring
+1.1
+1.2
+5.8
Post-hoc Teaching
−0.7
+0.5
+4.1
Table 2: Cross-model transfer and selective review. (a) Utility gain over No Skill (pp) on the shared lite subset (10 families, 77 cases). (b) Utility and Educator savings: Qwen main cohort (22 families, 155 cases); DeepSeek and Kimi lite subset. Savings and efficiency ratios compare Issue-Guided with Fixed-Window SSL. Qwen uses three repeats; DeepSeek and Kimi use one. Model pairs are defined in Appendix C.1 .
Figure 4: Teaching demand, Student development, and review cost. Qwen Fixed-Window SSL ( g=10 ), 22 families and 66 acquisition runs. Panels show (a) anchor scores and teaching rates at regular reviews, (b) teaching–silence differences in writing on the next tool call and Skill changes over ten calls, standardized by round with 95% family-bootstrap intervals, and (c) tokens per review before follow-up, with silent shares.
Verifier
Flagged
Selected line
Generic arithmetic
4
140
Financial relations
1
210
Table 3: Testing teaching advice. Qwen Fixed-Window SSL, HTML-finance family, repeat 1. Recorded anchor checks; the erroneous report entry is line 210.
Figure 5: Transfer utility and teaching efficiency across review schedules. Qwen main cohort: 22 families, 155 cases, 3 repeats. Panels compare (a) family-macro utility, (b) Educator tokens, and (c) transfer gain over No Skill per million Educator tokens, with shared schedule rows.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Protocol
Feedback enters through
Skill update mechanism
Trace2Skill
Local trajectory analysis
Extract, merge, and apply lessons
SkillGen
Candidate verification
Generate, verify, and refine packages
SKILL-KD
Teacher–Student trajectory contrast and Student reruns
Consolidation agent proposes and validates patches
SkillLearnBench: Teacher Feedback
Advice after failure, using a reference Skill
Learner revises the Skill before retrying
Fixed-Window SSL
Educator review during practice at fixed tool-call intervals
Student tests advice and writes the live package
Issue-Guided SSL
Review on unresolved issue transitions with local evidence
Student revisions and anchor feedback update persistent issue state
Appendix
Table 4: Feedback and package updates in representative Skill-learning protocols. SSL combines external guidance during ongoing practice with Student-owned tool use and package writing.
Domain
Family
Catalog cases
Main held-out
Systems
Deterministic tarball
12
6
ACL inheritance
13
6
SQLite truncation
14
6
DB WAL recovery
11
7
Git leak recovery
13
6
Recover obfuscated files
7
–
Appendix
Table 5: Curated task-family catalog and main evaluation cohort, showing case counts for each family within domain groupings and totals across the catalog.
Task volume, emergencies, priority rules, dependencies, API capabilities.
SPARQL professors
Rank semantics, affiliation paths, graph noise and duplicate bindings.
Audio STFT peaks
Signal recipe, STFT settings, peak selection and shell output contract.
PyPI server
Package layout, public API, index configuration, publication and installation.
Organization JSON
Entity relationships, field normalization, aggregation distributions.
Appendix
Table 6: Public variation axes for each family in the main evaluation cohort, summarizing the changes across cases declared in the family specifications.
Protocol
Operator / author
Review opportunity
Continuation
Expert Authoring
External model
No separate Educator
Author revises from anchor feedback
Post-hoc Teaching
Student
Once after the authoring trajectory in each round
Student checks and revises
Fixed-Window SSL
Student
Every g tool calls and an unreviewed tail
Student responds; one optional Educator follow-up
Issue-Guided SSL
Student
Unresolved issue transitions
Student investigates and commits revisions before further review
Appendix
Table 7: Authoring and teaching protocols. Each uses 3 acquisition rounds with fresh-solver anchor feedback between rounds.
Condition
Utility ↑
Gain (pp)
95% CI
Full-pass (%) ↑
Tokens/run (M)
No Skill
0.723
–
–
32.50
–
Expert Authoring
0.728
+0.46
[−5.46,6.56]
40.44
4.95
Post-hoc Teaching
0.748
+2.44
[−3.36,7.90]
40.37
8.71
Adapted Trace2Skill
0.761
+3.76
[−0.50,8.96]
39.11
–
Adapted SKILL-KD
0.751
+2.76
[−1.59,8.09]
41.65
–
Fixed-Window SSL ( g=10 )
0.801
+7.75
[3.53,12.68]
49.49
6.89
Appendix
Table 8: Utility, full-pass rate, and acquisition cost on the Qwen 22-family, 155-case main cohort (Avg@3, family-macro). Gain and 95% CI refer to utility differences from No Skill.
Family
No Skill
Expert Authoring
Post-hoc Teaching
Trace2Skill
SKILL-KD
Fixed- Window SSL
Issue- Guided SSL
ACL inheritance
0.933
0.728
0.858
0.885
0.933
0.933
0.881
API scheduler
0.767
0.730
0.745
0.897
0.921
0.791
0.830
Audio STFT peaks
0.563
0.631
0.652
0.608
0.664
0.643
0.656
Bank transaction filter
0.678
0.548
0.550
0.522
0.515
0.593
0.652
Cross-entropy method
0.553
0.556
0.642
0.617
0.556
0.667
0.656
DB WAL recovery
0.473
0.826
0.705
0.905
0.901
0.884
0.878
Appendix
Table 9: Per-family Avg@3 scores on the Qwen main cohort (22 families, 155 cases), including adapted Trace2Skill and SKILL-KD. Only the last row is a macro-average across task families.
Method
Improved
Unchanged
Regressed
All cases ( n=155 )
Expert Authoring
58
56
41
Post-hoc Teaching
57
53
45
Adapted Trace2Skill
59
65
31
Fixed-Window SSL
72
59
24
Issue-Guided SSL
71
63
21
Appendix
Table 10: Case-level Avg@3 transfer directions relative to No Skill on the Qwen main cohort (22 families, 155 cases), with non-ceiling and ceiling subsets for Fixed-Window SSL and Issue-Guided SSL.
Condition
Repeat 1
Repeat 2
Repeat 3
SD
Mean within-family SD
No Skill
0.719
0.730
0.721
0.006
0.036
Fixed-Window SSL
0.772
0.799
0.831
0.030
0.031
Issue-Guided SSL
0.767
0.810
0.838
0.036
0.037
Adapted SKILL-KD
0.746
0.756
0.750
0.005
0.039
Appendix
Table 11: Repeat variation on the Qwen main cohort (22 families, 155 cases): family-macro utility for each repeat, its standard deviation across repeats, and mean within-family variability.
DeepSeek
Kimi
Condition
Utility
Gain (pp)
Utility
Gain (pp)
No Skill
.754
–
.679
–
Expert Authoring
.766
+1.21
.738
+5.85
Post-hoc Teaching
.759
+0.54
.721
+4.15
Fixed-Window SSL
.784
+3.08
.762
+8.24
Issue-Guided SSL
.779
+2.50
.753
+7.36
Appendix
Table 12: Utility and gain over No Skill for DeepSeek and Kimi on the lite subset (10 families, 77 cases; single repeat).
Contrast
Qwen (shared)
DeepSeek
Kimi
Fixed-Window SSL − No Skill
[+1.02, +8.03]
[ − 4.63, +13.03]
[+2.48, +15.62]
Issue-Guided SSL − Fixed-Window SSL
[ − 0.84, +1.44]
[ − 5.34, +3.14]
[ − 4.72, +2.89]
Fixed-Window SSL − Expert Authoring
[ − 0.83, +7.14]
[ − 1.93, +7.15]
[ − 2.33, +7.98]
Expert Authoring − Post-hoc Teaching
[ − 1.28, +6.05]
[ − 0.89, +2.23]
[ − 1.13, +4.76]
Appendix
Table 13: Headline contrasts on the shared lite subset (10 families, 77 cases): 95% paired family-bootstrap intervals for Table 2 (a). Qwen averages 3 repeats; DeepSeek and Kimi use one.
Family
No Skill
Expert Authoring
Post-hoc Teaching
Fixed-Window SSL
Issue-Guided SSL
DeepSeek
Audio STFT peaks
.668
.657
.677
.637
.570
Filter JS from HTML
.315
.485
.452
.712
.529
Git leak recovery
.963
.891
.891
.912
.912
HTML finance verification
.833
1.000
1.000
1.000
1.000
JQ processing
.909
.891
.846
.817
.835
Appendix
Table 14: Per-family scores for DeepSeek and Kimi on the lite subset (10 families, 77 cases; single repeat). Qwen scores for these families appear in Table 9 .
Intent
Operational target
Opening
Follow-up
Knowledge transfer
Facts, mechanisms, APIs, formats, boundaries, or expert experience
101 (39.8%)
19 (20.9%)
Reasoning guidance
Representation, decomposition, causal reasoning, comparison, generalization, or decisions
168 (66.1%)
43 (47.3%)
Exploration/verification
Search, experimental design, evidence collection, validation, or tool-use procedure
23 (9.1%)
21 (23.1%)
Error diagnosis/recovery
Attribution of an observed error or stuck state and a recovery direction
99 (39.0%)
46 (50.5%)
Epistemic calibration
Unsupported assumptions, over-generalization, evidence levels, or confidence
39 (15.4%)
11 (12.1%)
Experience consolidation
Retrievable triggers, rules, workflows, scripts, or references from local experience
104 (40.9%)
25 (27.5%)
Appendix
Table 15: Intent definitions and tag frequencies for Qwen Fixed-Window SSL ( g=10 ; 22 families, 66 acquisition runs). Opening lessons and follow-up turns use separate denominators of 254 and 91.
Outcome and comparison
n
Adjusted difference (pp)
Teaching: observed tool error vs no error
432
−0.4 [ −9.2 , +8.1 ]
Teaching: shell failure vs no failure
413
−8.5 [ −20.8 , +2.6 ]
Teaching: observable vs unobservable package
233
−31.5 [ −41.7 , −22.8 ]
Teaching: preceding package change vs none
130
−10.2 [ −28.7 , +12.4 ]
Teaching: environment-dominant vs other work
432
+1.5 [ −8.2 , +10.1 ]
Teaching: write-dominant vs other work
432
+1.8 [ −21.4 , +25.2 ]
Appendix
Table 16: Round-standardized review-behavior contrasts for Qwen Fixed-Window SSL ( g=10 ; 22 families, 66 acquisition runs). Counts give supported observation windows, and brackets give cluster-bootstrap confidence intervals over task families.
Figure 6: Teaching emphasis and development after anchor success. Qwen Fixed-Window SSL ( g=10 ), 22 families and 66 acquisition runs. (a) Educator-reported intents in first-round versus later lessons; labels may overlap. (b) Package revisions in rounds following a perfect anchor score, grouped by whether a new lesson arrives.
Teaching schedule
Utility ↑
Tokens/run (M) ↓
Reviews
Lessons
Every 5 tool calls
0.776
9.56
1126
403
Every 10 tool calls
0.801
6.89
630
254
Every 20 tool calls
0.776
6.61
375
169
Issue transitions (Issue-Guided SSL)
0.805
5.72
329
135
Appendix
Table 17: Utility, acquisition cost, reviews, and opening lessons for the four Qwen schedules on the main cohort (22 families, 155 cases; 3 repeats and 66 acquisition runs per schedule).
Figure 7: Acquisition-cost allocation across the four teaching schedules. Qwen, 22 families and 66 acquisition runs per schedule. Stacked bars show mean Student-author, Educator, and anchor-evaluation tokens per run, with the total labeled for each schedule.
Pair
Runs
Educator saved
Author added
Anchor added
Net saved
Qwen
66
1.479
0.218
0.097
1.164
DeepSeek
10
4.175
5.276
–
− 1.101
Kimi
10
3.250
− 0.870
–
4.120
Appendix
Table 18: Role-level token changes and net acquisition savings for Issue-Guided SSL relative to Fixed-Window SSL, in millions of tokens per run. Qwen uses the 22 main-cohort families (3 repeats); DeepSeek and Kimi use the 10 shared families (one repeat). DeepSeek and Kimi rows cover the persistent author and Educator sessions.
Pair
Schedule
Student (M/run)
Educator (M/run)
Total (M/run)
DeepSeek
Fixed-Window SSL
13.46
6.55
20.01
Issue-Guided SSL
18.74
2.37
21.11
Kimi
Fixed-Window SSL
5.41
4.20
9.61
Issue-Guided SSL
4.54
0.95
5.49
Appendix
Table 19: Per-run acquisition tokens by session role for Fixed-Window SSL and Issue-Guided SSL for DeepSeek and Kimi on the 10-family lite subset (single repeat), covering the persistent author (Student) and Educator sessions.
Persistent issues
Count or summary
Persistent issue nodes
177
Issues/run: mean; median; range
2.68; 3; 1–6
Nodes reopened at least once
23/177 (13.0%)
Reopen events
28
Reopen events followed by a new lesson
28/28
Nodes closed by run end
175
Appendix
Table 20: Qwen Issue-Guided SSL lifecycle accounting across 22 families and 66 acquisition runs, listing persistent issues, reopen events, closure dispositions, and issue status at run end.
Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience. In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them. When agent-induced skills transfer reliably across tasks remains an open question. We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks. Specifically, we compare task-level with subtask-level skill induction and text with code skill formats, the two axes along which existing methods differ. Task-level skills mostly reduce the agent's performance below its no-memory baseline while subtask-level skills raise it above on average, and text skills transfer better than code skills. To further understand our findings, we examine two complementary properties of the induced skills: specificity, which measures how closely a skill matches real tasks, and abstractness, which measures how evenly its relevance spreads across tasks. Neither property alone predicts task success, but their combined effect does, which we propose as a skill utility score. The score correlates consistently with task success when skills are transferred, and subtask-level and text skills score higher. Computing skill utility only needs the skills and task descriptions, so our score serves as a lightweight diagnostic of a skill memory for practitioners.
Skill-based prompting has become a practical mechanism for improving LLM agents, yet existing methods often treat skills as summaries of the agent's own experience or of successful demonstrations. This creates a mismatch for weaker student agents. A failed trajectory may not reveal the missing knowledge or strategy, while a teacher trajectory may be too implicit to internalize. We propose SKILL-KD, a contrastive skill distillation framework that treats skills as an explicit distillation medium between agents of different capabilities. Given a student failure and the teacher trajectory on the same task, SKILL-KD distills their actionable discrepancy into a textual skill patch, evaluates it by re-running the student, and iteratively refines it when the student still fails. To prevent skill drift from repeated local updates, SKILL-KD maintains trace-linked edit histories and performs Drift-Aware Skill Consolidation to decide whether each patch is added, merged, or skipped. Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines.
Qiming Shi, Yibo Dou, Jiawen Zhu +5
State Key Lab of CAD&CG, Zhejiang University · School of Software and Microelectronics, Peking University · School of Software Technology, Zhejiang University +1
Large Language Model (LLM) agents increasingly rely on domain-specific skills, yet manually authoring such skills does not scale, and skills generated purely from parametric knowledge often miss critical operational pitfalls. We introduce Trace2Skill, a framework that consolidates broad execution trajectories in parallel into a unified skill directory through inductive reasoning over agent experience. Trace2Skill supports both deepening existing human-written skills and creating useful skills from weak LLM-generated drafts. Experiments demonstrate the effectiveness of Trace2Skill across diverse domains, including office workflows, math reasoning, and vision QA. Importantly, the evolved skills are not merely memorized artifacts of the trajectories used to create them: they often transfer across model scales, across model families, and to out-of-distribution settings. For example, skills evolved from Qwen3.5-35B trajectories improve a Qwen3.5-122B agent by up to 57.65 percentage points on WikiTableQuestions. Further analyses show that Trace2Skill outperforms sequential skill editing and ReasoningBank-style retrieval memories, compresses recurring failures and workarounds into standard operating procedures (SoPs), and yields portable skills that can be reused without parameter updates or test-time retrieval.
Jingwei Ni, Yihao Liu, Xinpeng Liu +7
ETH Zürich · University of Zurich · Peking University +2