Skills, as a useful abstraction for the procedural capabilities of large language models (LLMs), capture how models perform structured, multi-step reasoning and program execution. Existing approaches typically treat skills as explicit, surface-level constructs specified through prompts or programs, leaving open the question of how such procedural capabilities are represented inside the model and whether they can be manipulated as structured objects in latent space. In this empirical study, we investigate whether procedural LLM skills can be represented as directions in activation space and whether vector-space operations over these directions can express skill-level behaviors. We find that procedural skills admit a vector-space representation: individual skill directions can be activated to shift model behavior; independently extracted directions can compose to form higher-level skills. Contrastive directions yield context-conditioned algorithmic personalization and optimization trajectories over skill directions evolve non-monotonically, with intermediate states often surpassing fully optimized solutions. These results support a representation-level view of procedural LLM skills: they admit a latent vector-space organization that allows direct manipulation through internal interventions.
Figures & tables
Figure 1: Latent LLM Skills as Representation-Level Objects. We study four observable properties of skill-conditioned directions in activation space: steerability , composition , personalization , and evolution .
Figure 2: Per-skill accuracy on SkillSet-Math for Qwen3-0.6B: unsteered baseline vs. steered (PCA-extracted skill direction).
Model
0-shot
1-shot
2-shot
3-shot
4-shot
Steered
GSM8K
Qwen3-0.6B
74.00
46.70
65.35
69.37
64.74
75.66
Qwen3-1.7B
82.18
78.17
77.94
77.94
78.92
83.93
Qwen3-4B
93.56
92.42
90.30
90.37
91.13
92.34
Qwen3-8B
95.07
94.77
95.45
94.84
95.30
95.38
Qwen3-14B
95.83
95.60
95.98
96.97
95.98
96.21
Table 1: Accuracy (%) on real math benchmarks across the Qwen3 models from 0.6B to 14B. Steered is the model with one PCA-extracted SkillSet-Math skill direction added to hidden states at α=2.0 on the last 50% of layers.
Skill
Base
Txt-C
Txt-CoT
Vec
Skill
Base
Txt-C
Txt-CoT
Vec
multi_digit_mul
74
94
93
96 ↑
gcd
90
97
97
99 ↑
multi_digit_add
58
92
82
93 ↑
range_valid
0
11
11
5 ↓
multi_digit_sub
72
95
98
98=
linear_search
4
2
2
7 ↑
sum_of_range
67
99
97
96 ↓
reverse_list
58
57
58
52 ↓
decimal_add
83
91
93
95 ↑
alg_simplify
12
98
98
99 ↑
solve_sys_2eq
77
84
86
96 ↑
count_occur
73
96
96
94 ↓
Table 2: RQ2: text-based vs. vector-based composition on Qwen3-0.6B (accuracy %). Base : zero-shot. Txt-C / Txt-CoT : text-based composition (concise / chain-of-thought). Vec : composition of atomic skill directions in vector space. Highlighted cells indicate the best-performing method per skill.
Qwen3-0.6B
Qwen3-1.7B
Category
Base
Steered
Base
Steered
Graph Search
16.7
33.3
25.0
16.7
DP / Recursion
16.7
41.7
8.3
8.3
Data Struct. / String
16.7
45.8
8.3
20.8
Table 3: Approach match rate (%, controllability) by category, α=1.0 .
Qwen3-0.6B
Qwen3-1.7B
Cat.
Pair
Base
Peak
Step
Base
Peak
Step
Graph Search
DFS vs BFS
8
24 †
20
10
14 †
5
DFS vs UnionFind
6
22
200
22
28 †
15
BFS vs Dijkstra
22
22 †
0
18
22
200
BFS vs BiDirBFS
64
72 †
9
74
84 †
30
DP / Recursion
Recursion vs DP
0
24
200
2
6 †
100
Table 4: Approach match rate (%, controllability) at initialization ( Base , step 0) and at the best observed checkpoint during training ( Peak ), with the corresponding training step ( Step ). Bold = gain ≥5 pp over baseline. † Peak occurs strictly before the final checkpoint (step 200), indicating that continued optimization degrades performance.
Table 5: Skill inventory used in our experiments. atomic skills are atomic operations (single-digit arithmetic, list manipulation, comparisons); composite skills are composite procedures that each depend on 2–4 atomic sub-skills.
Table 6: Composite skill dependencies. Each composite skill is composed of 2–4 atomic sub-skills applied in sequence.
Figure 3: LLM-Based Data Generation
#
Pair
Approach A
Approach B
1
dfs_vs_bfs
DFS
BFS
2
recursion_vs_dp
Recursion + Memo
Bottom-Up DP
3
brute_force_vs_optimized
Brute Force O(n2)
Optimized O(n)
4
stack_vs_recursion
Explicit Stack
Recursive Descent
5
bfs_vs_dijkstra
BFS (unweighted)
Dijkstra (weighted)
6
dfs_vs_union_find
DFS traversal
Union-Find
Appendix
Table 7: The 9 contrastive approach pairs in SkillSet-Code .
Category
Task
Method A
Method B
Graph Search (6 tasks)
Number of Islands
DFS
BFS
Connected Components
DFS
Union-Find
Shortest Path (Grid)
BFS
Dijkstra
Word Ladder
BFS
Bidirectional BFS
Course Schedule
DFS
BFS (Kahn’s)
Bipartite Check
DFS
BFS
Appendix
Table 8: Overview of the SkillSet-Code evaluation dataset. Each task presents two alternative solution methods and two situational contexts that each favor one method.
Figure 4: In-Distribution Approach Match Rate Over Training
Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 64.1% fewer prefill tokens, and improves Search-QA exact match by 3.0 points with 72.2% lower skill-token overhead. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be precisely controlled via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.
Aofan Yu, Chenyu Zhou, Tianyi Xu +8
Shanghai Jiao Tong University · Sun Yat-Sen University · Shanghai Innovation Institute +1
Natural-language skills are textual procedural memories through which large language model (LLM) agents retain reusable task knowledge without updating model weights. Existing methods typically treat skills as either static artifacts or monolithic documents optimized using aggregate validation scores as feedback. However, representing a skill as a monolithic document restricts optimization to its textual content, without explicitly modeling the structure through which procedural knowledge is retrieved and executed. We identify a key distinction between learning what knowledge to retain and determining how to organize it: textual updates should first be validated through execution evidence, after which the retained knowledge should be structured according to its procedural dependencies and retrieval requirements. To this end, we introduce SkillSpec, a two-phase framework comprising consensus-gated evolution and representation specialization. In the consensus-gated phase, complementary editing intents generate complete candidate skills. An update is committed only when paired evaluations reach consensus, requiring sufficient overall improvement and non-negative aggregate paired gain in every repeated evaluation. In the specialization phase, signals of process and redundancy sensitivity derived from the full optimization trajectory, including accepted and rejected candidates, guide the selection of a flat, graph, or hybrid representation.Across six benchmarks and three target language models, SkillSpec improves average success rate over SkillOpt by 6.89%, averaged across the three models. These results demonstrate that reliable skill evolution and representation specialization address complementary objectives: deciding what knowledge to retain and how to structure it for inference.
Agent skills are commonly distributed as SKILL.md files: human-readable procedural documents that describe workflows, tools, resources, and domain conventions. While convenient for inspection and reuse, this design requires the same reusable procedure to be repeatedly injected into the runtime context. We propose Skill-to-LoRA(S2L), a behavior-centric skill representation that replaces runtime skill text with skill-specific LoRA adapters. Rather than compressing the skill document itself, S2L models the behavioral change induced by the skill text: offline, the complete SKILL.md is used to synthesize skill-guided demonstrations; online, the full document is omitted and the corresponding LoRA adapter is dynamically loaded to activate the learned skill behavior. We evaluate S2L with Qwen3.6-27B on a 21-skill subset of SWE-Skills-Bench. Compared with the no-skill and Full Skill Text baselines, S2L improves pass rate by 2.9 and 5.2 percentage points, respectively, while reducing per-step token cost by 6.6% relative to Full Skill Text prompting. S2L matches or improves Full Skill Text on 18/21 skills and the no-skill baseline on 15/21 skills. Control experiments further show that the gains depend on skill-specific adapter alignment: Wrong-LoRA and Shared-LoRA both reduce performance. These results suggest that many procedural agent skills can be converted from runtime instructions into trainable, dynamically loadable behavioral modules. Code will be released upon acceptance.