Toward Latent Language Model Skills Steering and Optimization: An Empirical Study

Xunyi Jiang; Junda Wu; Yuxin Xiong; Sheldon Yu; Tong Yu; David Arbour; Ritwik Sinha; Julian McAuley

cs.AIAug 29, 2026

Toward Latent Language Model Skills Steering and Optimization: An Empirical Study

Authors: Xunyi Jiang, Junda Wu, Yuxin Xiong, Sheldon Yu, Tong Yu, David Arbour, Ritwik Sinha, Julian McAuley,

Organizations: New York University · Adobe Research · UC San Diego

Abstract

Skills, as a useful abstraction for the procedural capabilities of large language models (LLMs), capture how models perform structured, multi-step reasoning and program execution. Existing approaches typically treat skills as explicit, surface-level constructs specified through prompts or programs, leaving open the question of how such procedural capabilities are represented inside the model and whether they can be manipulated as structured objects in latent space. In this empirical study, we investigate whether procedural LLM skills can be represented as directions in activation space and whether vector-space operations over these directions can express skill-level behaviors. We find that procedural skills admit a vector-space representation: individual skill directions can be activated to shift model behavior; independently extracted directions can compose to form higher-level skills. Contrastive directions yield context-conditioned algorithmic personalization and optimization trajectories over skill directions evolve non-monotonically, with intermediate states often surpassing fully optimized solutions. These results support a representation-level view of procedural LLM skills: they admit a latent vector-space organization that allows direct manipulation through internal interventions.

Figures & tables

Figure 1: Latent LLM Skills as Representation-Level Objects. We study four observable properties of skill-conditioned directions in activation space: steerability , composition , personalization , and evolution .

Figure 2: Per-skill accuracy on SkillSet-Math for Qwen3-0.6B: unsteered baseline vs. steered (PCA-extracted skill direction).

Model	0-shot	1-shot	2-shot	3-shot	4-shot	Steered
GSM8K
Qwen3-0.6B	74.00	46.70	65.35	69.37	64.74	75.66
Qwen3-1.7B	82.18	78.17	77.94	77.94	78.92	83.93
Qwen3-4B	93.56	92.42	90.30	90.37	91.13	92.34
Qwen3-8B	95.07	94.77	95.45	94.84	95.30	95.38
Qwen3-14B	95.83	95.60	95.98	96.97	95.98	96.21

Table 1: Accuracy (%) on real math benchmarks across the Qwen3 models from 0.6B to 14B. Steered is the model with one PCA-extracted SkillSet-Math skill direction added to hidden states at

\alpha=2.0

on the last 50% of layers.

Skill	Base	Txt-C	Txt-CoT	Vec	Skill	Base	Txt-C	Txt-CoT	Vec
multi_digit_mul	74	94	93	96 $\uparrow$	gcd	90	97	97	99 $\uparrow$
multi_digit_add	58	92	82	93 $\uparrow$	range_valid	0	11	11	5 $\downarrow$
multi_digit_sub	72	95	98	98=	linear_search	4	2	2	7 $\uparrow$
sum_of_range	67	99	97	96 $\downarrow$	reverse_list	58	57	58	52 $\downarrow$
decimal_add	83	91	93	95 $\uparrow$	alg_simplify	12	98	98	99 $\uparrow$
solve_sys_2eq	77	84	86	96 $\uparrow$	count_occur	73	96	96	94 $\downarrow$

Table 2: RQ2: text-based vs. vector-based composition on Qwen3-0.6B (accuracy %). Base : zero-shot. Txt-C / Txt-CoT : text-based composition (concise / chain-of-thought). Vec : composition of atomic skill directions in vector space. Highlighted cells indicate the best-performing method per skill.

	Qwen3-0.6B		Qwen3-1.7B
Category	Base	Steered	Base	Steered
Graph Search	16.7	33.3	25.0	16.7
DP / Recursion	16.7	41.7	8.3	8.3
Data Struct. / String	16.7	45.8	8.3	20.8

Table 3: Approach match rate (%, controllability) by category,

\alpha=1.0

		Qwen3-0.6B			Qwen3-1.7B
Cat.	Pair	Base	Peak	Step	Base	Peak	Step
Graph Search	DFS vs BFS	8	24 †	20	10	14 †	5
	DFS vs UnionFind	6	22	200	22	28 †	15
	BFS vs Dijkstra	22	22 †	0	18	22	200
	BFS vs BiDirBFS	64	72 †	9	74	84 †	30
DP / Recursion	Recursion vs DP	0	24	200	2	6 †	100

Table 4: Approach match rate (%, controllability) at initialization ( Base , step 0) and at the best observed checkpoint during training ( Peak ), with the corresponding training step ( Step ). Bold = gain

\geq 5

pp over baseline. † Peak occurs strictly before the final checkpoint (step 200), indicating that continued optimization degrades performance.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Level	#	Skills
atomic	16	extract_digits , reverse_digits , add_two_single_digits , subtract_two_single_digits , mul_two_single_digit_number , divide_single_digits , add_multiple_numbers , compare_values , detect_carry , detect_borrow , modulo_operation , get_place_value , shift_decimal , list_index , list_length , list_append
composite	18	multi_digit_addition , multi_digit_subtraction , multi_digit_multiplication , long_division , decimal_addition , sum_of_range , rounding_numbers , percentage_calculation , greatest_common_divisor , solve_linear_equation , solve_system_two_equations , check_equation_validity , inequality_solving , algebraic_simplification , range_validation , linear_search , reverse_list , count_occurrences

Appendix

Table 5: Skill inventory used in our experiments. atomic skills are atomic operations (single-digit arithmetic, list manipulation, comparisons); composite skills are composite procedures that each depend on 2–4 atomic sub-skills.

Composite Skill	Atomic skill Dependencies
multi_digit_addition	extract_digits , add_two_single_digits , detect_carry , reverse_digits
multi_digit_subtraction	extract_digits , subtract_two_single_digits , detect_borrow , reverse_digits
multi_digit_multiplication	extract_digits , mul_two_single_digit_number , add_multiple_numbers , shift_decimal
long_division	extract_digits , mul_two_single_digit_number , subtract_two_single_digits , compare_values
decimal_addition	extract_digits , add_two_single_digits , detect_carry , get_place_value
sum_of_range	add_multiple_numbers , compare_values , subtract_two_single_digits

Appendix

Table 6: Composite skill dependencies. Each composite skill is composed of 2–4 atomic sub-skills applied in sequence.

#	Pair	Approach A	Approach B
1	dfs_vs_bfs	DFS	BFS
2	recursion_vs_dp	Recursion + Memo	Bottom-Up DP
3	brute_force_vs_optimized	Brute Force $O(n^{2})$	Optimized $O(n)$
4	stack_vs_recursion	Explicit Stack	Recursive Descent
5	bfs_vs_dijkstra	BFS (unweighted)	Dijkstra (weighted)
6	dfs_vs_union_find	DFS traversal	Union-Find

Appendix

Table 7: The 9 contrastive approach pairs in SkillSet-Code .

Category	Task	Method A	Method B
Graph Search (6 tasks)	Number of Islands	DFS	BFS
	Connected Components	DFS	Union-Find
	Shortest Path (Grid)	BFS	Dijkstra
	Word Ladder	BFS	Bidirectional BFS
	Course Schedule	DFS	BFS (Kahn’s)
	Bipartite Check	DFS	BFS

Appendix

Table 8: Overview of the SkillSet-Code evaluation dataset. Each task presents two alternative solution methods and two situational contexts that each favor one method.

Figure 4: In-Distribution Approach Match Rate Over Training

Item	Setting
Extraction method	Single-class PCA (positive samples only; non-contrastive)
Principal component	First principal component (top-1)
Layer scope	Per-layer (independently extracted at every layer)
Token position	Last token of the formatted input
Sample size	Up to 500 examples per skill (capped at available count)
Prompt format	Instruction + Problem + Answer

Appendix

Table 9: RQ1/RQ2 extraction setting: single-class PCA on SkillSet-Math .

Item	Setting
Extraction method	Mean-difference contrastive: $v=\mu^{+}-\mu^{-}$
Layer scope	Per-layer (independently extracted at every layer)
Token position	Last token of the formatted input
Normalization	$\ell_{2}$ normalization of the direction
Sample size	200 positive + 200 negative per approach pair
Prompt format	“Problem: …\n\nSolution (Approach):\n <code> ”

Appendix

Table 10: RQ3 extraction setting: contrastive mean-difference on SkillSet-Code .

Composite skill	Base	Txt-C	Txt-CoT	Vec
multi_digit_addition	84	7	24	85
multi_digit_subtraction	74	11	21	71
multi_digit_multiplication	11	1	1	11
long_division	58	19	18	65
decimal_addition	78	65	36	77
sum_of_range	5	1	2	7

Appendix

Table 11: Skill composition on Llama-2-7B-chat. Accuracy (%) on the 18 composite skills of SkillSet-Math . Bold marks the best method per skill.

Skill	Base	$\alpha{=}0.5$	$\alpha{=}1.0$	$\alpha{=}2.0$	Best $\Delta$
subtract_two_single_digits	84	88	88	95	$+11$
multi_digit_subtraction	74	78	79	84	$+10$
inequality_solving	1	0	1	10	$+9$
compare_values	51	53	53	60	$+9$
solve_linear_equation	90	91	92	95	$+5$

Appendix

Table 12: Sensitivity to intervention strength

\alpha

on Llama-2-7B-chat, for five representative skills under single-class PCA extraction. Accuracy (%).

Benchmark	0-shot	Steered	$\Delta$
GSM8K	25.25	28.51	$+3.26$
MATH500	6.00	8.80	$+2.80$

Appendix

Table 13: Transfer of SkillSet-Math skill directions on Llama-2-7B-chat. Accuracy (%).

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
Jun 4, 2026Aofan Yu, Chenyu Zhou, Tianyi Xu +8LLM Agent Skill Learning In-Context Learning
Shanghai Jiao Tong University · Sun Yat-Sen University · Shanghai Innovation Institute +1
SkillSpec: Consensus-Gated Agent Skill Evolution via Representation Specialization
Sep 30, 2026Huancheng Chen, Xiaodi Sun, Zhaoqiong Huang +3LLM Agent Self-Improvement LLM Agent Skill Learning
Microsoft AI
Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents
Jun 15, 2026Tianyi Zhang, Zhonghao QiAI Coding Agents LLM Agent Skill Learning
1The Chinese University of Hong Kong

Jun 4, 2026J/K move · Enter open · S save

LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents

Aofan Yu, Chenyu Zhou, Tianyi Xu, Zihan Guo+7

Shanghai Jiao Tong University · Sun Yat-Sen University · Shanghai Innovation Institute+1

LLM Agent Skill Learning In-Context Learning

Open paper AskPDF arXiv

Agent systems increasingly use textual skills to encode reusable task procedures, but injecting these skills into the prompt at every step incurs substantial context overhead and exposes skill content as plaintext. We present LatentSkill, a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. LatentSkill stores skill knowledge in weight space rather than context space, removing per-step skill tokens while preserving modular loading, scaling, and composition. On ALFWorld and Search-QA, LatentSkill outperforms the corresponding in-context skill baseline while using substantially fewer prefill tokens: it improves ALFWorld success by 21.4 and 13.4 points on the seen and unseen splits with 64.1% fewer prefill tokens, and improves Search-QA exact match by 3.0 points with 72.2% lower skill-token overhead. Further analysis shows that generated skill LoRAs form a structured semantic geometry, can be precisely controlled via the LoRA scaling coefficient, and can be composed through parameter-space arithmetic when skill components are aligned. These findings suggest that weight-space skills provide an efficient, modular, and less exposed substrate for extending LLM agents.

Abstract

Language Modeling

Explore similar work

LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents

SkillSpec: Consensus-Gated Agent Skill Evolution via Representation Specialization

Skill-to-LoRA: From Using Skills to Learning Behaviors for Token-Efficient LLM Agents