CompSkill: Compiling Execution Experience for Skill Retrieval

Qi Shen; Ziyue Zhu; Yongqing Jiang; Zhen Xia; Li Shengjie; Shuai Zhao; Bo Cheng; Xiaobao Wu

cs.CLOct 31, 2026

CompSkill: Compiling Execution Experience for Skill Retrieval

Authors: Qi Shen, Ziyue Zhu, Yongqing Jiang, Zhen Xia, Li Shengjie, Shuai Zhao, Bo Cheng, Xiaobao Wu,

Organizations: Beijing University of Posts and Telecommunications · Nanyang Technological University · Sichuan University · Shanghai Jiao Tong University

Abstract

LLM agents increasingly rely on skill documents, yet most retrievers rank them by description alone and ignore what earlier executions revealed, even though a relevant skill can hurt execution. We present CompSkill, which indexes skills with learned 512-d keys and compiles matched same-task comparisons into skill offsets and capped key residuals (KeyOpt), so experience changes which document the executor receives, never its text. Under a logistic outcome model with unknown task difficulty, matched comparisons cancel that difficulty and only discordant ones carry information; on support executions, predictions improve with each discordant task. An experience hypergraph keeps each comparison with its executor and document versions, recalls missed skills and grounds a cited review. Compact keys raise SkillRet NDCG@10 from 64.0 to 70.9 at one eighth of the dimension. With one document, CompSkill completes 256 and 260 of 332 ToolQA tasks with GPT-5.6-luna and DeepSeek-V4.1-Flash, respectively, more than every baseline, and stays on par with the strongest baselines on AppWorld and ALFWorld. On 332 newly generated ToolQA tasks, it completes 256 with GPT-5.6-luna, significantly more than BM25, SkillRet-8B, Tool2Vec and RankGPT; over all 844 ToolQA tasks, it completes 24 more than the strongest baseline. Selection matters beyond relevance: pairing the right guide with a same-database sibling instead of another database's guide costs 8 of 96 tasks. Code, models and support data: https://github.com/satoshinji2992/CompSkill

Figures & tables

Figure 1: CompSkill on ToolQA: the semantic Top-10 offers a lookup guide; events recall the missing listings guide, which KeyOpt ranks first and cited review keeps.

Figure 2: Three challenges, CompSkill components and empirical evidence: SkillRet Recall@10 (compact keys: three-seed mean), a ToolQA Coffee ranking example, GPT-5.6-luna Hit@1 on the 332 ToolQA evaluation tasks, and paired review outcomes (9 wins, 3 losses) on the 332 newly generated tasks.

Method	Representation	Selection and adaptation mechanism
Dense / Vector Skills	Document vectors	Frozen cosine similarity retrieves original documents.
SkillRouter / SkillRet	Encoder / reranker	Relevance pairs train recall and reranking; original skill text stays fixed.
NUDGE	Document vectors	Relevance labels drive constrained vector edits; the encoder stays fixed.
RankGPT	Candidate list	Prompted LLM comparisons reorder retrieved documents.
GoS / GoSkills	Graph / skill groups	Recall prerequisites (GoS); compose role-aware groups (GoSkills).
SkillNet-Fabric	Typed graph + Wiki	Retrieve and expand typed relations; an Explorer selects from Wiki evidence.

Table 1: Skill retrieval and memory methods by representation and by selection and adaptation mechanism. CompSkill combines matched key calibration with attributed evidence while retaining original skill documents.

Figure 3: CompSkill links compact keys, KeyOpt calibration, attributed hypergraph evidence and native execution; cited review selects an original document and the bottom loop updates only from support tasks. The center shows how a query reads one key and what experience writes into each part (Recall@10 inset: SRA library).

			Success				CompSkill versus baseline
Executor	Method	#Docs	Tasks	Rate (%)	Complex (%)	Worst DB (%)	Gain [95% CI]	W/L	$p$
DeepSeek-V4.1-Flash	No Skills	0	200	60.2	37.1	41.4	$+$ 60 [ $42$ , $77$ ]	75/15	$<$ 0.001
	Vector Top-4	4	240	72.3	59.0	51.4	$+$ 20 [ $7$ , $34$ ]	34/14	0.006
	GoSkills	2.57	218	65.7	43.8	53.3	$+$ 42 [ $26$ , $59$ ]	59/17	$<$ 0.001
	Vector Top-1	1	236	71.1	57.3	55.7	$+$ 24 [ $13$ , $36$ ]	30/6	$<$ 0.001
	SkillRouter	1	226	68.1	52.8	51.4	$+$ 34 [ $21$ , $47$ ]	41/7	$<$ 0.001

Table 2: ToolQA results (

n=332

). Rows group No Skills, multi-document and single-document baselines; Docs: documents supplied per task (mean for GoSkills). Gain: additional tasks CompSkill completes, with a paired task-bootstrap 95% interval; W/L: tasks that only CompSkill or only the baseline completes;

p

: two-sided exact McNemar test. Complex: 178 complex queries; Worst DB: the weakest of four databases. Best in bold. Breakdown by query type, database and first-choice accuracy (Hit@1): Appendix G .

Figure 4: (a) ToolQA successes as calibration stages are added (GPT-5.6-luna; Development: the 332 evaluation tasks); from the KeyOpt index on, the pool also holds skills recalled by the event store; bracket: prespecified Holm-corrected test. (b) Net gains of CompSkill over the four prespecified baselines on the 332 new tasks, with 95% bootstrap intervals and Holm-corrected exact McNemar

P

. (c) Within-task success lost on the 332 evaluation tasks when a sibling or an unrelated document replaces the labeled guide (GPT: GPT-5.5). (d) Sibling guides supplied on the 180 held-out tasks by Vector Top-1 and the KeyOpt index (GPT fit).

Method	DS	GPT	DS	GPT	DS	GPT	Total
	ToolQA		AppWorld		ALFWorld
Support contrasts	87	72	8	2	1	0
No Skills	200	197	68	78	118	125	786
Vector Top-4	240	245	70	78	123	130	886
Vector Top-1	236	235	69	81	127	128	876
GoSkills	218	221	74	81	124	130	848

Table 3: Successes of the methods run on all three execution benchmarks (DS: DeepSeek-V4.1-Flash; GPT: GPT-5.6-luna with the GPT-5.5 fits on ToolQA, GPT-5.5 elsewhere). ToolQA: 332 tasks (Table 2 ); AppWorld: task completion on 90 Challenge tasks; ALFWorld: 134 valid_unseen games; Total: 1,112 tasks. Support contrasts: support tasks whose two executed documents differ in outcome; with eight or fewer, cross-validation keeps offsets at most, and AppWorld uses the calibrated index without recall or review. AppWorld SGC (of 30 templates), by row: DS 14, 14, 18, 20, 16, 19; GPT 22, 23, 25, 25, 23, 25. Bold: best per column.

Method	Docs	Eval.	New	All
		$n{=}332$	$n{=}332$	$n{=}844$
Vector Top-1	1	235	227	568
SkillRouter	1	225	225	556
BM25	1	223	214	543
SkillRet-8B	1	226	222	553
RankGPT	1	249	245	612

Table 4: ToolQA successes with GPT-5.6-luna and the GPT-5.5 fits on the 332 evaluation tasks (Table 2 ) and 332 newly generated tasks; All adds the 180 held-out tasks of Appendix H.1 . Bold: best in each column.

Selector	Supplied guidance	Signal behind the choice	Luna	DS
SkillRouter	a web hotel-booking skill	learned relevance	$\times$	$\times$
BM25	a Skiplagged travel skill	lexical overlap	$\times$	$\times$
Vector Top-1	DB lookup guide	embedding similarity	$\times$	$\times$
SkillRet-8B, RankGPT	DB lookup guide	dense retrieval, LLM reranking	$\times$	$\times$
Tool2Vec	DB listings guide	queries solved in support	✓	✓
Vector Top-4	lookup, Vrbo, listings, web skill (4 docs)	similarity, four documents	✓	✓

Table 5: Reference ToolQA task 00348. Three matched support events, in which the lookup guide failed and the listings guide succeeded, recall the listings guide that the Top-10 misses; KeyOpt ranks it first (reranker, offset and clipped conditional term:

.335{-}.032{+}.300

versus

.107{+}.032{-}.300

for the lookup guide, GPT fit), cited review keeps it, and both executors succeed. Quote and citations are from the DeepSeek review; Luna: GPT-5.6-luna.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Held-out log-loss for the winner of each discordant ToolQA support task, from five source-group folds, against the number

n

of discordant training tasks (mean and spread over 100 random subsets per

n

). The dashed curve fits

a+b/n

; the orange point is a pointwise success model fitted on concordant tasks only.

Symbol	Meaning
$D_{s}$ , $s\in\mathcal{S}$	versioned skill document and library
$x_{t}$ , $H$	public task context and frozen support history
$\pi_{H}$ , $U_{t}$ , $K$	selector, supplied skill set and its size cap
$\mathcal{E}$ , $B$ , $\xi$ , $R$	environment, execution budget, trajectory and native success
$E$ , $C_{\theta}$	frozen encoder and frozen compressor
$z_{s}$ , $a_{s}$ , $\omega$	4096-d embedding, its 512-d anchor and the compressor scale

Appendix

Table 6: Notation.

Compact keys
Setting	Value
Encoder	Qwen3-Embedding-8B
Key and query dimension	512 of 4,096
Compressor training	SkillRet TRAIN
Compressor	4096 $\to$ 512, 2 residual LN–MLP blocks
MLP expansion, activation	4,096, SiLU

Appendix

Table 7: Reference hyperparameters. Cross-validation on support selects

\kappa

c

\alpha

and residual use.

Benchmark	Eval.	Support	Library	Mean chars	Median chars	Actions / time	Success
ToolQA	332	318	785	6,446	4,142	20 / 900 s	answer exact match
AppWorld Challenge	90	90	469	1,139	961	30 / 900 s	unit tests (TGC, SGC)
ALFWorld	134	12	37	2,312	2,298	30 / 900 s	goal reached

Appendix

Table 8: Benchmark statistics; document lengths in characters (ALFWorld supplies at most 1,800 per skill).

ID	Title	Description (verbatim)	Chars
toolqa_002	US Domestic Flight Data — Single-Record Lookup Guide	Procedural guide for retrieving individual flight attributes from the 2022 US domestic flights dataset, covering schema, flight identifier conventions, time column encoding, boolean filters, and when to use SQL over FilterDB.	4,621
toolqa_003	US Domestic Flight Data — Multi-Record Aggregation Guide	Guide for answering multi-step aggregation and analysis questions about 2022 US domestic flight data, covering schema conventions, FilterDB pitfalls, and SQL workarounds.	5,167
toolqa_004	NYC Airbnb Listing Data — Single-Record Lookup	Guide for querying individual listing attributes from the NYC Airbnb dataset, covering FilterDB pitfalls and SQL workarounds.	3,863
toolqa_005	NYC Airbnb Listing Data — Aggregation and Derived Calculations	Guide for answering complex analytical questions about NYC Airbnb listing data, covering price parsing, multi-column derived calculations, and geographic queries.	4,905
toolqa_006	Yelp Business Data Lookup	Guide for retrieving attributes of individual Yelp businesses: address, city, state, postal code, star rating, review count, hours, appointment requirement, and coordinates.	3,473
toolqa_007	Yelp Business Data Aggregation and Spatial Queries	Guide for aggregation, category-based extremum search, and geospatial distance queries on Yelp business data, with SQL Haversine patterns for nearest-business and radius queries.	4,585

Appendix

Table 9: The eight ToolQA database guides in the 785-document library. Each domain has a lookup guide and an aggregation or range guide for the same database; choosing between them is ToolQA’s main selection problem.

Benchmark	Subset	Example task
ToolQA	Flights, lookup	What was the departure time of the AA2319 flight from MIA to LAS on 2022-06-05?
	Flights, complex	What percentage of the flights from JAC were delayed on 2022-02-24?
	Airbnb, lookup	What is the host’s name for Luxury Wellness Space - Breathtaking NYC Views in Long Island City?
	Airbnb, complex	What is the total price at least if you want to stay at Bright, modern room with panoramic window in Maspeth for 8 nights?
	Yelp, lookup	What is the address of Smilies in area with postal code T5V 1H9?
	Yelp, complex	How many Home Organization businesses are there in Franklin, TN?

Appendix

Table 10: Example evaluation tasks, quoted verbatim from the public task files.

Selector	Retrieval	Context supplied to the executor
No Skills	—	no guidance block
Vector Top-4	Qwen3-Embedding-8B, 4096-d cosine, top-4	four complete documents in rank order (ALFWorld: 1,800 characters per skill, 9,000 in total)
Vector Top-1	same encoder and similarity, top-1	one complete document
GoSkills	official group-structured retrieval	member documents of the selected group plan; on ALFWorld the official role contract with 1–4 skills
SkillRouter	released bi-encoder over the full library, top-20, then released cross-encoder	top-1 complete document
SkillNet-Fabric	37 skill contracts, 207 typed edges, task wiki and planner	task-specific wiki (ALFWorld only); skill text up to 1,700 characters

Appendix

Table 11: Selector configurations. All selectors use the released models and settings of their authors.

Setting	GPT-5.5	DeepSeek
Model	gpt-5.5	deepseek-flash
Reasoning	low effort	thinking, low
Reply limit (tokens)	16,384	16,384
Cumulative output gate	24,000	24,000
ToolQA actions / time	20 / 900 s	20 / 900 s
AppWorld actions / time	30 / 900 s	30 / 900 s

Appendix

Table 12: Default executor settings. GPT-5.6-luna runs ToolQA with the same low reasoning effort, 16,384-token reply limit and 20 actions / 900 s.

Method	Enc.	NDCG@10	R@10	Cmpl.@10
BM25	—	51.7	59.4	44.6
bge-large-en-v1.5	frozen	59.0	64.4	48.3
Qwen3-Emb.-0.6B	frozen	62.0	68.0	51.3
Qwen3-Emb.-8B	frozen	63.7	70.3	54.3
harrier-oss-0.6B	frozen	70.3	76.2	61.7
4096-d (same enc.)	frozen	64.0	70.6	54.4

Appendix

Table 13: SkillRet retrieval (%). Frozen encoders include learned compact keys; tuned references update encoder weights. Bold: best frozen-encoder value.

Projection	NDCG@10	R@10	Cmpl.@10
Truncated 128	53.6	60.5	44.4
Truncated 256	58.4	65.2	48.5
Truncated 512	62.2	68.8	52.4
Truncated 1024	63.3	69.6	53.0
Truncated 2048	64.1	70.5	54.2
Native 4096	64.0	70.6	54.4

Appendix

Table 14: Same-cache compression controls (%): truncation and random projection of the cached vectors, with one shared Gaussian matrix and no label fitting.

Fusion	NDCG@10	R@10	Cmpl.@10
BM25 + native 4096-d	62.2	70.3	55.3
BM25 + compact 512-d	65.8	74.5	60.0
BM25 + SkillRouter-Emb.	65.7	73.5	58.9
BM25 + SkillRet-Emb.-8B	70.0	78.9	66.1

Appendix

Table 15: Lexical–dense fusion on the SkillRet TEST split (%). Equal-weight reciprocal rank fusion (

k=60

) over cached Top-100 lists; compact uses seed 17.

Required skills	Queries	4096-d	512-d	$\Delta$ pp
One	2,143	84.2	89.7	+5.5
Two	1,703	32.0	45.1	+13.1
Three	546	7.0	11.3	+4.3
Two or three	2,249	25.9	36.9	+11.0
Unseen only	1,838	28.7	39.9	+11.2

Appendix

Table 16: SkillRet Completeness@10 (%) by number of relevant skills; unseen: queries without a relevant TRAIN match. Compact 512-d averages three seeds.

SRA (665 queries, 26K skills)
Index	R@1	R@5	R@10	MRR	nDCG
Compact, seed 17	44.4	64.9	73.9	60.9	60.9
Compact, seed 29	44.4	66.6	74.4	61.8	61.5
Compact, seed 43	45.3	67.4	74.9	62.5	62.5
Native 4096-d	46.1	66.8	71.8	62.5	62.1
Truncated	45.0	60.4	66.8	59.7	58.4

Appendix

Table 17: Retrieval quality (%) of compact keys, the embeddings they compress, truncation and a separately supervised compressor; MRR and nDCG at 10.

Figure 7: Recall@

k

on four libraries (SRA, its ToolQA subset, two AppWorld API sets): compact keys (three seeds, mean and range), the embeddings they compress, truncation and a separately supervised compressor.

Operation (CPU)	Median	P90
Compact Top-10	0.31 ms	0.40 ms
Default index and rendering	0.33 ms	0.59 ms
SkillRouter, cached scores	0.25 ms	0.32 ms
GoSkills retrieval and rendering	172 ms	181 ms
Storage	bytes
Compact keys (785)	1,607,680

Appendix

Table 18: Selection cost per ToolQA task (332 tasks, three repetitions) and index storage. SkillRouter uses cached encoder and reranker scores.

DeepSeek-V4.1-Flash
		Query type (%)		Database (%)
Method	Docs	Simple	Complex	Airbnb	Coffee	Flights	Yelp	Hit@1 (%)
No Skills	0	87.0	37.1	62.7	48.9	41.4	85.1	—
Vector Top-4	4	87.7	59.0	65.1	87.0	51.4	80.5	63.0
Vector Top-1	1	87.0	57.3	61.4	80.4	55.7	82.8	63.0
GoSkills	2.57	90.9	43.8	67.5	53.3	58.6	82.8	44.9

Appendix

Table 19: ToolQA success by query type (154 simple, 178 complex) and database (83 Airbnb, 92 Coffee, 70 Flights, 87 Yelp), and first-choice accuracy (Hit@1: the first supplied document is the labeled guide), for the runs of Table 2 . Tool2Vec, RankGPT and CompSkill select per executor; the others do not. Bold: column best per executor.

Supplied documents	Docs	Successes
Labeled guide	1	72
Labeled + sibling guide	2	67
Labeled + other-database guide	2	75

Appendix

Table 20: Adding a second guide on 96 held-out ToolQA tasks with GPT-5.6-luna (12 per labeled guide). The other-database guide has the sibling’s type; both two-document conditions keep each task’s document order.

Tasks	Comparison	W/L	Holm $p$
Held-out 180	CompSkill vs. SkillRet-8B	23/4	0.002
Held-out 180	CompSkill vs. Tool2Vec	8/9	1.000
Held-out 180	CompSkill vs. RankGPT	10/4	0.718
Held-out 180	CompSkill vs. BM25	23/5	0.006
Held-out 180	KeyOpt index vs. SkillRet-8B	22/8	0.097
Held-out 180	KeyOpt index vs. Tool2Vec	6/12	0.718

Appendix

Table 21: Paired comparisons with GPT-5.6-luna, one block per Holm family (wins/losses, exact two-sided McNemar on jointly known pairs). The held-out family contains eight tests: four for CompSkill and four for the KeyOpt index. Development 332: the 332 evaluation tasks; 6+4 pool: KeyOpt index on six semantic and four recalled candidates.

Cohort	Executor	Label $-$ sibling	Label $-$ unrelated	Unrelated $-$ sibling
ToolQA 332 (dev.)	GPT	$+0.24$ [ $+0.17$ , $+0.31$ ]	$+0.11$ [ $+0.07$ , $+0.16$ ]	$+0.09$ [ $+0.01$ , $+0.17$ ]
	DeepSeek	$+0.17$ [ $+0.10$ , $+0.24$ ]	$+0.08$ [ $+0.02$ , $+0.13$ ]	$+0.01$ [ $-0.04$ , $+0.08$ ]
ToolQA held-out 180	GPT	$+0.25$ [ $+0.13$ , $+0.36$ ]	$+0.09$ [ $0.00$ , $+0.18$ ]	$+0.11$ [ $0.00$ , $+0.26$ ]
	DeepSeek	$+0.25$ [ $+0.13$ , $+0.37$ ]	$+0.16$ [ $+0.04$ , $+0.28$ ]	$+0.09$ [ $0.00$ , $+0.20$ ]
ToolQA, three new databases	GPT	$+0.04$ [ $+0.02$ , $+0.06$ ]	$+0.09$ [ $+0.05$ , $+0.12$ ]	$+0.02$ [ $0.00$ , $+0.04$ ]
	DeepSeek	$+0.06$ [ $+0.02$ , $+0.10$ ]	$+0.11$ [ $+0.06$ , $+0.15$ ]	$+0.03$ [ $-0.01$ , $+0.08$ ]

Appendix

Table 22: Within-task success differences between supplied-document categories (95% intervals from 10,000 task resamples). GPT is GPT-5.5 on the 332 evaluation tasks and GPT-5.6-luna on the three other cohorts. Sibling: a guide of the same database (ToolQA) or a non-gold table of the same database (BIRD). Each column uses only the tasks on which both of its compared conditions were executed with the same executor.

Method ( $n=317$ )	GPT-5.6	DeepSeek-1017
No Skills	64	49
Vector Top-1	59	62
SkillRouter	61	58
BM25	59	57
SkillRet-8B	82	84
Tool2Vec	79	78

Appendix

Table 23: Three further ToolQA databases (SciREX, Agenda, DBLP; 317 evaluation tasks, 791 documents). Unknowns (at most one per cell) count as failures.

Method ( $n=622$ )	GPT-5.6	DeepSeek-1017
No Skills	233	308
Vector Top-1	275	325
SkillRouter	275	325
BM25	273	322
SkillRet-8B	271	332
Tool2Vec	285	331

Appendix

Table 24: BIRD-Skills (622 evaluation tasks, no evidence hints, 860 documents). Unknown outcomes (at most one per cell) count as failures.

Episodic Memory for LLM Agents

See all 8 papers

Generative Skill Composition for LLM Agents
Jun 30, 2026Xinyu Zhao, Zhen Tan, Vaishnav Tadiparthi +5AI Coding Agents LLM Agent Skill Learning
University of North Carolina at Chapel Hill · Arizona State University · Honda Research Institute USA
SkillTrace: Traversing a Query-Skill Graph for Composable LLM Agents
Aug 3, 2026Yue Yao, Shengyuan Wang, Xin Chen +4Graph-Based Retrieval LLM Agent Skill Retrieval
Shandong University · The Australian National University · Tsinghua University +1
Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose
Jun 16, 2026Xueping GaoTool-Augmented Language Model Agents LLM Planning
Alibaba Cloud Hangzhou, China

Jun 30, 2026J/K move · Enter open · S save

Generative Skill Composition for LLM Agents

Xinyu Zhao, Zhen Tan, Vaishnav Tadiparthi, Nakul Agarwal+4

University of North Carolina at Chapel Hill · Arizona State University · Honda Research Institute USA

AI Coding Agents LLM Agent Skill Learning LLM Agent Planning

Open paper AskPDF arXiv

Recent LLM agents benefit from skills for solving complex tasks. Skills encapsulate modular packages of procedural knowledge and instructions for performing specialized tasks, such as setting up a sandboxed environment, running a test suite, or refactoring a function across multiple files. As skill libraries grow and become reusable across tasks and domains, selecting an appropriate skill composition has emerged as a central bottleneck. Existing approaches fall into two categories. One exposes the agent's reasoning to the entire skill collection; the other performs skill retrieval via embeddings or LLM-based rerankers. Both provide useful insights; however, they miss the structural nature of skill composition, which is a joint decision over which skills, how many, and in what order -- three dimensions that cannot be decoupled. We formalize this as structured skill composition: given a task and a skill library, predict an executable skill plan that jointly specifies the activated subset, count, and execution order. We propose SkillComposer, which instantiates structured skill composition as task-conditioned skill sequence prediction. SkillComposer uses a constrained autoregressive decoder over skill identifiers, so subset, count, and order emerge jointly from a single decoding pass, and dependencies between successive skills are captured naturally. We build a training set of task-composition pairs from a real, human-curated skill library. We then evaluate SkillComposer along two axes: composition quality on a held-out test set, and downstream task success on SkillsBench across two production-grade coding agents. On GPT-5.2-Codex, Gemini-3-Pro-Preview, SkillComposer raises the pass rate by +23.1, +18.2pp over the no-skill baseline, surpassing top-3 retrieval and matching the gold-skill retrieval upper bound at lower prompt-token cost.

Abstract

Episodic Memory for LLM Agents

Explore similar work

Generative Skill Composition for LLM Agents

SkillTrace: Traversing a Query-Skill Graph for Composable LLM Agents

Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose