Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval

Lihui Ding; Zihan Guo; Bingwei Lu; Chenyu Zhou; Yuanjian Zhou; Weinan Zhang; Jianghao Lin; Dongdong Ge

cs.CLAug 17, 2026

Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrieval

Authors: Lihui Ding, Zihan Guo, Bingwei Lu, Chenyu Zhou, Yuanjian Zhou, Weinan Zhang, Jianghao Lin, Dongdong Ge

Organizations: Sun Yat-sen University Shenzhen, China · Shanghai Jiao Tong University Shanghai, China

Abstract

Pseudo-query generation can alleviate the supervision bottleneck for agent skill retrieval, but existing document-level approaches typically leave the rich internal relations among capabilities, parameters, and usage examples implicit. As a result, generated queries may be topically relevant to a skill while lacking capability grounding and parameter consistency, raising the question of whether explicitly exploiting a skill document's internal structure can produce more effective retrieval signals. We therefore propose Skill2Query, a framework that first parses a skill document into a Skill Knowledge Graph and then generates pseudo-queries through a three-stage process including style mimicking, query template generation, and parameter filling. The generated queries can be used for offline index augmentation, online query expansion, and retriever training. Four benchmarks (TheoremQA, LogicBench, ToolQA, and CHAMP) are used to evaluate Skill2Query with large-scale skill candidate pools across multiple downstream applications, including skill retrieval, retriever training, and end-to-end agent execution. Using nearly 30K skills across diverse domains, we generate 700K category-diverse pseudo-queries. Skill2Query consistently improves sparse, dense, and skill-routing retrieval, with an average Recall@1 gain of 6.70 percentage points across retrieval settings. Skill2Query-generated training data also achieves the best Recall@1 and nDCG@1 among the evaluated generation baselines. Further evaluations with multiple LLM backends demonstrate that improved skill retrieval translates into higher agent task success rates. Code and resources are available at https://github.com/MatZaharia/Skill2Query.

Figures & tables

Figure 1. Overview of the Skill2Query framework. The figure presents the overall workflow of Skill2Query, including Skill Knowledge Graph construction, three-stage progressive pseudo-query generation, offline index construction, and online retrieval with query expansion.

Figure 2. The node and edge types defined in the SKG. Solid arrows start from Skill nodes, and dashed arrows start from Example nodes. The figure illustrates the node categories and edge types in SKG, including Skill, Capability, Parameter, and Example nodes, as well as the semantic relationships among them.

Method	TheoremQA		LogicBench		ToolQA		CHAMP
	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
BM25 (full)	60.37 ( $\pm$ 0.00)	83.94 ( $\pm$ 0.00)	15.39 (+6.58)	45.39 (+21.19)	46.15 (+2.17)	81.61 (+2.45)	13.90 (-3.14)	36.29 (-6.12)
BM25 (name)	16.06 (+13.12)	29.45 (+22.22)	0.00 (+10.39)	1.84 (+31.19)	5.52 (+26.51)	24.27 (+37.27)	0.45 (+9.90)	1.72 (+23.71)
BM25 (name+desc)	22.22 (+12.45)	33.73 (+23.97)	3.16 (+6.31)	13.95 (+18.42)	19.44 (+13.43)	43.43 (+18.46)	3.81 (+6.28)	7.44 (+18.79)
BGE-Large-v1.5 (full)	63.86 (+0.93)	83.94 (+2.27)	7.89 (-1.97)	31.45 (-8.82)	34.76 (-2.17)	75.94 (-5.03)	10.74 (+0.10)	43.36 (-2.20)
BGE-Large-v1.5 (name)	36.81 (+16.07)	58.64 (+22.89)	2.76 (+1.85)	11.58 (+8.03)	15.52 (+15.11)	48.46 (+17.76)	5.31 (+5.79)	18.24 (+16.36)

Table 1. Recall performance of Skill2Query in the offline mode. Baseline results are shown outside parentheses, with offline gains in percentage points shown in parentheses.

Method	TheoremQA		LogicBench		ToolQA		CHAMP
	N@1	N@10	N@1	N@10	N@1	N@10	N@1	N@10
BM25 (full)	60.37 ( $\pm$ 0.00)	71.85 ( $\pm$ 0.00)	15.39 (+6.58)	28.37 (+14.62)	46.15 (+2.17)	64.92 (+3.43)	21.97 (-4.48)	27.91 (-5.43)
BM25 (name)	16.06 (+13.12)	22.44 (+17.18)	0.00 (+10.39)	0.88 (+20.15)	5.52 (+26.51)	14.86 (+32.92)	0.90 (+16.59)	1.25 (+18.14)
BM25 (name+desc)	22.22 (+12.45)	27.33 (+18.16)	3.16 (+6.31)	8.24 (+12.02)	19.44 (+13.43)	32.05 (+15.12)	4.48 (+12.56)	5.72 (+13.99)
BGE-Large-v1.5 (full)	63.86 (+0.93)	73.90 (+1.36)	7.89 (-1.97)	18.09 (-4.88)	34.76 (-2.17)	57.29 (-5.11)	14.35 (+0.45)	27.92 (-0.95)
BGE-Large-v1.5 (name)	36.81 (+16.07)	47.06 (+19.89)	2.76 (+1.85)	6.62 (+4.30)	15.52 (+15.11)	33.01 (+15.65)	6.73 (+10.31)	11.83 (+11.84)

Table 2. nDCG performance under offline index augmentation. Baseline scores are shown outside parentheses, and absolute gains in percentage points are shown in parentheses.

Figure 3. Pseudo-query generation quality comparison. Comparison of different pseudo-query generation methods using Exec-Pass, Func-Coverage, and Distinct-3 metrics.

Method	TheoremQA		LogicBench		ToolQA		CHAMP
	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
BM25 (baseline)	60.37	83.94	15.39	45.39	46.15	81.61	13.90	36.29
BM25 (offline)	60.37	83.94	21.97	66.58	48.32	84.06	10.76	30.17
BM25+S2Q online	65.19	89.42	12.76	37.50	46.43	81.19	18.39	46.67
SkillRouter (baseline)	82.73	91.40	22.37	40.13	35.80	49.44	26.64	56.89
SkillRouter (offline)	83.80	94.51	21.18	45.92	47.34	58.88	28.06	60.55

Table 3. Recall statistics of Skill2Query online query expansion. R@1 and R@10 are reported as percentages.

Method	TheoremQA		LogicBench		ToolQA		CHAMP
	N@1	N@10	N@1	N@10	N@1	N@10	N@1	N@10
BM25 (baseline)	60.37	71.85	15.39	28.37	46.15	64.92	21.97	27.91
BM25 (offline)	60.37	71.85	21.97	42.99	48.32	68.35	17.49	22.48
BM25+S2Q online	65.19	77.32	12.76	23.83	46.43	64.83	28.70	35.60
SkillRouter (baseline)	82.73	87.12	22.37	31.64	35.80	44.23	38.12	46.37
SkillRouter (offline)	83.80	89.29	21.18	32.65	47.34	54.07	39.91	48.45

Table 4. nDCG statistics of Skill2Query online query expansion. N@1 and N@10 are reported as percentages.

Method	TheoremQA		LogicBench		ToolQA		CHAMP
	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
Few-shot	82.06	96.25	18.68	38.55	40.91	47.20	30.04	57.06
SkillFlow-style	83.40	96.52	19.08	42.11	41.40	45.87	28.29	50.34
Skill2Query	85.27	95.58	21.97	49.34	45.03	52.31	31.73	62.22

Table 5. Recall results across four benchmarks (training data comparison).

Method	TheoremQA		LogicBench		ToolQA		CHAMP
	N@1	N@10	N@1	N@10	N@1	N@10	N@1	N@10
Few-shot	82.06	89.72	18.68	28.48	40.91	44.83	45.74	48.32
SkillFlow-style	83.40	90.41	19.08	30.39	41.40	44.22	42.60	43.77
Skill2Query	85.27	90.82	21.97	34.84	45.03	49.34	47.98	52.22

Table 6. nDCG results across four benchmarks (training data comparison).

Figure 4. Performance gains of Skill2Query across datasets and retrieval metrics. The heatmap illustrates the retrieval performance improvements of Skill2Query-generated training data compared with Few-shot LLM and SkillFlow-style methods across different datasets and evaluation metrics.

Agent	Retriever	No Skill	Baseline	Offline	Oracle
GPT-5.4	BM25	80.05	82.73	83.80	84.07
	SR-Emb-0.6B		80.32	82.58
	SkillRouter (FT)		83.80	83.53
DeepSeek-V4-Flash	BM25	73.76	76.31	76.17	85.69
	SR-Emb-0.6B		73.36	77.11
	SkillRouter (FT)		78.18	82.33

Table 7. End-to-end task success rates across LLM backends under different skill-retrieval configurations (%). FT denotes fine-tuning with Skill2Query-generated pseudo-queries.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Input Information	Description
Zero-shot LLM	name + description	Uses only concise skill metadata without structured information or query examples.
Few-shot LLM	name + description + query examples	Adds three randomly sampled query examples from the same skill document, without explicit capability or parameter schema information.
SkillFlow-style	name + description + body	Uses the complete skill document as unstructured text without explicitly decomposing capabilities, parameter schemas, or examples.
Skill2Query	parsed SKG	Uses structured capability, parameter schema, and example information extracted from SKG.

Appendix

Table 8. Input information configurations of different pseudo-query generation methods.

Setting	SR-Emb-0.6B	SR-Re-0.6B
Training objective	In-batch InfoNCE	Listwise cross-entropy
Temperature	0.05	1.0
Epochs	1	1
Batch size	8	1 candidate group
Gradient accumulation	1	16
Learning rate	$2\times 10^{-5}$	$1\times 10^{-5}$

Appendix

Table 11. Fine-tuning configurations of the two-stage skill retrieval model.

Variant	Exec-Pass (%)	Func-Coverage (%)	Distinct-3 (%)
Skill2Query	42.85	11.32	85.71
w/o SKG	22.63	2.41	96.74
w/o Param-Aware	21.66	11.25	85.54
w/o Exec-Verify	42.90	11.27	85.62

Appendix

Table 12. Ablation results of Skill2Query.

Figure 5. Normalized comparison of Skill2Query and its ablation variants across the three generation-quality metrics.

LLM Agent Skill Retrieval

Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.

See all 102 papers

SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests
Aug 9, 2026Donghong Jiang, Endian Lin, Luoping Cui +6CoT Supervision AI Agent Benchmarks
Beijing University of Posts and Telecommunications · Peking University · Beijing ZOYEN Technology Co., Ltd. +1
SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents
May 7, 2026Ryangkyung Kang, Hongcheol Cho, Youngeun KimAI Agent Benchmarks LLM Agent Skill Retrieval
ThakiCloud
SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale
Sep 30, 2026Guanqun Yang, Wenlong Zhang, Tian Shi +1LLM Agent Skill Retrieval Information Retrieval
Stevens Institute of Technology, Hoboken, NJ, USA · Independent Researcher

Aug 9, 2026J/K move · Enter open · S save

SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests

Donghong Jiang, Endian Lin, Luoping Cui, Hanqing Liu+5

Beijing University of Posts and Telecommunications · Peking University · Beijing ZOYEN Technology Co., Ltd.+1

CoT Supervision AI Agent Benchmarks LLM Agent Skill Retrieval

Open paper AskPDF arXiv

Large language model agents increasingly rely on reusable skills to extend their capabilities beyond parametric knowl- edge. However, retrieving the appropriate skill from a large- scale library remains challenging because realistic user re- quests are often concise and underspecified, stating only the task goal while leaving the required capabilities and execu- tion steps implicit. Existing benchmarks provide limited cov- erage of such requests. To address this gap, we introduce SkillReason-Bench, a large-scale cross-domain benchmark containing 3,729 queries and a retrieval corpus of 61,228 skills spanning nine domains. We further propose SkillRea- son, a two-stage framework that uses chain-of-thought rea- soning as training-time supervision for skill retrieval. In Stage I, capability reasoning traces generated by a stronger teacher provide explicit supervision through contrastive learning, re- trieval distribution alignment, and language modeling, en- couraging the retriever to internalize capability reasoning in its query representation. In Stage II, a retrieval-guided GRPO objective encourages the model to explore reasoning trajecto- ries better suited to its own capabilities and more effective for retrieval. At inference, SkillReason directly encodes the orig- inal query without autoregressive CoT generation, preserv- ing efficient query-only retrieval. Extensive experiments on SkillReason-Bench, SkillRet, and SRA-Bench show that Skill- Reason achieves state-of-the-art performance across all three benchmarks, demonstrating that reasoning-enhanced training better bridges the semantic gap between high-level task goals and skill capabilities.

Abstract

LLM Agent Skill Retrieval

Explore similar work

SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests

SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents

SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale