Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks. As skill ecosystems grow to tens of thousands of entries, exposing every skill at inference time becomes infeasible. This creates a skill-routing problem: given a user task, the system must identify relevant skills before downstream planning or execution. Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body. We examine this design choice on a SkillsBench-derived benchmark with approximately 80K candidate skills, targeting the practically important setting of large skill registries with heavy overlap. Across representative dense and reranking baselines on this setting, hiding the skill body causes a 37-44 percentage point drop in routing accuracy. Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart. Motivated by this finding, we present Skillrouter, a compact 1.2B body-aware retrieve-and-rerank pipeline. Skillrouter achieves 74.0% Hit@1 on our benchmark -- the strongest average top-1 routing performance among the baselines we evaluate -- while using 13× fewer parameters and running 5.8× faster than the strongest base pipeline. The ranking gains further generalize to a supplementary benchmark independently constructed from three skill sources. In a complementary end-to-end study across four coding agents, routing gains transfer to improved task success, with larger gains for more capable agents.
Figures & tables
Figure 1: Body access is a critical routing signal. Left: Averaged over Easy and Hard, removing body reduces Hit@1 by 37.3pp for Qwen3-Emb-0.6B, 38.7pp for Qwen3-Emb-8B, and 44.0pp for Qwen3-Emb-8B × Qwen3-Rank-8B. Right: A length-controlled diagnostic: the body occupies 96.5% of skill tokens, yet the 3.0%-token name field peaks at 26.3% attention in layer 19 before the final layer returns to 98.1% body attention.
System
Raw nd
Oracle nd
All-field
SkillRouter-1.2B pipeline
.273
.527
.740
Qwen3 0.6B × 0.6B
.267
.567
.640
Qwen3 8B × 8B
.240
.613
.680
Encoder-only, matched training for each input regime
Fine-tuned, nd input
.513
.320
–
Fine-tuned, all-field input
–
–
.653
Table 2: Stronger metadata controls on the main benchmark (Hit@1, Easy/Hard average). Oracle nd is a query-blind, body-informed replacement; Appendix E gives its exact construction. The fine-tuned encoders are trained separately on the same 37,979 pairs; .320 applies the GT-only Oracle replacement to the nd-trained encoder.
Figure 2: SkillRouter pipeline. A bi-encoder retrieves top-20 candidates from the ∼ 80K pool; a cross-encoder reranks them. Both stages use name, description, and body content, motivated by Section 3 .
Model
Params
E-Hit@1
H-Hit@1
A-Hit@1
A-MRR@10
A-R@20
BM25
–
.347
.280
.314
.365
.365
BGE-Large-v1.5
335M
.613
.587
.600
.653
.668
gemini-embedding-001
–
.613
.560
.587
.650
.687
text-embedding-3-large
–
.640
.600
.620
.658
.664
Qwen3-Emb-0.6B
0.6B
.587
.533
.560
.638
.637
Qwen3-Emb-8B
8B
.653
.627
.640
.698
.726
Table 3: Encoder-only retrieval results on the 80K skill-routing benchmark. The tuned 0.6B encoder (highlighted) outperforms the 13 × larger base encoder, showing that task-specific training compensates for scale in this setting. E/H/A denote Easy/Hard/Average. R@20 reflects candidate coverage for downstream reranking.
Encoder
Reranker
E-Hit@1
H-Hit@1
A-Hit@1
A-MRR@10
A-R@10
Qwen3-Emb-0.6B
Qwen3-Rank-0.6B
.653
.627
.640
.684
.604
Qwen3-Emb-8B
Qwen3-Rank-0.6B
.613
.547
.580
.672
.694
Qwen3-Emb-8B
Qwen3-Rank-8B
.680
.680
.680
.745
.692
SR-Emb-0.6B
Qwen3-Rank-0.6B
.720
.693
.707
.769
.724
SR-Emb-0.6B
Qwen3-Rank-8B
.720
.707
.714
.776
.727
SR-Emb-0.6B
SR-Rank-0.6B
.760
.720
.740
.791
.704
Table 4: End-to-end retrieve-and-rerank results (top-20 candidates). The compact 1.2B tuned pipeline (highlighted) reaches the highest Hit@1 among non-scaling configurations, exceeding the 16B base pipeline at 13 × fewer parameters. E/H/A denote Easy/Hard/Average.
Pipeline
Single
Multi
Hit@1
R@10
FC@10
Hit@1
R@10
FC@10
Qwen3-Emb-0.6B × Qwen3-Rank-0.6B
.625
.708
.708
.647
.556
.324
Qwen3-Emb-8B × Qwen3-Rank-8B
.667
.812
.812
.686
.636
.382
SR-Emb-0.6B × SR-Rank-0.6B
.729
.875
.875
.745
.624
.353
Table 5: Single- vs. multi-skill calibration for two base pipelines and our primary 1.2B pipeline. Hit@1 reflects top-1 routing, R@10 reflects shortlist coverage, and FC@10 reflects strict full coverage for multi-skill queries.
Component
Variant
Hit@1
MRR@10
R@10
Encoder training
SR-Emb-0.6B
Clean negatives
.653
.723
.688
SR-Emb-0.6B
Raw negatives
.613
.692
.672
Reranker training
SR-Emb-0.6B
Encoder-only (no reranking)
.653
.723
.688
SR-Emb-0.6B × Qwen3-Rank-0.6B
Base reranker
.707
.769
.724
Table 6: Key ablations. False-negative filtering contributes +4.0pp encoder Hit@1; listwise reranking is essential, outperforming the pointwise variant by +30.7pp. Top: encoder variants. Bottom: reranker variants using SR-Emb-0.6B as the retriever.
Encoder
Reranker
Hit@1
Hit@3
Hit@5
Hit@10
Hit@20
MRR@10
Qwen3-Emb-0.6B
Qwen3-Rank-0.6B
.516
.664
.707
.781
.812
.603
Qwen3-Emb-8B
Qwen3-Rank-8B
.637
.762
.801
.852
.863
.707
SR-Emb-0.6B
Qwen3-Rank-0.6B
.574
.707
.762
.805
.828
.652
SR-Emb-0.6B
SR-Rank-0.6B
.641
.746
.781
.809
.840
.699
SR-Emb-8B
Qwen3-Rank-8B
.699
.816
.855
.895
.914
.764
SR-Emb-8B
SR-Rank-8B
.730
.828
.859
.895
.910
.784
Table 7: SkillBench-Supp retrieve-and-rerank results on 256 single-label queries over a 77K pool. The compact 1.2B tuned pipeline (highlighted) reaches the highest Hit@1 among non-scaling configurations, while gray rows show 8B-component scaling variants. Bold and underlined values denote the best and second-best non-scaling results. Checkpoints are transferred without re-tuning.
Skill Condition
Router / Source
Top- K
Single Success
Multi Success
Overall Success
No skills
None
–
12.50%
16.01%
14.89%
Gold skills
Oracle ground-truth
GT
30.90%
33.50%
32.67%
Retrieved skills
Qwen3-Emb-8B × Qwen3-Rank-8B
1
26.74%
25.33%
25.78%
Retrieved skills
SR-Emb-0.6B × SR-Rank-0.6B
1
29.86%
26.47%
27.56%
Retrieved skills
Qwen3-Emb-8B × Qwen3-Rank-8B
10
20.49%
27.78%
25.45%
Retrieved skills
SR-Emb-0.6B × SR-Rank-0.6B
10
26.04%
28.60%
27.78%
Table 8: End-to-end agent evaluation on the 75-task core set using skills from the natural pool (without Hard-tier distractors). Results average over 3 trials × 4 coding agents. Gold skills are oracle upper bounds.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Value
Descriptions empty
0.12%
Descriptions < 10 words
18.66%
Descriptions < 25 words
59.22%
Median description length
21 words
Median body length
704 words
P90 body length
1,991 words
Appendix
Table 9: Metadata audit for the 78,361-skill Easy pool. Descriptions are usually present, but they are far more compressed than bodies.
Dimension
Value
Application domains
55 (8 super-categories)
Unique topic tags
244
SkillsBench task difficulty
easy 4 / medium 45 / hard 26
Single / multi-skill
24 / 51
GT skills per query
1–7 (mean 2.75)
Multi-skill types
pipeline 43% / substitute 25% / mixed 32%
Appendix
Table 10: Diversity profile of the 75 core benchmark queries.
Stage
Model
Input
Easy Hit@1
Hard Hit@1
Avg Hit@1
Encoder
Qwen3-Emb-0.6B
nd
.227
.147
.187
Encoder
Qwen3-Emb-0.6B
all-field
.587
.533
.560
Encoder
Qwen3-Emb-8B
nd
.307
.200
.253
Encoder
Qwen3-Emb-8B
all-field
.653
.627
.640
Pipeline
Qwen3 0.6B × 0.6B
nd
.360
.173
.267
Pipeline
Qwen3 0.6B × 0.6B
all-field
.653
.627
.640
Appendix
Table 11: Detailed nd/all-field Hit@1 on the same benchmark pool and 75 core queries used in the main text. The rows provide the per-tier values underlying the representative comparisons in Figure 1 .
System: You are a skill document writer for a coding agent platform. You produce SKILL.md-style documents that are plausible but address a DIFFERENT problem than the reference skill. Each distractor must look like a real, useful skill document but must NOT solve the same task as the reference.
User: I have a ground-truth skill used for the task(s): <task_ids>
Generate <num_distractors> HARD distractor skills. Each distractor must be a complete SKILL.md document that looks relevant to someone searching for this skill, but actually solves a different problem.
Use these distractor strategies (one per distractor):
Table 12: Distractor skill generation prompt for the Hard evaluation tier.
Layer (Group)
Name
Desc
Body
Layer groups (averaged)
Early (0–6)
2.3%
0.3%
97.3%
Middle (7–20)
9.6%
1.4%
89.0%
Late (21–27)
7.5%
0.8%
91.7%
Key individual layers
Layer 0
0.3%
0.1%
99.6%
Appendix
Table 13: Raw attention distribution across skill fields by layer group and key individual layers (SR-Rank-0.6B, 28 layers × 16 heads, 75 queries).
Field
Token Share
Overall Attn.
Layer 19
Layer 27
Name
3.0%
7.3%
26.3%
1.5%
Desc
0.5%
1.0%
1.8%
0.4%
Body
96.5%
91.7%
72.0%
98.1%
Appendix
Table 14: Length-controlled attention diagnostics for the same 75 query-skill pairs. Attention share is measured from the last input position used to predict the next token (and thus the yes/no decision), averaged over heads, and normalized within the name/description/body fields. If attention were explained mainly by field length, per-layer attention would remain close to field token share.
Figure 3: Length-controlled attention visualization for SR-Rank-0.6B on 75 query-skill pairs. Left: per-layer mean attention trajectories compared against each field’s token-share baseline; shaded bands denote ±1 standard deviation across the 75 query-skill pairs. The short name field spikes far above its 3.0% token-share baseline in the middle layers, while the final layer returns to body. Right: query-level final-layer body attention compared against each query’s body-token baseline. Most points lie above the diagonal, meaning the final layer attends to body more than a pure length-based baseline would predict.
Quartile
N
All-field Hit@1
ND Hit@1
Gap (pp)
Q1 ( ≤ 19w)
26
53.8%
26.9%
+26.9
Q2 (20–27w)
40
80.0%
30.0%
+50.0
Q3 (28–35w)
38
65.8%
26.3%
+39.5
Q4 ( > 35w)
44
52.3%
20.5%
+31.8
Overall
148
63.5%
25.7%
+37.8
Appendix
Table 15: nd versus all-field Hit@1 stratified by GT-description word count. The gap remains large in every quartile, including Q4 where descriptions are longest.
System
Stage
Raw nd
Oracle-ND
All-field
All-field − Oracle
SkillRouter-1.2B
Encoder
37.5
41.8
48.8
+7.0
Pipeline
47.3
55.9
64.1
+8.2
Qwen3-1.2B pipeline
Encoder
25.4
35.2
46.1
+10.9
Pipeline
32.4
46.5
51.6
+5.1
Qwen3-16B pipeline
Encoder
33.2
39.1
50.4
+11.3
Pipeline
43.0
51.6
63.7
+12.1
Appendix
Table 16: SkillBench-Supp Oracle-ND results (Hit@1, %). Oracle-ND recovers part of the raw-metadata deficit, while all-field routing retains a 5.1–12.1pp pipeline advantage and a 7.0–11.3pp encoder advantage.
System: You are an experienced user of AI assistants. You write clear, realistic task requests that describe what you need to accomplish.
User: Given the following skill specification, write a realistic task description that someone would ask an AI assistant to help with. The task should naturally require the capabilities described in this skill.
Skill name: <name>
Category: <category>
Description: <description>
Skill body: <body_preview>
Requirements: (1) Describe a concrete scenario with specific inputs/outputs. (2) Include enough detail that the skill would be clearly useful. (3) Do NOT mention the skill name “ <name> ” anywhere in the task.
Appendix
Table 17: Query generation prompt template (simplified). The LLM receives the skill’s metadata and produces a realistic task description without mentioning the skill name.
Setting
SR-Emb-0.6B
SR-Rank-0.6B
Objective
In-batch InfoNCE ( τ=0.05 )
Listwise CE ( τ=1.0 )
Input / prompt
Query instruction prefix; skill = name | description | body
flattened all-field query-document prompt over top-20 candidates
Field caps
query 1500 chars; desc 300 chars; body 2500 chars
desc 500 chars; body 2000 chars
Max len
2048
4096
Epoch
1
1
Batch
8
1 listwise group
Appendix
Table 18: Selected training hyperparameters for the reported SkillRouter models. Character caps are applied before tokenization.
Figure 4: Recall@ K candidate coverage for three encoder retrievers on Easy and Hard. The star marker at K=20 indicates the primary SR-Emb-0.6B × SR-Rank-0.6B pipeline’s end-to-end Hit@1 at the main operating point, and is shown only as a reference against the Recall@ K curves.
System
p50 (ms)
p95 (ms)
QPS
GPU Mem (MiB)
GPU-sec / 1K
SR-Emb-0.6B
19.8
20.8
50.5
9364
19.8
SR pipeline (1.2B)
495.8
871.4
1.83
18976
547.0
Qwen3-Emb-8B
60.4
81.3
18.7
22539
53.5
Qwen3 pipeline (16B)
2900.1
5676.5
0.32
22539
6189.9
Appendix
Table 20: Real-pool GPU serving benchmark on 80 timed queries (274–5109 chars). Encoder-only rows are encoder-bound; pipeline rows are rerank-forward-bound. GPU-sec / 1K is a benchmark compute-footprint measure rather than a dollar-cost estimate.
Model
Skill Condition
Router / Source
Top- K
Single
Multi
Overall
Kimi-K2.5
No skills
None
–
12.50
9.15
10.22
Gold skills
Oracle ground-truth
GT
20.83
25.49
24.00
Retrieved skills
Qwen3-Pipeline-16B
1
20.83
11.76
14.67
Retrieved skills
SkillRouter-1.2B
1
20.83
16.34
17.78
Retrieved skills
Qwen3-Pipeline-16B
10
16.67
21.57
20.00
Retrieved skills
SkillRouter-1.2B
10
19.44
17.65
18.22
Appendix
Table 25: Direct end-to-end agent success rates (%) by model and skill condition. Each task is evaluated three times per condition. Retrieved skills use the natural pool, i.e., the non-synthetic benchmark pool without Hard-tier distractors.
Large language model (LLM) agents increasingly rely on external skills, but routing user requests over large skill registries is difficult because many skills are functionally redundant while complex tasks often require complementary skill sets. Existing skill routers typically rank candidates independently by query relevance, which can waste context budget on redundant skills. We propose Diverse Skill Routing (DSR), a diversity-aware reranking framework that uses a Determinantal Point Process to balance relevance and non-redundancy. DSR introduces a query-residual diversity kernel that penalizes redundant skill overlap while reducing penalties caused only by shared query relevance. On the SkillRouter benchmark, DSR improves recall and full coverage over a strong pointwise reranking baseline, with larger gains on multi-skill queries. These results suggest that skill routing should be treated not only as relevance ranking, but also as complementary set selection.
Wang Wei, Tiankai Yang, Samyadeep Basu +7
Virginia Tech · University of Southern California · Adobe Research +3
Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. Deployed harnesses route by preloading every skill's metadata into the context, which disperses the agent's attention and caps the library size. Retrieval pipelines move the selection out of the context, but also out of the agent's capability. We show that the frozen agent LLM already carries the routing signal in its own forward passes, and that two linear maps suffice to read it out with no skill text in the context. Our Gavel (Glance And Verdict from a frozen LLM) reads it in two steps. A glance scores the full library by matching the task's mid-layer states against a compact bank that one forward pass builds for each skill at installation, with the two maps as the only trained parameters. A verdict then resumes each shortlisted skill's forward pass, reads the model's own likelihood and yes/no judgment, and fuses both with the glance as a product of experts. Trained once, Gavel transfers zero-shot to three public benchmarks and SkillTraj, our new benchmark of 372 simulated agent trajectories. On Qwen3-32B it outperforms progressive disclosure and retrieve-and-rerank pipelines that add 1.2B to 16B external parameters, by up to 13.4 points on written tasks and up to 21.9 when the need for a skill arises mid-rollout. Routing accuracy improves as the backbone does, and in a bash-agent harness Gavel lets the 32B trigger the right skill on Skill-Use more often than models of up to 1.6T parameters in Codex.
Ruishuo Chen, Xun Wang, Yu Chen +2
Institute for Interdisciplinary Information Sciences, Tsinghua University
Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We test whether this design scales by holding SkillsBench tasks and their golden skills fixed while adding irrelevant skills. We find skill routing collapse: agents increasingly load fewer or no skills, even though the relevant skills remain available. With 1,000 irrelevant skills, loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%, whereas explicit selection from the same library retains 73.9% recall. Loading other skills, including plausible alternatives to golden skills, is too rare to explain this decline. Masking only the descriptions of irrelevant skills largely restores skill loading, yet with the metadata intact, explicitly asking agents to plan skill use still yields lower recall than preselection. These findings point to semantic interference from descriptions and instruction interference when skill choice competes with task execution. We also evaluate explicit preselection as a practical mitigation, including its coverage and latency trade-offs. Our analysis informs skill protocols and evaluation for agents with growing skill libraries.
Jinlin Liu, Wei Chen, Xiaojin Zhang
The University of Hong Kong · Huazhong University of Science and Technology