cs.LGMar 23, 2026
SaveSkillRouter: Skill Routing for LLM Agents at Scale
Organizations: Alibaba Group, Hangzhou, China
Abstract
Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks. As skill ecosystems grow to tens of thousands of entries, exposing every skill at inference time becomes infeasible. This creates a skill-routing problem: given a user task, the system must identify relevant skills before downstream planning or execution. Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body. We examine this design choice on a SkillsBench-derived benchmark with approximately 80K candidate skills, targeting the practically important setting of large skill registries with heavy overlap. Across representative dense and reranking baselines on this setting, hiding the skill body causes a 37-44 percentage point drop in routing accuracy. Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart. Motivated by this finding, we present Skillrouter, a compact 1.2B body-aware retrieve-and-rerank pipeline. Skillrouter achieves 74.0% Hit@1 on our benchmark -- the strongest average top-1 routing performance among the baselines we evaluate -- while using 13 fewer parameters and running 5.8 faster than the strongest base pipeline. The ranking gains further generalize to a supplementary benchmark independently constructed from three skill sources. In a complementary end-to-end study across four coding agents, routing gains transfer to improved task success, with larger gains for more capable agents.
Figures & tables
Figure 1: Body access is a critical routing signal. Left: Averaged over Easy and Hard, removing body reduces Hit@1 by 37.3pp for Qwen3-Emb-0.6B, 38.7pp for Qwen3-Emb-8B, and 44.0pp for Qwen3-Emb-8B Qwen3-Rank-8B. Right: A length-controlled diagnostic: the body occupies 96.5% of skill tokens, yet the 3.0%-token name field peaks at 26.3% attention in layer 19 before the final layer returns to 98.1% body attention.
| System | Raw nd | Oracle nd | All-field |
| SkillRouter-1.2B pipeline | .273 | .527 | .740 |
| Qwen3 0.6B 0.6B | .267 | .567 | .640 |
| Qwen3 8B 8B | .240 | .613 | .680 |
| Encoder-only, matched training for each input regime | |||
| Fine-tuned, nd input | .513 | .320 | – |
| Fine-tuned, all-field input | – | – | .653 |
Table 2: Stronger metadata controls on the main benchmark (Hit@1, Easy/Hard average). Oracle nd is a query-blind, body-informed replacement; Appendix E gives its exact construction. The fine-tuned encoders are trained separately on the same 37,979 pairs; .320 applies the GT-only Oracle replacement to the nd-trained encoder.
Figure 2: SkillRouter pipeline. A bi-encoder retrieves top-20 candidates from the 80K pool; a cross-encoder reranks them. Both stages use name, description, and body content, motivated by Section 3 .
| Model | Params | E-Hit@1 | H-Hit@1 | A-Hit@1 | A-MRR@10 | A-R@20 |
| BM25 | – | .347 | .280 | .314 | .365 | .365 |
| BGE-Large-v1.5 | 335M | .613 | .587 | .600 | .653 | .668 |
| gemini-embedding-001 | – | .613 | .560 | .587 | .650 | .687 |
| text-embedding-3-large | – | .640 | .600 | .620 | .658 | .664 |
| Qwen3-Emb-0.6B | 0.6B | .587 | .533 | .560 | .638 | .637 |
| Qwen3-Emb-8B | 8B | .653 | .627 | .640 | .698 | .726 |
Table 3: Encoder-only retrieval results on the 80K skill-routing benchmark. The tuned 0.6B encoder (highlighted) outperforms the 13 larger base encoder, showing that task-specific training compensates for scale in this setting. E/H/A denote Easy/Hard/Average. R@20 reflects candidate coverage for downstream reranking.
| Encoder | Reranker | E-Hit@1 | H-Hit@1 | A-Hit@1 | A-MRR@10 | A-R@10 |
| Qwen3-Emb-0.6B | Qwen3-Rank-0.6B | .653 | .627 | .640 | .684 | .604 |
| Qwen3-Emb-8B | Qwen3-Rank-0.6B | .613 | .547 | .580 | .672 | .694 |
| Qwen3-Emb-8B | Qwen3-Rank-8B | .680 | .680 | .680 | .745 | .692 |
| SR-Emb-0.6B | Qwen3-Rank-0.6B | .720 | .693 | .707 | .769 | .724 |
| SR-Emb-0.6B | Qwen3-Rank-8B | .720 | .707 | .714 | .776 | .727 |
| SR-Emb-0.6B | SR-Rank-0.6B | .760 | .720 | .740 | .791 | .704 |
Table 4: End-to-end retrieve-and-rerank results (top-20 candidates). The compact 1.2B tuned pipeline (highlighted) reaches the highest Hit@1 among non-scaling configurations, exceeding the 16B base pipeline at 13 fewer parameters. E/H/A denote Easy/Hard/Average.
| Pipeline | Single | Multi | ||||
| Hit@1 | R@10 | FC@10 | Hit@1 | R@10 | FC@10 | |
| Qwen3-Emb-0.6B Qwen3-Rank-0.6B | .625 | .708 | .708 | .647 | .556 | .324 |
| Qwen3-Emb-8B Qwen3-Rank-8B | .667 | .812 | .812 | .686 | .636 | .382 |
| SR-Emb-0.6B SR-Rank-0.6B | .729 | .875 | .875 | .745 | .624 | .353 |
Table 5: Single- vs. multi-skill calibration for two base pipelines and our primary 1.2B pipeline. Hit@1 reflects top-1 routing, R@10 reflects shortlist coverage, and FC@10 reflects strict full coverage for multi-skill queries.
| Component | Variant | Hit@1 | MRR@10 | R@10 |
| Encoder training | ||||
| SR-Emb-0.6B | Clean negatives | .653 | .723 | .688 |
| SR-Emb-0.6B | Raw negatives | .613 | .692 | .672 |
| Reranker training | ||||
| SR-Emb-0.6B | Encoder-only (no reranking) | .653 | .723 | .688 |
| SR-Emb-0.6B Qwen3-Rank-0.6B | Base reranker | .707 | .769 | .724 |
Table 6: Key ablations. False-negative filtering contributes +4.0pp encoder Hit@1; listwise reranking is essential, outperforming the pointwise variant by +30.7pp. Top: encoder variants. Bottom: reranker variants using SR-Emb-0.6B as the retriever.
| Encoder | Reranker | Hit@1 | Hit@3 | Hit@5 | Hit@10 | Hit@20 | MRR@10 |
| Qwen3-Emb-0.6B | Qwen3-Rank-0.6B | .516 | .664 | .707 | .781 | .812 | .603 |
| Qwen3-Emb-8B | Qwen3-Rank-8B | .637 | .762 | .801 | .852 | .863 | .707 |
| SR-Emb-0.6B | Qwen3-Rank-0.6B | .574 | .707 | .762 | .805 | .828 | .652 |
| SR-Emb-0.6B | SR-Rank-0.6B | .641 | .746 | .781 | .809 | .840 | .699 |
| SR-Emb-8B | Qwen3-Rank-8B | .699 | .816 | .855 | .895 | .914 | .764 |
| SR-Emb-8B | SR-Rank-8B | .730 | .828 | .859 | .895 | .910 | .784 |
Table 7: SkillBench-Supp retrieve-and-rerank results on 256 single-label queries over a 77K pool. The compact 1.2B tuned pipeline (highlighted) reaches the highest Hit@1 among non-scaling configurations, while gray rows show 8B-component scaling variants. Bold and underlined values denote the best and second-best non-scaling results. Checkpoints are transferred without re-tuning.
| Skill Condition | Router / Source | Top- | Single Success | Multi Success | Overall Success |
| No skills | None | – | 12.50% | 16.01% | 14.89% |
| Gold skills | Oracle ground-truth | GT | 30.90% | 33.50% | 32.67% |
| Retrieved skills | Qwen3-Emb-8B Qwen3-Rank-8B | 1 | 26.74% | 25.33% | 25.78% |
| Retrieved skills | SR-Emb-0.6B SR-Rank-0.6B | 1 | 29.86% | 26.47% | 27.56% |
| Retrieved skills | Qwen3-Emb-8B Qwen3-Rank-8B | 10 | 20.49% | 27.78% | 25.45% |
| Retrieved skills | SR-Emb-0.6B SR-Rank-0.6B | 10 | 26.04% | 28.60% | 27.78% |
Table 8: End-to-end agent evaluation on the 75-task core set using skills from the natural pool (without Hard-tier distractors). Results average over 3 trials 4 coding agents. Gold skills are oracle upper bounds.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Statistic | Value |
| Descriptions empty | 0.12% |
| Descriptions 10 words | 18.66% |
| Descriptions 25 words | 59.22% |
| Median description length | 21 words |
| Median body length | 704 words |
| P90 body length | 1,991 words |
Table 9: Metadata audit for the 78,361-skill Easy pool. Descriptions are usually present, but they are far more compressed than bodies.
| Dimension | Value |
| Application domains | 55 (8 super-categories) |
| Unique topic tags | 244 |
| SkillsBench task difficulty | easy 4 / medium 45 / hard 26 |
| Single / multi-skill | 24 / 51 |
| GT skills per query | 1–7 (mean 2.75) |
| Multi-skill types | pipeline 43% / substitute 25% / mixed 32% |
Table 10: Diversity profile of the 75 core benchmark queries.
| Stage | Model | Input | Easy Hit@1 | Hard Hit@1 | Avg Hit@1 |
| Encoder | Qwen3-Emb-0.6B | nd | .227 | .147 | .187 |
| Encoder | Qwen3-Emb-0.6B | all-field | .587 | .533 | .560 |
| Encoder | Qwen3-Emb-8B | nd | .307 | .200 | .253 |
| Encoder | Qwen3-Emb-8B | all-field | .653 | .627 | .640 |
| Pipeline | Qwen3 0.6B 0.6B | nd | .360 | .173 | .267 |
| Pipeline | Qwen3 0.6B 0.6B | all-field | .653 | .627 | .640 |
Table 11: Detailed nd/all-field Hit@1 on the same benchmark pool and 75 core queries used in the main text. The rows provide the per-tier values underlying the representative comparisons in Figure 1 .
| System: You are a skill document writer for a coding agent platform. You produce SKILL.md-style documents that are plausible but address a DIFFERENT problem than the reference skill. Each distractor must look like a real, useful skill document but must NOT solve the same task as the reference. |
| User: I have a ground-truth skill used for the task(s): <task_ids> |
| Reference skill (name: <skill_name> , category: <category> ): |
| <body_truncated> |
| Generate <num_distractors> HARD distractor skills. Each distractor must be a complete SKILL.md document that looks relevant to someone searching for this skill, but actually solves a different problem. |
| Use these distractor strategies (one per distractor): |
| same-domain-diff-problem, same-tech-diff-use, over-generalized |
Table 12: Distractor skill generation prompt for the Hard evaluation tier.
| Layer (Group) | Name | Desc | Body |
| Layer groups (averaged) | |||
| Early (0–6) | 2.3% | 0.3% | 97.3% |
| Middle (7–20) | 9.6% | 1.4% | 89.0% |
| Late (21–27) | 7.5% | 0.8% | 91.7% |
| Key individual layers | |||
| Layer 0 | 0.3% | 0.1% | 99.6% |
Table 13: Raw attention distribution across skill fields by layer group and key individual layers (SR-Rank-0.6B, 28 layers 16 heads, 75 queries).
| Field | Token Share | Overall Attn. | Layer 19 | Layer 27 |
| Name | 3.0% | 7.3% | 26.3% | 1.5% |
| Desc | 0.5% | 1.0% | 1.8% | 0.4% |
| Body | 96.5% | 91.7% | 72.0% | 98.1% |
Table 14: Length-controlled attention diagnostics for the same 75 query-skill pairs. Attention share is measured from the last input position used to predict the next token (and thus the yes/no decision), averaged over heads, and normalized within the name/description/body fields. If attention were explained mainly by field length, per-layer attention would remain close to field token share.
Figure 3: Length-controlled attention visualization for SR-Rank-0.6B on 75 query-skill pairs. Left: per-layer mean attention trajectories compared against each field’s token-share baseline; shaded bands denote standard deviation across the 75 query-skill pairs. The short name field spikes far above its 3.0% token-share baseline in the middle layers, while the final layer returns to body. Right: query-level final-layer body attention compared against each query’s body-token baseline. Most points lie above the diagonal, meaning the final layer attends to body more than a pure length-based baseline would predict.
| Quartile | N | All-field Hit@1 | ND Hit@1 | Gap (pp) |
| Q1 ( 19w) | 26 | 53.8% | 26.9% | +26.9 |
| Q2 (20–27w) | 40 | 80.0% | 30.0% | +50.0 |
| Q3 (28–35w) | 38 | 65.8% | 26.3% | +39.5 |
| Q4 ( 35w) | 44 | 52.3% | 20.5% | +31.8 |
| Overall | 148 | 63.5% | 25.7% | +37.8 |
Table 15: nd versus all-field Hit@1 stratified by GT-description word count. The gap remains large in every quartile, including Q4 where descriptions are longest.
| System | Stage | Raw nd | Oracle-ND | All-field | All-field Oracle |
| SkillRouter-1.2B | Encoder | 37.5 | 41.8 | 48.8 | +7.0 |
| Pipeline | 47.3 | 55.9 | 64.1 | +8.2 | |
| Qwen3-1.2B pipeline | Encoder | 25.4 | 35.2 | 46.1 | +10.9 |
| Pipeline | 32.4 | 46.5 | 51.6 | +5.1 | |
| Qwen3-16B pipeline | Encoder | 33.2 | 39.1 | 50.4 | +11.3 |
| Pipeline | 43.0 | 51.6 | 63.7 | +12.1 |
Table 16: SkillBench-Supp Oracle-ND results (Hit@1, %). Oracle-ND recovers part of the raw-metadata deficit, while all-field routing retains a 5.1–12.1pp pipeline advantage and a 7.0–11.3pp encoder advantage.
| System: You are an experienced user of AI assistants. You write clear, realistic task requests that describe what you need to accomplish. |
| User: Given the following skill specification, write a realistic task description that someone would ask an AI assistant to help with. The task should naturally require the capabilities described in this skill. |
| Skill name: <name> |
| Category: <category> |
| Description: <description> |
| Skill body: <body_preview> |
| Requirements: (1) Describe a concrete scenario with specific inputs/outputs. (2) Include enough detail that the skill would be clearly useful. (3) Do NOT mention the skill name “ <name> ” anywhere in the task. |
Table 17: Query generation prompt template (simplified). The LLM receives the skill’s metadata and produces a realistic task description without mentioning the skill name.
| Setting | SR-Emb-0.6B | SR-Rank-0.6B |
| Objective | In-batch InfoNCE ( ) | Listwise CE ( ) |
| Input / prompt | Query instruction prefix; skill = name | description | body | flattened all-field query-document prompt over top-20 candidates |
| Field caps | query 1500 chars; desc 300 chars; body 2500 chars | desc 500 chars; body 2000 chars |
| Max len | 2048 | 4096 |
| Epoch | 1 | 1 |
| Batch | 8 | 1 listwise group |
Table 18: Selected training hyperparameters for the reported SkillRouter models. Character caps are applied before tokenization.
Figure 4: Recall@ candidate coverage for three encoder retrievers on Easy and Hard. The star marker at indicates the primary SR-Emb-0.6B SR-Rank-0.6B pipeline’s end-to-end Hit@1 at the main operating point, and is shown only as a reference against the Recall@ curves.
| System | p50 (ms) | p95 (ms) | QPS | GPU Mem (MiB) | GPU-sec / 1K |
| SR-Emb-0.6B | 19.8 | 20.8 | 50.5 | 9364 | 19.8 |
| SR pipeline (1.2B) | 495.8 | 871.4 | 1.83 | 18976 | 547.0 |
| Qwen3-Emb-8B | 60.4 | 81.3 | 18.7 | 22539 | 53.5 |
| Qwen3 pipeline (16B) | 2900.1 | 5676.5 | 0.32 | 22539 | 6189.9 |
Table 20: Real-pool GPU serving benchmark on 80 timed queries (274–5109 chars). Encoder-only rows are encoder-bound; pipeline rows are rerank-forward-bound. GPU-sec / 1K is a benchmark compute-footprint measure rather than a dollar-cost estimate.
| Model | Skill Condition | Router / Source | Top- | Single | Multi | Overall |
| Kimi-K2.5 | No skills | None | – | 12.50 | 9.15 | 10.22 |
| Gold skills | Oracle ground-truth | GT | 20.83 | 25.49 | 24.00 | |
| Retrieved skills | Qwen3-Pipeline-16B | 1 | 20.83 | 11.76 | 14.67 | |
| Retrieved skills | SkillRouter-1.2B | 1 | 20.83 | 16.34 | 17.78 | |
| Retrieved skills | Qwen3-Pipeline-16B | 10 | 16.67 | 21.57 | 20.00 | |
| Retrieved skills | SkillRouter-1.2B | 10 | 19.44 | 17.65 | 18.22 |
Table 25: Direct end-to-end agent success rates (%) by model and skill condition. Each task is evaluated three times per condition. Retrieved skills use the natural pool, i.e., the non-synthetic benchmark pool without Hard-tier distractors.
More in this topic
LLM Agent Skill Retrieval
Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.
See all 102 papers