Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale

Jinlin Liu; Wei Chen; Xiaojin Zhang

cs.AIOct 13, 2026

Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale

Authors: Jinlin Liu, Wei Chen, Xiaojin Zhang

Organizations: The University of Hong Kong · Huazhong University of Science and Technology

Abstract

Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We test whether this design scales by holding SkillsBench tasks and their golden skills fixed while adding irrelevant skills. We find skill routing collapse: agents increasingly load fewer or no skills, even though the relevant skills remain available. With 1,000 irrelevant skills, loaded-skill recall falls from 73.6% to 27.4% and the zero-load rate rises from 9.3% to 51.6%, whereas explicit selection from the same library retains 73.9% recall. Loading other skills, including plausible alternatives to golden skills, is too rare to explain this decline. Masking only the descriptions of irrelevant skills largely restores skill loading, yet with the metadata intact, explicitly asking agents to plan skill use still yields lower recall than preselection. These findings point to semantic interference from descriptions and instruction interference when skill choice competes with task execution. We also evaluate explicit preselection as a practical mitigation, including its coverage and latency trade-offs. Our analysis informs skill protocols and evaluation for agents with growing skill libraries.

Figures & tables

Figure 1: Lost in implicit routing. The relevant skills remain installed, but as irrelevant skills grow, agents increasingly fail to load them during implicit execution. Explicit routing recovers much of the missing skill recall.

Figure 2: Overview of the controlled skill-scaling setup and routing strategies. Starting from an existing SkillsBench task, we keep the instruction, files, verifier, and relevant skills fixed while varying only the number of added irrelevant skills.

Figure 3: Implicit-routing scaling by execution model. (a) End-to-end Pass@3. (b) Loaded-skill recall. (c) Zero-load rate. Only the number of added irrelevant skills changes.

$s$	Avg. skills loaded	ZeroLoad	Load Rec.
0	1.72	9.2	73.5
10	1.54	13.0	67.8
100	1.20	27.9	52.9
1000	0.66	50.8	27.6

Table 1: Skill-loading behavior as the number of irrelevant skills grows. ZeroLoad and loaded-skill recall are percentages. This analysis covers 86 tasks, a slightly broader set than the main study (Appendix I ).

Model	Measure	0	10	100	1k
GLM-5	Implicit loaded	79.6	71.8	55.0	20.9
GLM-5	Explicit selected	98.4	93.1	91.7	82.5
Kimi K2.5	Implicit loaded	78.8	74.4	63.2	40.5
Kimi K2.5	Explicit selected	98.5	93.3	90.7	80.7
MiniMax M2.5	Implicit loaded	62.3	58.2	41.6	21.5
MiniMax M2.5	Explicit selected	98.3	88.0	83.6	58.4

Table 2: Implicit loaded-skill recall (%) versus explicit LLM selected-skill recall (%) by model and number of added irrelevant skills.

Figure 4: Skill scaling curves by routing strategy. (a) End-to-end Pass@3. (b) Loaded-skill recall. (c) Zero-load rate.

Naive
Model	w/o	0	10	100	1k	10k
GLM-5	26.8	57.3	59.8	56.1	42.7	–
Kimi K2.5	34.1	52.4	56.8	50.6	43.9	–
MiniMax M2.5	26.8	45.1	46.3	41.5	29.3	–
LLM preselect
GLM-5	26.8	65.9	57.3	57.3	58.5	–

Table 3: Per-model Pass@3 (%) by routing strategy and number of added irrelevant skills. The w/o column disables skills entirely; “–” marks omitted infeasible settings.

Method	0	10	100	1k	10k
LLM	98.4	91.5	88.7	73.9	–
RAG	98.6	95.9	90.0	74.1	62.0
Hybrid	98.2	91.4	87.9	81.9	72.2

Table 4: Preselected-skill recall (%) by method and number of added irrelevant skills.

$s$	Strat.	A-In	A-Out	P-In	P-Out	Total
0	Naive	485k	13k	0k	0k	498k
0	LLM	481k	13k	0k	0k	494k
0	RAG	494k	13k	0k	0k	507k
0	Hybrid	499k	13k	0k	0k	513k
10	Naive	481k	13k	0k	0k	494k
10	LLM	495k	13k	1k	0k	509k

Table 5: Token decomposition by skill-pool size and strategy. A/P denote execution-agent/preselection tokens; In/Out denote input/output tokens.

$s$	SQ Rec.	Hybrid Rec.	$\Delta$ Rec.
0	99.4	99.4	+0.0
10	99.4	99.4	+0.0
100	99.4	99.4	+0.0
1000	93.6	96.3	+2.7
10000	88.2	91.8	+3.6
$s$	SQ All	Hybrid All	$\Delta$ All

Table 6: Single-query RAG@100 versus the hybrid candidate pool. Values are percentages. Top: individual recall; bottom: all-gold coverage.

Implicit routing, 1,000 irrelevant skills
Setting	Load Rec.	Zero	All-gold
No extra skills	73.5	5.6	52.2
Original	18.6	62.2	6.7
Masked	63.9	23.6	49.4
Plan-first prompt	44.6	15.7	25.8
Explicit preselection, 1,000 irrelevant skills

Table 7: Routing results (%) on the 10-task subset. All rows except the first use 1,000 irrelevant skills. Load Rec. is loaded-skill recall; Zero is the zero-load rate; All-gold is the share of runs that load or select every golden skill.

Setting	Avg. loaded	ZeroLoad	Load Rec.
Original metadata	0.50	60.0	18.3
Mask names	0.93	40.0	43.7
Mask descriptions	1.17	23.3	63.3
Mask locations	0.90	36.7	47.5
Mask names + locations	1.10	36.7	51.7
Mask all distractor (reference)	1.26	23.6	63.9

Table 8: Field-level masking of distractor metadata on the 10-task subset. Avg. loaded is the mean number of skills loaded; ZeroLoad and Load Rec. are percentages. The first five rows use one run per task and model; full masking is the three-run reference.

Setting	Recall	All-gold
Plan-first prompt	39.9	20.5
LLM preselection	70.6	54.4

Table 9: Golden-skill selection quality (%) in the Plan-first response versus isolated LLM preselection on the same 10-task subset. Recall is computed against the task’s golden skills.

Panel of Figure 5 — Figure 5: Two trace cases under the plan-first prompt. Actually loaded counts both load_skill calls and direct reads of skill files.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

$\|G\|$	Tasks
1	26
2	20
3	16
4	11
5	6
6	2

Appendix

Table 10: Distribution of golden-skill counts among the 82 adopted SkillsBench tasks.

|G|

is the number of golden skills for a task.

$s$	Runs	Time	Input toks	Output toks
0	730	15.3m	485k	13k
10	731	16.7m	481k	13k
100	728	15.4m	619k	13k
1000	727	17.0m	1.95M	13k

Appendix

Table 11: Average valid-run resource footprint under implicit routing by number of added irrelevant skills. Time is wall-clock execution time per run; token columns are LLM input and output tokens per run.

#	Task
1	fix-visual-stability
2	mhc-layer-impl
3	multilingual-video-dubbing
4	python-scala-translation
5	speaker-diarization-subtitles

Appendix

Table 12: Tasks excluded from the main controlled study for infrastructure or environment reasons.

#	Task
1	adaptive-cruise-control
2	data-to-d3
3	earthquake-phase-association
4	econ-detrending-correlation
5	enterprise-information-search
6	manufacturing-equipment-maintenance

Appendix

Table 13: Tasks used in the 10-task experimental subset.

Model	0	10	100	1k
GLM-5	57.3	59.8	56.1	42.7
Kimi K2.5	52.4	56.8	50.6	43.9
MiniMax M2.5	45.1	46.3	41.5	29.3

Appendix

Table 14: Values for Figure 3 (a): implicit-routing Pass@3 (%) by execution model and number of added irrelevant skills.

Model	0	10	100	1k
GLM-5	79.6	71.8	55.0	20.9
Kimi K2.5	78.8	74.4	63.2	40.5
MiniMax M2.5	62.3	58.2	41.6	21.5

Appendix

Table 15: Values for Figure 3 (b): implicit-routing loaded-skill recall (%) by execution model.

Model	0	10	100	1k
GLM-5	7.8	9.4	26.0	64.9
Kimi K2.5	7.0	7.9	16.7	31.3
MiniMax M2.5	13.1	20.7	39.3	58.4

Appendix

Table 16: Values for Figure 3 (c): implicit-routing zero-load rate (%) by execution model. Lower is better.

Strategy	0	10	100	1k	10k
Naive	51.6	54.3	49.4	38.6	–
LLM preselect	54.9	52.4	54.9	51.6	–
RAG	52.4	48.0	49.6	46.5	41.5
Hybrid preselection	52.8	53.7	50.0	52.8	50.4

Appendix

Table 17: Values for Figure 4 (a): aggregate Pass@3 (%) by routing strategy and number of added irrelevant skills. “–” indicates omitted infeasible settings.

Strategy	0	10	100	1k	10k
Naive	73.6	68.2	53.3	27.4	–
LLM preselect	74.0	73.6	73.0	62.0	–
RAG	74.6	70.1	69.3	58.6	48.5
Hybrid preselection	74.0	72.9	70.5	67.1	55.2

Appendix

Table 18: Values for Figure 4 (b): loaded-skill recall (%) by routing strategy.

Strategy	0	10	100	1k	10k
Naive	9.3	12.7	27.3	51.6	–
LLM preselect	8.5	8.4	9.5	9.8	–
RAG	8.1	11.7	12.3	13.2	18.0
Hybrid preselection	8.1	8.9	10.7	9.1	10.7

Appendix

Table 19: Values for Figure 4 (c): zero-load rate (%) by routing strategy. Lower is better.

Task	Task
3d-scan-calc	flink-query
adaptive-cruise-control	flood-risk-analysis
azure-bgp-oscillation-route-leak	gh-repo-analytics
citation-check	glm-lake-mendota
dynamic-object-aware-egomotion	gravitational-wave-detection
earthquake-phase-association	grid-dispatch-operator

Appendix

Table 20: Tasks used in the proprietary-model validation subset.

Model	0	10	100	1k
Claude Opus 4.6 $\dagger$	64.9	62.2	59.5	62.2
Gemini 3.1 Pro $\dagger$	59.5	51.4	48.6	51.4
GPT-5.4 $\dagger$	54.1	56.8	43.2	43.2
GLM-5	54.1	62.2	54.1	35.1
Kimi K2.5	48.6	58.3	50.0	45.9
MiniMax M2.5	35.1	45.9	37.8	21.6

Appendix

Table 21: Pass@3 (%) on the 37-task proprietary-model validation subset. Numeric columns give the number of added irrelevant skills on top of each task’s golden skills. Models marked with

\dagger

are proprietary; the remaining rows are the three open-weight execution models evaluated on the same task subset under implicit routing.

Model	0	10	100	1k
Claude Opus 4.6 $\dagger$	81.3	77.7	43.3	30.9
Gemini 3.1 Pro $\dagger$	77.4	70.4	57.9	43.0
GPT-5.4 $\dagger$	76.2	62.9	51.8	49.3

Appendix

Table 22: Loaded-skill recall (%) on the 37-task proprietary-model validation subset. The statistic counts golden skills whose full instructions are accessed during execution.

Naive
Model	0	10	100	1k	10k
GLM-5	99.6	99.5	97.1	88.6	–
Kimi K2.5	99.6	99.0	97.7	87.2	–
MiniMax M2.5	99.7	99.1	95.2	76.4	–
LLM preselect
GLM-5	99.6	99.6	97.7	90.8	–

Appendix

Table 23: Runtime loaded-skill precision (%) for open-weight models by routing strategy and number of added irrelevant skills.

Model	0	10	100	1k
Claude Opus 4.6 $\dagger$	100.0	100.0	100.0	95.2
Gemini 3.1 Pro $\dagger$	100.0	100.0	100.0	100.0
GPT-5.4 $\dagger$	100.0	97.3	90.5	82.7

Appendix

Table 24: Runtime loaded-skill precision (%) for proprietary models under implicit routing on the validation subset.

Method	0	10	100	1k	10k
LLM preselect	100.0	99.7	80.0	63.0	–
RAG	100.0	47.6	43.9	37.1	30.1
Hybrid preselection	100.0	99.8	84.5	72.0	52.6

Appendix

Table 25: Preselected-skill precision (%) by method and number of added irrelevant skills.

$s$	$\|G\|$	Tasks	Naive		LLM		RAG		Hybrid
			P3	Load	P3	Load	P3	Load	P3	Load
0	1	26	53.8	91.9	52.6	93.6	52.6	95.7	56.4	95.7
	2-3	36	52.8	68.7	55.6	68.1	55.6	68.7	50.0	67.2
	$\geq 4$	20	46.7	58.8	56.7	59.3	46.7	57.8	53.3	58.0
10	1	26	51.9	91.8	53.8	94.9	48.7	89.3	57.7	94.9
	2-3	36	58.3	60.9	52.8	67.6	49.1	63.3	53.7	66.6

Appendix

Table 26: Pass@3 and loaded-skill recall (%) by skill-pool size and number of golden skills. The row group

s

gives the number of added irrelevant skills;

|G|

bins tasks by their number of golden skills. Load is loaded-skill recall.

$s$	Agg.	Rec	All	Any
1000	Max	97.9	95.4	100.0
1000	Mean	93.9	89.7	97.7
1000	Quota	96.3	92.0	98.9
10000	Max	88.7	80.5	95.4
10000	Mean	88.7	78.2	95.4
10000	Quota	91.8	86.2	97.7

Appendix

Table 27: Hybrid candidate quality (%) at budget 100.

$s$	Non-golden load rate	$>0.8$ alternative rate
0	0.00	0.00
10	0.26	0.00
100	1.83	0.00
1000	7.19	3.79

Appendix

Table 28: Non-golden skill loading (%). Non-golden load rate measures whether any non-golden skill is loaded; alternative rate additionally requires similarity above 0.8 to a missed golden skill in the same evaluation.

Task	Model	Runs	Loaded non-golden $\rightarrow$ missed golden	Cosine
court-form-filling	Kimi K2.5	2	pdf-generator $\rightarrow$ pdf	0.922
financial-modeling-qa	Kimi K2.5	1, 2, 3	pdf-reader $\rightarrow$ pdf	0.903
financial-modeling-qa	MiniMax M2.5	1, 2	pdf-reader $\rightarrow$ pdf	0.903
invoice-fraud-detection	MiniMax M2.5	1, 3	pdf-reader $\rightarrow$ pdf	0.903
latex-formula-extraction	GLM-5	3	pdf-reader $\rightarrow$ pdf	0.903
latex-formula-extraction	MiniMax M2.5	1	pdf-reader $\rightarrow$ marker	0.826

Appendix

Table 29: All above-threshold skill-pair groups for the three open-weight models, all at

s=1000

. Runs lists which of the three repeated runs contain the pair. Similarities use full name-and-description embeddings; the same skill name can have task-specific descriptions.

Setting	Avg. golden exposed	Pass@3 (%)
All golden, no preselection	2.52	51.6
Naive-10	2.52	54.3
RAG-10	2.38	48.0
LLM-10	2.21	52.4
Hybrid-10	2.21	53.7
RAG-100	2.21	49.6

Appendix

Table 30: Fixed preselection trades reduced interference for missed golden-skill coverage. Method suffixes denote the number of added irrelevant skills.

Method	$s$	Mean (s)	Median (s)	P90 (s)
LLM	10	11.7	8.0	26
LLM	100	18.0	9.5	45
LLM	1000	35.4	19.0	73
Hybrid	10	38.9	33.0	75
Hybrid	100	43.7	37.0	89
Hybrid	1000	48.3	41.0	93

Appendix

Table 31: Observed routing-stage latency before task execution. P90 is the 90th percentile. The two methods were measured in separate experiment periods, so differences also reflect serving conditions.

Figure 6: Per-task Pass@3 for Naive exposure as the number of added irrelevant skills grows. Red means all trials fail and blue means all trials pass. Rows are tasks, sorted by the naive drop from no irrelevant skills to 1,000 irrelevant skills.

Figure 7: Per-task Pass@3 for LLM preselection. Compared with Naive exposure, many rows remain blue or light blue as the skill library grows, showing that explicit routing stabilizes task success when the full metadata list still fits in context.

Figure 8: Per-task Pass@3 for embedding RAG. RAG is feasible at the 10k scale, but the mixed colors show that semantic retrieval alone misses some task-specific skill combinations.

Figure 9: Per-task Pass@3 for hybrid preselection. The method keeps a compact candidate set while recovering more large-scale tasks than single-query RAG, especially when several capabilities must be retrieved together.

LLM Agent Skill Retrieval

Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.

See all 102 papers

SkillRouter: Skill Routing for LLM Agents at Scale
Mar 23, 2026YanZhao Zheng, ZhenTao Zhang, Chao Ma +8LLM Agent Skill Retrieval LLM Agent Routing
Alibaba Group, Hangzhou, China
Demystifying Agent Skills: Why They Work-Until They Don't
Aug 14, 2026Zhiyuan Jiang, Fangrui Huang, Hanwen Xing +6LLM Agent Skill Evaluation LLM Agents
Princeton University · Stanford University · University of Southern California +2
More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries
May 21, 2026Hongwen Song, Song WeiLLM Agent Evaluation LLM Agent Skill Learning
Databricks Inc.

Mar 23, 2026J/K move · Enter open · S save

SkillRouter: Skill Routing for LLM Agents at Scale

YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu+7

Alibaba Group, Hangzhou, China

LLM Agent Skill Retrieval LLM Agent Routing LLM Reranking

Open paper AskPDF arXiv

Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks. As skill ecosystems grow to tens of thousands of entries, exposing every skill at inference time becomes infeasible. This creates a skill-routing problem: given a user task, the system must identify relevant skills before downstream planning or execution. Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body. We examine this design choice on a SkillsBench-derived benchmark with approximately 80K candidate skills, targeting the practically important setting of large skill registries with heavy overlap. Across representative dense and reranking baselines on this setting, hiding the skill body causes a 37-44 percentage point drop in routing accuracy. Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart. Motivated by this finding, we present Skillrouter, a compact 1.2B body-aware retrieve-and-rerank pipeline. Skillrouter achieves 74.0% Hit@1 on our benchmark -- the strongest average top-1 routing performance among the baselines we evaluate -- while using 13

\times

fewer parameters and running 5.8

\times

faster than the strongest base pipeline. The ranking gains further generalize to a supplementary benchmark independently constructed from three skill sources. In a complementary end-to-end study across four coding agents, routing gains transfer to improved task success, with larger gains for more capable agents.

Abstract

LLM Agent Skill Retrieval

Explore similar work

SkillRouter: Skill Routing for LLM Agents at Scale

Demystifying Agent Skills: Why They Work-Until They Don't

More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries