Beyond Task-Only Matching: Personalized Skill Routing with Counterfactual Evaluation

Tianle Wang; Yanghe Zou; Xiang Liu; Ziyao Huang; Chenchen Fu; Weiwei Wu

cs.AIAug 28, 2026

Beyond Task-Only Matching: Personalized Skill Routing with Counterfactual Evaluation

Authors: Tianle Wang, Yanghe Zou, Xiang Liu, Ziyao Huang, Chenchen Fu, Weiwei Wu

Organizations: College of Software Engineering, Southeast University · School of Computer Science and Engineering, Southeast University · Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Computer Science, City University of Hong Kong

Abstract

The rapid expansion of reusable skill repositories makes skill routing a critical capability for large language model (LLM) agents. Existing methods treat routing as task-only semantic matching. However, when users with incompatible constraints issue an identical request, this assumption conflates task relevance with skill suitability: a task-only router can select a semantically plausible skill that is unsuitable for the requesting user. To expose this failure mode, we formulate personalized skill routing as profile-conditioned retrieval, in which relevance depends jointly on the task and the user profile. We first introduce a profile-counterfactual benchmark, in which the task is held fixed while changes in the user profile induce changes in the reference skill. We further construct paired counterfactual supervision and propose SkillFeed, a progressive retrieve-and-rerank framework that first establishes task--skill alignment and then learns profile-conditioned discrimination. By retrieving body-level evidence and reranking semantically similar but profile-conflicting candidates, SkillFeed identifies skills that satisfy both task requirements and user constraints. On SkillFeed-Bench, SkillFeed attains 75.1% top-1 retrieval accuracy, a 23.1-point improvement over the corresponding pretrained routing baseline. Adding profile conditioning yields a 35.1-point gain on queries where user profile changes the reference skill. This contrast shows that user profiles are most consequential precisely when they change skill suitability. Our website is publicly available at http://www.aiskillfeed.com .

Figures & tables

Figure 1: Motivation example of personalized skill recommendation. The same task, "planning a trip to Shanghai," is routed to a budget travel planner for a student and to a luxury itinerary planner for a business traveler. The target skill changes with user profile.

Figure 2: Overview of the dataset construction pipeline.

Figure 3: Overview of the proposed personalized skill routing framework. During offline training, user profile-aware supervision is constructed to optimize a dense retriever and a reranker. During online serving, the retriever recalls candidate skills from the repository, and the reranker produces the final personalized ranking.

Method	Input	All Hit@1	Sensitive Hit@1	Insensitive Hit@1	CF-Switch Acc.
Qwen3-Emb-0.6B + Qwen3-Reranker-0.6B	Task + Profile	0.520	0.426	0.611	0.143
SkillFeed Emb Phase 1	Task	0.483	0.340	0.623	0.052
SkillFeed Emb Phase 2	Task + Profile	0.617	0.512	0.719	0.286
SkillFeed Emb Phase 2 + SkillFeed Reranker	Task	0.644	0.444	0.838	0.026
SkillFeed Emb Phase 2 + SkillFeed Reranker	Task + Profile	0.733	0.630	0.832	0.377

Table 1: Effect of profile conditioning at the retrieval and reranking stages. “Task” denotes task-only conditioning, whereas “Task + Profile” denotes profile-aware conditioning. All metrics are computed on 329 test samples, 162 profile-sensitive samples and 167 profile-insensitive queries. CF-Switch Acc. denotes Hit@1 on 77 profile-counterfactual samples.

(1) Embedding Only (No Reranker)
Embedding Model	Reranker Model	Hit@1	Hit@3	Hit@5	Hit@10	MRR@20	nDCG@20
BM25 Only	—	0.410	0.538	0.578	0.611	0.485	0.528
E5-large-v2	—	0.322	0.474	0.508	0.565	0.410	0.459
BGE-large-en-v1.5	—	0.386	0.492	0.556	0.626	0.461	0.509
Qwen3-Emb-0.6B	—	0.416	0.559	0.644	0.717	0.514	0.575
SkillFeed Emb Phase 1 (Task-only FT)	—	0.483	0.638	0.699	0.787	0.583	0.642

Table 2: Main results on SkillFeed-Bench. We organize results by pipeline stage: (1) Embedding Only, (2) Embedding + Reranker, and (3) Full Pipeline. Bold indicates the best result, and underlining indicates the second-best result. Task/Profile Dense use SkillFeed Emb Phase 2 for respective inputs; Profile Chunk Dense chunks input.

Single-Stage Retrieval
Pipeline Configuration	Hit@1	Hit@5	Hit@10	MRR@20
BM25 Only	0.410	0.578	0.611	0.485
SkillFeed Emb Phase 2	0.617	0.796	0.833	0.697
Two-Stage: Dense + Reranker
SkillFeed Emb Phase 2 + Qwen3-Reranker-0.6B	0.559	0.818	0.882	0.668
SkillFeed Emb Phase 2 + SkillFeed Reranker	0.733	0.863	0.891	0.788

Table 3: Ablation on retrieval pipeline. The full pipeline defaults topk=50. “w/o” indicates specific components are removed.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: SkillFeed-Bench consists of tasks spanning 10 categories.

Statistic	Value
Total skill candidates	228,432
Test samples	329
– Profile-sensitive	162
– Profile counterfactual	77
– Standard (with constraints)	85
– Profile-insensitive	167

Appendix

Table 4: Key statistics of SkillFeed-Bench.

	Phase 1: Task-Only Training Data	Phase 2: Profile-Conditioned Training Data
Task	I want to plan a trip to Shanghai including attractions and a simple schedule.
User Profile	(Not included in query)	Occupation: Student
		Budget: Low budget
		Transportation: Subway preferred
Query Format	Task name: Shanghai Travel Planning Task description: Plan a trip to Shanghai including attractions and a simple itinerary.	Task name: Shanghai Travel Planning Task description: Plan a trip to Shanghai including attractions and a simple itinerary. User/context constraints: occupation: Student budget: Low budget transportation: Subway preferred
Ground Truth	city-travel-guide	budget-travel-planner

Appendix

Table 5: Training data examples for two-stage skill retrieval. Phase 1 learns general task-to-skill matching using only the task description, while Phase 2 incorporates user profile constraints to enable personalized skill selection.

By Country/Region
Group	N	Hit@1	Hit@5	MRR@20	nDCG@20
Global/Unknown	257	0.735	0.879	0.795	0.827
United States	46	0.717	0.804	0.760	0.785
China	14	0.571	0.714	0.648	0.696
Other Regions	12	0.917	0.917	0.917	0.917
By Language

Appendix

Table 6: Per-Profile-group analysis on SkillFeed Emb Phase 2 + SkillFeed Reranker with Profile. We evaluate performance across different persona groups: country/region, language, domain, and persona sensitivity. All 329 test samples are included.

Configuration	Hit@1	95% CI	MRR@20	95% CI
Full Pipeline	0.751	[0.706, 0.794]	0.803	[0.760, 0.846]
SkillFeed Emb Phase 2 + SkillFeed Reranker w Profile	0.733	[0.684, 0.778]	0.788	[0.744, 0.832]
SkillFeed Emb Phase 2 + SkillFeed Reranker w/o Profile	0.644	[0.596, 0.696]	0.712	[0.663, 0.761]
SkillFeed Emb Phase 2	0.617	[0.568, 0.669]	0.697	[0.647, 0.746]
SkillFeed Emb Phase 1	0.483	[0.429, 0.535]	0.583	[0.530, 0.636]

Appendix

Table 7: Bootstrap 95% confidence intervals for key configurations.

Comparison	Hit@1 $\Delta$	$p$ -value	Sig.
Full Pipeline vs Qwen3-Emb-0.6B + Qwen3-Reranker-0.6B	+0.231	$<0.0001$	∗∗∗
Embedding Only vs Embedding + SkillFeed Reranker	+0.116	0.001	∗∗∗
Reranker without vs with Profile	+0.089	$<0.01$	∗∗

Appendix

Table 8: Permutation test results for key comparisons. Statistical significance:

{}^{*}p<0.05

{}^{**}p<0.01

, and

{}^{***}p<0.001

LLM Agent Skill Retrieval

Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.

See all 102 papers

SkillRouter: Skill Routing for LLM Agents at Scale
Mar 23, 2026YanZhao Zheng, ZhenTao Zhang, Chao Ma +8LLM Agent Skill Retrieval LLM Agent Routing
Alibaba Group, Hangzhou, China
Beyond Top- $k$ Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents
Sep 5, 2026Wang Wei, Tiankai Yang, Samyadeep Basu +7LLM Agent Skill Retrieval LLM Agent Routing
Virginia Tech · University of Southern California · Adobe Research +3
Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale
Oct 13, 2026Jinlin Liu, Wei Chen, Xiaojin ZhangLLM Agent Skill Retrieval LLM Agent Evaluation
The University of Hong Kong · Huazhong University of Science and Technology

Mar 23, 2026J/K move · Enter open · S save

SkillRouter: Skill Routing for LLM Agents at Scale

YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu+7

Alibaba Group, Hangzhou, China

LLM Agent Skill Retrieval LLM Agent Routing LLM Reranking

Open paper AskPDF arXiv

Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks. As skill ecosystems grow to tens of thousands of entries, exposing every skill at inference time becomes infeasible. This creates a skill-routing problem: given a user task, the system must identify relevant skills before downstream planning or execution. Existing agent stacks often rely on progressive disclosure, exposing only skill names and descriptions while hiding the full implementation body. We examine this design choice on a SkillsBench-derived benchmark with approximately 80K candidate skills, targeting the practically important setting of large skill registries with heavy overlap. Across representative dense and reranking baselines on this setting, hiding the skill body causes a 37-44 percentage point drop in routing accuracy. Stronger controls show that the missing signal is body-resident rather than a simple length artifact: body-distilled descriptions recover part of the gap, but remain 7-21 points below direct all-field routing, while a metadata-only encoder trained with the same data remains 14.0 points below its all-field counterpart. Motivated by this finding, we present Skillrouter, a compact 1.2B body-aware retrieve-and-rerank pipeline. Skillrouter achieves 74.0% Hit@1 on our benchmark -- the strongest average top-1 routing performance among the baselines we evaluate -- while using 13

\times

fewer parameters and running 5.8

\times

faster than the strongest base pipeline. The ranking gains further generalize to a supplementary benchmark independently constructed from three skill sources. In a complementary end-to-end study across four coding agents, routing gains transfer to improved task success, with larger gains for more capable agents.

Abstract

LLM Agent Skill Retrieval

Explore similar work

SkillRouter: Skill Routing for LLM Agents at Scale

Beyond Top-kkk Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents

Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale

Beyond Top- $k$ Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents