TalentRL: Strategy-Guided Reinforcement Learning for Hierarchical Skill Routing
Organizations: Microsoft AI
Abstract
Natural-language skills enable large language model (LLM) agents to reuse procedures across tasks, yet most systems retrieve individual skills from a flat SkillBank without explicitly modeling the high-level strategy that governs their selection and use. We introduce TalentRL, a strategy-guided reinforcement learning framework for hierarchical routing over procedural memory. Coarse-grained talents represent recurring strategies for skill use, while fine-grained skills provide executable procedures. For each task, the policy model selects a talent that guides execution and softly prioritizes its associated skills during similarity-based retrieval. However, since task outcomes depend jointly on talent selection, retrieved skill context, and downstream execution, terminal rewards alone cannot isolate the contribution of the routing decision. TalentRL therefore trains talent routing with reward contrasts between rollouts of the same task under matched interventions on the talent-conditioned context, and optimizes downstream execution with group-relative advantages. Across seven open-domain and multi-hop question-answering benchmarks, talent-aware supervised fine-tuning improves average success rate by 7.53%, 6.71%, and 9.21% over the corresponding zero-shot baselines for Qwen2.5-7B, Qwen3-8B, and Llama-3.1-8B, respectively. Reinforcement learning further improves these SFT checkpoints by 7.58% to 9.32%, and TalentRL achieves the highest average success rate among all compared methods on all three backbones. On the agentic benchmarks ALFWorld and WebShop, TalentRL achieves the highest ALFWorld success rate (99.3%) and the highest WebShop score (90.7) among all compared methods.
Figures & tables
| Initial policy model ; environment ; task sets ; |
| talent catalog ; SkillBank ; skill budget |
| Model | Method | Single-hop | Multi-hop | Overall | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| NQ | TriviaQA | PopQA | Avg. | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Avg. | |||
| Qwen2.5-7B | Zero-shot | 25.32 | 50.19 | 25.07 | 33.53 | 27.63 | 25.13 | 9.76 | 27.20 | 22.43 | 27.19 |
| SkillRL-SFT | 34.07 | 52.13 | 32.87 | 39.69 | 32.91 | 23.71 | 12.62 | 32.80 | 25.51 | 31.59 | |
| TalentRL-SFT | 36.29 | 53.67 | 34.26 | 41.41 | 35.30 | 27.32 | 14.60 | 41.60 | 29.70 | 34.72 | |
| Qwen3-8B | Zero-shot | 26.12 | 54.72 | 28.51 | 36.45 | 26.17 | 29.73 | 6.70 | 35.20 | 24.45 | 29.60 |
| SkillRL-SFT | 37.23 | 55.03 | 34.05 | 42.10 | 35.84 | 27.70 | 15.02 | 40.00 | 29.64 | 34.98 | |
| Model | Method | Single-hop | Multi-hop | Overall | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| NQ | TriviaQA | PopQA | Avg. | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Avg. | |||
| Qwen2.5-7B | CoSkill | 44.24 | 61.39 | 38.09 | 47.91 | 37.87 | 39.09 | 14.81 | 39.20 | 32.74 | 39.24 |
| SAGE | 46.43 | 63.60 | 39.30 | 49.78 | 41.67 | 41.57 | 16.92 | 42.40 | 35.64 | 41.70 | |
| Skill1 | 46.15 | 62.66 | 38.70 | 49.17 | 39.41 | 39.44 | 15.52 | 38.40 | 33.19 | 40.04 | |
| ReSkill | 46.20 | 62.96 | 39.97 | 49.71 | 43.08 | 37.64 | 17.00 | 42.40 | 35.03 | 41.32 | |
| SkillRL | 48.01 | 63.24 | 40.70 | 50.65 | 43.04 | 36.90 | 17.38 | 44.00 | 35.33 | 41.90 | |
| Method | ALFWorld Success Rate (%) | WebShop | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Pick | Look | Clean | Heat | Cool | Pick2 | Overall | Score | Succ. | |
| MemRL | 62.8 | 38.5 | 22.2 | 12.5 | 8.0 | 0.0 | 21.4 | 29.5 | 9.2 |
| EvolveR | 64.9 | 33.3 | 46.4 | 13.3 | 33.3 | 33.3 | 43.8 | 42.5 | 17.6 |
| Mem0+GRPO | 78.1 | 54.8 | 56.1 | 31.0 | 65.0 | 26.9 | 54.7 | 58.1 | 37.5 |
| SimpleMem+GRPO | 89.5 | 36.3 | 60.0 | 50.0 | 64.9 | 26.3 | 62.5 | 67.8 | 46.9 |
| SkillRL | 97.9 | 71.4 | 90.0 | 90.0 | 95.5 | 87.5 | 89.9 | 85.2 | 72.7 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Hyperparameter | Value |
| SFT | Epochs | 2 |
| SFT | Learning rate | |
| SFT | Effective batch size | 16 |
| SFT | Maximum sequence length | 8,192 |
| RL | Policy updates | 200 |
| RL | Learning rate |
| Talent | Generation guidance | Associated skills |
|---|---|---|
| Task decomposition | Identify the dependency chain before searching, solve one subproblem at a time, and carry forward only verified intermediate facts. | gen_001 , mul_001 , mul_002 , gen_007 , gen_010 |
| Bridge-entity resolution | Resolve the bridge entity first, preserve relation direction, then issue a targeted follow-up query for the requested attribute. | gen_007 , mul_001 , mul_002 , gen_004 , gen_010 |
| Entity disambiguation | Use type, date, location, occupation, or related-work qualifiers and reject evidence that belongs to another referent. | gen_006 , mul_004 , ent_002 , ent_003 , gen_005 |
| Constraint application | Translate every clause into a checkable constraint, narrow candidates by intersection, and verify all constraints before answering. | com_004 , mul_001 , mul_002 , gen_005 , gen_010 |
| Evidence comparison | Collect the same attribute for every entity, normalize dates or units, and perform the comparison only after all values are verified. | gen_008 , mul_003 , com_001 , com_002 , com_003 , com_005 , gen_004 |
| Long-tail query expansion | When direct retrieval is weak, try aliases, related works, distinctive phrases, and progressively refined contextual queries. | dir_002 , dir_003 , ent_004 , gen_003 , mul_005 , gen_006 |
| Skill ID | Principle | When to apply |
|---|---|---|
| gen_001 | Break the question into minimal sub-questions (entities, attributes, relations) and handle each with its own query before synthesizing the final answer. | Any complex or multi-part question, especially multi-hop or comparative tasks. |
| gen_003 | If top results don’t yield direct evidence, rephrase by adding qualifiers (dates, roles, context) or alternate names instead of repeating the same query. | After scanning first result set and finding no definitive evidence. |
| gen_004 | Only state an answer once the supporting evidence (reliable source snippet) has been located and read; refrain from guessing or summarizing from memory. | Before committing to a final answer, especially for contentious or less-known facts. |
| gen_005 | Validate critical facts (dates, numbers, names) against at least two independent sources to avoid outdated or incorrect information. | When the fact may have changed over time or when initial sources disagree. |
| gen_006 | Detect ambiguous entity mentions and disambiguate by adding clarifiers (e.g., occupation, year, nationality) in the query or by exploring disambiguation pages. | When an entity name maps to multiple possible referents or results appear mixed. |
| gen_007 | For multi-hop tasks, first retrieve the intermediate entity (e.g., a film’s director) then run a second targeted query for that entity’s attribute (e.g., birthplace) before synthesizing. | Whenever the answer depends on an attribute of a related entity rather than the entity in the question. |
LLM Agent Routing
LLM agent routing selects models, tools, skills, or collaboration workflows for each task or step to balance capability, cost, latency, and reliability.
See all 105 papers