Just-in-Time Skill Retrieval via Long-Horizon State Tracking
Organizations: Xiamen University · Jilin University
Abstract
Large language model (LLM) agents increasingly rely on external skill libraries to support complex task execution. However, determining which skills are needed remains challenging in long-horizon tasks. Existing methods typically select a fixed skill set before execution, implicitly assuming that a task's procedural requirements can be determined from its initial description. This assumption is restrictive for long-horizon interaction, where new requirements emerge from intermediate observations, execution outcomes, and newly revealed constraints. This motivates just-in-time skill retrieval, which dynamically selects skills as execution unfolds. Just-in-time skill retrieval is promising but challenging: the utility of a skill depends not only on the current context, but also on the evolving task state shaped by prior steps. To this end, we propose JitSkill, a state-grounded framework that addresses this challenge through continuous task-state tracking. By connecting emerging requirements with accumulated task progress, JitSkill provides procedural support throughout execution without treating each retrieval decision independently. Experiments across three agent benchmarks demonstrate its effectiveness in improving long-horizon task completion. Our code is available at https://github.com/XMUDeepLIT/JitSkill.git/.
Figures & tables
| Method | SkillsBench | ALFWorld | -Bench | ||||||||||||
| Core | Extended | Extreme | SR | Pick | Look | Clean | Heat | Cool | Pick2 | SR | Airline | Retail | Telecom | SR | |
| Direct Execution (No Skills) | |||||||||||||||
| Qwen3-8B | 0.00 | 0.00 | 3.57 | 1.15 | 71.43 | 30.77 | 29.63 | 0.00 | 0.00 | 8.33 | 27.86 | 15.00 | 2.50 | 1.67 | 6.39 |
| Qwen3.8-27B | 0.00 | 8.18 | 3.57 | 6.13 | 88.57 | 84.62 | 85.19 | 6.25 | 52.00 | 75.00 | 69.29 | 38.33 | 56.67 | 20.83 | 38.61 |
| Pre-Execution Skill Retrieval | |||||||||||||||
| SkillTrace | 11.11 | 13.84 | 8.33 | 11.88 | 98.10 | 76.92 | 74.07 | 58.33 | 70.67 | 93.06 | 81.19 | 48.33 | 61.67 | 33.33 | 47.78 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Method description | Differences from ours |
| Pre-execution fixed skill retrieval | ||
| SkillFlow ( Li et al., 2025 ) ; SkillRouter ( Zheng et al., 2026 ) | Retrieve skills from initial task requirements; SkillRouter additionally uses procedural information in skill bodies. | Determine the retrieved skill set before execution, without updating it as task progress changes. |
| Graph-of-Skills (GoS) ( Liu et al., 2026 ) ; GoSkills ( Zeng et al., 2026 ) ; CaSKG ( Li et al., 2026f ) | Expand the retrieved skill set through dependencies, role-labeled groups, or calibrated skill relations. | Improve the initial support set, but do not condition retrieval on evolving task state. |
| SkillAligner ( Li et al., 2026c ) | Adapt retrieved skill fragments to task requirements and interfaces, resolving conflicts before execution. | Improves pre-execution compatibility; later feedback does not trigger renewed support selection. |
| SkillTrace ( Yao et al., 2026 ) | Decompose a task into subqueries, match them to skills, and add dependency-based supporting skills. | A static skill-context baseline: it fixes the retrieved skill set before execution rather than revising it from interaction feedback. |
| Execution-time local-signal skill retrieval | ||
| Setting | Value | Setting | Value |
| Encoder base model | Qwen3-8B (frozen) | Tokens per update (new block / encoder window) | 1,536 / 4,096 |
| Executor | Qwen3.8-27B (frozen) | Skill and record embedder | all-MiniLM-L6-v2 |
| LoRA rank / / dropout | 32 / 64 / 0.05 | Epochs | 3 |
| LoRA target modules | projections | Learning rate (encoder / navigator) | / |
| Latent slots | 16 | Held-out validation scenarios | 5 |
| Method | Hit@3 | TGC D1 | TGC D2 | TGC D3 | Avg. query tokens |
| Initial Retrieval | 0.115 | 85.4 | 69.4 | 57.7 | 81 |
| Online Retrieval | 0.385 | 88.3 | 75.0 | 55.6 | 328 |
| Online Retrieval + Full History | 0.346 | 87.7 | 65.3 | 56.1 | 2,924 |
| Model | SkillsBench | ALFWorld | -Bench | ||||||||||||
| Core | Ext. | Extr. | SR | Pick | Look | Clean | Heat | Cool | Pick2 | SR | Airline | Retail | Telecom | Avg. | |
| Qwen3-8B | 0.00 | 0.00 | 3.57 | 1.15 | 71.43 | 30.77 | 29.63 | 0.00 | 0.00 | 8.33 | 27.86 | 15.00 | 2.50 | 1.67 | 6.39 |
| Llama-3.1-8B-Instruct | 0.00 | 0.00 | 0.00 | 0.00 | 11.43 | 15.38 | 0.00 | 6.25 | 0.00 | 8.33 | 6.43 | 11.67 | 3.33 | 7.50 | 7.50 |
| Qwen3.8-27B | 0.00 | 8.18 | 3.57 | 6.13 | 88.57 | 84.62 | 85.19 | 6.25 | 52.00 | 75.00 | 69.29 | 38.33 | 56.67 | 20.83 | 38.61 |
| Gemma-4-31B-IT | 16.67 | 6.29 | 3.57 | 6.13 | 95.24 | 66.67 | 65.43 | 0.00 | 50.67 | 77.78 | 65.00 | 41.62 | 64.17 | 22.50 | 42.76 |
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | SR |
| StructAgent | 95.24 | 53.85 | 80.25 | 81.25 | 52.00 | 87.50 | 77.86 |
| FlowState | 98.10 | 89.74 | 83.95 | 72.92 | 69.33 | 95.83 | 86.19 |
| SKILL.state | 64.76 | 41.03 | 62.96 | 52.08 | 42.67 | 38.89 | 52.38 |
| JitSkill | 98.10 | 94.87 | 90.12 | 83.33 | 89.33 | 94.44 | 92.38 |
| Method | TGC D1 | TGC D2 | TGC D3 |
| No skills | 83.0 | 70.1 | 61.9 |
| Pre-execution retrieval | 85.4 | 69.4 | 57.7 |
| Step-wise retrieval | 88.3 | 75.0 | 55.6 |
| without instruction | 87.7 | 69.4 | 48.7 |
| + raw textual history | 86.5 | 68.8 | 57.7 |
| + full history | 87.7 | 65.3 | 56.1 |
| Method | Steps / game | SR (%) |
| No skills | 17.57 | 70.24 |
| SGDR | 15.03 | 87.14 |
| SkillTrace | 15.53 | 81.19 |
| SKILL.state | 21.10 | 52.38 |
| JitSkill | 13.35 | 92.38 |
| Method | Steps / ep. | Skill tokens / step | Skill tokens / ep. |
| No skills | 17.8 | 0 | 0 |
| Whole library | 4.4 | 27,828 | 122,465 |
| Pre-execution retrieval | 15.5 | 1,072 | 16,605 |
| Step-wise retrieval | 14.6 | 1,107 | 16,193 |
| + raw textual history | 14.7 | 1,212 | 17,765 |
| JitSkill | 15.8 | 1,209 | 19,051 |
| Variant | Affected quantity | Default | Variant |
| w/o | alignment JSD | 0.085 | 0.500 |
| w/o | episode retrieval top-1 (chance ) | 0.337 | 0.072 |
| reconstruction NLL | 0.537 | 0.588 | |
| Full objective (default weights) | 0.523 | 0.518–0.620 | |
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | SR |
| No skills (Qwen3.8-27B) | 91.7 | 85.2 | 81.7 | 2.9 | 54.0 | 88.2 | 66.92 |
| SGDR | 88.9 | 85.2 | 87.1 | 71.0 | 90.5 | 86.3 | 84.83 |
| WebXSkill | 95.8 | 96.3 | 90.3 | 73.9 | 90.5 | 82.4 | 88.31 |
| SkillTrace | 97.2 | 90.7 | 82.8 | 63.8 | 84.1 | 94.1 | 84.83 |
| SoftSkill | 91.7 | 88.9 | 90.3 | 72.5 | 74.6 | 70.6 | 82.34 |
| CodeSkill | 95.8 | 96.3 | 88.2 | 85.5 | 74.6 | 90.2 | 88.31 |
| Method | Action calls / solved task | Completion after feedback (%) |
| No skills | 31.6 | 71.6 |
| Pre-execution retrieval | 32.7 | 68.8 |
| Step-wise retrieval | 32.5 | 64.2 |
| + raw textual history | 30.3 | 62.7 |
| JitSkill | 27.4 | 72.8 |
| Benchmark | Test episodes | Breakdown | Step limit | #Skills | Skill length (mean / median) | Skill length (max) |
| ALFWorld | / | seen / unseen environments | / | |||
| SkillsBench | Core , Extended , Extreme | task-specific | / | |||
| -Bench | Airline , Retail , Telecom | ( / / ) | / |
| Benchmark | Run 1 | Run 2 | Run 3 | Mean std |
| ALFWorld (seen) | 92.14 | 92.86 | 92.14 | |
| ALFWorld (unseen) | 90.30 | 88.81 | 85.82 | |
| SkillsBench | 14.94 | 14.94 | 17.24 | |
| -Bench (avg.) | 46.67 | 51.67 | 49.17 |
| Benchmark | Method | Skills used in library | Switches / episode | Distinct skills / episode |
| ALFWorld (seen) | Pre-execution retrieval | 18/37 | 0.00 | 1.73 |
| Step-wise retrieval | 34/37 | 3.33 | 4.34 | |
| JitSkill | 36/37 | 7.21 | 7.31 | |
| SkillsBench | JitSkill | 178/195 | 11.27 | – |
| -Bench | JitSkill | 33/33 | 6.22 | 8.42 |
| Stages | Type | No skills | JitSkill | |
| Single | Pick | 88.57 | 98.10 | |
| Look | 84.62 | 94.87 | ||
| Combined | 87.50 | 97.22 | ||
| Multiple | Clean | 85.19 | 90.12 | |
| Heat | 6.25 | 83.33 | ||
| Cool | 52.00 | 89.33 |
| Step | Injected skills | Executor action | Observation |
| 1 | heat-object-with-appliance | go to countertop 1 | sees apple 1 and apple 2 |
| 2 | object-locator | take apple 1 from countertop 1 | picks up apple 1 |
| 3 | object-locator | go to microwave 1 | the microwave is closed |
| 4 | object-heater, appliance-navigator | open microwave 1 | the microwave is open |
| 5 | appliance-navigator, object-heater | heat apple 1 with microwave 1 | “You heat the apple 1” |
| 6 | object-heater, appliance-navigator | go to fridge 1 | the fridge is closed |
LLM Agent Skill Retrieval
Large language model agent skill retrieval finds and selects reusable procedures to help agents complete tasks.
See all 109 papers