cs.LGSep 2, 2026
SaveToollery: Scaling LLM Agents to Thousands of Skills and Tools
Organizations: Laboratories, Huawei
Abstract
As LLM agents are exposed to hundreds to tens of thousands of skills, tools, and API functions, full-library prompting becomes costly, slow, and less reliable: each added candidate increases prompt tokens and latency, while longer candidate lists introduce more distractors for LLM selection. We present Toollery, a training-free candidate-compression framework for scalable LLM skill/tool selection. Following established document-side query expansion, Toollery generates user-intent queries from each skill/tool specification and builds a retrieval index that maps real user requests to compact candidate sets before final LLM decision-making. By treating high-level skills and atomic tools as selectable capabilities, Toollery can be applied to both skill libraries and tool registries. We evaluate Toollery on the roughly 79K-capability SkillRouter benchmark, BFCL-V4 with over 440 atomic tools, and 3,396 proprietary smart-cockpit requests over 220 tools. Across these settings, Toollery keeps online selection bounded to a compact top-K candidate set and improves recall over ordinary specification retrieval. At a fixed top-10 budget, Toollery improves end-to-end selection on the cockpit dataset, and maintains comparable AST Accuracy on BFCL-V4. These results support Toollery as a practical candidate-compression framework for large and evolving agent capability libraries, while showing that quality and cost gains depend on workload coverage and provider caching.
Figures & tables
Figure 1: Overview of Toollery. Toollery constructs a manual by generating user-intent queries from each capability specification. At inference time, it retrieves a compact candidate set using the manual and passes only the retrieved candidate specifications to the final LLM for decision making.
| Retrieval Setting | Retriever | Online ms/query | Hit@1 | Recall@10 | Recall@50 | MRR@10 | nDCG@10 | FullCoverage@10 |
| Raw specification | BM25 | 309.4 | 0.553 | 0.571 | 0.726 | 0.625 | 0.512 | 0.413 |
| Raw specification | RAGAnything | 38210.6 | 0.600 | 0.568 | 0.712 | 0.663 | 0.500 | 0.393 |
| Raw specification | SkillRouter † | 495.8 | 0.740 | 0.704 | 0.814 | 0.791 | 0.670 | 0.520 |
| Toollery (manual only) | BM25 | 218.9 | 0.840 | 0.880 | 0.951 | 0.881 | 0.846 | 0.760 |
| Toollery (manual only) | RAGAnything | 21505.9 | 0.900 | 0.922 | 0.953 | 0.925 | 0.882 | 0.827 |
| Toollery (manual + specification) | BM25 | 623.0 | 0.920 | 0.942 | 0.968 | 0.942 | 0.880 | 0.853 |
Table 1: SkillRouter benchmark retrieval over roughly 79K skills, averaged over the easy and hard splits. Raw specification retrieves from original skill specifications only. Full-library prompting at this scale would require roughly 43.1M input tokens per request; Toollery top-10 prompting requires roughly 27.7K tokens, over three orders of magnitude smaller. † denotes the SkillRouter reference using SR-Emb-0.6B SR-Rank-0.6B.
| Retrieval Setting | Retriever | Online ms/query | Hit@1 | Recall@10 | Recall@50 | MRR@10 | nDCG@10 | FullCoverage@10 |
| Raw specification | BM25 | 3.6 | 0.643 | 0.931 | 0.976 | 0.744 | 0.790 | 0.931 |
| Raw specification | RAGAnything | 152.5 | 0.737 | 0.976 | 0.995 | 0.831 | 0.867 | 0.976 |
| Toollery (manual only) | BM25 | 0.3 | 0.468 | 0.822 | 0.934 | 0.584 | 0.642 | 0.822 |
| Toollery (manual only) | RAGAnything | 81.8 | 0.543 | 0.855 | 0.929 | 0.653 | 0.702 | 0.855 |
| Toollery (manual + specification) | BM25 | 5.0 | 0.731 | 0.948 | 0.986 | 0.811 | 0.845 | 0.948 |
| Toollery (manual + specification) | RAGAnything | 139.7 | 0.776 | 0.980 | 0.997 | 0.856 | 0.887 | 0.980 |
Table 2: BFCL-V4 live multiple retrieval over the complete 457-tool pool.
| Retrieval Setting | Retriever | Online ms/query | Hit@1 | Recall@10 | Recall@50 | MRR@10 | nDCG@10 | FullCoverage@10 |
| Raw specification | BM25 | 14.4 | 0.780 | 0.970 | 0.995 | 0.854 | 0.883 | 0.970 |
| Raw specification | RAGAnything | 524.6 | 0.825 | 1.000 | 1.000 | 0.895 | 0.922 | 1.000 |
| Toollery (manual only) | BM25 | 0.2 | 0.555 | 0.885 | 0.990 | 0.668 | 0.721 | 0.885 |
| Toollery (manual only) | RAGAnything | 77.9 | 0.605 | 0.925 | 0.985 | 0.722 | 0.772 | 0.925 |
| Toollery (manual + specification) | BM25 | 25.9 | 0.815 | 1.000 | 1.000 | 0.887 | 0.915 | 1.000 |
| Toollery (manual + specification) | RAGAnything | 657.3 | 0.845 | 1.000 | 1.000 | 0.904 | 0.928 | 1.000 |
Table 3: BFCL-V4 non-live multiple retrieval over the complete 443-tool pool.
| Dataset | Selection Setting | # Candidates | Input Tokens | Latency | Accuracy |
| BFCL live | Full-library prompt | 457 | 84.1K | 11.8s | .770 |
| BFCL live | Raw specification top-10 | 10 | 2.7K | 2.8s | .834 |
| BFCL live | Toollery top-10 | 10 | 2.1K | 1.6s | .837 |
| BFCL non-live | Full-library prompt | 443 | 57.0K | 8.4s | .703 |
| BFCL non-live | Raw specification top-10 | 10 | 1.9K | 2.2s | .825 |
| BFCL non-live | Toollery top-10 | 10 | 1.3K | 1.5s | .830 |
Table 4: End-to-end selection under full-library prompting and fixed top-10 retrieval. The BFCL full-library rows retain the DeepSeek V3.2 results as scaling references, whereas all fixed top-10 rows use DeepSeek-V4-Pro. BFCL reports official AST Accuracy and Cockpit reports exact-set accuracy. Input tokens and latency are per-call means.
| Data | Raw specification | Re-Invoke † | Toollery |
| ( ) | ( ) | ( ) | |
| Cockpit | .545 / .224 | .593 / .240 | .684 / .381 |
| BFCL non-live | 1.000 / 1.000 | 1.000 / 1.000 | 1.000 / 1.000 |
| BFCL live | .976 / .976 | .976 / .976 | .980 / .980 |
Table 5: Retrieval coverage under the same top-10 candidate budget. Each cell reports Recall@10 / FullCoverage@10, and is the number of generated queries per capability. Cockpit uses BM25, whereas BFCL uses Qwen3-Embedding-0.6B dense retrieval. † Expansion-only Re-Invoke reimplementation without its online intent model.
| Retrieval Setting | Retriever | Online ms/query | Hit@1 | Recall@10 | Recall@20 | Recall@50 | MRR@10 | nDCG@10 | FullCoverage@10 |
| Raw specification | BM25 | 6.4 | 0.430 | 0.545 | 0.696 | 0.867 | 0.576 | 0.446 | 0.224 |
| Raw specification | RAGAnything | 144.5 | 0.408 | 0.582 | 0.718 | 0.871 | 0.588 | 0.464 | 0.229 |
| Toollery (manual only) | BM25 | 0.7 | 0.481 | 0.670 | 0.784 | 0.918 | 0.648 | 0.544 | 0.358 |
| Toollery (manual only) | RAGAnything | 221.6 | 0.453 | 0.666 | 0.781 | 0.902 | 0.633 | 0.538 | 0.357 |
| Toollery (manual + specification) | BM25 | 69.7 | 0.484 | 0.684 | 0.803 | 0.926 | 0.654 | 0.553 | 0.381 |
| Toollery (manual + specification) | RAGAnything | 253.4 | 0.403 | 0.630 | 0.758 | 0.893 | 0.591 | 0.500 | 0.316 |
Table 6: Proprietary smart cockpit retrieval over the complete 220-tool pool. Results are averaged over 3,396 real company-internal requests with two or three gold tools. The proprietary dataset is used only for deployment-facing evaluation and is not released.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Value used in reported runs |
| SkillRouter pools | 78,361 easy capabilities; 79,141 hard capabilities; 75 requests per pool |
| BFCL-live end-to-end pool | 457 tools; 1,053 live multiple requests; final column uses the complete pool |
| BFCL-non-live end-to-end pool | 443 tools; 200 multiple requests; final column uses the complete pool |
| Manual construction | DeepSeek-V4-Pro through an OpenAI-compatible API; 3 user-intent queries per capability; seed 31; 4 workers; batch size 4 |
| Source-capability recovery verifier | asks DeepSeek-V4-Pro to choose the matching capability id from the source capability plus distractors, or NONE if no candidate fits |
| Distractor construction | verifier pool uses the source capability plus 8 randomly sampled non-gold distractors; scaled BFCL candidate pools sample the gold tool plus non-gold tools without replacement |
Table 7: Reproducibility details for the main retrieval and efficiency results. Secrets and local API endpoints are omitted; paths in the released run configurations identify the exact output artifacts.
Figure 2: Toollery pipeline walkthrough. Toollery converts capability specifications into user-intent queries during offline manual construction. During inference, the user request retrieves matching manual entries and corresponding capability specifications, which are aggregated into a compact candidate set for final LLM decision making.
| Setting | Skills | Cand. | Ret. | Tok. | Cost |
| SkillRouter easy | 78,361 | 235K | 228K | 361.8M | $168.70 |
| SkillRouter hard | 79,141 | 237K | 230K | 365.4M | $170.38 |
Table 8: Offline construction cost for generated-query construction and verifier filtering. The measured rows use the SkillRouter easy and hard pools with three generated queries per capability. Cost uses official DeepSeek-V4-Pro pricing: 0.87/M output tokens; cache-hit discounts are not applied.
| Dataset | Indexed representation | BM25 | Dense (Qwen3) | Hybrid RRF | |||
| R@10 | FC@10 | R@10 | FC@10 | R@10 | FC@10 | ||
| BFCL non-live | Specification | 0.970 | 0.970 | 1.000 | 1.000 | 0.990 | 0.990 |
| Generated queries | 0.885 | 0.885 | 0.925 | 0.925 | 0.975 | 0.975 | |
| Specification + generated queries | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | |
| BFCL live | Specification | 0.931 | 0.931 | 0.976 | 0.976 | 0.976 | 0.976 |
| Generated queries | 0.822 | 0.822 | 0.855 | 0.855 | 0.916 | 0.916 | |
Table 9: Retrieval ablation across indexed representations and retrievers. Results report Recall@10 (R@10) and FullCoverage@10 (FC@10); bold marks the best result in each dataset–retriever–metric group, including ties. SkillRouter averages the easy and hard pools, and Hybrid RRF fuses BM25 and dense rankings.
| Dataset | R@10 | FC@10 | Query-text tokens (est.) | |
| BFCL non-live | 1 | 0.980 | 0.980 | 10,835 |
| 3 | 0.995 | 0.995 | 33,582 | |
| 5 | 0.995 | 0.995 | 56,012 | |
| 10 | 0.995 | 0.995 | 111,042 | |
| BFCL live | 1 | 0.932 | 0.932 | 11,829 |
| 3 | 0.942 | 0.942 | 36,078 |
Table 10: Generated-query budget sweep at . Bold marks the default configuration. Rows use nested prefixes from a separately generated batch; this arm is therefore distinct from the main-run artifact.
| Filter | BFCL-NL | BFCL-L | Cockpit |
| No filter | 0.995 / 0.995 | 0.941 / 0.941 | 0.623 / 0.310 |
| Random neg. | 0.995 / 0.995 | 0.942 / 0.942 | 0.680 / 0.365 |
| BM25 hard neg. | 0.995 / 0.995 | 0.941 / 0.941 | 0.602 / 0.288 |
Table 11: Recovery-filter ablation at with eight verifier negatives. Cells report Recall@10 / FullCoverage@10; bold marks the best result per dataset, including ties. BFCL-NL and BFCL-L denote non-live and live. Hard-negative verification is diagnostic only.
| Recall / FC | Candidate tokens | ms/query | |
| 1 | 0.700 | 254 | 0.15 |
| 3 | 0.864 | 664 | 0.15 |
| 5 | 0.911 | 1,063 | 0.15 |
| 10 | 0.942 | 2,039 | 0.15 |
| 20 | 0.962 | 3,965 | 0.15 |
| 50 | 0.979 | 9,734 | 0.17 |
Table 12: Candidate-budget sweep on BFCL live using combined BM25 retrieval. Bold marks the default configuration. Each request has one gold function, so Recall equals FullCoverage.
| Cockpit route ( ) | R@10 | FC@10 |
| Held-out hot only | 0.080 | 0.004 |
| Raw BM25 | 0.545 | 0.224 |
| Toollery | 0.684 | 0.381 |
| 3 hot + Toollery | 0.628 | 0.302 |
| 5 hot + Toollery | 0.563 | 0.230 |
Table 13: Five-fold held-out evaluation of static popularity routing. Hot rankings are computed on four folds and evaluated on the unseen fold; bold marks the best route.
| Split | Requests | Capabilities | Exact | Max 5-gram Jaccard | Unit |
| SkillRouter easy | 75 | 78,361 | 0 | 0.063 | token |
| SkillRouter hard | 75 | 79,141 | 0 | 0.060 | token |
| BFCL non-live | 200 | 443 | 0 | 0.455 | token |
| BFCL live | 1,053 | 457 | 0 | 0.229 | token |
| Cockpit | 3,396 | 220 | 0 | 0.417 | character |
Table 14: Evaluation-to-generated-query overlap audit. Exact denotes normalized exact matches; English splits use token 5-grams and cockpit uses character 5-grams.
| Cache regime | Method | Calls | Candidates | AST | Latency | Billed input cost | Input tokens | Hit rate |
| Natural replay | Full library | 600 | 443–457 | 0.787 | 3.76 s | $0.000749 | 87,074 | 0.989 |
| Toollery | 600 | 10 | 0.840 | 2.33 s | $0.000819 | 2,196 | 0.144 | |
| Cache-suppressed | Full library | 300 | 443–457 | 0.797 | 9.03 s | $0.037879 | 87,078 | 0.000 |
| Toollery | 300 | 10 | 0.817 | 2.15 s | $0.000951 | 2,187 | 0.000 | |
| Forced-warm exact | Full library | 300 | 443–457 | 0.807 | 3.43 s | $0.000328 | 87,077 | 1.000 |
| Toollery | 300 | 10 | 0.853 | 2.14 s | $0.000036 | 2,188 | 0.970 |
Table 15: Controlled cache-aware BFCL replay. Latency, billed input cost, input tokens, and hit rate are per-request means. Forced-warm values exclude the paid priming call and represent a boundary condition rather than a deployable cost forecast.
| Model | 10 | 20 | 30 | 40 | 50 | 100 | 457 | |
| Qwen3-4B (Prompt) | 85.50 | 86.00 | 88.00 | 82.50 | 78.00 | 69.50 | 24.88 | |
| Qwen3-4B (Prompt) + Toollery | 87.00 | 84.50 | 87.00 | 86.50 | 82.50 | 81.00 | 83.75 | |
| Qwen3-8B (Prompt) | 97.00 | 93.50 | 92.00 | 84.00 | 83.00 | 84.50 | 38.50 | |
| Qwen3-8B (Prompt) + Toollery | 95.50 | 98.50 | 98.50 | 92.50 | 94.50 | 86.50 | 79.38 | |
| DeepSeek V3.2 (Prompt) | 94.50 | 80.00 | 84.00 | 89.50 | 79.50 | 63.00 | 77.00 | |
| DeepSeek V3.2 (Prompt) + Toollery | 98.00 | 78.50 | 94.50 | 99.50 | 89.50 | 89.00 | 83.13 |
Table 16: BFCL-V4 live multiple results
| Model | 10 | 20 | 30 | 40 | 50 | 100 | 443 |
| Qwen3-4B (Prompt) | 95.50 | 99.50 | 97.50 | 96.50 | 95.50 | 90.00 | 24.25 |
| Qwen3-4B (Prompt) + Toollery | 100.00 | 97.50 | 98.00 | 95.00 | 98.00 | 91.50 | 83.63 |
| Qwen3-8B (Prompt) | 97.50 | 98.50 | 99.50 | 97.50 | 97.00 | 89.50 | 37.63 |
| Qwen3-8B (Prompt) + Toollery | 99.50 | 100.00 | 97.50 | 96.00 | 98.00 | 91.00 | 80.63 |
| DeepSeek V3.2 (Prompt) | 86.00 | 99.50 | 97.00 | 94.50 | 96.50 | 85.50 | 70.25 |
| DeepSeek V3.2 (Prompt) + Toollery | 97.50 | 97.50 | 95.00 | 98.00 | 98.50 | 89.50 | 81.38 |
Table 17: BFCL-V4 non-live multiple results
| Dataset | Query / task | Gold capability | Top-5 retrieval |
| SkillRouter | You need to fix build errors in a Java codebase. The repository is located in … | gt/maven-build-lifecycle gt/maven-dependency-management gt/maven-plugin-configuration | gt/analyze-ci gt/uv-package-manager development/ci-debug other/creating-commit other/github-pr-fix |
| SkillRouter | Build an itinerary for the user according to the following requirements: We require a 7-day travel itinerary for two leaving from Minneapolis … | gt/search-accommodations gt/search-attractions gt/search-cities gt/search-driving-distance gt/search-flights gt/search-restaurants | other/travel-planner-ai-claude-skills other/tour-operations gt/search-accommodations development/travel-planner other/travel-requirements-expert |
| SkillRouter | In this task, you are given a clip of screen recording of a player playing the game Super Mario. Your goal is to analyze the key frames in this video and count how many coins / enemies / turtles ever showing up in each frame and write the results into a CSV file. | gt/ffmpeg gt/image_editing gt/object_counter | gt/object_counter gt/ffmpeg gt/videoframeextraction other/videoprocessing data/videoframereader |
| BFCL-live | I’m having trouble understanding how to use the list_files functionality. Help me. | dartfx_help | list_files list_directory_contents help list_projects adriel_list_projects |
| BFCL-live | search flight prices from NYC to NEw Delhi use tools, finally respond in json | search_web_tool | get_trip Flights_4_SearchRoundtripFlights Flights_4_SearchOnewayFlight render_ui_finish Trains_1_GetTrainTickets |
| BFCL-live | Find a recent news article about corona virus | search_web_tool | search_engine.query get_news_report HNA_WQA.search HNA_WQA.query HNA_NEWS.search |
Table 18: Representative Toollery failure cases. Skill-routing failures often miss implementation-specific or complementary capabilities, while BFCL-live failures often arise from near-duplicate tool interfaces or underspecified requests.