Typed Federated Artifacts for the Agentic Web:Sharing Tool-Routing Knowledge Across Frozen,Heterogeneous LLM Agents

Abhijit Chakraborty; Ni Trieu; Vivek Gupta

cs.CLSep 6, 2026

Typed Federated Artifacts for the Agentic Web:Sharing Tool-Routing Knowledge Across Frozen,Heterogeneous LLM Agents

Authors: Abhijit Chakraborty, Ni Trieu, Vivek Gupta

Organizations: Arizona State University

Abstract

An open, networked web will allow agents to run frozen models from multiple vendors, keep their history private, and teach each other which tool to call and when. Flat text (prompts, example pools) makes it difficult for the protocol to distinguish between noise statistics, merging rules, and documentation. Weights and adapters cannot transfer that knowledge between platforms. We suggest sharing typed federated artifacts, schema-validated objects with well-defined fields for per-field privacy (described here, but measured), dispute resolution, and cross-model transfer, and instantiating them as SYNAPSE1, a common tool-routing knowledge. After deleting 192 garbage entries and 1,916 training items that duplicate or almost duplicate test queries, a federated compendium routes within 1.1 points of a centralized one at 20 MB of JSON per client each round on StableToolBench (3,180 tools). The same experience merged and shown to the router as typed fields rather than one flat string is worth 8.5 points on clean data and 7.4 under 60% injected contradiction. Crossing merge and rendering shows the halves are inseparable (the typed merge shown flat is the worst arm), while three conflict policies are indistinguishable, so the conflict log that motivated this work is not the On τ-bench retail, each compendium arm improves GPT-4o agents' per-step tool-call accuracy by at least 6.7 points, attributed to format rather than federated experience. Two cautionary findings conclude the paper: on a topic-labeled math proxy and StableToolBench, a TF-IDF classifier over the same labeled experience beats every LLM routing arm (by 48 and 26 points, mostly retrieval recall) because the benchmark's pool holds labeled queries for every supposedly unseen tool and every test query verbatim before our filter. It cannot measure routing to tools without labels, which routing exists for.

Figures & tables

Panel of Figure 1 — Figure 1: Typed vs. untyped (flat JSON) under injected contradiction on StableToolBench G1+G2-instruction ( $n$ =258; mean $\pm$ SD over seeds 42/123/456, round-delayed and the 20% column from two seeds; identical injected set across arms per (rate, seed), verified by hash). (a) Routing accuracy vs. fraction of contradictory contributions. (b) The same data with recall@5 and seed counts; an untyped conflict log has no precaution field to attach dissent to. The typed/untyped gap is already present at 0%.

	G1 (single tool)			G2 (multi-tool)		G3
Arm	instr. (153)	tool † (152)	cat. † (134)	instr. (105)	cat. † (124)	instr. (61)	all (729)
Local-only	$0.484$	$0.457$	$0.464$	$0.300$	$0.278$	$0.252$	$0.394$
Docs-only	$0.562$	$0.546$	$0.537$	$\mathbf{0.533}$	$\mathbf{0.460}$	$0.508$	$0.528$
Synapse (federated)	$\mathbf{0.601}$	$\mathbf{0.594}$	$\mathbf{0.614}$	$0.514$	$0.438$	$0.519$	$\mathbf{0.555}$
Centralized	$0.599$	$0.583$	$\mathbf{0.614}$	$0.495$	$0.435$	$\mathbf{0.530}$	$0.550$
Query classifier ‡	$0.837$	$0.827$	$0.838$	$0.816$	$0.780$	$0.771$	$0.817$

Table 1: StableToolBench routing accuracy (alias-collapsed), mean over 3 seeds;

K

=5 clients, 5,000 items per client from the leak-filtered pool; SDs and recall@5 in App. C . † unseen relative to ToolLLaMA’s training split only: the instruction pool holds labeled queries for every test tool (§ 5 ). ‡ TF-IDF+SVM on the same 25,000 labeled items per seed, no federation; 0 on any tool without labeled queries. Bold: best compendium arm.

Arm	Tool-call step acc.	Task success	Steps / seed
Plain (no compendium)	$0.596\pm 0.036$	$0.297\pm 0.046$	264
Docs-only	$0.679\pm 0.012$	$0.364\pm 0.048$	264
Synapse (federated)	$0.663\pm 0.025$	$0.370\pm 0.010$	264
Centralized	$0.672\pm 0.019$	$0.358\pm 0.038$	264

Table 2:

\tau

-bench retail, 55 held-out tasks, GPT-4o ( gpt-4o-2024-08-06 ) agent and simulator, mean

\pm

SD over 3 seeds. Every compendium arm is

\geq

6.7 points above Plain ; the three compendium arms are within one SD. Plain is our own agent loop in the vendored harness and is not comparable to the leaderboard (App. C ).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: One round of Synapse . Each organization turns its private tool-use experience into a typed compendium and shares only that artifact (numeric fields clipped and noised, text masked; about20 MB of JSON). An edge node validates each artifact against the schema and merges field by field, keeping disagreements as a typed conflict log rather than voting them away, and broadcasts one global compendium. At inference, any frozen LLM retrieves five candidate tools by similarity to descriptions and pooled scenarios and picks one by reading each candidate’s “when” and “when not” fields. The outcome of the call becomes next round’s experience.

	G1-Instr. (153)		G1-Tool (152)		G1-Cat. (134)
Arm	Acc	R@5	Acc	R@5	Acc	R@5
Synapse	$.601\pm.000$	$.684\pm.008$	$.594\pm.036$	$.654\pm.056$	$.614\pm.024$	$.684\pm.024$
Centralized	$.599\pm.004$	$.667\pm.011$	$.583\pm.034$	$.634\pm.031$	$.614\pm.024$	$.672\pm.020$
Local-only	$.484\pm.002$	$.556\pm.010$	$.457\pm.005$	$.513\pm.002$	$.464\pm.013$	$.546\pm.008$
Docs-only	$.562$	$.706$	$.546$	$.691$	$.537$	$.709$
	G2-Instr. (105)		G2-Cat. (124)		G3-Instr. (61)

Appendix

Table 3: Full StableToolBench results, mean

\pm

SD over seeds 42/123/456 ( Docs-only is deterministic; Local-only is the mean over the five clients, so its SD is over client-averaged values rather than over one 153-query count). Generated from the three combined_summary.json files listed in tables/manifest.json . The query-classifier arm of this run had been fit on the unfiltered pool (0.93–0.98 on every split) and is excluded; the row in Tab. 1 is a separate run fit on the filtered pool with an asserted zero exact/near-duplicate overlap ( stabletoolbench_query_classifier_clean_r1 ).

	all (729)		oracle retrieval, per group
Arm	real retr.	oracle	G1-instr.	G1-tool	G1-cat.	G2-instr.	G2-cat.	G3-instr.
Synapse	$.555$	$.762\pm.014$	$.767\pm.023$	$.750\pm.023$	$.749\pm.034$	$.771\pm.019$	$.750\pm.050$	$.820\pm.028$
Centralized	$.550$	$.770\pm.014$	$.776\pm.016$	$.752\pm.033$	$.769\pm.039$	$.781\pm.019$	$.753\pm.065$	$.825\pm.025$
Docs-only	$.528$	$.658$	$.667$	$.632$	$.657$	$.657$	$.637$	$.754$
Query classifier	$.817$	–	(no retrieval stage)

Appendix

Table 4: Routing accuracy with real retrieval (Tab. 1 ) and with oracle retrieval (gold tool forced into the five candidates), mean

\pm

SD over seeds 42/123/456; Docs-only is deterministic. Run stabletoolbench_oracle_r1 , hashes and commit in tables/manifest.json .

		0% contradiction		60% contradiction
Merge	Rendered to reranker as	Acc	R@5	Acc	R@5
typed (conflict log)	typed fields ( Synapse )	$.583\pm.018$	$.764$	$.575\pm.026$	$.769$
typed (conflict log)	one flat string	$.460\pm.006$	$.738$	$.416\pm.025$	$.749$
flat (majority)	typed fields (decoded)	$.531\pm.014$	$.820$	$.539\pm.014$	$.827$
flat (majority)	one flat string	$.497\pm.026$	$.811$	$.501\pm.018$	$.822$

Appendix

Table 5: Merge and rendering crossed (StableToolBench G1+G2-instruction,

n

=258, mean

\pm

SD over seeds 42/123/456; run stabletoolbench_2x2_renderswap_r1 , commit and per-cell hashes in tables/manifest.json ). The two baseline rows reproduce Tab. 1(b) to three decimals. The decoded row re-parses the flat merge’s stored string into fields; its re-serialization is not byte-identical to the stored string, so it is an approximation of the flat merge’s content. The typed merge helps only when the reranker is shown its fields; shown flat, it is worse than the flat merge, and worst under contradiction.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 290 papers

Sep 13, 2026

Cartograph: Federated Tool Discovery with Operator-Attested Retrieval for AI Agents

The Model Context Protocol (MCP) enables AI agents to discover and call tools, but loading every definition becomes expensive as connected catalogs grow. We present Cartograph, a federated MCP proxy that changes agent-visible tool discovery from

O(n)

catalog traversal to

O(k)

progressive disclosure. Cartograph combines three mechanisms: (1) operator-attested capability cards, Ed25519-signed descriptions generated under the deploying operator's control rather than ranked publisher copy; (2) Rift, a three-layer confusable-cluster analysis comprising density clustering, query-margin analysis, and token diagnosis; and (3) two-stage retrieval, which ranks servers before tools. On a 22-server, 374-tool deployment, Cartograph exposes three proxy tools instead of 374 definitions. A 49-query author-constructed benchmark yields R@5 of 0.816, compared with 0.592 for a Jaccard keyword baseline, while a measured top-5 discovery exchange uses 475 tokens rather than 42,450 under the stated full-catalog accounting. Rift identifies 49 confusable clusters, including four HIGH-risk clusters in bootstrap-generated cards. An exploratory comparison of 119 LLM-generated descriptions removes the observed zero-distance cluster but shows that mixing card-generation regimes can reduce R@5. Gateway measurements over ten trials add 5ms mean latency (0.8%) relative to direct stdio MCP calls. Cartograph is complementary to code-execution approaches: it controls which tool descriptions are surfaced and records the provenance of the descriptions used for ranking for each query.

Justice Owusu Agyemang, Michael Agyare, Kwame Opuni-Boachie Obour Agyekum +3

VIA Cybersecurity Lab, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana · Quantum and Assistive Technologies Lab, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana · Ghana Communication Technology University, Ghana

LLM Tool Use Agentic Retrieval Model Context Protocol Information Retrieval

Ask PDF

Jul 13, 2026

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory

Large language model (LLM) agents increasingly rely on external tools served by shared providers and accessed by heterogeneous downstream agents. Existing approaches improve tool use on the agent side through parameter updates, prompt refinement, or agent-side memory, making tool knowledge difficult to share and limited to behaviors observed in past tasks. We argue that reusable tool knowledge should instead be maintained by the tool provider. We introduce ToolAtlas, a graph-based framework that builds a persistent provider-side tool memory of tool capabilities, failure boundaries, and cross-tool compositions through execution-verified probing. At inference time, agents query the tool memory via adaptive graph traversal. Across two MCP-based benchmarks spanning eight services, ToolAtlas outperforms existing tool-side optimization and agent-side memory baselines by up to 21.61% in pass@1 and 18.61% in pass@4. The same tool memory also transfers across environment instances and agent frameworks without retraining or task-time exploration, yielding up to 24.16%/16.22% and 17.49%/14.27% relative gains in pass@1/pass@4, respectively. Ablation studies show that these gains arise from combining tool-centered memory organization with capability-guided execution probing. These results establish provider-side tool memory as an effective and reusable paradigm for tool servers. Our code is in: https://github.com/PuppyKnightUniversity/ToolAtlas.

Yue Fang, Zhibang Yang, Fangkai Yang +5

School of Computer Science, Peking University, Beijing, China · Microsoft

LLM Tool Use

Ask PDF

Jul 6, 2026

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

AI Agent Reliability Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks LLM Agent Reliability Failure Attribution for LLM Agents

Ask PDF

Abstract

Tool-Using Agents

Explore similar work

Cartograph: Federated Tool Discovery with Operator-Attested Retrieval for AI Agents

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents