ToolFlood: Beyond Selection -- Hiding Valid Tools from LLM Agents via Semantic Covering

Hussein Jawad; Nicolas J-B Brunel

cs.CLMar 14, 2026

ToolFlood: Beyond Selection -- Hiding Valid Tools from LLM Agents via Semantic Covering

Authors: Hussein Jawad, Nicolas J-B Brunel

Organizations: Capgemini Invent, Paris, France · LaMME, University Paris-Saclay, Evry, France · ENSIIE, Evry, France

Abstract

Large Language Model (LLM) agents increasingly use external tools for complex tasks and rely on embedding-based retrieval to select a small top-k subset for reasoning. As these systems scale, the robustness of this retrieval stage is underexplored, even though prior work has examined attacks on tool selection. This paper introduces ToolFlood, a retrieval-layer attack on tool-augmented LLM agents. Rather than altering which tool is chosen after retrieval, ToolFlood overwhelms retrieval itself by injecting a few attacker-controlled tools whose metadata is carefully placed by exploiting the geometry of embedding space. These tools semantically span many user queries, dominate the top-k results, and push all benign tools out of the agent's context. ToolFlood uses a two-phase adversarial tool generation strategy. It first samples subsets of target queries and uses an LLM to iteratively generate diverse tool names and descriptions. It then runs an iterative greedy selection that chooses tools maximizing coverage of remaining queries in embedding space under a cosine-distance threshold, stopping when all queries are covered or a budget is reached. We provide theoretical analysis of retrieval saturation and show on standard benchmarks that ToolFlood achieves up to a 95% attack success rate with a low injection rate (1% in ToolBench). The code will be made publicly available at the following link: https://github.com/as1-prog/ToolFlood

Figures & tables

Dataset	Attack	Avg. B.	Avg. PR	TDR	ASR (%)
Dataset	Attack	Avg. B.	Avg. PR	TDR	GPT-4o	GPT-5-mini	GPT-4o-mini
MetaTool	Random-Sybil Injection	200	100%	1.6%	4.6%	3.8%	4.0%
	PoisonedRAG	100	50%	95%	98.1%	98.3%	98.1%
	ToolFlood	171	86.3%	97.2%	99.6%	99.4%	99%
ToolBench	Random-Sybil Injection	200	1.7%	0%	1%	0.4%	0.2%
	PoisonedRAG	100	0.85%	82.0%	91.4%	90.9%	92.0%

Table 1: ToolFlood effectiveness on MetaTool and ToolBench (test split) using retrieval with text-embedding-3-small : Top-k Domination Rate (TDR) and Attack Success Rate (ASR), with average injection budget (Avg. B) and poisoning rate (Avg. PR); ASR reported for selector LLMs (GPT-4o, GPT-5-mini, GPT-4o-mini).

Figure 1: ASR vs. injection budget (B level) per task on ToolBench (test split). Higher B increases top-k domination opportunities, yielding higher ASR (retrieval: text-embedding-3-small, selector LLM: GPT-4o-mini).

Target Retriever	Shadow Retriever
	TE-3-S		MiniLM
	TDR	ASR	TDR	ASR
TE-3-S	91.0%	94.6%	84%	91.2%
TE-3-L	91.8%	95%	88.2%	93.7%
MiniLM	78%	92.7%	82%	92%

Table 2: ToolFlood transferability across embedding retrievers on ToolBench (test split): TDR/ASR when optimizing with a shadow retriever and evaluating with a (possibly different) target retriever (k = 5; LLM selector: GPT-4o-mini). Results are shown using abbreviated retriever names (TE-3-S: text-embedding-3-small, TE-3-L: text-embedding-3-large, MiniLM: all-MiniLM-L6-v2)

Defense	TDR (%) $\downarrow$	ASR (%) $\downarrow$
None (ToolFlood)	91%	96.1%
+ MMR reranking	44.8%	91.0%
+ Llama Prompt Guard	91%	96.1%

Table 3: Retrieval-time defenses against ToolFlood on ToolBench (test split): impact of MMR reranking and Llama Prompt Guard filtering on TDR and ASR (retrieval: text-embedding-3-small, selector LLM: GPT-4o-mini).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Notes / where used
$q$	Natural-language query / instruction	Input to agent pipeline
$Q$	Space/set of possible queries	Domain of $\phi$
$Q_{\text{tar}}$	Target query distribution / set	Used for coverage, TDR/ASR
$Q_{\text{gen}}$	Generation/selection split	“No-contamination” split
$Q_{\text{test}}$	Held-out evaluation split	Reported metrics
$t$	A tool	$t\in T$

Appendix

Table 4: List of symbols used in ToolFlood.

Task	Example queries
1. Space images	“Find high-resolution pictures of distant galaxies using ImageSearch.” “Can you find space-themed wallpapers for a computer background?” “Provide images depicting the surface of Mars from recent missions.” “I need detailed visuals of Saturn’s rings for a presentation.”
2. Website performance + SEO	“How can I improve my website’s overall performance for better SEO results?” “What tools are recommended for analyzing effective SEO keywords?” “How can I enhance my website’s loading speed to improve SEO rankings?” “Which keywords are most effective for targeting my niche audience?”
3. Financial insights / portfolio	“What are the current performance metrics of my investment portfolio?” “How can I balance risk and return in my financial investments?” “How can I diversify my investments to minimize risk?” “What are the recommended investment strategies for long-term growth?”
4. Resume + job outreach emails	“How can I optimize my resume for a software development position?” “What strategies should I use to write effective job outreach emails?” “How can I improve my resume to better align with job descriptions?” “Please suggest methods to effectively polish my resume for a managerial role.”
5. Daily workout + reminders	“Can you help me design a daily workout routine and set up reminders to stay consistent?” “I need assistance in creating a habit plan for exercising and scheduling notifications.” “How can I establish a daily workout routine with consistent reminders?” “Suggest tools for planning and tracking my fitness habits effectively.”
6. Travel itinerary + booking	“What are some recommended travel itineraries for a week-long trip to Japan?” “Can you help me book accommodations and plan activities for a family vacation in Paris?” “Can you help me plan a week-long trip to Italy, including flights, accommodations, and sightseeing tours?” “I’m looking for restaurant recommendations and reservations in Paris for a romantic dinner.”

Appendix

Table 5: MetaTool tasks and example queries (from train/test sets).

Task	Example queries
1. Email validation and deliverability	“How can I verify if an email address is valid?” “What tools are available for testing email bounce rates?” “How do I test an email address to confirm if it will bounce?” “Is there a method to validate email addresses in bulk for a mailing list?” “Can you validate if this email is disposable or not?”
2. Financial insights and risk assessment	“What are the latest cryptocurrency market trends and their potential impact on investments?” “Can you provide a risk assessment for climate change on property investments in a specific region?” “What are the projected trends for global stock markets in the next quarter?” “Provide a summary of current cryptocurrency market movements.”
3. Fitness plans and health tracking	“What is the recommended daily caloric intake for a 30-year-old male with moderate activity levels?” “How can I track my macronutrient intake for a balanced diet?” “Can you suggest a customized workout plan tailored to weight loss?” “How do I calculate the calories burned during a 30-minute running session?”
4. SMS communications	“Can you provide the history of messages received by a virtual number?” “What is the current balance available for bulk SMS sending?” “How can I dispatch an SMS using the branded SMS service in Pakistan?” “Check the delivery status of a message sent through the D7SMS service.”
5. Food and recipe management	“What are some easy-to-make keto recipes for beginners?” “Please provide the nutritional breakdown of a medium-sized apple.” “What are some low-calorie dinner recipes suitable for vegetarians?” “Can you calculate the daily caloric intake for someone weighing 70 kg?” “Suggest easy-to-prepare keto recipes for a beginner.”
6. Medical and health services	“What are the current global statistics on COVID-19 cases and recoveries?” “Can I assess my genetic predisposition to specific health conditions?” “How has grocery store mobility been affected by the pandemic?” “What resources are available for virtual triage consultations?”

Appendix

Table 6: ToolBench tasks and example queries (from train/test sets).

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 320 papers

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs
Jun 4, 2026Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber +1Tool-Augmented Language Model Agents LLM Auditing
SAP Labs
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer
May 23, 2026Vyzantinos Repantis, Ameya Gawde, Harshvardhan Singh +1Tool-Use Evaluation Tool-Augmented Language Model Agents
Meta Platforms
Tools Are Not Islands: Set-Level Tool Retrieval for LLM Agents via Query-Conditioned Hyperedge Prediction
Jul 28, 2026Xinyi Hong, Pinjun Dong, Xinyang Yu +1Tool-Augmented Language Model Agents LLM Tool Use
School of Mathematical Sciences, Shanghai Jiao Tong University · Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University

Jun 4, 2026J/K move · Enter open · S save

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

Ashutosh Hathidara, Sai Shruthi Sistla, Sebastian Schreiber, Sahil Bansal

SAP Labs

Tool-Augmented Language Model Agents LLM Auditing LLM Agent Evaluation LLM Tool Use Language Model Probing Tool Discovery for LLM Agents

Open paper AskPDF arXiv

Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck. As embedding-based retrieval approaches rely on compact encoders that may under-capture specialized tool semantics, parametric tool retrieval addresses this by encoding each tool as a virtual token appended to the LLM vocabulary, fine-tuned in two stages (memorization then retrieval SFT) to use the LLM as a retriever, achieving strong performance on standard ToolBench retrieval benchmarks. Yet these benchmarks use verbose, fully-specified queries, and their evaluation applies constrained decoding that restricts outputs to valid token paths, neither reveals whether the model actually understands its tools. We introduce ToolSense, an open-source LLM-powered diagnostic framework that takes any tool catalog as input and automatically generates three benchmarks: a Realistic Retrieval Benchmark (RRB) with queries at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark. Applying ToolSense to ToolBench (~47k tools) and evaluating five parametric model training configurations reveals a knowledge-retrieval dissociation: on RRB queries, several configurations collapse by ~50-64 percentage points compared to fully-specified ToolBench benchmarks, falling below the embedding-model baseline. Additionally, despite strong retrieval performance, some models score near-random on factual probes, suggesting a knowledge-retrieval dissociation. We open-source the ToolSense framework and the ToolBench diagnostic benchmarks at https://github.com/SAP/toolsense.

Abstract

LLM Agent Security

Explore similar work

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

How Many Tools Should an LLM Agent See? A Chance-Corrected Answer

Tools Are Not Islands: Set-Level Tool Retrieval for LLM Agents via Query-Conditioned Hyperedge Prediction