Large language models (LLMs) increasingly serve as autonomous agents that invoke external tools. However, this capability introduces tool hallucination, selecting incorrect tools or generating invalid calls. Existing mitigation methods report substantial improvements, yet we identify a previously overlooked failure mode that we term Hallucination Escape. These methods reduce hallucination on the tool configuration they are tuned on but increase it on other configurations, canceling out the gain. We further investigate this phenomenon and find that hallucination rises sharply when a model's intrinsic tool-use tendencies conflict with the current tool configuration, and that existing methods reinforce rather than suppress these tendencies, which in turn contributes to hallucination escape. Building on these findings, we propose EscapeGuard, a training-free inference-time method that combines conflict-aware gating with configuration-derived attention enhancement to mitigate tool hallucination while preventing hallucination escape. Across six benchmarks on various models, EscapeGuard reduces tool-selection hallucination by 9.0 pp and suppresses hallucination escape, lowering the cross-configuration mean by 23.7 pp and achieving an 89.1% net improvement in paired-query evaluation. We hope this work can encourage evaluation beyond a single tool configuration and pave the way for more reliable tool-using LLM agents.
Figures & tables
Figure 1: Hallucination escape and repair. (a) Under the standard configuration ( C1 ), an existing mitigation method produces the correct call. (b) After the tool is renamed ( C2 ), the same method still calls the original name. (c) EscapeGuard adapts to the renamed configuration and produces the correct call.
Figure 2: Tool configuration changes (C 1 –C 6 ) illustrated with a weather API example. Each variant modifies one or more aspects of the correct tool while ensuring a valid solution remains available.
Figure 3: Cross-query hallucination escape (%).
Figure 4: Hallucination rate by prior-strength quintile under aligned and conflict settings. (a) tool-selection conflict under Btool ; (b) argument-name and (c) value conflict under Barg . Q1-Q5:weakest to strongest tendency.
Figure 5: Overview of EscapeGuard . Given a query and tool list, EscapeGuard records tool-related token spans and extracts hidden states from the frozen LLM (❶). Conflict probes then score each input per stage and activate intervention only above a threshold (❷). When activated, attention is enhanced toward tool names and descriptions during selection, or toward the selected tool’s schema during argument generation (❸).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Tool and description
Original argument schema
currency_conversion.convert Converts a specified amount of money from one currency to another at the latest rate.
from_currency (string): The currency that you want to convert from. to_currency (string): The currency that you want to convert to. amount (float): The amount of money that you want to convert. Required: from_currency , to_currency , amount .
currency_conversion.get_latest_rate Get the latest currency conversion rate from one currency to another.
from_currency (string): The currency that you want to convert from. to_currency (string): The currency that you want to convert to. Required: from_currency , to_currency .
currency_conversion.get_rate Get the currency conversion rate from one currency to another at a specified date.
from_currency (string): The currency that you want to convert from. to_currency (string): The currency that you want to convert to. date (string): The date at which the conversion rate applies. Default is the current date. Required: from_currency , to_currency .
Appendix
Table 6
Tool and description
Original argument schema
send_email Send an email to a recipient
recipient (string): The email address of the recipient subject (string): The subject of the email message (string): The content of the email Required: recipient , subject , message .
search_recipes Search for recipes based on ingredients
ingredients (array of string): The ingredients to search for Required: ingredients .
Appendix
Table 7
Tool and description
Original argument schema
translateAmharic Translate text to Amharic language Field: Foreign Languages/Amharic
text (str): The text to be translated to Amharic Required: text .
translateToHebrew Translate a text to Hebrew Field: Foreign Languages/Hebrew
text (str): The text to be translated language (str): The source language of the text (e.g., English, Spanish, French) Required: text .
translateHausa Translate text from Hausa to English or vice versa Field: Foreign Languages/Hausa
text (str): The text to be translated Required: text .
pronounceXhosaPhrase Get the pronunciation of a given Xhosa phrase Field: Foreign Languages/Xhosa
phrase (str): The Xhosa phrase for which you want to get the pronunciation Required: phrase .
translateIgbo Translate a word or phrase from English to Igbo Field: Foreign Languages/Igbo
word (str): The word or phrase to be translated from English to Igbo Required: word .
Appendix
Table 8
Tool and description
Original argument schema
weather.get_by_city_date Retrieves the historical weather data based on city and date.
city (string): The city for which to retrieve the weather. date (string): The date for which to retrieve the historical weather data in the format YYYY-MM-DD. Required: city , date .
weather.get_forecast_by_coordinates Get the weather forecast for a specific geographical coordinates.
coordinates (tuple): The geographical coordinates (latitude, longitude). days_ahead (integer): Number of days to forecast from current date (default is 7). Required: coordinates .
weather.get_by_coordinates_date Retrieves the historical weather data based on coordinates and date.
coordinates (tuple): The geographical coordinates (latitude, longitude). date (string): The date for which to retrieve the historical weather data in the format YYYY-MM-DD. Required: coordinates , date .
Appendix
Table 9
Tool and description
Original argument schema
book_flight Book a flight for a specific destination
destination (string): The desired destination departure_date (string): The date of departure return_date (string): The date of return passenger_count (integer): The number of passengers flying Required: destination , departure_date , return_date , passenger_count .
create_calendar_event Create a new calendar event
event_title (string): The title of the event start_time (string): The start time of the event end_time (string): The end time of the event Required: event_title , start_time , end_time .
Appendix
Table 10
7B/9B models
Other models †
Base
+EscapeGuard
Overhead
Base
+EscapeGuard
Overhead
LLaMA
42.3
44.8
+5.9%
156.7
163.2
+4.1%
Qwen
40.1
42.4
+5.7%
148.3
154.6
+4.2%
Mistral
41.6
44.0
+5.8%
—
—
—
Mixtral †
—
—
—
89.4
93.7
+4.8%
Appendix
Table H.12: Inference latency (ms per token) and relative overhead of EscapeGuard.
Tool-augmented large language model (LLM) agents fail in a way no tool-selection or tool-security method addresses: they call tools that do not exist and pass arguments no schema declares. Existing defenses either pick the right tool (selection) or constrain what an agent may do with real tools (gating), both of which presuppose the emitted call refers to a real tool at all. We show this is a structural blind spot: a hallucinated call is by construction not a decision any gate made, so no gate can reject it. This paper is primarily a measurement and benchmark study. We give a five-class taxonomy of tool hallucination (H1-H5) and, as a reference point, the Resolution Rung: a training-free, closed-world resolver (registry membership plus a signature check) whose interest is where it must sit, not what it computes. We prove hallucination defense must precede any causal gate, and characterize the one irreducible residue (borrowed arguments schema-indistinguishable from a valid call). Across ten hosted models under two invocation surfaces we measure 322 genuine hallucinations; fabricated-tool calls concentrate on the unconstrained raw-JSON surface (34 vs. 3), and model scale does not help (a 675B model matches a 7-8B one). We then extend to the Model Context Protocol, where merging several servers into one namespace creates hallucination surfaces a single registry cannot express (a second taxonomy, M1-M5); on the live MCP surface we measure 154 hallucinations, including from frontier models that were clean on the single-registry surface, because collisions and shadowing are structural to the merge. We release the versioned Hallucinated-Tools Benchmark (HTB) so any resolver is comparable across submissions.
Tool-augmented language models ground their responses in external tool outputs, improving reliability beyond parametric knowledge alone. However, even when the correct tool is called and returns accurate information, models may still contradict such outputs, omit critical details, or generate unsupported claims. Existing hallucination benchmarks for tool-augmented dialogue primarily focus on the tool-calling stage, evaluating whether the correct tool and arguments are selected, while overlooking errors that arise during post-tool response generation. To address this gap, we introduce ToolHACE, a span-level benchmark for hallucination detection in tool-augmented LLM responses, covering five fine-grained hallucination categories. Through extensive experiments with modern hallucination detectors, we show that post-tool-call hallucinations are widespread and remain challenging to detect. Our results further demonstrate that effective detection requires training on specialized tool-augmented corpora, while transfer from tool-agnostic hallucination datasets performs poorly in this setting. These findings highlight the need for dedicated benchmarks and models for reliable hallucination detection in tool-augmented language systems.
Empowered by large language models (LLMs), intelligent agents have become a popular paradigm for interacting with open environments to facilitate AI deployment. However, hallucinations generated by LLMs-where outputs are inconsistent with facts-pose a significant challenge, undermining the credibility of intelligent agents. Only if hallucinations can be mitigated, the intelligent agents can be used in real-world without any catastrophic risk. Therefore, effective detection and mitigation of hallucinations are crucial to ensure the dependability of agents. Unfortunately, the related approaches either depend on white-box access to LLMs or fail to accurately identify hallucinations. To address the challenge posed by hallucinations of intelligent agents, we present HalMit, a novel black-box watchdog framework that models the generalization bound of LLM-empowered agents and thus detect hallucinations without requiring internal knowledge of the LLM's architecture. Specifically, a probabilistic fractal sampling technique is proposed to generate a sufficient number of queries to trigger the incredible responses in parallel, efficiently identifying the generalization bound of the target agent. Experimental evaluations demonstrate that HalMit significantly outperforms existing approaches in hallucination monitoring. Its black-box nature and superior performance make HalMit a promising solution for enhancing the dependability of LLM-powered systems.
Siyuan Liu, Wenjing Liu, Zhiwei Xu +3
College of Computer Science, Nankai University · College of intelligent Science and Technology, Inner Mongolia University of Technology · Institute of Computing Technology, Chinese Academy of Sciences +3