Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents

Peigui Qi; Kunsheng Tang; Yide Song; Weiming Zhang; Nenghai Yu

cs.CLOct 3, 2026

Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents

Authors: Peigui Qi, Kunsheng Tang, Yide Song, Weiming Zhang, Nenghai Yu

Organizations: University of Science and Technology of China · University of Washington

Abstract

Large language models (LLMs) increasingly serve as autonomous agents that invoke external tools. However, this capability introduces tool hallucination, selecting incorrect tools or generating invalid calls. Existing mitigation methods report substantial improvements, yet we identify a previously overlooked failure mode that we term Hallucination Escape. These methods reduce hallucination on the tool configuration they are tuned on but increase it on other configurations, canceling out the gain. We further investigate this phenomenon and find that hallucination rises sharply when a model's intrinsic tool-use tendencies conflict with the current tool configuration, and that existing methods reinforce rather than suppress these tendencies, which in turn contributes to hallucination escape. Building on these findings, we propose EscapeGuard, a training-free inference-time method that combines conflict-aware gating with configuration-derived attention enhancement to mitigate tool hallucination while preventing hallucination escape. Across six benchmarks on various models, EscapeGuard reduces tool-selection hallucination by 9.0 pp and suppresses hallucination escape, lowering the cross-configuration mean by 23.7 pp and achieving an 89.1% net improvement in paired-query evaluation. We hope this work can encourage evaluation beyond a single tool configuration and pave the way for more reliable tool-using LLM agents.

Figures & tables

Figure 1: Hallucination escape and repair. (a) Under the standard configuration (

C_{1}

), an existing mitigation method produces the correct call. (b) After the tool is renamed (

C_{2}

), the same method still calls the original name. (c) EscapeGuard adapts to the renamed configuration and produces the correct call.

Figure 2: Tool configuration changes (C 1 –C 6 ) illustrated with a weather API example. Each variant modifies one or more aspects of the correct tool while ensuring a valid solution remains available.

Figure 3: Cross-query hallucination escape (%).

Figure 4: Hallucination rate by prior-strength quintile under aligned and conflict settings. (a) tool-selection conflict under

B_{\mathrm{tool}}

; (b) argument-name and (c) value conflict under

B_{\mathrm{arg}}

. Q1-Q5:weakest to strongest tendency.

Figure 5: Overview of EscapeGuard . Given a query and tool list, EscapeGuard records tool-related token spans and extracts hidden states from the frozen LLM (❶). Conflict probes then score each input per stage and activate intervention only above a threshold (❷). When activated, attention is enhanced toward tool names and descriptions during selection, or toward the selected tool’s schema during argument generation (❸).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Tool and description	Original argument schema
currency_conversion.convert Converts a specified amount of money from one currency to another at the latest rate.	from_currency (string): The currency that you want to convert from. to_currency (string): The currency that you want to convert to. amount (float): The amount of money that you want to convert. Required: from_currency , to_currency , amount .
currency_conversion.get_latest_rate Get the latest currency conversion rate from one currency to another.	from_currency (string): The currency that you want to convert from. to_currency (string): The currency that you want to convert to. Required: from_currency , to_currency .
currency_conversion.get_rate Get the currency conversion rate from one currency to another at a specified date.	from_currency (string): The currency that you want to convert from. to_currency (string): The currency that you want to convert to. date (string): The date at which the conversion rate applies. Default is the current date. Required: from_currency , to_currency .

Appendix

Table 6

Tool and description	Original argument schema
send_email Send an email to a recipient	recipient (string): The email address of the recipient subject (string): The subject of the email message (string): The content of the email Required: recipient , subject , message .
search_recipes Search for recipes based on ingredients	ingredients (array of string): The ingredients to search for Required: ingredients .

Appendix

Table 7

Tool and description	Original argument schema
translateAmharic Translate text to Amharic language Field: Foreign Languages/Amharic	text (str): The text to be translated to Amharic Required: text .
translateToHebrew Translate a text to Hebrew Field: Foreign Languages/Hebrew	text (str): The text to be translated language (str): The source language of the text (e.g., English, Spanish, French) Required: text .
translateHausa Translate text from Hausa to English or vice versa Field: Foreign Languages/Hausa	text (str): The text to be translated Required: text .
pronounceXhosaPhrase Get the pronunciation of a given Xhosa phrase Field: Foreign Languages/Xhosa	phrase (str): The Xhosa phrase for which you want to get the pronunciation Required: phrase .
translateIgbo Translate a word or phrase from English to Igbo Field: Foreign Languages/Igbo	word (str): The word or phrase to be translated from English to Igbo Required: word .

Appendix

Table 8

Tool and description	Original argument schema
weather.get_by_city_date Retrieves the historical weather data based on city and date.	city (string): The city for which to retrieve the weather. date (string): The date for which to retrieve the historical weather data in the format YYYY-MM-DD. Required: city , date .
weather.get_forecast_by_coordinates Get the weather forecast for a specific geographical coordinates.	coordinates (tuple): The geographical coordinates (latitude, longitude). days_ahead (integer): Number of days to forecast from current date (default is 7). Required: coordinates .
weather.get_by_coordinates_date Retrieves the historical weather data based on coordinates and date.	coordinates (tuple): The geographical coordinates (latitude, longitude). date (string): The date for which to retrieve the historical weather data in the format YYYY-MM-DD. Required: coordinates , date .

Appendix

Table 9

Tool and description	Original argument schema
book_flight Book a flight for a specific destination	destination (string): The desired destination departure_date (string): The date of departure return_date (string): The date of return passenger_count (integer): The number of passengers flying Required: destination , departure_date , return_date , passenger_count .
create_calendar_event Create a new calendar event	event_title (string): The title of the event start_time (string): The start time of the event end_time (string): The end time of the event Required: event_title , start_time , end_time .

Appendix

Table 10

	7B/9B models			Other models $\dagger$
	Base	+EscapeGuard	Overhead	Base	+EscapeGuard	Overhead
LLaMA	42.3	44.8	+5.9%	156.7	163.2	+4.1%
Qwen	40.1	42.4	+5.7%	148.3	154.6	+4.2%
Mistral	41.6	44.0	+5.8%	—	—	—
Mixtral $\dagger$	—	—	—	89.4	93.7	+4.8%

Appendix

Table H.12: Inference latency (ms per token) and relative overhead of EscapeGuard.

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 179 papers

Closed-World Resolution Against Tool Hallucination in LLM Agents
Sep 16, 2026Laxmipriya Ganesh IyerLLM Agent Evaluation LLM Tool Use
Independent Researcher
ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses
Oct 12, 2026Nikolay Ivanov, Sergey Pletenev, Elisei Rykov +8Hallucination Detection LLM Hallucination Detection
Skoltech · AIRI · T-Tech +1
Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor
Jul 21, 2025Siyuan Liu, Wenjing Liu, Zhiwei Xu +3LLM Hallucination Mitigation Hallucination in Language Models
College of Computer Science, Nankai University · College of intelligent Science and Technology, Inner Mongolia University of Technology · Institute of Computing Technology, Chinese Academy of Sciences +3

Sep 16, 2026J/K move · Enter open · S save

Closed-World Resolution Against Tool Hallucination in LLM Agents

Laxmipriya Ganesh Iyer

Independent Researcher

LLM Agent Evaluation LLM Tool Use

Open paper AskPDF arXiv

Tool-augmented large language model (LLM) agents fail in a way no tool-selection or tool-security method addresses: they call tools that do not exist and pass arguments no schema declares. Existing defenses either pick the right tool (selection) or constrain what an agent may do with real tools (gating), both of which presuppose the emitted call refers to a real tool at all. We show this is a structural blind spot: a hallucinated call is by construction not a decision any gate made, so no gate can reject it. This paper is primarily a measurement and benchmark study. We give a five-class taxonomy of tool hallucination (H1-H5) and, as a reference point, the Resolution Rung: a training-free, closed-world resolver (registry membership plus a signature check) whose interest is where it must sit, not what it computes. We prove hallucination defense must precede any causal gate, and characterize the one irreducible residue (borrowed arguments schema-indistinguishable from a valid call). Across ten hosted models under two invocation surfaces we measure 322 genuine hallucinations; fabricated-tool calls concentrate on the unconstrained raw-JSON surface (34 vs. 3), and model scale does not help (a 675B model matches a 7-8B one). We then extend to the Model Context Protocol, where merging several servers into one namespace creates hallucination surfaces a single registry cannot express (a second taxonomy, M1-M5); on the live MCP surface we measure 154 hallucinations, including from frontier models that were clean on the single-registry surface, because collisions and shadowing are structural to the merge. We release the versioned Hallucinated-Tools Benchmark (HTB) so any resolver is comparable across submissions.

Abstract

LLM Hallucination Mitigation

Explore similar work

Closed-World Resolution Against Tool Hallucination in LLM Agents

ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses

Towards Mitigation of Hallucination for LLM-empowered Agents: Progressive Generalization Bound Exploration and Watchdog Monitor