ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses
Authors: Nikolay Ivanov, Sergey Pletenev, Elisei Rykov, Anatolii Frolov, Bogdan Monogov, Alexander Potekhin, Maria Marina, Daniil Moskovskiy, Viktor Moskvoretskii, Alexander Panchenko, Ilseyar Alimova
Tool-augmented language models ground their responses in external tool outputs, improving reliability beyond parametric knowledge alone. However, even when the correct tool is called and returns accurate information, models may still contradict such outputs, omit critical details, or generate unsupported claims. Existing hallucination benchmarks for tool-augmented dialogue primarily focus on the tool-calling stage, evaluating whether the correct tool and arguments are selected, while overlooking errors that arise during post-tool response generation. To address this gap, we introduce ToolHACE, a span-level benchmark for hallucination detection in tool-augmented LLM responses, covering five fine-grained hallucination categories. Through extensive experiments with modern hallucination detectors, we show that post-tool-call hallucinations are widespread and remain challenging to detect. Our results further demonstrate that effective detection requires training on specialized tool-augmented corpora, while transfer from tool-agnostic hallucination datasets performs poorly in this setting. These findings highlight the need for dedicated benchmarks and models for reliable hallucination detection in tool-augmented language systems.
Figures & tables
Figure 1: Main types of generation errors in tool-augmented LLM responses. A model can produce a correct answer, contradict the tool output, add unsupported information, omit a tool-provided fact, or introduce an unrelated tool/action request.
Figure 2: Dataset generation pipeline. ToolACE dialogues are split into single-hop and multi-hop traces. Hallucinations are collected through LLM-based synthetic edits, programmatic JSON perturbations, and naturally occurring ( wild ) generations from small LLMs. The resulting examples are filtered by multiple judges and subsequently refined through quality-control, split isolation, and normalization.
Resource
Tool-calling
Span-level
setting
hallucination labels
ToolBench, BFCL, ToolACE, APIBench,
APIGen, ToolAlpaca, StableToolBench,
ToolHop, T-Eval, MetaTool
ToolEmu, InjecAgent, ToolBH
RAGTruth, FAVA, PsiloQA, HAD
Table 1: Existing resources cover either the tool-calling setting or span-level hallucination annotation, but not both. Our resource combines them.
Class
Definition
Correct
Answer is faithful to the tool response and fully addresses the query.
Answer Mismatch
Answer asserts a value or fact that the tool response contradicts .
Overgeneration
Answer adds content with no support in the tool response (e.g. elaboration, background, general knowledge), without claiming any tool-mediated action.
Missing Tool
Answer references an action or capability that no available tool can supply (e.g. offering to book, send, fetch, or compute when no such tool exists).
Undergeneration
Answer omits information from the tool response that the query required.
Table 2: The five classes in ToolHACE. Boundaries are designed to be mutually exclusive: Answer Mismatch requires a contradicting value in the tool response; Overgeneration adds free-form content; Missing Tool specifically claims tool-mediated capability.
Figure 3: Examples of Route B Hallucination Insertion in ToolHACE. Type 1a (Answer Mismatch): individual leaf values in the tool response are replaced with random values from other samples—each field is corrupted independently. Type 1b (Answer Mismatch): all values within a JSON subtree are replaced together, preserving sibling branches. Type 2 (Overgeneration): fields are deleted from the tool response, causing the model to fabricate values it can no longer ground. Type 3 (Missing Tool): the model appends a factually unsupported sentence referencing context absent from the tool response.
Type
Train
Test
clean
16,335
5,252
overgeneration
7,511
2,316
missing_tool
4,931
1,521
answer_mismatch
3,702
1,196
undergeneration
4,656
1,321
Total
37,135
11,606
Table 3: Overall ToolHACE dataset statistics across hallucination types and train/test splits.
Span-level
Response-level
Setting
Data
Model
Mis-match
Over-gen.
Missing tool
Avg.
Correct
Mis-match
Over-gen.
Missing tool
Avg.
Under-gen. ∗
Zero-Shot
–
GPT-4o
50.99
68.87
77.52
65.79
76.41
55.15
68.89
59.37
64.95
57.18
GPT-5.4-mini
52.77
66.82
76.94
65.51
76.22
54.69
67.89
59.20
64.50
37.79
Kimi 2.6
56.07
73.84
73.60
67.84
77.46
55.81
70.77
59.58
65.90
58.13
Few-Shot
–
GPT-4o
53.21
68.65
76.44
66.10
75.21
53.88
68.56
57.65
63.83
57.17
GPT-5.4-mini
52.83
64.07
73.75
63.55
63.47
45.57
61.62
50.35
55.25
41.69
Table 4: F1 performance on the ToolHACE test dataset by hallucination type. At the span level, Correct is not evaluated, as span-level metrics are computed only for hallucinated examples where the model correctly identifies the hallucination type. Undergen. is reported only for decoder-based models, as undergeneration has no corresponding annotated span and encoder-based models therefore produce an empty span for this category.
Type
Synth (%)
Real-World (%)
overgeneration
35.0
29.8
missing_tool
32.2
23.3
undergeneration
22.2
32.7
answer_mismatch
10.6
14.2
# hallucinated samples
4,693
2,155
Table 5: Type distribution among positives: ToolHACE test split vs. real-world hallucinated responses.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Span-level
Response-level
Data
Classifier
Mis-match
Over-gen.
Missing tool
Avg.
Correct
Mis-match
Over-gen.
Missing tool
Avg.
Under-gen. ∗
CNNDM
Span-level (LR)
0.08
0
0
0.02
37.55
1.42
1.04
1.02
8.94
-
NQ
Span-level (LR)
4.11
4.33
7.40
3.19
0
18.53
51.76
22.33
24.97
-
CNNDM
Window-level (LR)
4.45
8.41
12.7
5.81
2.02
18.63
52.12
22.49
25.32
-
NQ
Window-level (LR)
0.78
2.27
3.05
1.79
35.73
19.75
38.41
19.00
25.42
-
ToolHACE split
Span-level (LR)
5.08
3.73
8.87
3.19
0.08
18.54
51.78
22.34
24.98
-
Appendix
Table 6: Ablation study results of the Lookback-Lens method. LR stands for the logistic regression.