Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents

Abhinaba Basu

cs.CRMar 9, 2026

Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents

Authors: Abhinaba Basu

Abstract

AI agents that execute tasks via tool calls frequently hallucinate results - fabricating tool executions, misstating output counts, or presenting inferences as facts. Recent approaches to verifiable AI inference rely on zero-knowledge proofs, which provide cryptographic guarantees but impose minutes of proving time per query, making them impractical for interactive agents. We propose NabaOS, a lightweight verification framework inspired by Indian epistemology (Nyaya Shastra), which classifies every claim in an LLM response by its epistemic source (pramana): direct tool output (pratyaksha), inference (anumana), external testimony (shabda), absence (abhava), or ungrounded opinion. Our runtime generates HMAC-signed tool execution receipts that the LLM cannot forge, then cross-references claims against these receipts to detect hallucinations in real time. We evaluate on NyayaVerifyBench, a new benchmark of 1,800 agent response scenarios across four languages with injected hallucinations of six types. NabaOS detects 94.2% of fabricated tool references, 87.6% of count misstatements, and 91.3% of false absence claims, with <15ms verification overhead per response. For deep delegation (agents performing multi-step web tasks), our cross-checking protocol catches 78.4% of URL fabrications via independent re-fetching. We compare against five approaches: zkLLM (cryptographic proofs, 180s/query), TOPLOC (locality-sensitive hashing), SPEX (sampling-based proof of execution), tensor commitments, and self-consistency checking. NabaOS achieves the best cost-latency-coverage trade-off for interactive agents: 94.2% coverage at <15ms versus zkLLM's near-perfect coverage at 180,000ms. For interactive agents, practical receipt-based verification provides better cost-benefit than cryptographic proofs, and epistemic classification gives users actionable trust signals rather than binary judgments.

Figures & tables

Pramāṇa	Source	Verification Method	Confidence
Pratyakṣa	Direct tool output	Check against receipt	Highest
Anumāna	Inference from tool data	Check that premises exist	Medium
Upamāna	Analogy or comparison	Check that comparison is valid	Lower
Śabda	External source cited	Check that source was fetched	Depends on source
Abhāva	Absence claim	Check tool returned empty	Medium
Ungrounded	No evidence	Cannot verify—flag	Lowest

Table 1: Pramāṇa classification of LLM agent claims. Each claim is assigned a category based on its relationship to tool execution evidence, which determines the verification method applied.

Type	Description	Count/Lang
Fabricated Tool Call	LLM references a tool that was never executed	50
Count Mismatch	Tool returned $N$ results; LLM reports $M\neq N$	50
Fact Mismatch	Tool output states $X$ ; LLM reports $Y$ for the same field	50
Inference-as-Fact	LLM presents an inference as if directly from tool output	50
False Absence	Tool returned results; LLM claims “nothing found”	50
Source Fabrication	LLM cites a source URL it never fetched	50

Table 2: Hallucination types in NyayaVerifyBench . Each type appears 50 times per language, totaling 200 instances across the four languages.

Method	Detection Rate	FPR	Latency	Cost/req
No Verification	0%	0%	0 ms	$0
Self-Consistency	45%	12%	+3–5 s	+$0.03
RAG-Grounding	52%	18%	+1–2 s	+$0.01
Output Regex	35%	8%	+2 ms	$0
SVIP-style	68%	10%	+50 ms	$0
NabaOS (ours)	91%	4%	+12 ms	$0

Table 3: Main results on NyayaVerifyBench . Detection Rate is the percentage of injected hallucinations correctly identified. FPR is the false positive rate on clean control scenarios. Latency is the additional time per response. Cost is the additional monetary cost per request.

Hallucination Type	NabaOS	Self-Cons.	SVIP	Regex
Fabricated Tool Call	94.2%	38%	65%	20%
Count Mismatch	87.6%	52%	71%	45%
Fact Mismatch	89.1%	41%	68%	30%
Inference-as-Fact	82.3%	55%	72%	15%
False Absence	91.3%	35%	60%	25%
Source Fabrication	78.4%	48%	58%	12%

Table 4: Detection rate by hallucination type across methods.

Method	EN	HI	ZH	ES	Avg
NabaOS	92.8%	90.1%	88.7%	91.5%	90.8%
SVIP-style	72.4%	58.3%	55.1%	64.7%	62.6%
Self-Consistency	48.2%	39.7%	36.8%	43.1%	42.0%

Table 5: Detection rates by language. NabaOS ’s advantage is largest in non-English settings, where text-based methods degrade.

Trust Level	Actual Correctness	Calibration Error
Fully Verified	98.7%	1.3%
Mostly Verified	94.2%	3.1%
Partial	82.5%	5.8%
Unreliable	23.4%	N/A (correctly flagged)
Ungrounded	71.2%	N/A (opinion, no ground truth)

Table 6: Trust level calibration. Actual Correctness is the fraction of claims at each trust level that are factually correct according to ground truth.

Property	ZK Proofs	NabaOS Receipts
Proves that…	Model ran correctly	Claims are grounded
Verification overhead	Minutes per query	$<$ 15 ms per response
Hardware requirements	Specialized GPU	Any machine
Catches hallucination	No ∗	Yes
Language-independent	N/A	Yes
User-facing signal	Binary (valid/invalid)	5 trust levels

Table 7: Comparison of zero-knowledge proof approaches and NabaOS receipt-based verification.

Hallucination Detection

Hallucination detection identifies unsupported, fabricated, or evidence-inconsistent content in language and multimodal model outputs.

See all 78 papers

Aug 7, 2026

NiyamAI - An Intent-Bound AI Agent with Cryptographically Verifiable Guardrails using Zero-Knowledge Proofs

Giving an AI agent the ability to send emails, query databases, or execute commands is useful--until the agent is tricked into doing something it shouldn't. Prompt injection, hallucinated reasoning, and unsafe tool calls form the primary attack surface for autonomous LLM agents. Existing defenses rely on software checks like system prompts or policy filters running on the same machine the attacker targets, offering no verifiable proof of execution. We introduce Niyam-AI, a framework that makes safety enforcement provable. At session start, permitted tools and constraints are locked into an Intent Contract committed via SHA-256. Every tool call is intercepted and validated by an isolated Judge model; upon passing, a zk-SNARK proof is generated via EZKL. The tool executes only after proof verification, allowing third parties to confirm enforcement without accessing Judge model weights. Evaluating Niyam-AI on 2,000 real-world scenarios from Agent-SafetyBench against NeMo Guardrails, Meta's Llama Prompt Guard 2, and OpenAI's GPT-OSS-Safeguard using 5-fold stratified cross-validation yields an F1 score of 88.5% with a 1.1% false-positive rate (bootstrap 95% CI: [85.19%, 91.88%], N=1000). McNemar's exact paired test confirms significant improvement: Niyam-AI wins 390 discordant scenarios against NeMo (vs 20 losses), 115 against Prompt Guard 2 (vs 13), and 384 against GPT-OSS-Safeguard (vs 19) with p < 0.0001 in all cases. Proof generation adds 2260.6 +/- 218.4 ms per approved action, while verification takes 53.1 +/- 11.8 ms. Niyam-AI provides a guardrail that is both highly accurate and mathematically verifiable--though this reflects a classifier adapted to Agent-SafetyBench evaluated against zero-shot baselines, a distinction discussed in Section IV.C.

Aditya Katkar, Om Karkele, Kartik Mandhane +2

Department of Computer Engineering, Vishwakarma Institute of Technology, Pune, Maharashtra, India

Tool-Augmented Language Model Agents LLM Guardrails Zero-Knowledge ML Safety Filtering Runtime Enforcement for AI Agents Prompt Injection Defense+3

Ask PDF

Jul 20, 2026

Zero Hallucination, by Construction: Hallucination-Aware Layered Oversight for Trustworthy Enterprise AI

Enterprises will not deploy AI agents they cannot trust, and the most-cited reason for distrust is hallucination: confident, fluent output that is simply not true. The common response is to wait for a model that does not hallucinate. We argue that this is the wrong target. Large language models are, by construction, capable of generating unsupported text, and no amount of scale removes the possibility; a faithfulness judge bolted onto a raw model catches some errors but still ships others, and even well-curated retrieval pipelines have been shown to fabricate citations. We reframe the goal: "zero hallucination" is not a property a model possesses but a property a system enforces. We present HALO (Hallucination-Aware Layered Oversight), an assurance architecture which treats hallucination as a containable failure mode rather than an eliminable one. HALO composes six layers of defense: grounded generation over retrieved, approved content; constrained, deterministic execution that bounds where the model can err; multi-signal verification that scores every output for groundedness and hallucination using both an LLM judge and evidence-based checks against the source text; calibrated abstention, so the system declines rather than guesses when grounding is insufficient; total traceability of every retrieval, tool call, and generation; and continuous oversight that detects drift, alerts on threshold breaches, and closes the loop by regenerating and statistically validating improved agents. We detail each layer, give particular attention to evidence-based confidence (which verifies extractions against the source document rather than trusting the model's self-reported certainty), and illustrate the architecture on a regulated claims-extraction workload.

Bogdan Raduta, Horia Velicu, Alexandru Preda +1

AI Agent Reliability LLM Hallucination Mitigation LLM Guardrails LLM Hallucination Detection Trustworthy AI

Ask PDF

Oct 12, 2026

ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses

Tool-augmented language models ground their responses in external tool outputs, improving reliability beyond parametric knowledge alone. However, even when the correct tool is called and returns accurate information, models may still contradict such outputs, omit critical details, or generate unsupported claims. Existing hallucination benchmarks for tool-augmented dialogue primarily focus on the tool-calling stage, evaluating whether the correct tool and arguments are selected, while overlooking errors that arise during post-tool response generation. To address this gap, we introduce ToolHACE, a span-level benchmark for hallucination detection in tool-augmented LLM responses, covering five fine-grained hallucination categories. Through extensive experiments with modern hallucination detectors, we show that post-tool-call hallucinations are widespread and remain challenging to detect. Our results further demonstrate that effective detection requires training on specialized tool-augmented corpora, while transfer from tool-agnostic hallucination datasets performs poorly in this setting. These findings highlight the need for dedicated benchmarks and models for reliable hallucination detection in tool-augmented language systems.

Nikolay Ivanov, Sergey Pletenev, Elisei Rykov +8

Skoltech · AIRI · T-Tech +1

Hallucination Detection LLM Hallucination Detection LLM Tool Use

Ask PDF

Abstract

Hallucination Detection

Explore similar work

NiyamAI - An Intent-Bound AI Agent with Cryptographically Verifiable Guardrails using Zero-Knowledge Proofs

Zero Hallucination, by Construction: Hallucination-Aware Layered Oversight for Trustworthy Enterprise AI

ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses