MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents
Organizations: Peking University · Zhongguancun Academy · Meituan‡ · The Hong Kong University of Science and Technology ·
Abstract
Large language model agents are increasingly deployed through the Model Context Protocol (MCP), which allows them to discover and invoke tools across multiple servers at runtime. Existing MCP benchmarks primarily measure task completion; where they diagnose failures, they do so after execution rather than building specific reasoning, execution, and memory failure risks into task construction. We introduce MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory. MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers. Each task is designed with a structural trigger that makes one of four subtypes the dominant failure risk: Branch Collapse, Unreachable Goal, Tool Misuse, or Context Forgetting. This design enables subtype-level analysis rather than aggregate task success alone. We evaluate fourteen frontier LLM agents from eight vendors on 5{,}012 trajectories. Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642 on a scale (higher is better). Failures concentrate on Unreachable Goal tasks, where the average score drops to 0.458 and 23.2% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals. Subtype rankings diverge sharply: a model's rank shifts by 6.9 of 13 positions on average across subtypes, and Branch Collapse is uncorrelated with Unreachable Goal across models (). MCPHallu is released at https://github.com/J1zz/mcp_hallu.
Figures & tables
| Benchmark | Multi-server | Stateful | Long-horizon | Subtype-level attribution |
|---|---|---|---|---|
| ToolBH | ✗ | ✗ | ✗ | partial |
| RelyToolBench | ✗ | ✗ | ✗ | partial |
| -bench | ✗ | ✓ | ✓ | ✗ |
| WebArena | ✗ | ✓ | ✓ | ✗ |
| AgentBench | ✗ | partial | ✓ | ✗ |
| MCP-Atlas ( Bandi et al., 2026 ) | ✓ | partial | ✓ | post-hoc |
| Subtype | Judge | Outcome | Dependency | Early-Stop | Confusion |
|---|---|---|---|---|---|
| Branch Collapse | |||||
| Context Forgetting | |||||
| Unreachable Goal | |||||
| Tool Misuse |
| Calls | Runs | Pass% | Mean | Comp.% |
|---|---|---|---|---|
| 0 | 320 | 65.0 | 0.778 | 0.6 |
| 1–3 | 455 | 27.9 | 0.569 | 13.8 |
| 4–7 | 266 | 0.0 | 0.342 | 30.1 |
| 8–15 | 158 | 0.0 | 0.201 | 36.7 |
| 16+ | 243 | 0.0 | 0.121 | 19.8 |
| Type | Runs | Pass% | Mean | Steps | Conf. calls |
|---|---|---|---|---|---|
| Correct only | 825 | 84.7 | 0.927 | 3.54 | 0.00 |
| Correct + conf. | 260 | 32.3 | 0.649 | 10.70 | 3.42 |
| Conf. only | 16 | 0.0 | 0.348 | 3.38 | 1.63 |
| Neither | 75 | 8.0 | 0.364 | 0.37 | 0.00 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Intended / Observed | BC | UG | TM | CF | |
|---|---|---|---|---|---|
| Branch Collapse | 16 | 1 | 2 | 2 | 21 |
| Unreachable Goal | 0 | 21 | 0 | 0 | 21 |
| Tool Misuse | 0 | 4 | 13 | 0 | 17 |
| Context Forgetting | 0 | 3 | 1 | 17 | 21 |
| Role | Model | Stage |
|---|---|---|
| Task generator | Claude Sonnet 4.6 | Adversarial synthesis (§ 4.3 ) |
| Repair and filtering solver | Claude Code (Claude Sonnet 4.6) | Automated filtering (Appendix D.3 ) |
| Semantic judge | GPT-4o | Evaluation (§ 4.4 ) |
| Cross-validation judge | Claude Sonnet 4.5 | Judge validation (Appendix H ) |
| Use case | Tasks | Share |
|---|---|---|
| Financial / market lookup | 73 | 20% |
| Code review / repository operations | 73 | 20% |
| Trip / local-information planning | 72 | 20% |
| Reporting / data analysis | 63 | 18% |
| Literature / paper search | 55 | 15% |
| Record / file / calendar writes | 16 | 4% |
| Model | API Provider | Native context | Temp |
|---|---|---|---|
| GPT-4o | OpenAI | 128K | 1.0 |
| GPT-5-chat | OpenAI | 400K | 1.0 |
| Claude Haiku 4.5 | Anthropic | 200K | 1.0 |
| Claude Sonnet 4.5 | Anthropic | 200K | 1.0 |
| Claude Sonnet 4.6 | Anthropic | 200K | 1.0 |
| Claude Opus 4.5 | Anthropic | 200K | 1.0 |
| Agent behavior | #Runs | Pass% |
|---|---|---|
| Keeps acting, no infeasibility declared (4+ calls) | 307 | 0.0 |
| Keeps acting, no infeasibility declared (1–3 calls) | 210 | 0.0 |
| Stops and declares infeasibility | 448 | 48.0 |
| Final-response class | #Runs | Pass% | Mean score | Avg. steps |
|---|---|---|---|---|
| Completion only | 195 | 2.1 | 0.199 | 9.98 |
| Limitation only | 304 | 44.4 | 0.668 | 4.82 |
| Completion + limitation | 83 | 18.1 | 0.514 | 9.08 |
| Neither | 784 | 23.0 | 0.442 | 7.64 |
| Regime | #Runs | Pass% | Mean score | Avg. steps |
|---|---|---|---|---|
| Stateless ( General / Analytics / Financial ) | 644 | 59.2 | 0.713 | 7.17 |
| Stateful ( Productivity / Coding ) | 504 | 24.6 | 0.431 | 11.46 |
| Model | Branch | Context | Unreach. | Tool | Spread |
|---|---|---|---|---|---|
| Claude Haiku 4.5 | 3 | 9 | 12 | 7 | 9 |
| Claude Opus 4.5 | 1 | 7 | 7 | 6 | 6 |
| Claude Opus 4.6 | 7 | 1 | 2 | 1 | 6 |
| Claude Sonnet 4.5 | 6 | 4 | 3 | 2 | 4 |
| Claude Sonnet 4.6 | 9 | 3 | 1 | 3 | 8 |
| DeepSeek-V3.1 | 5 | 2 | 11 | 10 | 9 |
| Subtype | General | Analytics | Financial | Productivity | Coding |
|---|---|---|---|---|---|
| Tool Misuse | 0.886 | 0.859 | 0.886 | 0.605 | 0.775 |
| Context Forgetting | 0.599 | 0.832 | 0.695 | 0.396 | 0.474 |
| Branch Collapse | 0.705 | 0.687 | 0.808 | 0.598 | 0.809 |
| Unreachable Goal | 0.562 | 0.421 | 0.542 | 0.565 | 0.246 |
| Failure mode | Requirement |
|---|---|
| Faulty invocation (appropriate tool, invalid arguments) | Current harness |
| Feasible near-miss controls for Unreachable Goal | Current harness |
| Cross-server protocol mismatch | Current harness |
| Authentication-state confusion | Current harness |
| Planning beyond the 20-call budget | Current harness, larger call budget |
| Multi-agent coordination | Multi-agent runtime |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,304 papers