cs.SEOct 23, 2026

Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance

Authors: Antonio Ken Iannillo, Joshua Segun Owotogbe, Roberto Natella, Francesco Avallone, Kristina Kudryavtseva, Indika Kumara

Organizations: SnT, University of Luxembourg · Tilburg University · Gran Sasso Science Institute · University of Naples Federico II

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM agents increasingly rely on external tools exposed via the Model Context Protocol (MCP), yet current benchmarks assume tools respond correctly and on time, leaving agent resilience to realistic tool failures untested. We introduce MCP-Injector, a drop-in, protocol-aware fault-injection middleware that interposes on the MCP tool-call path with controlled persistence (permanent, transient, intermittent) and seedable schedules. Its fault model is calibrated to the observed incident distribution (failure mode x persistence) of the public MCP ecosystem, mined from 57,240 issue and pull-request threads across 1,774 open-source MCP repositories. In a 1,350-cell experiment crossing two backbone sizes (20B, 120B), three agent stacks (custom multi-round plan-and-execute, OpenAI Agents SDK, LangChain), and three fault injection conditions, we find that faults drawn from this permanent-heavy stress profile cause severe quality degradation (task-completion score falls by 60%, from 8.08 to 3.23 out of 10), and that scale does not purchase fault tolerance for the two gpt-oss backbones we test: the 120B retains no larger share of its fault-free score under faults (resilience-ratio model, p=0.96p = 0.96), and its completion rate under faults is, if anything, lower (91.1% vs. 95.6%; difference 4.4 points, 95% CI [−0.2,9.3][-0.2, 9.3]). Agent stacks differ in how well they absorb faults, which suggests that framework-level error handling matters more than model scale. These results demonstrate that capability benchmarks alone can mask operational brittleness, motivating protocol-aware fault injection as a complementary evaluation dimension for tool-calling agents.

Figures & tables

Figure 1: MCP-Injector architecture. The middleware transparently interposes on the MCP tool-call path. Each invocation passes through four stages: Observe (read tool name, arguments, and context), Decide (select an operator based on the fault profile, persistence state, and seeded randomness), Act (forward the call to the real server and optionally mutate the response), and Record (emit a structured log entry for post-hoc analysis).
IDOperatorClassEffect
OP0Null injectorn/aPassthrough (no mutation)
OP1Minor mistakeContentRandom character replacement
OP2Incorrect outputContentWord-level substitution
OP3Invalid outputContentType coercion (e.g., str →\to int)
OP4Explicit errorContentReturns MCP error message
OP5Delayed responseTimingArtificial sleep before return
Table 1: Fault operators implemented by MCP-Injector. Content operators act on the server’s response; timing operators act before or instead of forwarding.
Figure 2: Fault-profile calibration pipeline: from repository discovery to a seedable injection profile. Numbers indicate corpus size after each stage.
Failure ModeInterm.Perm.Trans.Uncert.Total
DELAYED_RESPONSE51157124233
EXPLICIT_ERROR6699,0759755210,393
INCORRECT_OUTPUT1551,7331501,939
INVALID_OUTPUT47511125584
MINOR_MISTAKE15177010202
SILENT_DROP1679634831,217
Table 2: Joint distribution of failure modes and persistence categories across 57,240 mined threads. NON_FAILURE and UNCERTAIN persistence entries are classified but excluded from the injection profile.
Figure 3: Failure mode distribution across all 450 OPP cells, stratified by stack and model. Partial degradation is the most common outcome (44 to 71% of cells); non-completion, catastrophic completion, graceful degradation, and tool bypass together account for 21 to 39%.
Figure 4: Stack sensitivity analysis. (a) OPP completion rates by stack and model. (b) Primary and completed-only resilience ratios by stack. (c) Stack ×\times condition interaction: MCPBench and OpenAI degrade more steeply than LangChain. (d) Post-failure response strategy differs markedly across stacks.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Operator (mode)PersistenceCountprob.act. pDerivation of act. p
OP5 (DELAYED_RESPONSE)INTERMITTENT510.00370.09750.3⋅51/1570.3\cdot 51/157
 PERMANENT1570.0114n/an/a
 TRANSIENT10.00011.0fixed
OP4 (EXPLICIT_ERROR)INTERMITTENT6690.04840.02210.3⋅669/90750.3\cdot 669/9075
 PERMANENT90750.6565n/an/a
 TRANSIENT970.00701.0fixed
Appendix
Table 3: Derivation of every entry in the OPP profile ( fault_mode_1.json ). Selection weights are count/N\mathrm{count}/N with N=13,824N=13{,}824 . INTERMITTENT activation probabilities follow Equation ( 2 ) with d=0.3d=0.3 ; TRANSIENT activation is fixed at 1.01.0 . Small residuals reflect four-decimal rounding in the shipped JSON.
Figure 5: Distribution of execution-time deltas (OP0 −- proxy-off) per task, by stack. Distributions are centered near zero with no systematic positive bias, confirming minimal proxy overhead.
Figure 6: Specified selection weights vs. realized share of fired events (1,905 events from 395 OPP cells) across (operator ×\times persistence) entries. Because a PERMANENT draw fires on every subsequent tool call, entries that sustain longer tool-call sequences (notably OP2/PERMANENT) are over-represented per event, and INTERMITTENT entries rarely fire. The title reports the per-event goodness-of-fit test.
ModelStackProxy-offOP0OPP
20bLangChain100.0 [95.1, 100.0]100.0 [95.1, 100.0]97.3 [90.8, 99.3]
MCPBench100.0 [95.1, 100.0]100.0 [95.1, 100.0]94.7 [87.1, 97.9]
OpenAI96.0 [88.9, 98.6]97.3 [90.8, 99.3]94.7 [87.1, 97.9]
Pooled98.7 [96.2, 99.5]99.1 [96.8, 99.8]95.6 [92.0, 97.6]
120bLangChain94.7 [87.1, 97.9]96.0 [88.9, 98.6]94.7 [87.1, 97.9]
MCPBench100.0 [95.1, 100.0]100.0 [95.1, 100.0]90.7 [82.0, 95.4]
Appendix
Table 4: Task completion rates (%) by model, stack, and proxy condition, with Wilson 95% CIs ( n=75n=75 cells per stack row, 225 per pooled row). Per-stack intervals overlap widely; the largest OPP drops are for the 120b under MCPBench and OpenAI.
MetricOP0OPPΔ\Delta% Drop
Task completion8.083.23−- 4.85−- 60
Tool selection8.103.36−- 4.74−- 59
Planning eff.7.512.49−- 5.02−- 67
Task fulfillment8.073.37−- 4.70−- 58
Grounded fulfillment †7.702.83−- 4.87−- 63
Grounding8.093.10−- 5.00−- 62
Appendix
Table 5: Score degradation under fault injection (OPP vs. OP0), pooled across stacks and models. Paired drops have 95% CIs within ±0.4\pm 0.4 points and Benjamini-Hochberg corrected Wilcoxon p<0.001p<0.001 for every metric. † Task fulfillment counted only when tool appropriateness is at least 2 (Section 6.1 ).
Figure 7: Per-task score degradation (OPP vs. OP0, % change) across all nine evaluation metrics. Tasks are ordered alphabetically. Darker cells indicate more severe degradation.
 MCPBenchOpenAI (Agents SDK)LangChain ( create_agent )
Orchestration patternMulti-round plan-and-execute with JSON planner outputSingle message loop with native function callingReAct state graph on LangGraph (agent/tools nodes)
Tool invocationStructured JSON prompt ( planned_tools list)Native function calling ( tool_choice required)Native tool calling on ChatOpenAI
Parallel tool callsExplicit per-round, per-server semaphoreSDK-level fan-outasyncio.gather in ToolNode
Tool-error handlingError text added to accumulated information bufferReturned as isError tool resultCaught by middleware, returned as tool message
Tool-call retry policyLLM decides (next round)LLM decides (next turn)LLM decides (next agent step)
Stop decisionExplicit should_continue Boolean per roundImplicit (assistant message with no tool calls)Implicit (AIMessage with no tool calls)
Appendix
Table 6: Intrinsic orchestration differences across the three stacks. Items listed as “LLM decides” indicate the stack has no built-in mechanism and delegates the decision to the model on the next orchestration step.
Figure 8: Post-failure behavior by stack. (a) Distribution of the immediate agent response after each fault-adjacent tool call, in percent of failed calls per stack; MCPBench is switch-dominant while LangChain and OpenAI are fallback-leaning. (b) Mean tool calls per task under OP0 versus OPP; within each stack the difference is small and non-significant (paired Wilcoxon, p≥0.09p\geq 0.09 ).
Figure 9: Task-domain sensitivity. (a) Per-task resilience ratio (median of OPP/OP0 across seeds) by stack, with tasks sorted by chain depth. (b) Task-median resilience ratio versus chain depth, labeled by task; the observed Spearman correlation is +0.29+0.29 , p=0.17p=0.17 . (c) Distribution of per-cell resilience ratios (capped at 2.0) by domain class; real-time API, stateful financial, and developer-tooling tasks cluster at the low-resilience end.
More in this topic

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 826 papers

Explore similar work

CardsList
Jul 13, 2026

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

Aritra Mazumder, Nusrat jahan Lia

University of Utah · University of Dhaka

AI Agent ReliabilityLLM Agent EvaluationLLM Agent VerificationTool-Using AgentsLLM Agent Reliability
AskPDF
Dec 31, 2025

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.

Zixiang Liu, Wenrui Liu, Elsie Dai +5

Peking University · ZTE

LLM Agent EvaluationAI Agent BenchmarksLLM Tool UseModel Context Protocol
AskPDF
Sep 21, 2026

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents

As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet underexplored factor. How funψtionality is decomposed into tools affects whether an agent can select the right tool and construct valid arguments. This choice is especially consequential at the edge, where resource constraints limit which models can run locally and scaling up is often not an option. We present MCP-GRANITE, an open-source extensible benchmark framework that treats tool-interface granularity as a controlled variable for MCP-based agents, evaluated under edge and IoT scenarios. It comprises 81 multi-step scenarios across 9 domains, instantiated at 4 granularity levels from fine-grained primitive tools to a single tool. We evaluate 9 locally deployed models (268M-20.9B parameters) across 8,748 trials using task completion, tool selection F1, argument accuracy, latency, and resource-usage metrics. Results show that a 4-tool interface offers the best trade-off, improving task completion by 16.4% over fine-grained primitives and 33.6% over a single monolithic tool, while nearly doubling argument accuracy. Model size is only weakly correlated with task completion and strongly with latency, while its association with argument accuracy is less robust, and a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one. These findings identify tool-interface granularity as a key design parameter for MCP-based agents.

Demetris Paschalides, Moysis Symeonides, George Pallis +1

Department of Computer Science University of Cyprus Nicosia, Cyprus

AI Agent BenchmarksTool-Using Agents
AskPDF