Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance

Antonio Ken Iannillo; Joshua Segun Owotogbe; Roberto Natella; Francesco Avallone; Kristina Kudryavtseva; Indika Kumara

cs.SEOct 23, 2026

Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance

Authors: Antonio Ken Iannillo, Joshua Segun Owotogbe, Roberto Natella, Francesco Avallone, Kristina Kudryavtseva, Indika Kumara

Organizations: SnT, University of Luxembourg · Tilburg University · Gran Sasso Science Institute · University of Naples Federico II

Abstract

LLM agents increasingly rely on external tools exposed via the Model Context Protocol (MCP), yet current benchmarks assume tools respond correctly and on time, leaving agent resilience to realistic tool failures untested. We introduce MCP-Injector, a drop-in, protocol-aware fault-injection middleware that interposes on the MCP tool-call path with controlled persistence (permanent, transient, intermittent) and seedable schedules. Its fault model is calibrated to the observed incident distribution (failure mode x persistence) of the public MCP ecosystem, mined from 57,240 issue and pull-request threads across 1,774 open-source MCP repositories. In a 1,350-cell experiment crossing two backbone sizes (20B, 120B), three agent stacks (custom multi-round plan-and-execute, OpenAI Agents SDK, LangChain), and three fault injection conditions, we find that faults drawn from this permanent-heavy stress profile cause severe quality degradation (task-completion score falls by 60%, from 8.08 to 3.23 out of 10), and that scale does not purchase fault tolerance for the two gpt-oss backbones we test: the 120B retains no larger share of its fault-free score under faults (resilience-ratio model, $p = 0.96$ ), and its completion rate under faults is, if anything, lower (91.1% vs. 95.6%; difference 4.4 points, 95% CI $[-0.2, 9.3]$ ). Agent stacks differ in how well they absorb faults, which suggests that framework-level error handling matters more than model scale. These results demonstrate that capability benchmarks alone can mask operational brittleness, motivating protocol-aware fault injection as a complementary evaluation dimension for tool-calling agents.

Figures & tables

Figure 1: MCP-Injector architecture. The middleware transparently interposes on the MCP tool-call path. Each invocation passes through four stages: Observe (read tool name, arguments, and context), Decide (select an operator based on the fault profile, persistence state, and seeded randomness), Act (forward the call to the real server and optionally mutate the response), and Record (emit a structured log entry for post-hoc analysis).

ID	Operator	Class	Effect
OP0	Null injector	n/a	Passthrough (no mutation)
OP1	Minor mistake	Content	Random character replacement
OP2	Incorrect output	Content	Word-level substitution
OP3	Invalid output	Content	Type coercion (e.g., str $\to$ int)
OP4	Explicit error	Content	Returns MCP error message
OP5	Delayed response	Timing	Artificial sleep before return

Table 1: Fault operators implemented by MCP-Injector. Content operators act on the server’s response; timing operators act before or instead of forwarding.

Figure 2: Fault-profile calibration pipeline: from repository discovery to a seedable injection profile. Numbers indicate corpus size after each stage.

Failure Mode	Interm.	Perm.	Trans.	Uncert.	Total
DELAYED_RESPONSE	51	157	1	24	233
EXPLICIT_ERROR	669	9,075	97	552	10,393
INCORRECT_OUTPUT	155	1,733	1	50	1,939
INVALID_OUTPUT	47	511	1	25	584
MINOR_MISTAKE	15	177	0	10	202
SILENT_DROP	167	963	4	83	1,217

Table 2: Joint distribution of failure modes and persistence categories across 57,240 mined threads. NON_FAILURE and UNCERTAIN persistence entries are classified but excluded from the injection profile.

Figure 3: Failure mode distribution across all 450 OPP cells, stratified by stack and model. Partial degradation is the most common outcome (44 to 71% of cells); non-completion, catastrophic completion, graceful degradation, and tool bypass together account for 21 to 39%.

Figure 4: Stack sensitivity analysis. (a) OPP completion rates by stack and model. (b) Primary and completed-only resilience ratios by stack. (c) Stack

\times

condition interaction: MCPBench and OpenAI degrade more steeply than LangChain. (d) Post-failure response strategy differs markedly across stacks.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Operator (mode)	Persistence	Count	prob.	act. p	Derivation of act. p
OP5 (DELAYED_RESPONSE)	INTERMITTENT	51	0.0037	0.0975	$0.3\cdot 51/157$
	PERMANENT	157	0.0114	n/a	n/a
	TRANSIENT	1	0.0001	1.0	fixed
OP4 (EXPLICIT_ERROR)	INTERMITTENT	669	0.0484	0.0221	$0.3\cdot 669/9075$
	PERMANENT	9075	0.6565	n/a	n/a
	TRANSIENT	97	0.0070	1.0	fixed

Appendix

Table 3: Derivation of every entry in the OPP profile ( fault_mode_1.json ). Selection weights are

\mathrm{count}/N

with

N=13{,}824

. INTERMITTENT activation probabilities follow Equation ( 2 ) with

d=0.3

; TRANSIENT activation is fixed at

1.0

. Small residuals reflect four-decimal rounding in the shipped JSON.

Figure 5: Distribution of execution-time deltas (OP0

-

proxy-off) per task, by stack. Distributions are centered near zero with no systematic positive bias, confirming minimal proxy overhead.

Figure 6: Specified selection weights vs. realized share of fired events (1,905 events from 395 OPP cells) across (operator

\times

persistence) entries. Because a PERMANENT draw fires on every subsequent tool call, entries that sustain longer tool-call sequences (notably OP2/PERMANENT) are over-represented per event, and INTERMITTENT entries rarely fire. The title reports the per-event goodness-of-fit test.

Model	Stack	Proxy-off	OP0	OPP
20b	LangChain	100.0 [95.1, 100.0]	100.0 [95.1, 100.0]	97.3 [90.8, 99.3]
	MCPBench	100.0 [95.1, 100.0]	100.0 [95.1, 100.0]	94.7 [87.1, 97.9]
	OpenAI	96.0 [88.9, 98.6]	97.3 [90.8, 99.3]	94.7 [87.1, 97.9]
	Pooled	98.7 [96.2, 99.5]	99.1 [96.8, 99.8]	95.6 [92.0, 97.6]
120b	LangChain	94.7 [87.1, 97.9]	96.0 [88.9, 98.6]	94.7 [87.1, 97.9]
	MCPBench	100.0 [95.1, 100.0]	100.0 [95.1, 100.0]	90.7 [82.0, 95.4]

Appendix

Table 4: Task completion rates (%) by model, stack, and proxy condition, with Wilson 95% CIs (

n=75

cells per stack row, 225 per pooled row). Per-stack intervals overlap widely; the largest OPP drops are for the 120b under MCPBench and OpenAI.

Metric	OP0	OPP	$\Delta$	% Drop
Task completion	8.08	3.23	$-$ 4.85	$-$ 60
Tool selection	8.10	3.36	$-$ 4.74	$-$ 59
Planning eff.	7.51	2.49	$-$ 5.02	$-$ 67
Task fulfillment	8.07	3.37	$-$ 4.70	$-$ 58
Grounded fulfillment †	7.70	2.83	$-$ 4.87	$-$ 63
Grounding	8.09	3.10	$-$ 5.00	$-$ 62

Appendix

Table 5: Score degradation under fault injection (OPP vs. OP0), pooled across stacks and models. Paired drops have 95% CIs within

\pm 0.4

points and Benjamini-Hochberg corrected Wilcoxon

p<0.001

for every metric. † Task fulfillment counted only when tool appropriateness is at least 2 (Section 6.1 ).

Figure 7: Per-task score degradation (OPP vs. OP0, % change) across all nine evaluation metrics. Tasks are ordered alphabetically. Darker cells indicate more severe degradation.

	MCPBench	OpenAI (Agents SDK)	LangChain ( create_agent )
Orchestration pattern	Multi-round plan-and-execute with JSON planner output	Single message loop with native function calling	ReAct state graph on LangGraph (agent/tools nodes)
Tool invocation	Structured JSON prompt ( planned_tools list)	Native function calling ( tool_choice required)	Native tool calling on ChatOpenAI
Parallel tool calls	Explicit per-round, per-server semaphore	SDK-level fan-out	asyncio.gather in ToolNode
Tool-error handling	Error text added to accumulated information buffer	Returned as isError tool result	Caught by middleware, returned as tool message
Tool-call retry policy	LLM decides (next round)	LLM decides (next turn)	LLM decides (next agent step)
Stop decision	Explicit should_continue Boolean per round	Implicit (assistant message with no tool calls)	Implicit (AIMessage with no tool calls)

Appendix

Table 6: Intrinsic orchestration differences across the three stacks. Items listed as “LLM decides” indicate the stack has no built-in mechanism and delegates the decision to the model on the next orchestration step.

Figure 8: Post-failure behavior by stack. (a) Distribution of the immediate agent response after each fault-adjacent tool call, in percent of failed calls per stack; MCPBench is switch-dominant while LangChain and OpenAI are fallback-leaning. (b) Mean tool calls per task under OP0 versus OPP; within each stack the difference is small and non-significant (paired Wilcoxon, $p\geq 0.09$ ).

Figure 9: Task-domain sensitivity. (a) Per-task resilience ratio (median of OPP/OP0 across seeds) by stack, with tasks sorted by chain depth. (b) Task-median resilience ratio versus chain depth, labeled by task; the observed Spearman correlation is $+0.29$ , $p=0.17$ . (c) Distribution of per-cell resilience ratios (capped at 2.0) by domain class; real-time API, stateful financial, and developer-tooling tasks cluster at the low-resilience end.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 832 papers

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP
Jul 13, 2026Aritra Mazumder, Nusrat jahan LiaAI Agent Reliability LLM Agent Evaluation
University of Utah · University of Dhaka
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Dec 31, 2025Zixiang Liu, Wenrui Liu, Elsie Dai +5LLM Agent Evaluation AI Agent Benchmarks
Peking University · ZTE
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
Sep 21, 2026Demetris Paschalides, Moysis Symeonides, George Pallis +1AI Agent Benchmarks Tool-Using Agents
Department of Computer Science University of Cyprus Nicosia, Cyprus

Jul 13, 2026J/K move · Enter open · S save

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

Aritra Mazumder, Nusrat jahan Lia

University of Utah · University of Dhaka

AI Agent Reliability LLM Agent Evaluation LLM Agent Verification Tool-Using Agents LLM Agent Reliability

Open paper AskPDF arXiv

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

Abstract

LLM Agent Reliability

Explore similar work

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents