GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways

Yitian Zheng; Zhaorun Chen; Haoyu Li; Bo Li

cs.AIOct 16, 2026

GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways

Authors: Yitian Zheng, Zhaorun Chen, Haoyu Li, Bo Li

Organizations: Tsinghua University · University of Chicago · University of Illinois Urbana-Champaign

Abstract

AI agents have increasingly interacted with external environments by invoking tools, with Model Context Protocol (MCP) emerging as a standard interface for connecting agents to heterogeneous tools and services. As organizations deploy agents at scale, MCP gateways have become a critical middleware layer for aggregating, governing, and shaping the tool surface exposed to agents. Despite the growing importance of MCP gateways in agentic systems, no prior work has systematically studied how gateway design affects agent capability, efficiency, and security. To address this gap, we introduce GatewayBench, the first unified evaluation framework and benchmark for MCP gateways. We characterize gateway architectures into two major paradigms: flattened gateways and hierarchical gateways. GatewayBench comprises 524 tasks across six domains, together with a pool of 11,118 real-world tools from 251 authentic MCP servers. To emulate large-scale deployments, we systematically inject task-irrelevant tools sampled from this pool into each agent environment, enabling controlled evaluation as the accessible tool surface scales. Based on GatewayBench, we conduct extensive experiments across seven widely deployed MCP gateways and diverse agent configurations, covering both benign and adversarial tasks. Our results reveal substantial differences across gateway designs in task completion, tool-use efficiency, and resilience to attacks, and further demonstrate that these effects interact strongly with the underlying model and agent harness. These findings highlight the MCP gateway as a consequential yet underexplored component of the agentic AI stack. By enabling controlled evaluation of gateway architectures under realistic tool ecosystems, GatewayBench provides both practical guidance for MCP gateway design and a rigorous testbed for developing more capable, efficient, and secure agentic systems.

Figures & tables

Figure 1: Overview of the GatewayBench pipeline. Each sampled benign or malicious task exposes its MCP servers, plus injected task-irrelevant noisy tools, to the agent through the gateway under test. A verifiable judge inspects the final environment state to compute task success rate (TSR), attack success rate (ASR), task time, and tool-call latency.

Figure 2: Flattened vs. hierarchical MCP gateways on a travel-booking task example. The flattened gateway exposes tools from all connected servers as a single flat list. The hierarchical gateway instead exposes tool-discovery meta-tools, letting the agent reach the relevant tools through two alternative mechanisms: (A) layered discovery, or (B) direct keyword search over the task intent.

Mechanism	Gateway	Open Source	Tool Name Collision Handling	Server Ordering	Token Efficiency	Search Tool
Flattened	agentcore	✗	✓	✗	✗	✓
	agentgateway	✓	✓	✗	✗	✗
	litellm	✓	✓	✓	✗	✗
	lunar	✓	✓	✓	✗	✗
Hierarchical	enkrypt	$\triangle$	/	/	$\triangle$	✗
	strata	✓	/	/	✓	✓

Table 1: Comparison of MCP gateways evaluated in this work. The table summarizes the key design characteristics of each gateway under its default configuration.

\triangle

denotes partial or limited support, and "/" denotes a dimension that does not apply.

Figure 3: Overview of GatewayBench benchmark. The GatewayBench comprises a task dataset of 524 tasks spanning 6 domains from two datasets, DTap and MCP-Universe, alongside a real-world tool pool of 11,118 noisy tools from 251 authentic MCP servers, enabling systematic evaluation across diverse workflows and adversarial scenarios.

Gateway	OpenAI SDK (GPT-5.1)				OpenAI SDK (GPT-5.4)				OpenClaw (GPT-5.1)
Gateway	TSR	ASR d	TSR ind	ASR ind	TSR	ASR d	TSR ind	ASR ind	TSR	ASR d	TSR ind	ASR ind
Flattened
agentcore	78.0	79.3	54.0	58.7	85.3	50.0	60.7	48.7	89.3	48.0	60.0	42.7
agentgateway	73.3	73.3	54.7	61.3	86.7	43.3	56.7	46.0	84.0	50.7	64.0	41.3
litellm	75.3	76.0	52.0	63.3	87.3	40.7	57.3	46.7	86.0	50.0	63.3	40.7
lunar	74.7	77.3	52.0	58.7	89.3	42.7	57.3	47.3	87.3	48.0	64.7	45.3

Table 2: Main results: agent capability and security under each gateway, with no noisy tools injected. All values are percentages; Mean rows average the gateways within each architecture type. Bold marks the best value per column: highest TSR/TSR ind and lowest ASR d /ASR ind .

	OpenAI SDK (GPT-5.1)				OpenAI SDK (GPT-5.4)				OpenClaw (GPT-5.1)
	0	500	1000	1500	0	500	1000	1500	0	500	1000	1500
agentcore	78.0	75.3	72.0	69.3	85.3	89.3	88.7	85.3	89.3	86.7	82.7	77.3
agentgateway	73.3	78.7	76.7	72.0	86.7	87.3	86.7	87.3	84.0	89.3	82.7	84.7
litellm	75.3	70.7	72.0	57.3	87.3	88.0	82.0	82.7	86.0	86.7	80.7	77.3
lunar	74.7	75.3	67.3	62.7	89.3	84.0	89.3	78.0	87.3	90.7	85.3	83.3
Mean	75.3	75.0	72.0	65.3	87.2	87.2	86.7	83.3	86.7	88.4	82.9	80.7

Table 3: Benign task success rate (%) versus the number of injected noisy tools. Within each agent configuration, the sub-columns

0

500

1{,}000

, and

1{,}500

give the number of injected noisy tools. Mean rows average the gateways within each architecture type.

Panel of Figure 4 — Figure 4: Indirect attack success rate for environment injection and tool injection under OpenClaw with GPT-5.1, aggregated into the two architecture types. Each bar is the architecture mean; the error bar spans the per-gateway minimum and maximum within the architecture.

Panel of Figure 5 — Figure 5: Interaction cost as the tool set scales. task duration (left) and average tokens per turn (right) of different gateways under the OpenAI Agents SDK with GPT-5.4.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Gateway	158 Noisy Tools		254 Noisy Tools		358 Noisy Tools
Gateway	Score	TSR (%)	Score	TSR (%)	Score	TSR (%)
agentgateway	0.73	52.3	0.66	52.0	0.75	54.0
enkrypt	0.60	34.7	0.62	36.0	0.52	23.3
strata	0.68	44.7	0.68	46.0	0.68	43.7

Appendix

Table 4: Real-world tool scaling on MCP-Universe (OpenAI Agents SDK, GPT-5.1) for three representative gateways. TSR is the fraction of tasks that pass all execution-based checkpoints, and Score is the average fraction of per-task checkpoints passed (partial credit); higher is better for both.

Panel of Figure 6 — Figure 6: Indirect attack success rate by injection type as the tool set scales: environment injection (top) and tool injection (bottom), for each agent configuration (columns). One line per gateway; flattened gateways are dashed, hierarchical gateways solid.

Panel of Figure 7 — Figure 7: Interaction cost as the tool set scales, under the OpenAI Agents SDK with GPT-5.1 (top) and GPT-5.4 (bottom): task time (left), average tool-call latency (middle), and tokens per turn (right). Flattened gateways are dashed, hierarchical gateways solid.

Agent	Search Used
OpenAI SDK (GPT-5.1)	11
OpenAI SDK (GPT-5.4)	19
OpenClaw (GPT-5.1)	1

Appendix

Table 5: Number of task runs in which the AgentCore search tool was invoked. Each agent includes 1,800 task runs.

Model Context Protocol

Model Context Protocol (MCP) connects AI agents to external tools and data through a shared interface.

See all 32 papers

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Dec 31, 2025Zixiang Liu, Wenrui Liu, Elsie Dai +5LLM Agent Evaluation AI Agent Benchmarks
Peking University · ZTE
DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers
Jul 10, 2026Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov +4Computer-Use Agent Benchmarks LLM Agent Evaluation
ITMO University · Central University
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
Sep 21, 2026Demetris Paschalides, Moysis Symeonides, George Pallis +1AI Agent Benchmarks Tool-Using Agents
Department of Computer Science University of Cyprus Nicosia, Cyprus

Dec 31, 2025J/K move · Enter open · S save

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

Zixiang Liu, Wenrui Liu, Elsie Dai, Wenhan Yu+4

Peking University · ZTE

LLM Agent Evaluation AI Agent Benchmarks LLM Tool Use Model Context Protocol

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.

Abstract

Model Context Protocol

Explore similar work

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents