AI agents have increasingly interacted with external environments by invoking tools, with Model Context Protocol (MCP) emerging as a standard interface for connecting agents to heterogeneous tools and services. As organizations deploy agents at scale, MCP gateways have become a critical middleware layer for aggregating, governing, and shaping the tool surface exposed to agents. Despite the growing importance of MCP gateways in agentic systems, no prior work has systematically studied how gateway design affects agent capability, efficiency, and security. To address this gap, we introduce GatewayBench, the first unified evaluation framework and benchmark for MCP gateways. We characterize gateway architectures into two major paradigms: flattened gateways and hierarchical gateways. GatewayBench comprises 524 tasks across six domains, together with a pool of 11,118 real-world tools from 251 authentic MCP servers. To emulate large-scale deployments, we systematically inject task-irrelevant tools sampled from this pool into each agent environment, enabling controlled evaluation as the accessible tool surface scales. Based on GatewayBench, we conduct extensive experiments across seven widely deployed MCP gateways and diverse agent configurations, covering both benign and adversarial tasks. Our results reveal substantial differences across gateway designs in task completion, tool-use efficiency, and resilience to attacks, and further demonstrate that these effects interact strongly with the underlying model and agent harness. These findings highlight the MCP gateway as a consequential yet underexplored component of the agentic AI stack. By enabling controlled evaluation of gateway architectures under realistic tool ecosystems, GatewayBench provides both practical guidance for MCP gateway design and a rigorous testbed for developing more capable, efficient, and secure agentic systems.
Figures & tables
Figure 1: Overview of the GatewayBench pipeline. Each sampled benign or malicious task exposes its MCP servers, plus injected task-irrelevant noisy tools, to the agent through the gateway under test. A verifiable judge inspects the final environment state to compute task success rate (TSR), attack success rate (ASR), task time, and tool-call latency.
Figure 2: Flattened vs. hierarchical MCP gateways on a travel-booking task example. The flattened gateway exposes tools from all connected servers as a single flat list. The hierarchical gateway instead exposes tool-discovery meta-tools, letting the agent reach the relevant tools through two alternative mechanisms: (A) layered discovery, or (B) direct keyword search over the task intent.
Mechanism
Gateway
Open Source
Tool Name Collision Handling
Server Ordering
Token Efficiency
Search Tool
Flattened
agentcore
✗
✓
✗
✗
✓
agentgateway
✓
✓
✗
✗
✗
litellm
✓
✓
✓
✗
✗
lunar
✓
✓
✓
✗
✗
Hierarchical
enkrypt
△
/
/
△
✗
strata
✓
/
/
✓
✓
Table 1: Comparison of MCP gateways evaluated in this work. The table summarizes the key design characteristics of each gateway under its default configuration. △ denotes partial or limited support, and "/" denotes a dimension that does not apply.
Figure 3: Overview of GatewayBench benchmark. The GatewayBench comprises a task dataset of 524 tasks spanning 6 domains from two datasets, DTap and MCP-Universe, alongside a real-world tool pool of 11,118 noisy tools from 251 authentic MCP servers, enabling systematic evaluation across diverse workflows and adversarial scenarios.
Gateway
OpenAI SDK (GPT-5.1)
OpenAI SDK (GPT-5.4)
OpenClaw (GPT-5.1)
TSR
ASR d
TSR ind
ASR ind
TSR
ASR d
TSR ind
ASR ind
TSR
ASR d
TSR ind
ASR ind
Flattened
agentcore
78.0
79.3
54.0
58.7
85.3
50.0
60.7
48.7
89.3
48.0
60.0
42.7
agentgateway
73.3
73.3
54.7
61.3
86.7
43.3
56.7
46.0
84.0
50.7
64.0
41.3
litellm
75.3
76.0
52.0
63.3
87.3
40.7
57.3
46.7
86.0
50.0
63.3
40.7
lunar
74.7
77.3
52.0
58.7
89.3
42.7
57.3
47.3
87.3
48.0
64.7
45.3
Table 2: Main results: agent capability and security under each gateway, with no noisy tools injected. All values are percentages; Mean rows average the gateways within each architecture type. Bold marks the best value per column: highest TSR/TSR ind and lowest ASR d /ASR ind .
OpenAI SDK (GPT-5.1)
OpenAI SDK (GPT-5.4)
OpenClaw (GPT-5.1)
0
500
1000
1500
0
500
1000
1500
0
500
1000
1500
agentcore
78.0
75.3
72.0
69.3
85.3
89.3
88.7
85.3
89.3
86.7
82.7
77.3
agentgateway
73.3
78.7
76.7
72.0
86.7
87.3
86.7
87.3
84.0
89.3
82.7
84.7
litellm
75.3
70.7
72.0
57.3
87.3
88.0
82.0
82.7
86.0
86.7
80.7
77.3
lunar
74.7
75.3
67.3
62.7
89.3
84.0
89.3
78.0
87.3
90.7
85.3
83.3
Mean
75.3
75.0
72.0
65.3
87.2
87.2
86.7
83.3
86.7
88.4
82.9
80.7
Table 3: Benign task success rate (%) versus the number of injected noisy tools. Within each agent configuration, the sub-columns 0 , 500 , 1,000 , and 1,500 give the number of injected noisy tools. Mean rows average the gateways within each architecture type.
Figure 4: Indirect attack success rate for environment injection and tool injection under OpenClaw with GPT-5.1, aggregated into the two architecture types. Each bar is the architecture mean; the error bar spans the per-gateway minimum and maximum within the architecture.
Figure 5: Interaction cost as the tool set scales. task duration (left) and average tokens per turn (right) of different gateways under the OpenAI Agents SDK with GPT-5.4.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Gateway
158 Noisy Tools
254 Noisy Tools
358 Noisy Tools
Score
TSR (%)
Score
TSR (%)
Score
TSR (%)
agentgateway
0.73
52.3
0.66
52.0
0.75
54.0
enkrypt
0.60
34.7
0.62
36.0
0.52
23.3
strata
0.68
44.7
0.68
46.0
0.68
43.7
Appendix
Table 4: Real-world tool scaling on MCP-Universe (OpenAI Agents SDK, GPT-5.1) for three representative gateways. TSR is the fraction of tasks that pass all execution-based checkpoints, and Score is the average fraction of per-task checkpoints passed (partial credit); higher is better for both.
Figure 6: Indirect attack success rate by injection type as the tool set scales: environment injection (top) and tool injection (bottom), for each agent configuration (columns). One line per gateway; flattened gateways are dashed, hierarchical gateways solid.
Figure 7: Interaction cost as the tool set scales, under the OpenAI Agents SDK with GPT-5.1 (top) and GPT-5.4 (bottom): task time (left), average tool-call latency (middle), and tokens per turn (right). Flattened gateways are dashed, hierarchical gateways solid.
Agent
Search Used
OpenAI SDK (GPT-5.1)
11
OpenAI SDK (GPT-5.4)
19
OpenClaw (GPT-5.1)
1
Appendix
Table 5: Number of task runs in which the AgentCore search tool was invoked. Each agent includes 1,800 task runs.
Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.
Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful. We present DynamicMCPBench, a reusable framework rather than a fixed dataset. A practitioner can run it on their own MCP servers to test models on their own tasks, or let it collect servers automatically to measure a model's general ability to solve agentic tasks. Given the servers and any set of models, it generates realistic goals, pursues each one live to record a successful trajectory, distills that trajectory into path-agnostic effect checkpoints, and scores an agent on whether it reproduces those effects, never on the final answer. To show what the framework reveals, we run it at scale: 24 models over 121 servers and 750 tasks spread evenly over 15 task categories (50 each), where each category targets a distinct tool-use challenge of the generated questions. Each task is scored by pass^3: it counts as solved only if all three independent attempts succeed. Even the strongest agents solve only about half of the tasks, 31% of tasks are solved by no model at all, and accuracy collapses as the required tool chain grows longer (from 39% on the shortest chains to 13% on the longest). A human validation study confirms the automatic scoring is reliable (chance-corrected agreement of 0.76). DynamicMCPBench thus turns benchmark construction into something practitioners can rerun on their own servers and models, while exposing a consistent inability of current agents to handle long, multi-step agentic tasks.
Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov +4
As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet underexplored factor. How funψtionality is decomposed into tools affects whether an agent can select the right tool and construct valid arguments. This choice is especially consequential at the edge, where resource constraints limit which models can run locally and scaling up is often not an option. We present MCP-GRANITE, an open-source extensible benchmark framework that treats tool-interface granularity as a controlled variable for MCP-based agents, evaluated under edge and IoT scenarios. It comprises 81 multi-step scenarios across 9 domains, instantiated at 4 granularity levels from fine-grained primitive tools to a single tool. We evaluate 9 locally deployed models (268M-20.9B parameters) across 8,748 trials using task completion, tool selection F1, argument accuracy, latency, and resource-usage metrics. Results show that a 4-tool interface offers the best trade-off, improving task completion by 16.4% over fine-grained primitives and 33.6% over a single monolithic tool, while nearly doubling argument accuracy. Model size is only weakly correlated with task completion and strongly with latency, while its association with argument accuracy is less robust, and a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one. These findings identify tool-interface granularity as a key design parameter for MCP-based agents.
Demetris Paschalides, Moysis Symeonides, George Pallis +1
Department of Computer Science University of Cyprus Nicosia, Cyprus