Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

Leonardo Liparulo; Francesco Pierri

cs.AIAug 25, 2026

Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

Authors: Leonardo Liparulo, Francesco Pierri

Organizations: Politecnico di Milano

Abstract

We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations---such as creating components, adding ports, and wiring connections---through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.

Figures & tables

Figure 1: A component in the target application domain. Subcomponents, ports, and connections form a dependency-ordered structure: each element must be created before it can be referenced by subsequent operations.

Group	Tools
Adder	create_component , add_signal_port , add_transactional_port , add_user_parameter , add_generator_parameter , add_subcomponent , add_subcomponent_connection
Getter	get_component_details , get_element , list_components
Other	delete_element , delete_component , save_component , load_component

Table 1: MCP server tool set.

Figure 2: Evaluation pipeline. Each task prompt is submitted to the agent, which issues tool calls to the MCP server and receives results. The produced call sequence is compared against annotated expected calls to compute the metrics of Section 3.5 .

Figure 3: Best-configuration ECC per model and task suite under ReAct.

	Config	ECC $\uparrow$	EVCR $\downarrow$	TFR $\downarrow$
Gemma 4 31B	R/MD/C	0.990	0.015	0.011
Gemma 4 26B	R/MD/C	0.958	0.021	0.061
GPT-OSS 20B	R/none/C	0.951	0.073	0.049
Qwen 3.5 27B	R/MD/M	0.880	0.036	0.048
Qwen 3.6 27B	R/MD/M	0.858	0.086	0.065
Gemma 4 E4B	R/MD/C	0.811	0.023	0.058

Table 2: Best single ReAct configuration per model, averaged over the six core task suites and ordered by ECC. Config notation: history / prompt / tools, where R = run, T = task, C = comprehensive, M = minimal, fs = few-shot.

	No-call (Errors)			Noise (Hard)
	NCA	EVCR	TFR	clean	noisy
Gemma 4 31B	0.80	0.00	0.05	—	—
Gemma 4 26B	0.40	0.05	0.55	0.00	0.16
GPT-OSS 20B	0.25	0.58	0.13	—	—
Qwen 3.5 27B	0.70	0.15	0.17	—	—
Qwen 3.6 27B	0.45	0.20	0.37	—	—

Table 3: NCA, EVCR, and TFR on the 20 no-call tasks in Errors, at each model’s best configuration. Right: EVCR on Hard vs. Hard-Noise under task scope and matched configuration for the three models tested. ECC changes are small and omitted for space.

Worker	Suite	ReAct	Plan-and-Act
Llama 3.1 8B	Avg. (6 core)	0.554	0.718
Llama 3.1 8B	History	0.113	0.650
Llama 3.1 8B	Cross	0.166	0.450
Gemma 4 26B	Hard-Noise (60)	0.858	0.950

Table 4: ECC under ReAct vs. Plan-and-Act, at each worker’s best configuration (averaged over the six core suites). The Hard-Noise row uses a fixed run-scope configuration, isolating the effect of session length.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: ReAct sequence diagram. The host submits the task prompt and tool schemas to the LLM. Within a turn-bounded loop, the LLM returns a response that may include a tool call; the host executes the first call, appends the result, and repeats. The loop terminates when no further tool call is produced or the turn limit is reached.

Figure 5: Plan-and-Act sequence diagram. The planner produces a <STEP> -delimited plan; the validator checks it with one replan permitted. Each step is executed by an independent ReAct worker with a compact trace of prior results. A synthesis call assembles the final answer.

	$T=0$			$T=0.5$			$T{=}\text{default}$
Model	ECC	EVCR	TFR	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 26B	0.896	0.055	0.094	0.889	0.055	0.111	0.881	0.056	0.105
Gemma 4 E4B	0.539	0.044	0.129	0.523	0.042	0.127	0.553	0.046	0.144
Llama 3.1 8B	0.247	0.058	0.268	0.235	0.052	0.264	0.243	0.056	0.268

Appendix

Table 5: ECC and TFR across temperatures 0, 0.5, and each model’s default (1.0, except 0.8 for Llama 3.1 8B), averaged over the three datasets covered by the sweep (Cross, Errors, Hard) and over all 16 system-prompt

\times

tool-description

\times

history-scope configurations per model. Rankings and absolute performance are stable across temperatures for every model; lowering temperature does not consistently reduce TFR.

Model	$\bar{\sigma}_{\mathrm{ECC}}$	$\sigma_{\max,\mathrm{ECC}}$
Gemma 4 26B	0.025	0.076
Llama 3.1 8B	0.051	0.219
Gemma 4 E4B	0.059	0.494

Appendix

Table 6: Mean and maximum standard deviation of ECC across two repeated runs of every system-prompt

\times

tool-description

\times

history-scope configuration (16 per model), on Cross, Errors, and Hard — the three most demanding suites. The Gemma 4 E4B outlier (

\sigma_{\max}=0.494

) is the run -scope, few-shot, minimal-tools configuration, the same one responsible for the few-shot collapse reported in Section 4.2 , confirming that effect is genuine rather than a sampling artefact.

Model	Config	ECC	EVCR	TFR
Gemma 4 26B	best	0.973	0.027	0.013
Gemma 4 26B	worst	0.829	0.017	0.067
Llama 3.1 8B	best	0.495	0.106	0.291
Llama 3.1 8B	worst	0.034	0.001	0.028

Appendix

Table 7: Mean ECC, EVCR, and TFR across 10 repeated runs of each model’s best and worst configuration, averaged over all six core suites (10 tasks each). Standard deviations of ECC across runs: 0.008 (Gemma best), 0.038 (Gemma worst), 0.033 (Llama best), 0.109 (Llama worst) — the latter driven by near-total collapse (9 of 10 runs at ECC

\,\approx\,

0) rather than ordinary run-to-run noise.

	none			basic			MD			few-shot
Model	ECC	EVCR	TFR	ECC	EVCR	TFR	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 31B	0.935	0.039	0.076	0.913	0.010	0.056	0.956	0.021	0.061	0.571	0.006	0.032
Gemma 4 26B	0.907	0.044	0.086	0.910	0.020	0.063	0.938	0.027	0.063	0.897	0.034	0.071
GPT-OSS 20B	0.861	0.078	0.104	0.784	0.030	0.068	0.859	0.042	0.061	0.741	0.030	0.072
Qwen 3.5 27B	0.847	0.063	0.068	0.827	0.031	0.058	0.852	0.034	0.057	0.825	0.032	0.054
Qwen 3.6 27B	0.839	0.063	0.106	0.798	0.030	0.063	0.853	0.058	0.060	0.801	0.036	0.056

Appendix

Table 8: System-prompt ablation, averaged over the six core suites, tool-description formats, and history scopes. The few-shot collapse for Gemma 4 31B (

0.956\!\to\!0.571

) and Gemma 4 E4B (

0.731\!\to\!0.179

) is visible as a coverage drop with EVCR and TFR falling together (inaction, not error).

	comprehensive			minimal
Model	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 31B	0.819	0.019	0.031	0.868	0.020	0.081
Gemma 4 26B	0.932	0.028	0.048	0.894	0.034	0.093
GPT-OSS 20B	0.859	0.046	0.057	0.764	0.044	0.096
Qwen 3.5 27B	0.845	0.034	0.040	0.830	0.046	0.078
Qwen 3.6 27B	0.823	0.037	0.046	0.822	0.057	0.097

Appendix

Table 9: Tool-description ablation, averaged over the six core suites, system prompts, and history scopes. Minimal descriptions raise TFR for every model (roughly doubling it in most cases); the ECC effect is less uniform.

	run			task
Model	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 31B	0.858	0.014	0.037	0.829	0.024	0.075
Gemma 4 26B	0.918	0.029	0.071	0.909	0.034	0.071
GPT-OSS 20B	0.862	0.051	0.061	0.760	0.039	0.091
Qwen 3.5 27B	0.864	0.047	0.038	0.811	0.033	0.081
Qwen 3.6 27B	0.823	0.051	0.060	0.822	0.042	0.083

Appendix

Table 10: History-scope ablation, averaged over the six core suites, system prompts, and tool-description formats. Llama 3.1 8B is the only model that prefers per-task scope; all others tolerate or benefit from cumulative history at this session length.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 277 papers

Apr 16, 2026

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench, the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project's native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents.

Fan Cui, Hongyuan Hou, Zizhang Luo +2

Peking University · Beijing, China

Automated Program Repair Electronic Design Automation LLM Agent Evaluation AI Agent Benchmarks

Ask PDF

Jul 20, 2026

Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows

Large language model (LLM) agents are extending electronic design automation (EDA) beyond static RTL generation toward long-horizon, tool-interactive workflows. Yet it remains unclear whether general-purpose coding agents, even with domain-specific EDA skills, can reliably execute an end-to-end RTL-to-GDS flow encompassing synthesis, physical implementation, and engineering change order (ECO) optimization. We evaluate AI agents on a PicoRV32 RTL-to-GDS flow using commercial EDA tools under two timing targets. Their performance is assessed using end-to-end design score, stage completion, and Token ROI, a cost-efficiency metric relating design quality to runtime and cost. Comparing three agent architectures and four foundation models, we derive three practical lessons. First, domain-specific skills improve agents' understanding of individual subtasks but do not ensure reliable completion of a long-horizon EDA flow. Second, agents that achieve similar design progress can still differ by up to 141 times in Token ROI, revealing substantial differences in runtime and cost efficiency. Third, low-level tool-interface mismatches are a major source of physical design failures, particularly when Tcl commands depend on the tool version or execution mode. These results suggest that robust Agentic EDA requires not only stronger models but also structured tool interfaces, persistent design context, controlled execution, and process-level evaluation.

Jinyuan Deng, Zhengrui Chen, Xufeng Wei +4

1Zhejiang University, China · 2ChipFlux, China

Electronic Design Automation LLM Agent Evaluation AI Agent Benchmarks Long-Horizon LLM Agents

Ask PDF

Sep 21, 2026

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents

As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet underexplored factor. How funψtionality is decomposed into tools affects whether an agent can select the right tool and construct valid arguments. This choice is especially consequential at the edge, where resource constraints limit which models can run locally and scaling up is often not an option. We present MCP-GRANITE, an open-source extensible benchmark framework that treats tool-interface granularity as a controlled variable for MCP-based agents, evaluated under edge and IoT scenarios. It comprises 81 multi-step scenarios across 9 domains, instantiated at 4 granularity levels from fine-grained primitive tools to a single tool. We evaluate 9 locally deployed models (268M-20.9B parameters) across 8,748 trials using task completion, tool selection F1, argument accuracy, latency, and resource-usage metrics. Results show that a 4-tool interface offers the best trade-off, improving task completion by 16.4% over fine-grained primitives and 33.6% over a single monolithic tool, while nearly doubling argument accuracy. Model size is only weakly correlated with task completion and strongly with latency, while its association with argument accuracy is less robust, and a 3.2B model at the optimal granularity outperforms a 20.9B model at a mismatched one. These findings identify tool-interface granularity as a key design parameter for MCP-based agents.

Demetris Paschalides, Moysis Symeonides, George Pallis +1

Department of Computer Science University of Cyprus Nicosia, Cyprus

AI Agent Benchmarks Tool-Using Agents

Ask PDF

Abstract

Tool-Using Agents

Explore similar work

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents