Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

Leonardo Liparulo; Francesco Pierri

cs.AIAug 25, 2026

Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

Authors: Leonardo Liparulo, Francesco Pierri

Organizations: Politecnico di Milano

Abstract

We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations---such as creating components, adding ports, and wiring connections---through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.

Figures & tables

Figure 1: A component in the target application domain. Subcomponents, ports, and connections form a dependency-ordered structure: each element must be created before it can be referenced by subsequent operations.

Group	Tools
Adder	create_component , add_signal_port , add_transactional_port , add_user_parameter , add_generator_parameter , add_subcomponent , add_subcomponent_connection
Getter	get_component_details , get_element , list_components
Other	delete_element , delete_component , save_component , load_component

Table 1: MCP server tool set.

Figure 2: Evaluation pipeline. Each task prompt is submitted to the agent, which issues tool calls to the MCP server and receives results. The produced call sequence is compared against annotated expected calls to compute the metrics of Section 3.5 .

Figure 3: Best-configuration ECC per model and task suite under ReAct.

	Config	ECC $\uparrow$	EVCR $\downarrow$	TFR $\downarrow$
Gemma 4 31B	R/MD/C	0.990	0.015	0.011
Gemma 4 26B	R/MD/C	0.958	0.021	0.061
GPT-OSS 20B	R/none/C	0.951	0.073	0.049
Qwen 3.5 27B	R/MD/M	0.880	0.036	0.048
Qwen 3.6 27B	R/MD/M	0.858	0.086	0.065
Gemma 4 E4B	R/MD/C	0.811	0.023	0.058

Table 2: Best single ReAct configuration per model, averaged over the six core task suites and ordered by ECC. Config notation: history / prompt / tools, where R = run, T = task, C = comprehensive, M = minimal, fs = few-shot.

	No-call (Errors)			Noise (Hard)
	NCA	EVCR	TFR	clean	noisy
Gemma 4 31B	0.80	0.00	0.05	—	—
Gemma 4 26B	0.40	0.05	0.55	0.00	0.16
GPT-OSS 20B	0.25	0.58	0.13	—	—
Qwen 3.5 27B	0.70	0.15	0.17	—	—
Qwen 3.6 27B	0.45	0.20	0.37	—	—

Table 3: NCA, EVCR, and TFR on the 20 no-call tasks in Errors, at each model’s best configuration. Right: EVCR on Hard vs. Hard-Noise under task scope and matched configuration for the three models tested. ECC changes are small and omitted for space.

Worker	Suite	ReAct	Plan-and-Act
Llama 3.1 8B	Avg. (6 core)	0.554	0.718
Llama 3.1 8B	History	0.113	0.650
Llama 3.1 8B	Cross	0.166	0.450
Gemma 4 26B	Hard-Noise (60)	0.858	0.950

Table 4: ECC under ReAct vs. Plan-and-Act, at each worker’s best configuration (averaged over the six core suites). The Hard-Noise row uses a fixed run-scope configuration, isolating the effect of session length.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: ReAct sequence diagram. The host submits the task prompt and tool schemas to the LLM. Within a turn-bounded loop, the LLM returns a response that may include a tool call; the host executes the first call, appends the result, and repeats. The loop terminates when no further tool call is produced or the turn limit is reached.

Figure 5: Plan-and-Act sequence diagram. The planner produces a <STEP> -delimited plan; the validator checks it with one replan permitted. Each step is executed by an independent ReAct worker with a compact trace of prior results. A synthesis call assembles the final answer.

	$T=0$			$T=0.5$			$T{=}\text{default}$
Model	ECC	EVCR	TFR	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 26B	0.896	0.055	0.094	0.889	0.055	0.111	0.881	0.056	0.105
Gemma 4 E4B	0.539	0.044	0.129	0.523	0.042	0.127	0.553	0.046	0.144
Llama 3.1 8B	0.247	0.058	0.268	0.235	0.052	0.264	0.243	0.056	0.268

Appendix

Table 5: ECC and TFR across temperatures 0, 0.5, and each model’s default (1.0, except 0.8 for Llama 3.1 8B), averaged over the three datasets covered by the sweep (Cross, Errors, Hard) and over all 16 system-prompt

\times

tool-description

\times

history-scope configurations per model. Rankings and absolute performance are stable across temperatures for every model; lowering temperature does not consistently reduce TFR.

Model	$\bar{\sigma}_{\mathrm{ECC}}$	$\sigma_{\max,\mathrm{ECC}}$
Gemma 4 26B	0.025	0.076
Llama 3.1 8B	0.051	0.219
Gemma 4 E4B	0.059	0.494

Appendix

Table 6: Mean and maximum standard deviation of ECC across two repeated runs of every system-prompt

\times

tool-description

\times

history-scope configuration (16 per model), on Cross, Errors, and Hard — the three most demanding suites. The Gemma 4 E4B outlier (

\sigma_{\max}=0.494

) is the run -scope, few-shot, minimal-tools configuration, the same one responsible for the few-shot collapse reported in Section 4.2 , confirming that effect is genuine rather than a sampling artefact.

Model	Config	ECC	EVCR	TFR
Gemma 4 26B	best	0.973	0.027	0.013
Gemma 4 26B	worst	0.829	0.017	0.067
Llama 3.1 8B	best	0.495	0.106	0.291
Llama 3.1 8B	worst	0.034	0.001	0.028

Appendix

Table 7: Mean ECC, EVCR, and TFR across 10 repeated runs of each model’s best and worst configuration, averaged over all six core suites (10 tasks each). Standard deviations of ECC across runs: 0.008 (Gemma best), 0.038 (Gemma worst), 0.033 (Llama best), 0.109 (Llama worst) — the latter driven by near-total collapse (9 of 10 runs at ECC

\,\approx\,

0) rather than ordinary run-to-run noise.

	none			basic			MD			few-shot
Model	ECC	EVCR	TFR	ECC	EVCR	TFR	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 31B	0.935	0.039	0.076	0.913	0.010	0.056	0.956	0.021	0.061	0.571	0.006	0.032
Gemma 4 26B	0.907	0.044	0.086	0.910	0.020	0.063	0.938	0.027	0.063	0.897	0.034	0.071
GPT-OSS 20B	0.861	0.078	0.104	0.784	0.030	0.068	0.859	0.042	0.061	0.741	0.030	0.072
Qwen 3.5 27B	0.847	0.063	0.068	0.827	0.031	0.058	0.852	0.034	0.057	0.825	0.032	0.054
Qwen 3.6 27B	0.839	0.063	0.106	0.798	0.030	0.063	0.853	0.058	0.060	0.801	0.036	0.056

Appendix

Table 8: System-prompt ablation, averaged over the six core suites, tool-description formats, and history scopes. The few-shot collapse for Gemma 4 31B (

0.956\!\to\!0.571

) and Gemma 4 E4B (

0.731\!\to\!0.179

) is visible as a coverage drop with EVCR and TFR falling together (inaction, not error).

	comprehensive			minimal
Model	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 31B	0.819	0.019	0.031	0.868	0.020	0.081
Gemma 4 26B	0.932	0.028	0.048	0.894	0.034	0.093
GPT-OSS 20B	0.859	0.046	0.057	0.764	0.044	0.096
Qwen 3.5 27B	0.845	0.034	0.040	0.830	0.046	0.078
Qwen 3.6 27B	0.823	0.037	0.046	0.822	0.057	0.097

Appendix

Table 9: Tool-description ablation, averaged over the six core suites, system prompts, and history scopes. Minimal descriptions raise TFR for every model (roughly doubling it in most cases); the ECC effect is less uniform.

	run			task
Model	ECC	EVCR	TFR	ECC	EVCR	TFR
Gemma 4 31B	0.858	0.014	0.037	0.829	0.024	0.075
Gemma 4 26B	0.918	0.029	0.071	0.909	0.034	0.071
GPT-OSS 20B	0.862	0.051	0.061	0.760	0.039	0.091
Qwen 3.5 27B	0.864	0.047	0.038	0.811	0.033	0.081
Qwen 3.6 27B	0.823	0.051	0.060	0.822	0.042	0.083

Appendix

Table 10: History-scope ablation, averaged over the six core suites, system prompts, and tool-description formats. Llama 3.1 8B is the only model that prefers per-task scope; all others tolerate or benefit from cumulative history at this session length.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 277 papers

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
Apr 16, 2026Fan Cui, Hongyuan Hou, Zizhang Luo +2Automated Program Repair Electronic Design Automation
Peking University · Beijing, China
Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows
Jul 20, 2026Jinyuan Deng, Zhengrui Chen, Xufeng Wei +4Electronic Design Automation LLM Agent Evaluation
1Zhejiang University, China · 2ChipFlux, China
MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
Sep 21, 2026Demetris Paschalides, Moysis Symeonides, George Pallis +1AI Agent Benchmarks Tool-Using Agents
Department of Computer Science University of Cyprus Nicosia, Cyprus

Apr 16, 2026J/K move · Enter open · S save

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

Fan Cui, Hongyuan Hou, Zizhang Luo, Chenyun Yin+1

Peking University · Beijing, China

Automated Program Repair Electronic Design Automation LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench, the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project's native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents.

Abstract

Tool-Using Agents

Explore similar work

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

Can AI Agents Really Complete RTL-to-GDS? Lessons from Benchmarking Tool-Interactive EDA Workflows

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents