cs.AROct 2, 2026

An Executable Benchmark for LLM-Based HLS Repair:Design Complexity and Repair Underconstraint

Authors: Maisha Mastora, Dean Sullivan

Organizations: Department of Electrical and Computer Engineering University of New Hampshire Durham, NH, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Automated repair of High-Level Synthesis (HLS) designs using large language models (LLMs) is an emerging but underexplored problem. While LLM-based repair shows strong results on register-transfer level (RTL) Verilog, the only prior systematic study of HLS logic repair reports just 10.5% correction accuracy for GPT-4, with no analysis of why repair fails or what drives difficulty. We present the first comprehensive evaluation of LLM-based HLS repair across four models (GPT-4o, GPT-4o-mini, GPT-5.4, and Claude Opus 4.6) on 125 benchmark instances spanning eight logic bug types across three open-source HLS suites (CHStone, MachSuite, Polybench). We construct the first executable APR-style HLS repair benchmark with suite-specific functional oracles, enabling pass@k evaluation rather than the string-match approximations used in prior work. Design context and scale, rather than bug type alone, dominate repair difficulty: repair rates range from 6-45% on complex cryptographic kernels (CHStone) to 81-93% on compact algorithmic kernels (MachSuite) despite identical bug type distributions. We introduce solution multiplicity, the fraction of distinct patches generated across repeated repair attempts, as an empirical measure of repair underconstraint, and show it strongly predicts repair failure across all four models (Spearman rho = -0.393, p = 6.61 x 10^-20). Frontier models reduce underconstrained instances from 41-46 (GPT-4o, GPT-4o-mini) to just 3 (Claude Opus 4.6), with pass@1 improving from 42-53% to 73-76%. SHFT bugs remain consistently hard across all models, and semantic bug classes such as buffer indexing become reliably repairable only at frontier scale. These findings show that future APR benchmarks must include complex, executable, and weakly identifiable designs that remain challenging after frontier LLM repair.

Figures & tables

Fig. 1: System overview. HLS benchmark suites are paired with executable oracles and validated via bug injection to produce a benchmark of correct and buggy instance pairs. Validated instances are repaired zero-shot by LLMs (N=5 attempts per design); each candidate patch is recompiled and checked against a suite-specific functional oracle, then scored by pass@k and solution multiplicity. Failure modes and repair outcomes are aggregated for empirical complexity and identifiability characterization.
Bug TypeCHStoneMachSuitePolybenchTotal
MLU710926
ZERO771024
INF751022
OOB78318
USE431017
SHFT7108
TABLE I: Bug type distribution across 125 benchmark instances.
Modelpass@1pass@5Multiplicity
GPT-4o-mini42.4%63.2%0.970
GPT-4o53.3%70.4%0.954
GPT-5.473.4%78.4%0.771
Claude Opus 4.675.7%78.4%0.310
TABLE II: Overall pass@1, pass@5, and solution multiplicity by model.
SuiteGPT-4o-miniGPT-4oGPT-5.4Opus 4.6
CHStone21.0%13.7%37.6%44.9%
MachSuite48.5%74.0%92.5%90.5%
Polybench56.8%71.4%89.5%90.9%
TABLE III: pass@1 by suite and model. The CHStone vs. MachSuite gap holds across all models despite identical bug type distributions.
Fig. 2: Repair difficulty versus target-file LOC within each suite. Each point is one benchmark instance, colored by bug type. The y-axis reports mean empirical pass@1 across models. CHStone shows a significant negative correlation ( ρ=−0.389\rho=-0.389 , p=0.012p=0.012 ); MachSuite and Polybench show weaker non-significant negative trends due to narrower LOC ranges.
ModelCHStoneMachSuitePolybenchTotal
GPT-4o-mini23121046
GPT-4o255535
GPT-5.4200323
Opus 4.62103
TABLE IV: Underconstrained instances (multiplicity ≥0.8\geq 0.8 , pass@5 =0=0 ) by model and suite.
Fig. 3: Repair rate by model and bug type. Columns sorted by overall average empirical pass@1. Model scaling improves most bug types, but SHFT remains the hardest category. BUF shows strong improvement in larger models, suggesting some semantic bug classes become repairable only at frontier scale.
Fig. 4: Repair rate by bug type and suite. Each cell reports mean empirical pass@1 across models, with the number of instances in parentheses. The same bug type can have very different repair rates across suites, confirming that benchmark context substantially mediates bug-type difficulty.
Fig. 5: Comparison of repair difficulty predictors. LOC and mean solution multiplicity are both strongly negatively correlated with mean empirical pass@1, while cyclomatic complexity is not a significant negative predictor.
More in this topic

Automated Program Repair

Automated program repair develops methods that locate software faults and produce validated fixes with limited human effort.

See all 158 papers

Explore similar work

CardsList
Sep 4, 2026

Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair

Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation. We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.

Xuemeng Cai, Jiakun Liu, Linhan Yang +2

School of Computing and Information Systems, Singapore Management University, Singapore · Harbin Institute of Technology, Harbin, China

Automated Program RepairHallucination in Language Models
AskPDF
Jul 30, 2026

Open-Source LLM-Driven Formal Verification: A Multi-Agent Pipeline for RTL Repair

Verification consumes the majority of modern chip design effort, yet the formal verification tools that provide mathematical guarantees of correctness remain expensive and restrictively licensed. While large language models (LLMs) have shown promise for hardware design, existing approaches to RTL repair validate their results through simulation - which exercises only a subset of inputs - or rely on commercial tools, and few combine formal proof with an entirely open-source toolchain. In this paper, we present a multi-agent pipeline that couples an LLM with an open-source formal backend (Yosys, SymbiYosys, and Z3) to repair RTL through counterexample-guided iteration: the framework generates formal properties, verifies the design, and feeds counterexamples back to the LLM until the design is proved correct by k-induction or an iteration budget is exhausted. Through an ALU case study, we show that the pipeline can detect and repair a real functional bug with a formal proof of correctness. Across a six-benchmark suite, one design is repaired reliably, and we characterize four distinct failure modes: bounded-cover vacuity, specification ambiguity, temporal-logic bugs, and multi-property pressure. We frame this work as a feasibility study with a detailed failure analysis, and additionally report a practical limitation of the Yosys bind directive relevant to the open-source formal verification community.

Ha Trung Tran

Independent Researcher · Seattle, WA, USA

Formal Hardware VerificationAutomated Program RepairProgram SynthesisFormal Verification
AskPDF
Apr 16, 2026

HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks

Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench, the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project's native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents.

Fan Cui, Hongyuan Hou, Zizhang Luo +2

Peking University · Beijing, China

Automated Program RepairElectronic Design AutomationLLM Agent EvaluationAI Agent Benchmarks
AskPDF