cs.PLOct 4, 2026

VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generation

Authors: Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav, Ehsan Degan, Vandana Mukherjee

Organizations: IBM Research San Jose, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages. However, most existing benchmarks focus on Verilog, with limited evaluation of VHDL, which remains widely used in industry and academia for FPGA and safety-critical systems. To address this gap, we introduce VHDL-REPOBENCH, a large-scale, cross-file, repository-level benchmark for assessing LLM capabilities on realistic VHDL design generation and analysis tasks. VHDL-REPOBENCH curates ~100 open-source VHDL repositories, encompassing ~2.5k VHDL files and ~500 testbenches, and provides structured problem statements, module stubs, and self-verifying testbenches. The benchmark enables comprehensive evaluation across syntax, semantic correctness, hierarchical reasoning, cross-file dependency resolution, and functional verification. We evaluate several state-of-the-art models, including GPT-4o, Llama-3-70B, Qwen2.5-72B, CodeLlama-70B, and multi-step reasoning approaches such as Reflexion and CoDes. Results reveal that while current LLMs achieve moderate line- and block-level accuracy, substantial challenges remain in multi-file reasoning, hierarchical design understanding, and specification-to-module generation. VHDL-REPOBENCH represents the first large-scale VHDL-focused benchmark and provides a valuable resource for the hardware design community to evaluate, compare, and advance LLM capabilities for practical VHDL development.

Figures & tables

Fig. 1: Overview of Vhdl-RepoBench . Public GitHub repositories are processed via a preprocessing pipeline to construct the dataset. The benchmark supports 4 tasks: Next-Line Prediction (LP), Block Completion (BC), Specification-to-Module Generation (SMG), & Repository Summarization (RS). Verification module enables evaluation of LLM performance on multi-file VHDL reasoning & generation.
DatasetSizeFunct.Cross-FileLang.
HumanEval [ 13 ]264✓✗Python
RepoCoder [ 14 ]2k✓✗Python
VerilogEval [ 15 ]156✓✗Verilog
Vhdl-Eval [ 16 ]202✓✗VHDL
RTLLM [ 17 ]30✓✗Verilog
RTLRewriter [ 18 ]95✗✓Verilog
TABLE I: Comparison of Vhdl-RepoBench with prior RTL datasets. Dataset size indicates the number of files or problems as reported. Funct. denotes functional verifiability and Lang. indicates the programming language.
Fig. 2: Sample cross-file dependency graph for Midori64 block cipher [ 22 ] , where the top-level file Midori64Enc.vhd instantiates submodules such as FMulti.vhd and MC.vhd .
StatisticValue
# Repositories∼\sim 100
# VHDL files∼\sim 2.5k
# Testbenches∼500\sim 500
Avg. designs per repository1.26
Avg. Cross-file edges per repo29.6
Avg. Max dependency depth3.4
TABLE II: Key statistics of the Vhdl-RepoBench dataset. Files, entities, architectures, and testbenches are counted across all ∼\sim 100 repositories.
Fig. 3: Example task formulations for Vhdl-RepoBench showing the prompt provided to the LLM, input context from the PositAdder repository, and the expected output for each task: Next-Line Prediction (LP), Block Completion (BC), Spec.-to-Module Generation (SMG-R), and Repository Summarization (RS).
TaskSize (approx.)InputOutput
Next-Line Prediction (LP)∼7\sim 7 kPreceding lines (local/cross-file context)Next line of code
Block Completion (BC)∼1\sim 1 kPartial file or masked blockCompleted block/file
Repository Summarization (RS)850Repository filesNL summary
SMG-Module (SMG-M)∼\sim 1.2kEntity stub + Specification + Cross-file ContextArchitecture Body
SMG-Repo (SMG-R)20System-Level Spec + Entity SignaturesComplete RTL source
TABLE III: Summary of tasks in Vhdl-RepoBench , including approximate size, input, and output format. Here, SMG refers to Specification-to-Module Generation (SMG)
ModelContext (tokens)LPBC (Pass@1)SMG (Pass@1)RS
  Exact MatchPass@1 SMG-RSMG-MROUGE-LPR
GPT-4o128K59.260.945.119.841.344.50.49
Qwen2.5-72B128K48.753.138.215.034.837.60.40
Llama-3-70B128K48.348.936.914.233.138.40.41
CodeLlama-70B16K39.240.329.810.221.428.70.26
Verigen2K26.424.817.56.512.8--
TABLE IV: Performance of evaluated LLMs on Vhdl-RepoBench . LP: Next-Line Prediction, BC: Block Completion, SMG: Specification-to-Module Generation, RS: Repository Summarization. Context indicates maximum tokens the model can process. CoDes [ 26 ] and Reflexion [ 27 ] show competitive improvements over base Qwen2.5-72B.

Explore similar work

CardsList

  1. VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation

    Jun 11, 2026Yijun Shen, Minghao Shao, Yichen Zhao +4High-Level SynthesisEdge Hardware

    Center for Data Science, NYU Shanghai, China · NYU Tandon School of Engineering, USA · NYU Abu Dhabi, UAE

  2. Benchmarking LLMs for Verilog Design Flows

    Jul 23, 2026Angshuman Chakravertty, Rahul Koshti, Buddhi Prakash Sharma +1High-Level SynthesisHierarchical Register Transfer Level Generation

    School of Technology Management and Engineering, NMIMS, Hyderabad, India · Department of Electronics and Communication Engineering, Indian Institute of Technology, Roorkee, India · Department of Electrical and Electronics Engineering and the Anuradha and Prashanth Palakurthi Centre for Artificial Intelligence Research (APPCAIR), Birla Institute of Technology and Science, Pilani (BITS Pilani), Pilani 333031, India

  3. RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

    Jun 8, 2026Jing Wang, Shang Liu, Wenji Fang +3Register Transfer LevelLarge Language Model Benchmarks

    Hong Kong University of Science and Technology