ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness
Organizations: Imperial College London United Kingdom
Abstract
Field-Programmable Gate Arrays (FPGAs) are widely used in modern hardware design, yet writing Hardware Description Language (HDL) code for FPGA implementation remains a complex and time-consuming task. Large Language Models (LLMs) have emerged as a promising tool for HDL generation, but existing benchmarks for LLM-based code generation primarily focus on functional correctness while overlooking hardware resource usage. Furthermore, current benchmarks offer limited diversity and do not fully represent the wide range of real-world FPGA applications. To address these shortcomings, we introduce ResBench, the first resource-focused benchmark explicitly designed to distinguish between resource-optimized and inefficient LLM-generated HDL code. ResBench consists of 56 problems across 12 categories, covering applications from finite state machines to financial computing. Our open-source evaluation framework automatically tests LLMs by generating Verilog code, verifying correctness, and measuring resource usage. The experiments, which primarily analyze Lookup Table (LUT) usage, reveal significant differences among LLMs, demonstrating ResBench's capability to identify models that generate more resource-optimized FPGA designs.
Figures & tables
| Benchmark | Size | PL | Type | Features |
|---|---|---|---|---|
| VerilogEval ( Liu et al., 2023 ) | 156 | Verilog | Verilog code generation tasks | Covers a wide range of tasks from simple combinational circuits to finite state machines; includes automatic functional correctness testing. |
| HDLEval ( Kashanaki et al., 2024 ) | 100 | Multiple | Language-agnostic HDL evaluation | Evaluates LLMs across multiple HDLs using standardized testbenches and formal verification; categorizes problems into combinational and pipelined tests. |
| PyHDL-Eval ( Batten et al., 2024 ) | 168 | Python-embedded DSLs | Specification-to-RTL tasks | Focuses on Python-embedded DSLs for hardware design; includes Verilog reference solutions and testbenches; evaluates LLMs’ ability to handle specification-to-RTL translations. |
| RTLLM ( Liu et al., 2024b ) | 50 | Verilog, VHDL, Chisel | Design RTL generation | Supports evaluation across multiple HDL formats; spans various design complexities and scales; includes an automated evaluation framework. |
| VHDL-Eval ( Vijayaraghavan et al., 2024 ) | 202 | VHDL | VHDL code generation tasks | Aggregates translated Verilog problems and publicly available VHDL problems; utilizes self-verifying testbenches for functional correctness validation. |
| GenBen ( Wan et al., 2025 ) | 351 | Verilog | Fundamental hardware design and debugging tasks | Evaluates synthesizability, power consumption, area utilization, and timing performance to ensure real-world applicability. |
| Category | # Problems | Example Problems |
|---|---|---|
| Combinational Logic | 8 | parity_8bit, mux4to1, bin_to_gray |
| Finite State Machines | 4 | fsm_3state, traffic_light, elevator_controller |
| Mathematical Functions | 5 | int_sqrt, fibonacci, mod_exp |
| Basic Arithmetic Operations | 5 | add_8bit, mult_4bit, abs_diff |
| Bitwise and Logical Operations | 4 | bitwise_ops, left_shift, rotate_left |
| Pipelining | 5 | pipelined_adder, pipelined_multiplier, pipelined_fir |
| Benchmark | Year | Hardware | Optimization | Problem Diversity |
|---|---|---|---|---|
| Awareness | ||||
| VerilogEval ( Thakur et al., 2024 ) | 2023 | General HDL | No | Logic, FSMs, arithmetic |
| HDLEval ( Kashanaki et al., 2024 ) | 2024 | General HDL | No | Digital circuits, control logic |
| PyHDL-Eval ( Batten et al., 2024 ) | 2024 | General HDL | No | Python-based HDL, small designs |
| RTLLM ( Lu et al., 2024 ) | 2024 | General HDL | No | RTL, bus protocols, DSP |
| VHDL-Eval ( Vijayaraghavan et al., 2024 ) | 2024 | General HDL | No | VHDL logic, sequential circuits |
| GPT-3.5 | GPT-4 | GPT-4o | GPT-o1 | Llama3.1 | Qwen-max | Qwen-plus | Qwen2.5-coder | Codestral | |
| turbo | mini | 405B | 32B | ||||||
| fsm 3state | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| traffic light | 1 | 1 | 2 | 0 | 0 | 2 | 3 | 2 | |
| elevator controller | 3 | 3 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
| vending machine | 1 | 1 | 2 | 1 | 2 | 1 | 1 | 2 | 1 |
| int sqrt | 68 | 177 | 64 | 229 | 173 |
Code Generation Benchmarks
Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.
See all 169 papers