Organizations: School of Computer Science, Peking University · SiliconMind · Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education · aiXcoder · Tsinghua University · Renmin University of China · Omnicom-tech
Design verification (DV) remains a major bottleneck in hardware development, with large-scale verification using the Universal Verification Methodology (UVM) being highly labor-intensive. Recent advances in large language models (LLMs) and agentic systems offer new opportunities for DV automation. However, existing benchmarks primarily construct mutually decoupled subtasks, which deviates from the real-world workflow. In this paper, we present DVBench, the first end-to-end benchmark for Real-World UVM design verification agents. DVBench comprises 95 tasks across 9 functional domains and 5 difficulty levels, covering diverse verification scenarios and design scales. Each task requires agents to independently complete the full verification workflow. DVBench evaluates final testbenches via compilation, coverage, and bug detection, while assessing intermediate artifacts including feature lists, test plans, and reference models. We evaluate frontier LLMs (e.g. Claude Opus 5.5, GPT-6 Astra) and agent frameworks (e.g. Claude Code, Codex, OpenCode). Experimental results show that the best LLM-based agent (Codex & GPT-6 Astra) can only achieve an overall success rate of 21.05%, revealing substantial room for improvement in complex verification tasks.
Figures & tables
Figure 3: DVBench diversity across 95 tasks. (a) Primary functional domains, stacked by difficulty level. (b) Per-task RTL SLOC and source module definitions, including supplied dependencies; both axes use logarithmic scales. Arrows mark the maximum on each axis. The shared legend names each difficulty level and reports its task count.
Figure 4: DVBench evaluation protocol. We take synchronous FIFO as an example. The final submission undergoes feature list assessment, test plan assessment, reference model accuracy checks, and complete testbench evaluation. Diagnostic scores provide insights into agentic capabilities at earlier stages of the verification workflow. Overall success requires compilation, golden-RTL acceptance, mutant detection, sufficient DUT coverage, and strict reference model correctness. Detailed evaluation criteria are described from Section 4.2 to Section 4.5 .
Agent
LLM
Overall success
Task-level criteria
Testbench
Mean DUT coverage
Mean F1
Kill all 10 mutants
Ref. model exact
All cov. ≥90%
Compile
Golden pass
Mutant kill (mean)
Line
Cond.
Branch
Toggle
Feature List
Test Plan
Claude Code
GLM 5.3
4.21
8.42
42.11
14.74
71.58
41.05
32.00
59.77
54.57
54.90
49.46
48.52
14.90
DeepSeek V4.1 Flash 1017
14.74
46.32
58.95
29.47
100.00
96.84
83.47
91.34
85.46
86.67
80.73
71.61
30.01
Claude Opus 5.5
16.84
43.16
71.58
35.79
98.95
93.68
80.63
90.82
85.72
86.91
83.37
78.50
42.95
GPT-6.1 Sol
17.89
47.37
68.42
36.84
98.95
91.58
81.58
91.49
86.04
87.39
82.92
94.11
71.43
GPT-6 Astra
18.95
51.58
70.53
35.79
100.00
93.68
84.32
92.72
87.06
88.70
83.60
92.49
70.37
Table 1: Main results on 95 tasks. All values are percentages; higher is better. Bold and underlined values denote the best and second-best distinct values across all configurations in each column, including ties.
Figure 5: Overall success by difficulty for five models using Claude Code. Each group follows the legend order; n denotes the number of tasks.
Figure 6: Primary failure categories in 50 selected unsuccessful traces from Codex with GPT-6 Astra from an earlier evaluation snapshot. Labels show counts and percentages.
Agent
LLM
Mean tokens ( 106 )
Mean time (min)
Claude Code
GLM 5.3
17.27
258.55
DeepSeek V4.1 Flash 1017
22.51
37.96
Claude Opus 5.5
2.37
32.83
GPT-6.1 Sol
1.36
21.75
GPT-6 Astra
1.08
12.43
OpenCode
GLM 5.3
1.46
566.61
Table 2: Mean token usage and elapsed time per task over 95 tasks.
Model
Recovered
Recovery rate
GPT-6.1 Sol
6/120
5.0%
Claude Opus 5.5
3/120
2.5%
GPT-6 Astra
4/120
3.3%
DeepSeek V4.1 Flash 1017
0/120
0.0%
Table 3: Exact specification recovery on 60 tasks, with 120 answers per model. Recovered counts are measured after answer extraction. Lower recovery indicates lower benchmark-specific contamination risk under this probe.
Figure 7: Three trajectories distinguish short refinement, long-horizon convergence, and long-horizon non-convergence. Curves join measured cumulative token counts; markers identify engineering milestones. All panels use millions of tokens (M), with independent axis ranges. Golden RTL passes denote successful execution of the agent’s testbench against golden RTL during development, using seed 1; final outcomes in subtitles come from frozen-submission assessment. The fetch target queue achieves a Golden RTL pass after restoring a snapshot, but its reference model fails independent evaluation. Hidden outcomes are not feedback available at the plotted turns.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
ID
Framework
Model
Solved
Rate (%)
Tokens (M)
Time (min)
C01
Claude Code
GLM 5.3
4/95
4.21
17.27
258.55
C02
Claude Code
DeepSeek V4.1 Flash 1017
14/95
14.74
22.51
37.96
C03
Claude Code
Claude Opus 5.5
16/95
16.84
2.37
32.83
C04
Claude Code
GPT-6.1 Sol
17/95
17.89
1.36
21.75
C05
Claude Code
GPT-6 Astra
18/95
18.95
1.08
12.43
C06
Codex
GLM 5.3
2/95
2.11
10.19
99.18
Appendix
Table 9
ID
Compile
Golden
Kill all
Exact model
Cov. ≥ 90
Overall
C01
68
39
8
40
14
4
C02
95
92
44
56
28
14
C03
94
89
41
68
34
16
C04
94
87
45
65
35
17
C05
95
89
49
67
34
18
C06
79
35
7
23
13
2
Appendix
Table 10
ID
L1
L2
L3
L4
L5
C01
3/6
1/7
0/32
0/29
0/21
C02
4/6
2/7
4/32
3/29
1/21
C03
5/6
4/7
1/32
3/29
3/21
C04
5/6
3/7
4/32
5/29
0/21
C05
5/6
3/7
4/32
4/29
2/21
C06
1/6
1/7
0/32
0/29
0/21
Appendix
Table 11
Domain
N
C01
C02
C03
C04
C05
C06
C07
C08
C09
C10
C11
C12
C13
Arithmetic
8
2
3
4
4
4
1
3
4
4
3
4
4
4
Memory
13
0
2
2
1
1
0
1
3
3
0
1
1
2
Digital design
24
1
3
3
5
5
0
4
5
5
1
3
4
5
Interconnect
8
1
1
1
1
1
1
1
1
1
1
1
1
1
Control
7
0
0
1
1
2
0
0
1
1
0
1
1
2
Communication
5
0
1
0
0
1
0
0
1
1
0
0
0
0
Appendix
Table 12
Task
Lv.
C01
C02
C03
C04
C05
C06
C07
C08
C09
C10
C11
C12
C13
dv-001
L1
1
1
1
1
1
1
1
1
1
1
1
1
1
dv-002
L1
0
1
1
1
1
0
1
1
1
1
1
1
1
dv-003
L1
1
0
1
1
1
0
1
1
1
1
1
1
1
dv-004
L1
0
0
0
0
0
0
0
0
0
0
0
0
0
dv-005
L1
1
1
1
1
1
0
1
1
1
1
1
1
1
dv-006
L1
0
1
1
1
1
0
0
1
1
0
1
1
1
Appendix
Table 13
Figure 8: The generated FIFO verification architecture. DUT and model receive the same inputs. The model predicts independently of DUT outputs; the monitor packages observations for the scoreboard. The diagram summarizes the frozen submission.
Task
Mutant
Semantic target
Discriminating scenario
dv-044
m4
Rounding bit
Discarded/retained bits differ
dv-037
m2
Unsigned comparison
Opposite operand sign bits
dv-008
m1
Gray encoding
Adjacent binary ones
dv-016
m7
Any-match reduction
Two simultaneous matches
dv-017
m7
Response ownership
Two ports, two returns
dv-009
m1
FIFO multiplicity
Consecutive equal pushes
Appendix
Table 9: Representative mutants and the verification distinctions they require.
Integrated Circuit (IC) verification consumes nearly 70% of the IC development cycle, and recent research leverages Large Language Models (LLMs) to automatically generate testbenches and reduce verification overhead. However, LLMs have difficulty generating testbenches correctly. Unlike high-level programming languages, Hardware Description Languages (HDLs) are extremely rare in LLMs training data, leading LLMs to produce incorrect code. To overcome challenges when using LLMs to generate Universal Verification Methodology (UVM) testbenches and sequences, wepropose HAVEN (Hybrid Automated Verification ENgine) to prevent LLMs from writing HDL directly. For UVM testbench generation, HAVEN utilizes LLM agents to analyze design specifications to produce a structured architectural plan. The HAVEN Template Engine then combines with predefined and protocol-specific templates to generate all UVM components with correct bus-handshake timing. For UVM sequence generation, HAVEN introduces a Protocol-Aware Sequence Domain-Specific Language (DSL) that decomposes sequences into fine-grained step types. A set of predefined DSL patterns first establishes sequences that achieve a high coverage rate without LLM involvement. HAVEN continues to improve the coverage rate by iteratively leveraging LLM agents to analyze coverage gap reports and compose additional targeted DSL sequences. Unlike previous works, HAVEN is the first system that utilizes pre-defined, protocol-specific Jinja2 templates to generate all UVM components and UVM sequences using our proposed Protocol-Aware DSL and rule-based code generator. Our experimental results on 19 open-source IP designs spanning three interface protocols (Direct, Wishbone, AXI4-Lite) show that HAVEN achieves 100% compilation success, 90.6% code coverage, and 87.9% functional coverage on average, and is SOTA among LLM-assisted testbench generation systems.
Chang-Chih Meng, Yu-Ren Lu, Guan-Yu Lin +3
National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Large language models (LLMs) have made rapid advancements in code generation for popular languages such as Python and C++. Many of these recent gains can be attributed to the use of ``agents'' that wrap domain-relevant tools alongside LLMs. Hardware design languages such as Verilog have also seen improved code generation in recent years, but the impact of agentic frameworks on Verilog code generation tasks remains unclear. In this work, we present the first systematic evaluation of agentic LLMs for Verilog generation, using the recently introduced CVDP benchmark. We also introduce several open-source hardware design agent harnesses, providing a model-agnostic baseline for future work. Through controlled experiments across frontier models, we study how structured prompting and tool design affect performance, analyze agent failure modes and tool usage patterns, compare open-source and closed-source models, and provide qualitative examples of successful and failed agent runs. Our results show that naive agentic wrapping around frontier models can degrade performance (relative to standard forward passes with optimized prompts), but that structured harnesses meaningfully match and in some cases exceed non-agentic baselines. We find that the performance gap between open and closed source models is driven by both higher crash rates and weaker tool output interpretation. Our exploration illuminates the path towards designing special-purpose agents for verilog generation in the future.
Patrick Yubeaton, Siddharth Garg, Chinmay Hegde
New York University, Tandon School of Engineering, Brooklyn, NY
Existing benchmarks for hardware design primarily evaluate Large Language Models (LLMs) on isolated, component-level tasks such as generating HDL modules from specifications, leaving repository-scale evaluation unaddressed. We introduce HWE-Bench, the first large-scale, repository-level benchmark for evaluating LLM agents on real-world hardware bug repair tasks. HWE-Bench comprises 417 task instances derived from real historical bug-fix pull requests across six major open-source projects spanning both Verilog/SystemVerilog and Chisel, covering RISC-V cores, SoCs, and security roots-of-trust. Each task is grounded in a fully containerized environment where the agent must resolve a real bug report, with correctness validated through the project's native simulation and regression flows. The benchmark is built through a largely automated pipeline that enables efficient expansion to new repositories. We evaluate seven LLMs with four agent frameworks and find that the best agent resolves 70.7% of tasks overall, with performance exceeding 90% on smaller cores but dropping below 65% on complex SoC-level projects. We observe larger performance gaps across models than commonly reported on software benchmarks, and difficulty is driven by project scope and bug-type distribution rather than code size alone. Our failure analysis traces agent failures to three stages of the debugging process: fault localization, hardware-semantic reasoning, and cross-artifact coordination across RTL, configuration, and verification components, providing concrete directions for developing more capable hardware-aware agents.