Large language models have improved Verilog generation, yet functional correctness remains difficult to optimize because simulation commonly provides a coarse module-level pass/fail outcome. Such feedback discards useful evidence from partially correct RTL modules and cannot determine whether a source statement is functionally responsible for an observed output behavior. We present a new method named Causal Signal Credit Assignment (CSCA), which is a counterfactual preference-optimization framework for Verilog generation. CSCA first performs continuous signal-level behavioral verification on sampled candidate modules. It then locates output-driving assignment spans and applies five interventions while holding the remaining module and testbench fixed. The resulting changes in signal-level errors are combined with behavioral agreement to form a continuous statement credit, which is mapped to response-token weights for strict weighted direct preference optimization. This enables CSCA to learn from locally supported statements without uniformly reinforcing an otherwise incorrect response. Built on Icarus Verilog and a CodeV-derived corpus, CSCA is evaluated on VerilogEval2.0 and RTLLMv2.0, including both code completion and specification-to-RTL tasks. With a Qwen2.5-Coder-7B-Instruct backbone, CSCA leads the evaluated 7B-scale models and remains comparable to the much larger GPT-5.4-mini on the reported Verilog-generation benchmarks.
Figures & tables
Figure 1. Overview of CSCA. A naive code model is first fine-tuned on prompt–reference-code pairs to obtain an SFT model. The SFT model samples candidate RTL implementations, which are assessed by continuous behavioral verification and counterfactual causal attribution. The resulting statement-level credits are used to localize weighted DPO updates.
Figure 2. Examples of CSCA’s five local counterfactual interventions. Starting from the same output-driving statement, CSCA creates Zero, One, Hold, Invert, and Delete variants by modifying only the target assignment while preserving the remaining statements in the local begin...end block. Each variant is subsequently executed using the original testbench to measure its signal-level behavioral impact.
Type
Model
Size
VerilogEval (Completion)
VerilogEval (Specification)
RTLLM v2.0
T=0
T=0.8
T=0
T=0.8
T=0
T=0.8
Pass@1
Pass@1
Pass@5
Pass@1
Pass@1
Pass@5
Pass@1
Pass@1
Pass@5
Foundation General Model
Llama3.1
8B
2.6
2.1
6.4
19.1
16.8
32.1
6.0
6.0
16.0
Qwen2.5-7B
7B
21.8
12.8
35.3
24.4
21.4
42.3
24.0
19.6
42.0
Qwen3-30B-A3B
30B
43.3
43.5
51.9
44.2
46.2
57.1
38.0
39.6
52.0
DeepSeek-v3
671B
68.0
66.1
71.8
68.8
66.9
72.4
56.0
59.1
71.5
Table 1. Main Results on VerilogEval2.0 and RTLLM v2.0 Benchmarks (%). T=0 denotes greedy decoding, for which only Pass@1 can be evaluated since a single sample is generated. For T=0.8 , we report both Pass@1 and Pass@5. Bold values indicate the best performance among Verilog-specific models.
Configuration
VerilogEval 2.0
RTLLM v2.0
Code Completion
Spec.-to-RTL
Syntax Pass@1
Syntax Pass@5
Pass@1
Pass@5
Pass@1
Pass@5
Pass@1
Pass@5
Base Model (Qwen2.5-Coder)
27.9
43.0
22.0
50.6
74.0
88.0
33.2
52.5
+ SFT
32.7
44.2
31.0
53.8
75.2
96.2
35.5
50.0
+ SFT + Counterfactual Effect
49.7
65.6
43.2
62.0
84.5
94.0
41.0
55.3
+ SFT + Behavioral Correctness
49.3
65.2
42.8
61.3
84.8
94.0
41.2
56.0
Table 2. Credit component ablation study of CSCA ( T=0.8 ). All variants share the same base model, prompt set, and simulation budget. VerilogEval 2.0 is split into Code Completion and Specification-to-RTL subtasks. The largest improvement among all ablations for each metric is marked in bold.
Figure 5
Temperature
VerilogEval 2.0
RTLLM v2.0
Code Completion
Spec.-to-RTL
Syntax Pass@1
Syntax Pass@5
Pass@1
Pass@5
Pass@1
Pass@5
Pass@1
Pass@5
T=0.0†
56.4
—
49.3
—
88.0
—
50.0
—
T=0.1
53.7
57.7
46.3
52.6
87.2
90.0
45.2
54.0
T=0.2
54.1
59.6
45.1
51.9
87.2
92.0
45.6
56.0
T=0.3
55.1
60.9
45.4
56.4
86.8
94.0
47.2
60.0
Table 3. Functional and syntax pass rates (%) over temperature T . T=0 denotes greedy decoding ( N=1 ), where Pass@5 metrics are not applicable. For T≥0 , N=5 samples are used.
Figure 5. Counterfactual gain statistics from 893225 RTL intervention samples. (a) Distribution of intervention outcomes. (b) Histogram of nonzero counterfactual gains for informative records.
Task
Verilog generated by SFT
Verilog generated by CSCA
Prob030_popcount255
assign count_8bit = in[7:0] + in[15:8] + ⋯ + in[255:248]; Adds byte values rather than their constituent bits.
always @(*) begin pop_count = 0; for (integer i=0; i<255; i=i+1) pop_count = pop_count + in[i]; end
Testbench
0/215
215/215
Pass rate
0.0%
100.0%
Prob037_ review2015_count1k
always @(posedge clk or posedge reset) if (reset) counter <= 0; else counter <= counter + 1; Uses asynchronous reset and never wraps at 1000.
always @(posedge clk) begin if (reset) q <= 0; else if (q == 10’d999) q <= 0; else q <= q + 1; end
Testbench
4896/8027
8027/8027
Pass rate
61.0%
100.0%
Table 4. Representative functional improvements over the SFT baseline. Both candidates compile. The benchmark testbench reports passing samples and pass rate.
Large language models (LLMs) have made rapid advancements in code generation for popular languages such as Python and C++. Many of these recent gains can be attributed to the use of ``agents'' that wrap domain-relevant tools alongside LLMs. Hardware design languages such as Verilog have also seen improved code generation in recent years, but the impact of agentic frameworks on Verilog code generation tasks remains unclear. In this work, we present the first systematic evaluation of agentic LLMs for Verilog generation, using the recently introduced CVDP benchmark. We also introduce several open-source hardware design agent harnesses, providing a model-agnostic baseline for future work. Through controlled experiments across frontier models, we study how structured prompting and tool design affect performance, analyze agent failure modes and tool usage patterns, compare open-source and closed-source models, and provide qualitative examples of successful and failed agent runs. Our results show that naive agentic wrapping around frontier models can degrade performance (relative to standard forward passes with optimized prompts), but that structured harnesses meaningfully match and in some cases exceed non-agentic baselines. We find that the performance gap between open and closed source models is driven by both higher crash rates and weaker tool output interpretation. Our exploration illuminates the path towards designing special-purpose agents for verilog generation in the future.
Patrick Yubeaton, Siddharth Garg, Chinmay Hegde
New York University, Tandon School of Engineering, Brooklyn, NY
Large language models (LLMs) have demonstrated remarkable capabilities in generating Verilog code from natural language descriptions. However, Verilog code inherently encodes the structural characteristics of hardware circuits, including data path organization and interconnections among components. Effectively leveraging these structural characteristics to enhance the functional and syntactic correctness of LLM-based Verilog code generation remains a significant challenge. To address this challenge, we propose VeriGRAG, a novel framework that converts Verilog code into data path graphs and employs graph neural networks (GNNs) to encode their structural information. A multimodal retriever, trained through knowledge distillation from a cross-attention encoder to a dual encoder, then selects the graph embeddings most relevant to the input hardware description. The VeriFormer module uses learnable query tokens and two-stage training to extract code-relevant structural features and align them with the LLM's input embedding space, producing structure-aware soft prompts. Experiments on VerilogEval and RTLLM show improvements in both syntactic and functional correctness. With Qwen2.5-Coder-32B as its backbone, VeriGRAG achieves the highest functional pass@1 and pass@5 scores among the compared methods on both benchmarks.
Recent advances in large language models have improved code generation, but their use in hardware description languages is still limited. Moreover, training data and testbenches for these models are often scarce. This paper presents a workflow that uses multi-agent models to generate testbenches for high-quality fine-tuning data. By automating testbench creation, the fine-tuned model for the specification-to-Verilog task achieves performance comparable to state-of-the-art methods on the refined VerilogEval v2 benchmark while using less training data. This study provides a basis for future work on LLM-based HDL generation and automated verification.
Mu-Chi Chen, Po-Hsuan Huang, Yu-Hung Kao +6
Academia Sinica, Taipei, Taiwan · National Taiwan University, Taipei, Taiwan · National Cheng Kung University, Tainan, Taiwan