Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models

Koki Takeshita; Kenji Uebayashi; Hikaru Tomonari; Masaya Tsunokake; Koichiro Yawata; Tatsuya Sasaki; Yusuke Ohtsubo; Kota Dohi

cs.AIOct 16, 2026

Pareto-Optimal RTL Code Generation via Multi-Objective Reinforcement Learning with Large Language Models

Authors: Koki Takeshita, Kenji Uebayashi, Hikaru Tomonari, Masaya Tsunokake, Koichiro Yawata, Tatsuya Sasaki, Yusuke Ohtsubo, Kota Dohi,

Organizations: Research & Development Group, Hitachi, Ltd.

Abstract

Recent studies have proposed reinforcement learning with verifiable rewards (RLVR) methods that enable LLMs to generate high-quality RTL circuits from natural-language specifications. Existing approaches optimize circuit PPA (Power, Performance, Area) by aggregating multiple objectives into a single scalar reward, making it difficult to simultaneously generate diverse Pareto-optimal solutions across different trade-offs such as low power, low latency, and small area. However, in practical design, the relative importance of each objective varies depending on product requirements and design constraints; thus, obtaining a diverse and high-quality candidate set is beneficial in that designers can choose a preferred trade-off a posteriori. In this work, we are the first to propose a learning method for LLM-based circuit generation that optimizes, not the quality of a single circuit, but the quality and diversity of the entire circuit set obtained through multiple generations in the PPA objective space. The proposed method consists of (1) a generation-based training pipeline that reformulates coverage improvement of the entire Pareto solution set into a sequential circuit generation problem, and (2) a reward function based on Hypervolume Contribution (HVC) that measures how much a generated circuit contributes to the Pareto front. Evaluation on the RTLLM v2 benchmark shows that the proposed method achieves +13.3 percentage points in relative Hypervolume over a baseline without PPA optimization and +14.4 percentage points over a single-objective reinforcement learning method, while also demonstrating superiority in circuit diversity.

Figures & tables

Figure 1: Overview of the proposed method. By alternating between reinforcement learning with HVC reward (top) and dataset reconstruction (bottom), the model learns Pareto solution exploration capabilities while progressively extending circuit history.

Figure 2: Example of Hypervolume Contribution (HVC). In the objective function space, the HVC is the Hypervolume increment with origin as the reference point when adding candidate

i

. A larger HVC means the candidate contributes more to the Pareto front.

Base Model	Method	rHV (iter=1)	rHV (iter=5)	rHV (iter=10)
VeriReason-3B	Base	13.1%	17.5%	19.1%
	Single-Objective PPA	10.4%	15.4%	18.0%
	Sequential TB	15.5%	18.5%	19.2%
	Sequential TB+PPA	17.2%	20.6%	21.6%
	Sequential HVC (Ours)	22.9%	30.7%	32.4%
CodeV-R1-Distil-7B	Base	23.1%	34.1%	38.4%

Table 1: Relative Hypervolume (rHV) for each model. Each column shows the rHV measured after a given number of sequential generation iterations ( iter ), where the model generates a new circuit conditioned on the circuit history accumulated so far.

Panel of Figure 3 — Figure 3: Win Rate progression per generation count.

Panel of Figure 4 — Figure 4: Circuit diversity comparison.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	VeriReason-3B	CodeV-R1-Distil-7B
Num. GPUs	4	4
GPU	H200	H200
Batch Size (per device)	2	2
Gradient Accumulation Steps	4	4
Learning Rate	$1.0\times 10^{-6}$	$1.0\times 10^{-6}$
Max Steps	15,000	1,500

Appendix

Table 2: Training hyperparameters. Common GRPO settings are shared across all methods on both base models; Max Steps differs by base model.

Base Model	Comparison	rHV Diff (95% CI)
VeriReason-3B	Ours $-$ Base	$+13.3\ [+2.1,\ +24.8]$
	Ours $-$ Single-Objective PPA	$+14.5\ [+2.5,\ +27.0]$
	Ours $-$ Sequential TB	$+13.2\ [+4.7,\ +22.7]$
	Ours $-$ Sequential TB+PPA	$+10.8\ [+1.2,\ +20.9]$
CodeV-R1-Distil-7B	Ours $-$ Base	$+4.8\ [+0.6,\ +9.0]$
	Ours $-$ Single-Objective PPA	$+5.4\ [+1.4,\ +9.9]$

Appendix

Table 3: Bootstrap confidence intervals (95%) of the rHV difference between Sequential HVC (Ours) and each baseline at iter=10.

Base Model	Method	rHV (iter=1)	rHV (iter=5)	rHV (iter=10)
VeriReason-3B	Base	2.2%	6.4%	10.3%
	Single-Objective PPA	0.9%	5.3%	10.3%
	Sequential TB	0.0%	0.2%	0.3%
	Sequential TB+PPA	0.1%	0.9%	3.4%
	Sequential HVC (Ours)	3.0%	15.1%	19.4%
CodeV-R1-Distil-7B	Base	11.5%	23.9%	32.1%

Appendix

Table 4: Median relative Hypervolume (rHV) computed with respect to the baseline-only archive. Each column shows the median rHV after a given number of sequential generation iterations ( iter ).

Panel of Figure 5 — Figure 5: Scatter plots of the generated solutions in the objective space for each base model. Visualizing the solutions in the objective space clarifies how each method covers different trade-off regions of the Pareto front. Sequential HVC (Ours) attains a broader coverage of the objective space than the baselines.

Panel of Figure 6 — Figure 6: Effect of adding an explicit repetition penalty to the sequential baseline, evaluated in terms of relative Hypervolume (top), Win Rate (bottom left), and circuit diversity (bottom right). Although the repetition penalty reduces duplicate generations, it does not match the improvement of Sequential HVC, indicating that the gains of our method arise from directly optimizing the Pareto-front contribution rather than merely avoiding duplicates.

Panel of Figure 7 — Figure 7: Relative Hypervolume (rHV) evaluated on the union of solutions generated across multiple trials for each method. Aggregating the solutions from independent trials measures the overall Pareto-front coverage attainable by each method. Sequential HVC (Ours) retains its advantage over the baselines for smaller numbers of unioned trials. The results for CodeV-R1-Distil-7B are shown on the following page.

Panel of Figure 8 — Figure 8: (continued) Relative Hypervolume (rHV) evaluated on the union of solutions generated across multiple trials for each method.

RTL Code Generation

Register-transfer-level (RTL) code generation turns hardware specifications into synthesizable circuits, balancing functional correctness with area, timing, and power.

See all 31 papers

COEVO: Co-Evolutionary Framework for Joint Functional Correctness and PPA Optimization in LLM-Based RTL Generation
Apr 16, 2026Heng Ping, Peiyu Zhang, Shixuan Li +5Multi-Objective Evolutionary Optimization Code Generation
University of Southern California United States
POET: Power-Oriented Evolutionary Tuning for LLM-Based RTL PPA Optimization
Mar 19, 2026Heng Ping, Peiyu Zhang, Zhenkun Wang +5Large Language Model-Guided Optimization Evolutionary Optimization
University of Southern California, United States
RTLSeek: Boosting the LLM-Based RTL Generation with Multi-Stage Diversity-Oriented Reinforcement Learning
Mar 29, 2026Xinyu Zhang, Zhiteng Chao, Yonghao Wang +6RTL Code Generation LLM Post-Training
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, China · University of Chinese Academy of Sciences, China · Peking University, China +1

Apr 16, 2026J/K move · Enter open · S save

COEVO: Co-Evolutionary Framework for Joint Functional Correctness and PPA Optimization in LLM-Based RTL Generation

Heng Ping, Peiyu Zhang, Shixuan Li, Wei Yang+4

University of Southern California United States

Multi-Objective Evolutionary Optimization Code Generation LLM-Based Program Synthesis RTL Code Generation Code Language Models

Open paper AskPDF arXiv

LLM-based RTL code generation methods increasingly target both functional correctness and PPA quality, yet existing approaches universally decouple the two objectives, optimizing PPA only after correctness is fully achieved. Whether through sequential multi-agent pipelines, evolutionary search with binary correctness gates, or hierarchical reward dependencies, partially correct but architecturally promising candidates are systematically discarded. Moreover, existing methods reduce the multi-objective PPA space to a single scalar fitness, obscuring the trade-offs among area, delay, and power. To address these limitations, we propose COEVO, a co-evolutionary framework that unifies correctness and PPA optimization within a single evolutionary loop. COEVO formulates correctness as a continuous co-optimization dimension alongside area, delay, and power, enabled by an enhanced testbench that provides fine-grained scoring and detailed diagnostic feedback. An adaptive correctness gate with annealing allows PPA-promising but partially correct candidates to guide the search toward jointly optimal solutions. To preserve the full PPA trade-off structure, COEVO employs four-dimensional Pareto-based non-dominated sorting with configurable intra-level sorting, replacing scalar fitness without manual weight tuning. Evaluated on VerilogEval 2.0 and RTLLM 2.0, COEVO achieves 97.5% and 94.5% Pass@1 with GPT-5.4-mini, surpassing all agentic baselines across four LLM backbones, while attaining the best PPA on 43 out of 49 synthesizable RTLLM designs.

Abstract

RTL Code Generation

Explore similar work

COEVO: Co-Evolutionary Framework for Joint Functional Correctness and PPA Optimization in LLM-Based RTL Generation

POET: Power-Oriented Evolutionary Tuning for LLM-Based RTL PPA Optimization

RTLSeek: Boosting the LLM-Based RTL Generation with Multi-Stage Diversity-Oriented Reinforcement Learning