FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification
Organizations: University of Illinois Urbana-Champaign · NVIDIA · University of Maryland, College Park
Abstract
The remarkable reasoning and code generation capabilities of large language models (LLMs) have recently motivated increasing interest in automating formal verification (FV), a process that ensures hardware correctness through mathematically precise assertions but remains highly labor-intensive, particularly through the translation of natural language into SystemVerilog Assertions (NL-to-SVA). However, LLMs still struggle with SVA generation due to limited training data and the intrinsic complexity of FV operators. Consequently, a more efficient and robust methodology for ensuring correct SVA operator selection is essential for producing functionally correct assertions. To address these challenges, we introduce FVRuleLearner, an Operator-Level Rule (Op-Rule) learning framework built on a novel Operator Reasoning Tree (Op-Tree), which models SVA generation as structured, interpretable reasoning. FVRuleLearner operates in two complementary phases: (1) Training: it constructs Op-Tree that decomposes NL-to-SVA alignment into fine-grained, operator-aware questions, combining reasoning paths that lead to correct assertions; and (2) Testing: it performs operator-aligned retrieval to fetch relevant reasoning traces from the learned Op-Tree and generate new rules for unseen specifications. In the comprehensive studies, the proposed FVRuleLearner outperforms the state-of-the-art baseline by 3.95% in syntax correctness and by 31.17% in functional correctness on average. Moreover, FVRuleLearner successfully reduces an average of 70.33% of SVA functional failures across diverse operator categories through a functional taxonomy analysis, showing the effectiveness of applying learned Op-Tree to the Op-Rule generations for unseen NL-to-SVA tasks. These results establish FVRuleLearner as a new paradigm for domain-specific reasoning and rule learning in formal verification.
Figures & tables
| Operator Category | Representative Operators |
|---|---|
| Temporal Implication Operators | |-> , |=> |
| Temporal Delay Operators | ##m , [m:n] |
| Temporal Sampling Operators | rose , stable |
| Temporal Liveness Operators | s_eventually , s_always , strong |
| Combinational Logic Operators | && , || , ! , == , !== , ˆ |
| Miscellaneous Differences | @(posedge clk) , disable iff(rst) |
| Model / Method | NL2SVA-Human | NL2SVA-Machine | NL2SVA-OpenCore | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU | Syn | Func | R.Func | BLEU | Syn | Func | R.Func | BLEU | Syn | Func | R.Func | |
| GPT-4o | ||||||||||||
| FVEval-0-shot | 0.5203 | 0.9333 | 0.5333 | 0.6667 | 0.6054 | 0.9500 | 0.4167 | 0.4500 | 0.6145 | 0.9250 | 0.7050 | 0.7200 |
| FVEval-3-shot | 0.5823 | 0.9333 | 0.4000 | 0.6000 | 0.6461 | 0.9500 | 0.4333 | 0.5167 | 0.6187 | 0.8950 | 0.6750 | 0.6950 |
| Ensemble | 0.5876 | 0.8667 | 0.4000 | 0.4667 | 0.6562 | 0.9667 | 0.5000 | 0.6000 | 0.5922 | 0.9550 | 0.7000 | 0.7150 |
| FVRuleLearner | 0.6846 | 1.0000 | 0.8000 | 0.8000 | 0.7072 | 0.9667 | 0.7500 | 0.8167 | 0.6976 | 0.9900 | 0.8500 | 0.8750 |
| Model / Method | Syn | Func | R.Func |
|---|---|---|---|
| o3-mini | |||
| FVEval-3-shot | 0.9500 | 0.4833 | 0.5833 |
| General Rules | 0.9500 | 0.6667 | 0.7500 |
| Static Op-Rules | 0.9500 | 0.7167 | 0.7833 |
| FVRuleLearner | 0.9667 | 0.7833 | 0.8333 |
| Op. Category | NL2SVA-Human | NL2SVA-Machine | NL2SVA-OpenCore | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Base | Ours | Red. | Base | Ours | Red. | Base | Ours | Red. | |
| Temp. Impl. | 4 | 1 | -75% | 19 | 4 | -79% | 41 | 5 | -88% |
| Temp. Delay | 2 | 0 | -100% | 1 | 0 | -100% | 20 | 9 | -55% |
| Temp. Sampl. | 0 | 0 | – | 3 | 2 | -33% | 2 | 1 | -50% |
| Temp. Live. | 1 | 0 | -100% | 4 | 3 | -25% | 0 | 0 | – |
| Comb. Logic | 0 | 0 | – | 2 | 2 | 0% | 0 | 0 | – |