RTLReviewBench: LLM Hardware Reviewers Rarely Miss Bugs but Routinely Reject Correct RTL
Organizations: University of Florida
Abstract
Hardware teams increasingly ask large language models (LLMs) to review register-transfer level (RTL) code, yet it is unclear whether a model can tell a correct module from a buggy one without simulating it. We introduce RTLReviewBench, a benchmark of 114 matched pairs of buggy and correct Verilog modules. Each pair shares one specification, most buggy modules are real failures of LLM code generators, and every label is confirmed by simulation. Evaluating 12 commercial and open-weight models, we find a consistent asymmetry: reviewers rarely accept buggy code but often reject correct code, producing more than five false rejections for every missed bug. Synthetic mutants hide part of the problem: on the same designs, models miss far fewer injected bugs than real ones. Asking a model to reason before it commits to a verdict sharply reduces false rejections for weaker reviewers, and a controlled ablation shows that where the reasoning appears in the output matters more than the instruction to reason, whereas sampling and majority voting do not help. A blinded audit further shows that some correct "buggy" verdicts rest on false technical claims. We release the benchmark, a harder split for future models, all model responses, and the evaluation code at https://github.com/yasirmahmud/RTLReviewBench
Figures & tables
| Benchmark | Domain | Task | Matched pairs | FAR & FRR |
|---|---|---|---|---|
| VerilogEval v2 [ 13 ] | RTL | Generate RTL from a specification | – | – |
| ChipBench [ 21 ] | RTL | Generate, debug, and write reference models | – | – |
| VeriBugBench [ 12 ] | RTL | Localize and repair injected faults | – | – |
| RTLRepairBench [ 25 ] | RTL | Repair real and synthetic faults | partial | – |
| CodeJudgeBench [ 6 ] | Software | Rank two candidate responses | ✓ | – |
| SWR-Bench [ 22 ] | Software | Comment on pull requests | ✓ | ✓ |
| Split | Designs | Purpose |
|---|---|---|
| Clean-114 | 114 | Primary evaluation: 97 real LLM failures (RTLRepairBench) and 17 hand-written faults (ChipBench); 80 sequential designs |
| Full-120 | 120 | Sensitivity check: Clean-114 plus the six Artifact-6 pairs |
| SynMut-50 | 50 | Single-operator synthetic mutants (six operator types) |
| Matched-42 | 42 | Designs present in both Clean-114 and SynMut-50 |
| Hard | 22 | Pairs that at least 6 of 12 reviewers get wrong; headroom for future models |
| Reviewer | Accuracy | FAR | FRR | Pair |
|---|---|---|---|---|
| Claude Haiku 5.5 (thinking) | 97.4 [94.4, 99.0] | 2.6 | 1.8 | 94.7 |
| Claude Haiku 5.5 | 97.4 [94.4, 99.0] | 3.5 | 1.8 | 94.7 |
| Gemini 3.1 Pro | 96.9 [93.8, 98.8] | 0.9 | 5.3 | 93.9 |
| Gemini 3.8 Flash | 93.4 [89.4, 96.3] | 0.0 | 13.2 | 86.8 |
| Claude Sonnet 4.6 | 93.0 [88.9, 95.9] | 2.6 | 11.4 | 87.7 |
| gpt-oss-120b | 91.7 [87.3, 94.9] | 1.8 | 14.9 | 84.2 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Pipeline Gate / Disposition | Count | Exclusion / Selection Basis ( Section B.2 ) |
| Core Track — Source 1: ChipBench (MIT; Sequential Multi-Always Pool) | ||
| Upstream debugging rows scanned | 89 | state_machine (6), timing (29), arithmetic (24), assignment (30) |
| -- Excluded (incomplete spec / spec–ref conflict / non-seq) | Missing standalone specs ( ), conflicts ( ), comb. slice ( ) | |
| -- Excluded by Icarus 11.0 compile / TB gate | Buggy compile fails ( ), golden reference TB mismatches ( ) | |
| Eligible Clean ChipBench Retained | 49 17 | 17 seq / 0 comb ( arithmetic : 11, assignment : 6; artifacts) |
| Core Track — Source 2: RTLRepairBench RealBugs (Apache-2.0 / MIT; VerilogEval v2 Pool) | ||
| Dataset Slice | Pairs ( ) | Cases ( ) | Seq / Comb | RTL Lines of Code | Specification Words | Bug-1st | ||
| Mean | Med [Min–Max] | Mean | Med [Min–Max] | Order | ||||
| ChipBench (Sequential, Clean) | 17 | 34 | 17 / 0 | 48.6 | 43.5 [11–107] | 249.9 | 254.0 [144–374] | 6 / 17 |
| RTLRepairBench Base (Clean) | 90 | 180 | 61 / 29 | 26.2 | 20.0 [6–134] | 148.2 | 117.5 [45–496] | 57 / 90 |
| RTLRepairBench Tuned (Clean) | 7 | 14 | 2 / 5 | 12.8 | 11.5 [6–27] | 85.7 | 65.0 [41–170] | 5 / 7 |
| Clean Sequential / Combinational | 80 / 34 | 160 / 68 | 80 / 34 | 33.5 / 17.5 | 27.0 / 13.0 | 175.2 / 122.6 | 144.5 / 110.0 | 47/80 ; 21/34 |
| Clean-114 Primary Track | 114 | 228 | 80 / 34 | 28.7 | 22.0 [6–134] | 159.5 | 132.0 [41–496] | 68 / 114 |
| Source Asset | Origin / Role | Commit SHA | License | Retained Pairs |
|---|---|---|---|---|
| ChipBench [ 21 ] | Debug candidates + self-contained specs | b4ba3a067598... | MIT | 17 Clean (34c) |
| RTLRepairBench RealBugs [ 25 ] | LLM fails + VerilogEval v2 refs | 17709198a893... | Apache-2.0 / MIT | 97 Clean (+6 Art.) |
| RTLRepairBench Sem85 [ 25 ] | Single-op mutants ( SynMut-50 ) | 17709198a893... | Apache-2.0 / MIT | 50 (100c) |
| Clean-114 + SynMut-50 | Clean Matched ( Art. overlap, Syn-only) | — | — | 164 (+6 Art.) |
| Model ID | Deployment Paradigm | Reasoning | Core | SynMut | Reason- | Pairwise / | SC@5 |
|---|---|---|---|---|---|---|---|
| Tokens † | 228 [240] | (100) | First (228) | Audit (1,309) | (1,140) | ||
| claude-haiku-5.5 | Budget Commercial (Anthropic) | None (0) | ✓ | ✓ | ✓ | ✓/ ✓ | ✓ |
| gemini-3.8-flash † | Budget Commercial (Google) | Adaptive ( – ) | ✓ | ✓ | ✓ | ✓/ ✓ | ✓ |
| gpt-4.1-mini | Budget Commercial (OpenAI) | None (0) | ✓ | ✓ | ✓ | ✓/ ✓ | — |
| claude-4.6-sonnet | Flagship Commercial (Anthropic) | Prose / Ext. (0) | ✓ | ✓ | ✓ | ✓/ ✓ | — |
| gemini-3.1-pro † | Flagship Commercial (Google) | Native ( Clean) | ✓ | ✓ | ✓ | ✓/ ✓ | — |
| Model / Gate | Clean Acc. ( , 95% CI) | FAR (Miss) | FRR (Over-Rej.) | Pair: Both / 1 / 0 | Brier | Calib. | Med. Lat. | Med. Tok. | Full-120 Sens. | |
| ( ) | ( ) | ( Pairs) | Score | Gap | (ms) | (Ok / Err) | Acc. | FAR | ||
| Single Models ( , Sorted by Clean-114 Strict Accuracy then Brier Score) | ||||||||||
| claude-haiku-5.5-thk ‡ | 222/228 (97.4%) [94.4, 99.0] | 3/114 (2.6%) | 2/114 (1.8% ‡ ) | 108 / 6 / 0 (94.7%) | 0.0280 | 8998 | 555.0 / 1403.5 | 95.0% | 7.5% | |
| claude-haiku-5.5 | 222/228 (97.4%) [94.4, 99.0] | 4/114 (3.5%) | 2/114 (1.8%) | 108 / 6 / 0 (94.7%) | 0.0310 | 3617 | 476.5 / 1595.0 | 95.0% | 8.3% | |
| gemini-3.1-pro | 221/228 (96.9%) [93.8, 98.8] | 1/114 (0.9%) | 6/114 (5.3%) | 107 / 7 / 0 (93.9%) | 0.0299 | 12237 | 903.0 / 1878.0 | 95.0% | 5.0% | |
| gemini-3.8-flash | 213/228 (93.4%) [89.4, 96.3] | 0/114 (0.0%) | 15/114 (13.2%) | 99 / 15 / 0 (86.8%) | 0.0590 | 5505 | 431.0 / 1218.0 | 92.5% | 2.5% | |
| Model | Clean-114 Acc. | Acc. | FAR | FRR | Pair | Reason-First Acc. |
|---|---|---|---|---|---|---|
| claude-haiku-5.5 | 97.4% | 88.6% | 18.2% | 4.5% | 77.3% | 90.9% |
| claude-haiku-5.5-thinking | 97.4% | 88.6% | 13.6% | 9.1% | 77.3% | 90.9% |
| gemini-3.1-pro | 96.9% | 84.1% | 4.5% | 27.3% | 68.2% | 90.9% |
| gpt-oss-120b | 91.7% | 75.0% | 9.1% | 40.9% | 54.5% | 72.7% |
| gemini-3.8-flash | 93.4% | 72.7% | 0.0% | 54.5% | 45.5% | 77.3% |
| claude-4.6-sonnet | 93.0% | 70.5% | 13.6% | 45.5% | 50.0% | 77.3% |
| Model | SynMut-50 Full Track ( ) | Clean Matched-42 Base Designs ( Cases Each) | |||||
| Acc. (95% CI) | FAR ( ) | FRR ( ) | Real Syn Acc. | / | Real Syn FAR | Real Syn FRR | |
| claude-haiku-5.5-thinking | 99.0% [94.6,100] | 0.0% (0) | 2.0% (1) | / | |||
| gemini-3.1-pro | 99.0% [94.6,100] | 0.0% (0) | 2.0% (1) | / | |||
| claude-haiku-5.5 | 97.0% [91.5, 99.4] | 2.0% (1) | 4.0% (2) | / | |||
| gemini-3.8-flash | 96.0% [90.1, 98.9] | 2.0% (1) | 6.0% (3) | / | |||
| o4-mini | 96.0% [90.1, 98.9] | 0.0% (0) | 8.0% (4) | / | |||
| Model | Prompt Intervention: Verdict-First Reason-First (Clean ) | Contrastive Pairwise (Clean Pairs ) | |||||||||||
| Acc. (VF RF) | Acc | / | FAR (VF RF) | FRR (VF RF) | Brier | Tok. | Pool | Swap | Iso | Flips (A/B) | Bias | Neith. | |
| gpt-4.1 | / | 6 (5/1) | 40 | ||||||||||
| gpt-4.1-mini | / | 11 (9/2) | 46 | ||||||||||
| gemma-4-31b-it | / | ( / ) | 34 | ||||||||||
| gpt-5.4 | / | ( / ) | 43 | ||||||||||
| o4-mini | / | ( / ) | 14 | ||||||||||
| Model | Cell (Order, Trace) | Clean Acc. (95% CI) | FAR | FRR | Pair-Both | Brier | Tok. § | VF ( ) |
| gpt-4.1 | VF ( V C R , no) | 159/228 (69.7%) [63.3, 75.6] | 1.8% | 58.8% | 40.4% | 0.2716 | 83.0 | — |
| A: vf_trace ( V C R , yes) | 153/228 (67.1%) [60.6, 73.2] | 3.5% | 61.4% | 36.8% | 0.2994 | 238.0 | ( ) | |
| B: rf_order_only ( R V C , no) | 177/228 (77.6%) [71.7, 82.9] | 10.5% | 33.3% | 58.8% | 0.2017 | 110.0 | ( ) | |
| RF ( R V C , yes) | 196/228 (86.0%) [80.8, 90.2] | 8.8% | 19.3% | 73.7% | 0.1346 | 270.5 | ( ) | |
| gpt-4.1-mini | VF ( V C R , no) | 164/228 (71.9%) [65.6, 77.7] | 2.6% | 53.5% | 45.6% | 0.2348 | 100.0 | — |
| A: vf_trace ( V C R , yes) | 157/228 (68.9%) [62.4, 74.8] | 3.5% | 58.8% | 40.4% | 0.2618 | 315.0 | ( ) |
| Model / Regime | 5-Draw Mean SD | maj@5 Acc. (95% CI) | maj@5 FAR | maj@5 FRR | Pair Both | Vote Brier | 5:0 Ok / 5:0 Err / Split | vs. T=0 / RF | vs. 3x Div. |
|---|---|---|---|---|---|---|---|---|---|
| claude-haiku-5.5 ( ) | 96.1% [92.6, 98.2] | 4.4% [1.4,9.9] | 3.5% [1.0,8.7] | 92.1% | 0.025 | 213 / 2 / 13 ( ) | / | ( ) | |
| gemini-3.8-flash † ( ) | 93.4% [89.4, 96.3] | 0.0% [0.0,3.2] | 13.2% [7.6,20.8] | 86.8% | 0.055 | 205 / 9 / 14 ( ) | / | ( ) | |
| gpt-5.4 ( ) | 83.3% [77.8, 87.9] | 3.5% [1.0,8.7] | 29.8% [21.6,39.1] | 67.5% | 0.146 | 184 / 28 / 16 ( ) | / ( ) | ( ) | |
| Top-3 Diverse Maj. ( ) | — ( ) | 98.2% [95.6, 99.5] | 0.9% [0.0,4.8] | 2.6% [0.5,7.5] | 96.5% | — | — | — | Reference |
| Reviewer Model | TPs Audited | Grounded | Spurious | Vague | Disp. | Grounded | Nominal | Eff. Grounded |
|---|---|---|---|---|---|---|---|---|
| ( ) | Claim | Prec./TP | TPR | Sens. ( Gap) | ||||
| gemini-3.1-pro | 113 | 113 | 0 | 0 | 0 | 100.0% [96.8, 100.0] | 99.1% | 99.1% [95.2, 100.0] ( ) |
| gemini-3.8-flash † | 114 | 111 | 3 | 0 | 0 | 97.4% [92.5, 99.5] | 100.0% | 97.4% [92.5, 99.5] ( ) |
| claude-haiku-5.5 | 110 | 110 | 0 | 0 | 0 | 100.0% [96.7, 100.0] | 96.5% | 96.5% [91.3, 99.0] ( ) |
| claude-4.6-sonnet | 111 | 110 | 1 | 0 | 0 | 99.1% [95.1, 100.0] | 97.4% | 96.5% [91.3, 99.0] ( ) |
| claude-haiku-5.5-thinking † | 111 | 110 | 1 | 0 | 0 | 99.1% [95.1, 100.0] | 97.4% | 96.5% [91.3, 99.0] ( ) |
| Class | Hardware Failure Mechanism (Representative Case IDs) | Haiku 5.5 | Hk-Thk | Gem 3.1P | Gem 3.8F | GPT-4.1m |
| Clean-114 False Acceptances (FN: Buggy RTL Accepted as correct ) | ||||||
| FN-1 | Clean simulation-only / init artifacts (excluded via Artifact-6 ) | 0 | 0 | 0 | 0 | 0 |
| FN-2 | Sequential control, shift slice, pipeline, & async reset / FSM ( C003 , C020 , C209 ) | 3 | 3 | 1 | 0 | 2 |
| FN-3 | Combinational priority order & bit indexing ( C184 ) | 1 | 0 | 0 | 0 | 1 |
| Clean Sub. FN | 4 | 3 | 1 | 0 | 3 | |
| Clean-114 False Rejections (FP) & Schema Errors ( I ) | ||||||
| Study / Experimental Matrix | Queries | Prompt Tok. | Compl. Tok. | Reas. Tok. † | Total Tok. | Decoding |
|---|---|---|---|---|---|---|
| Study 1: Core Verdict-First ( ) | 2,880 | 1,871,093 | 1,637,156 | 1,019,865 | 3,508,249 | Greedy ( ) |
| Study 2: SynMut-50 Synthetic ( ) | 1,200 | 722,676 | 605,158 | 426,848 | 1,327,834 | Greedy ( ) |
| Study 3a: Reason-First Prompting ( ) | 2,880 | 1,967,333 | 2,368,805 | 1,363,141 | 4,336,138 | Greedy ( ) |
| Study 3b: Self-Consistency maj@5 ( ) | 3,600 | 2,445,800 | 1,732,738 | 786,419 | 4,178,538 | Sample ( ) |
| Study 3c: Reason-First Ablation ( ) | 1,440 | 893,836 | 413,708 | 0 | 1,307,544 | Greedy ( ) |
| Study 4: Bidirectional Pairwise ( ) | 2,880 | 2,576,278 | 1,731,229 | 1,211,111 | 4,307,507 | Greedy ( ) |
Automated Code Review
Automated code review uses software analysis and AI to find defects, assess changes, and support pull-request decisions.
See all 57 papers