MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
Organizations: School of Computing, National University of Singapore.
Abstract
Large language models (LLMs) have achieved strong performance on code completion in general-purpose programming languages, but existing repository-level benchmarks focus almost exclusively on software code and largely overlook hardware description languages (HDLs). We present MHRC-Bench, consisting of MHRC-Bench-Train and MHRC-Bench-Eval, the first benchmark for multilingual hardware code completion at the repository level with functional evaluation. MHRC-Bench covers three major hardware design coding styles and annotates each completion target with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree (CST) analysis. Post-training billion-scale code LLMs on MHRC-Bench-Train enables smaller open-weight models to outperform much larger general-purpose models under matched context windows. Beyond aggregate accuracy, our analysis surfaces four findings that distinguish hardware from software code completion: (i)~performance varies sharply across HDLs, with high-level synthesis (HLS) easiest and VHDL hardest in most settings across functional pass rate, exact match (EM), and edit similarity (ES); (ii)~hardware accuracy is non-monotonic in code-structure depth, in contrast with the monotonic depth--accuracy curve known for software; (iii)~accuracy plateaus once the input context exceeds 2{,}048 tokens, while software completion continues to benefit from longer context; and (iv)~certain repository-level retrievers tuned on software underperform a no-retrieval baseline on HDL repositories because their structural and dependency assumptions are violated by hardware code.
Figures & tables
| Benchmark | Level | Task Type | Input | Languages | FG | Train | Output | Data types | Samples |
| VerilogEval | Module | Code Gen | NL Spec | RTL | × | ✓ | Module Code | Synthetic and Real-world | 8,502 |
| RTLLM | Module | Code Gen | NL Spec | RTL | × | × | Module Code | Real-world | 30 |
| HLS-Eval | Module | Code Gen | NL Spec or C Code | HLS | × | × | Module Code | Real-world | 94 |
| HDLEval | Module | Code Gen | NL Spec | RTL, DSL | × | × | Module Code | Real-world | 461 |
| RTL-Repo | Repository | Code Completion | Repository Code | RTL | × | ✓ | Single-line | Real-world | 4098 |
| Ours | Repository | Code Completion | Repository Code | RTL,HLS,DSL | ✓ | ✓ | Single & Multi-line | Real-world | 47,200 |
| Model | Retrieval Method | Preceding context | Full context | ||||||||||||||
| Chisel | V/SV | VHDL | HLS | Chisel | V/SV | VHDL | HLS | ||||||||||
| EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | ||
| GPT-5 | No RAG | 15.2 | 51.6 | 21.9 | 55.4 | 20.8 | 58.5 | 12.1 | 46.7 | 24.8 | 56.7 | 34.4 | 60.9 | 15.3 | 52.0 | 31.4 | 63.7 |
| RepoCoder | 18.3 | 54.1 | 22.4 | 56.1 | 22.4 | 60.3 | 14.7 | 47.2 | 27.6 | 59.6 | 35.4 | 61.3 | 16.7 | 54.1 | 34.4 | 63.9 | |
| GraphCoder | 11.1 | 17.3 | 14.4 | 35.4 | 7.9 | 33.1 | 16.2 | 25.2 | 13.9 | 44.2 | 15.7 | 40.1 | 7.5 | 35.7 | 25.0 | 49.5 | |
| RLCoder | 18.0 | 55.3 | 23.8 | 55.9 | 26.6 | 61.4 | 17.3 | 56.2 | 27.9 | 60.2 | 36.2 | 61.7 | 17.3 | 55.3 | 36.8 | 72.9 | |
| Model | Preceding context | Full context | ||||||||||||||
| Chisel | V/SV | VHDL | HLS | Chisel | V/SV | VHDL | HLS | |||||||||
| EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | EM | ES | |
| Qwen2.5-Coder-3B | 0.0 | 15.5 | 0.0 | 19.0 | 0.0 | 22.8 | 0.2 | 15.7 | 12.38 | 46.55 | 17.32 | 48.66 | 9.68 | 48.45 | 17.11 | 48.91 |
| +Tuning | 17.3 | 56.1 | 28.9 | 61.5 | 26.6 | 64.7 | 17.1 | 54.0 | 24.77 | 54.30 | 34.84 | 65.50 | 28.23 | 59.10 | 33.11 | 55.00 |
| +Retrieval & Tuning | 18.9 | 57.6 | 30.1 | 63.4 | 28.2 | 66.2 | 18.2 | 56.0 | 27.55 | 63.95 | 38.78 | 69.94 | 31.45 | 71.88 | 40.35 | 72.63 |
| Qwen2.5-Coder-7B | 0.0 | 9.8 | 0.0 | 12.8 | 0.2 | 14.0 | 0.0 | 9.7 | 10.22 | 45.79 | 15.55 | 47.00 | 10.48 | 49.26 | 12.72 | 39.51 |
| Model | Bundle budget | Chisel (EM/ES) | V/SV (EM/ES) | VHDL (EM/ES) | HLS (EM/ES) | ||||
| GPT-5 | 400,000 | 30.96 | 63.57 | 38.58 | 62.95 | 14.95 | 50.57 | 44.30 | 71.50 |
| Gemini-2.5 | 1,000,000 | 32.20 | 64.11 | 36.81 | 61.17 | 11.90 | 43.63 | 46.37 | 71.12 |
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | SystemVerilog / Verilog | VHDL | Chisel | HLS C/C++ |
| Design Structure | “module”, “package” | “entity”, “architecture” | “Module”, “Bundle” | “Top Function”, “Kernel” |
| Declaration and Definition | “wire”, “reg” | “signal”, “generic” | “IO”, “Wire” | “Function Signature” |
| Storage Block | “reg”, “always_ff” | “clocked signal”, “process” | “Reg”, “Mem” | “Array”, “hls::stream” |
| Computation Block | “Arithmetic Op”, “Bit Slice” | “Arithmetic Op”, “Type Conversion” | “Arithmetic Op”, “Mux” | “Arithmetic Expression”, “Array Access” |
| Control Flow Block | “always”, “if” | “process”, “if” | “when”, “switch” | “for loop”, “if” |
| Interface | “Port List”, “Ready/Valid” | “Port Map”, “Component Instantiation” | “Decoupled”, “Valid” | “AXI Interface”, “Stream Interface” |
| Language | High ( ) | Medium ( ) | Lower ( ) |
| SystemVerilog | 38.6 | 41.9 | 19.5 |
| VHDL | 55.6 | 38.1 | 6.2 |
| Chisel | 55.4 | 18.9 | 25.7 |
| HLS C/C++ | 41.4 | 35.5 | 23.0 |
| Rule | Pattern (and context) | Subcategory | Conf. |
| Priority 1 – Property and Assertion Specification | |||
| SV-R1 | assert property , cover property , assume property | Assertion Definition | 0.90 |
| SV-R2 | property <name> + endproperty | Assertion Definition | 0.85 |
| SV-R3 | property <name> ; nearby context contains assert property | Assertion Definition | 0.80 |
| Priority 2 – Monitoring and Checking Logic | |||
| SV-R4 | ‘define / ‘ASSERT + error / fatal | Error Reporting and Logging | 0.85 |
| Rule | Pattern (and context) | Subcategory | Conf. |
| Priority 1 – Testbench / Monitoring | |||
| VH-R1 | wait for | Stimulus Generation | 0.85 |
| VH-R2 | wait until | Stimulus Generation | 0.80 |
| VH-R3 | assert + report | Error Reporting and Logging | 0.85 |
| VH-R4 | report "..." | Error Reporting and Logging | 0.75 |
| VH-R5 | assert + severity | Error Reporting and Logging | 0.80 |
| Rule | Pattern (and context) | Subcategory | Conf. |
| Priority 1 – Monitoring / Testbench | |||
| CH-R1 | expect( | Error Reporting and Logging | 0.85 |
| CH-R2 | poke( / peek( / step( | Stimulus Generation | 0.85 |
| Priority 2 – Design Structure | |||
| CH-R3 | package <name> | Package and Namespace Definition | 0.85 |
| CH-R4 | import <pkg>. (standalone) | Package and Namespace Definition | 0.75 |
| Rule | Pattern (and context) | Subcategory | Conf. |
| Priority 1 – Testbench / Monitoring | |||
| HL-R1 | testbench or golden / reference / expected comparison in a loop | Stimulus Generation | 0.75 |
| HL-R2 | sc_main / sc_start / sc_trace | Stimulus Generation | 0.80 |
| HL-R3 | int main( + golden / reference / test in context | Stimulus Generation | 0.70 |
| HL-R4 | golden / reference / expected + != / == | Error Reporting and Logging | 0.75 |
| HL-R5 | printf / fprintf / cout + error / fail / mismatch | Error Reporting and Logging | 0.75 |
| Language | Tool |
| HLS C/C++ | Vitis HLS / g++ with ac_int |
| SystemVerilog | Icarus Verilog / Verilator |
| VHDL | GHDL |
| Chisel | sbt + ChiselTest |
| Retriever | # Snippets (center 90%) | Mean Length (chars) | Std. Dev. (chars) |
| GraphCoder | 1595 | 2868.10 | 3223.60 |
| RepoCoder | 1596 | 4536.98 | 1060.49 |
| Language | Hit@1 | Hit@5 |
| Scala | 0.03 | 0.05 |
| VHDL | 0.11 | 0.16 |
| SystemVerilog | 0.09 | 0.12 |
| HLS | 0.19 | 0.25 |
| Python (API-level) | 0.29 | 0.35 |
| Python (Line-level) | 0.27 | 0.32 |
| Model | Ctx | Chisel | V/SV | VHDL | HLS | ||||
| EM | ES | EM | ES | EM | ES | EM | ES | ||
| FT DS-Coder-6.7B | 512 | 16.10 | 50.64 | 24.41 | 54.34 | 17.14 | 51.33 | 25.00 | 55.00 |
| 1,024 | 22.29 | 61.99 | 34.25 | 69.36 | 26.61 | 67.59 | 31.80 | 67.70 | |
| 2,048 | 25.70 | 56.50 | 37.80 | 67.40 | 31.45 | 61.10 | 36.84 | 60.40 | |
| 4,096 | 19.81 | 59.97 | 30.12 | 64.59 | 23.79 | 61.45 | 28.29 | 62.36 | |
| 8,192 | 19.81 | 57.40 | 30.12 | 62.62 | 22.58 | 58.38 | 28.73 | 61.47 | |
| Language | Model | Tasks | Sent (M) | Recv. (M) | Avg./Inst. |
| Chisel | Gemini-2.5-Pro | 323 | 4.02 | 2.00 | 18,638 |
| Chisel | GPT-5 | 323 | 4.71 | 0.92 | 17,426 |
| HLS | Gemini-2.5-Pro | 456 | 7.73 | 2.48 | 22,397 |
| HLS | GPT-5 | 456 | 10.17 | 1.45 | 25,471 |
| SystemVerilog | Gemini-2.5-Pro | 508 | 6.13 | 1.85 | 15,709 |
| SystemVerilog | GPT-5 | 508 | 10.77 | 1.93 | 25,005 |
| Agent / harness | Model | Chisel EM | Chisel ES | V/SV EM | V/SV ES | VHDL EM | VHDL ES | HLS EM | HLS ES |
| Aider | GPT-5 | 42.41 | 78.40 | 43.11 | 72.57 | 43.15 | 81.41 | 44.52 | 75.25 |
| Aider | Gemini-2.5-Pro | 46.44 | 80.46 | 39.17 | 67.30 | 45.77 | 81.61 | 48.46 | 75.59 |
| Model | Few-shot | CoT | Our Prompt | |||
| EM | ES | EM | ES | EM | ES | |
| Proprietary LLMs | ||||||
| Gemini-2.5 Pro | 27.8 | 60.5 | 28.1 | 60.4 | 28.0 | 59.1 |
| GPT-5 | 26.2 | 57.9 | 26.0 | 59.5 | 26.5 | 58.3 |
| DeepSeek-V3.2 | 21.6 | 51.3 | 22.9 | 53.8 | 23.7 | 51.4 |
| Fine-tuned Models (LoRA) | ||||||
| Model | Setting | Ctx | Chisel EM | Chisel ES | V/SV EM | V/SV ES | VHDL EM | VHDL ES | HLS EM | HLS ES |
| GPT-5 | Long-context FIM with relevant files | 400K | 30.96 | 63.57 | 38.58 | 62.95 | 14.95 | 50.57 | 44.30 | 71.50 |
| Gemini-2.5-Pro | Long-context FIM with relevant files | 1M | 32.20 | 64.11 | 36.81 | 61.17 | 11.90 | 43.63 | 46.37 | 71.12 |
| Qwen2.5-Coder-14B | Tuning + retrieval | 2,048 | 33.13 | 70.60 | 43.90 | 76.10 | 37.10 | 75.40 | 39.91 | 73.20 |
| Model | Method | Ctx | Chisel EM | Chisel ES | V/SV EM | V/SV ES | VHDL EM | VHDL ES | HLS EM | HLS ES |
| FT DS-Coder-6.7B | Fine-tuned | 512 | 16.10 | 50.64 | 24.41 | 54.34 | 17.14 | 51.33 | 25.00 | 55.00 |
| 1,024 | 22.29 | 61.99 | 34.25 | 69.36 | 26.61 | 67.59 | 31.80 | 67.70 | ||
| 2,048 | 25.70 | 56.50 | 37.80 | 67.40 | 31.45 | 61.10 | 36.84 | 60.40 | ||
| 4,096 | 19.81 | 59.97 | 30.12 | 64.59 | 23.79 | 61.45 | 28.29 | 62.36 | ||
| 8,192 | 19.81 | 57.40 | 30.12 | 62.62 | 22.58 | 58.38 | 28.73 | 61.47 | ||
| FT Qwen-14B | Fine-tuned | 512 | 26.93 | 64.83 | 32.48 | 67.39 | 26.41 | 64.92 | 30.70 | 66.50 |
| Setting | Model | Chisel EM | Chisel ES | V/SV EM | V/SV ES | VHDL EM | VHDL ES | HLS EM | HLS ES |
| Pretrained native FIM | Deepseek-5.7B | 15.48 | 50.28 | 21.06 | 55.57 | 11.90 | 56.18 | 22.81 | 56.36 |
| Deepseek-6.7B | 10.53 | 43.95 | 20.28 | 52.67 | 9.27 | 52.00 | 17.32 | 49.79 | |
| Qwen-Coder-3B | 12.38 | 46.55 | 17.32 | 48.66 | 9.68 | 48.45 | 17.11 | 48.91 | |
| Qwen-Coder-7B | 10.22 | 45.79 | 15.55 | 47.00 | 10.48 | 49.26 | 12.72 | 39.51 | |
| Qwen-Coder-14B | 17.96 | 57.29 | 22.44 | 55.99 | 11.09 | 51.44 | 24.34 | 55.66 | |
| GPT-5 | 24.8 | 56.7 | 34.4 | 60.9 | 15.3 | 52.0 | 31.4 | 63.7 |
| Model | Few-shot EM | Few-shot ES | CoT EM | CoT ES | Task EM | Task ES |
| Gemini-2.5 Pro | 27.8 | 60.5 | 28.1 | 60.4 | 28.0 | 59.1 |
| GPT-5 | 26.2 | 57.9 | 26.0 | 59.5 | 26.5 | 58.3 |
| DeepSeek-V3.2 | 21.6 | 51.3 | 22.9 | 53.8 | 23.7 | 51.4 |
| Qwen-14B-LoRA | 35.1 | 70.7 | 34.9 | 70.6 | 36.7 | 72.2 |
| Qwen-7B-LoRA | 32.4 | 68.2 | 32.7 | 68.0 | 33.9 | 69.7 |
| DS-6.7B-LoRA | 31.3 | 59.9 | 31.6 | 60.1 | 32.9 | 61.4 |
| Lang. | Repo Name | Ground Truth | Pre-Train Output (Failure) |
| Verilog | AXI-SDCard-High-Speed-Controller | input logic[2:0] axilite_arprot, | .waddr(waddr), .wdata(wdata), .we(we)... (Instantiation Hallucination) |
| Verilog | custom_uvm_report_server | super.build_phase(phase); | // Copyright 2007-2010 Mentor Graphics... (Header Regurgitation) |
| Verilog | SystemVerilogAssertions | logic gnt3; | // Write assertions for a simple Round Robin... (Non-code Content Hallucination) |
| Verilog | virtio | always_ff @(posedge aclk...) | _EVENT_IDX: begin tx.tvalid <= 1’b1; ... (Logic Hallucination) |
| HLS | hls-spmv | int col_left = 0; | #include "spmv.h" #include <hls_stream.h>... (Header Regurgitation) |
| HLS | hls-spmv | L1: for (int i = 0; ... | void spmv(int rowPtr[NUM_ROWS+1]... (Function Definition Hallucination) |
Code Language Models
Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.
See all 125 papers