cs.AIFeb 18, 2026
SaveLLM4Cov: Execution-Aware Agentic Learning for High-coverage Testbench Generation
Organizations: University of California San Diego, USA · NVIDIA · Agentrys
Abstract
Execution-aware LLM agents offer a promising paradigm for learning from tool feedback, but such feedback can be expensive and slow to obtain, making online reinforcement learning (RL) less practical in certain scenarios. High-coverage hardware verification exemplifies this challenge due to its reliance on industrial simulators and non-differentiable execution signals. We propose LLM4Cov, an offline agent-learning framework that models verification as single-step state transitions guided by deterministic evaluators. Building on this formulation, we introduce execution-validated data curation, policy-aware agentic data synthesis, and worst-state-prioritized sampling to enable scalable learning under execution constraints. We further curate a reality-aligned benchmark adapted from an existing verification suite through a revised evaluation protocol. Using the proposed pipeline, a compact 4B-parameter model achieves 69.2% pass rate and 90.4% average coverage in CVDP-ECov under agentic evaluation, outperforming its teacher by 5.3% and 10.5%, demonstrating competitive performance against models an order of magnitude larger.
Figures & tables
Figure 1: Execution-aware verification loop and its dominant cost in modern hardware design. ( Hegde, 2025 ; Foster, 2025 ; Foster, 2017 )
Figure 2: Coverage pass rates of existing LLMs. Results are measured in agentic setting on our benchmark (Section 4.1 ).
Figure 3: Main components of LLM4Cov . (a) The framework samples trajectories to SFT the student model, and the process is separated into stages to align with the evolving student distribution. (b) Coverage-Guided Agentic Rejection Fine-tuning retains low-coverage drafts and their most coverage-improving revisions, concentrating supervision on recovery behaviors. (c) Verification-Conditioned Progressive Learning generates and trains on staged synthetic trajectories conditioned on the current student, yielding progressively stronger agentic performance and more stable final coverage.
| Pass@1 (higher is better) | ||
|---|---|---|
| Model | Vanilla | Single-Step |
| Qwen3-Coder-30B-A3B-Instruct | 59.5% | 63.9% |
| Qwen3-4B-Instruct-2507 | 28.2% | 28.4% |
Table 1: Effect of the single-step formulation on agents. Results are measured on our verification benchmark (Section 4.1 ).
Figure 4: Student-grounded trajectory synthesis. Intermediate states are sampled by , while the final transition by .
Figure 5: Staged data synthesis v.s. vanilla augmentation. Stage-specific supervision align with current student states, while mixing data together may dilute later stage signals.
| CVDP-ECov | AutoEval-ECov | ||||||||
| Agentic | Direct Inference | Agentic | Direct Inference | ||||||
| Type | Model | Pass % | Avg Cov | Pass % | Avg Cov | Pass % | Avg Cov | Pass % | Avg Cov |
| General Purpose | llama-4-maverick (400B) | 60.2% | 81.7% | 23.1% | 47.3% | 32.9% | 48.7% | 21.4% | 41.9% |
| Qwen3-235b-A22b-2507 | 69.9% | 83.2% | 29.9% | 47.0% | 84.0% | 92.6% | 70.6% | 83.4% | |
| Qwen2.5-72B | 42.9% | 64.1% | 16.4% | 32.2% | 63.8% | 87.6% | 20.9% | 73.3% | |
| Qwen3-30B-A3B-2507 | 50.6% | 68.0% | 30.1% | 52.2% | 81.9% | 91.6% | 72.3% | 86.1% | |
Table 2: Evaluation results on CVDP-ECov and AutoEval-ECov benchmarks. Best metrics are bolded and second runners are underscored . Following the original CVDP evaluation, we treat coverage pass rate as the primary metric. Because all training stages optimize agentic execution—the primary metric and training objective—direct-inference results are reported only as a reference and may slightly degrade in later stages. All models evaluated in instruct mode and recommended config (reported in Appendix B ). CorrectBench Baseline is an exception as it is a mult-agent framework by itself and therefore has no Direct Inference result. As Claude 3.5 Sonnet is no long provided by Anthropic, the original authors kindly sent us their evaluation results which we calculated AutoEval-ECov metrics upon. See Appendix D for more details on baseline comparisons.
Figure 6: Agentic trace taxonomy under intermediate-state distribution drift. Full-teacher traces are omitted in Stage 2 since the relative gap between imitation-style and full-teacher supervision arises from state-distribution mismatch, and is not expected to vary qualitatively with the teacher–student performance gap.
Figure 7: Comparison between intermediate state selection strategies in Stage 1. Evaluated under the agentic setting. Syn Pass means percentage of repositories where syntax is correct.
Figure 8: Stage-Conditioned Progression, evaluated in agentic setting. Data-Aug stands for naive data augmentation.
| Pass Rate (%) | ||||
| Round Number | ||||
| 30B Teacher | 29.6% | 49.6% | 58.8% | 63.9% |
| 4B Base | 13.7% | 21.4% | 26.3% | 28.4% |
| Stage 0 | 45.5% | 55.2% | 58.6% | 60.2% |
| Stage 1 | 48.7% | 59.3% | 63.6% | 67.0% |
| Stage 2 | 48.2% | 62.7% | 67.5% | 69.2% |
Table 3: Pass Rate across agentic rounds for teacher, base, and LLM4Cov stages. 30B teacher stands for Qwen3-30B-A3B-2507, and 4B Base stands for Qwen3-4B-2507 (Base).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Model | Temperature | Top P |
| General Purpose | llama-4-maverick (400B) | 0.7 | 0.8 |
| Qwen3-235b-A22b-2507 | 0.7 | 0.8 | |
| Qwen2.5-72B | 0.7 | 0.8 | |
| Qwen3-30B-A3B-2507 | 0.7 | 0.8 | |
| Coding Specialized | Qwen2.5-Coder-32B | 0.7 | 0.8 |
| Qwen3-Coder-30B-A3B | 0.7 | 0.8 |
Table 4: Model Evaluation Config. Recommended config is used at best effort.
Figure 9: Simulator pass rates of existing LLMs on verification stimulus generation, all in instruct mode. Stage 0 model is used here as LLM4Cov. All baseline models has Syn Pass metric below 70%, while our execution-validated dataset enables over 85% pass rate.
| Syn Pass (%) | ||
| Type | Direct Infer | Agentic |
| Teacher | 53.3% | 85.1% |
| Teacher with augmented spec | 71.1% | 85.8% |
| Base | 45.8% | 61.0% |
| SFT: Data without filtering | 70.6% | 79.5% |
| SFT: Data with Execution filtering | 83.9% | 87.7% |
Table 5: Ablation on execution-based dataset curation. All datasets contain the same number of samples.
Figure 10: Syntax Improvement by Execution-Validated Dataset on models from different families. Fine-tuned on instruct versions.
| Method | Model | Eval setup | Avg Cov% |
|---|---|---|---|
| TBnTB | 14B (Claude-gen. data) | custom, Aldec Riviera-Pro | 25% (reported) |
| LLM4Cov | 4B (30B teacher) | CVDP-ECov, Cadence Xcelium | 90.4% |
Table 6: Reported results from TBnTB compared to LLM4Cov . The two methods use different toolchains and evaluation protocols; numbers are not strictly comparable but illustrate a large gap in favor of LLM4Cov .
| Method | Avg Cov% | Pass Rate (Full Cov Rate) |
|---|---|---|
| CorrectBench (Claude 3.5 Sonnet) | 89.9% | 61.5% |
| LLM4Cov | 96.5% | 84.0% |
Table 7: Re-evaluation of CorrectBench’s released testbenches and LLM4Cov under the unified AutoEval-ECov protocol on Verilog-Eval.
| Direct Pass Rate | Agentic Pass Rate | |||
|---|---|---|---|---|
| SFT Stage | Pass@1 | Pass@5 | Pass@1 | Pass@5 |
| Stage 0 | 45.5% | 68.7% | 60.2% | 77.1% |
| Stage 1 | 48.2% | 68.7% | 67.0% | 84.3% |
| Stage 2 | 50.6% | 72.3% | 69.2% | 81.9% |
Table 8: Output diversity check for each SFT stage