From FIPS to FPGA with LLM Agents: Measuring What Tests and Reviews Miss in Post-Quantum Cryptographic Hardware
Organizations: Lucid Motors, Newark, CA, USA · EYL Inc., Seoul, Republic of Korea
Abstract
As LLM agents increasingly generate RTL alongside the tests and reviews that validate it, self-generated evidence is a weak foundation for trustworthy hardware. We present our seven-stage verification-gated workflow, evolved during a post-quantum design port, in which a gate counts only if shown able to fail on negative controls and fault mutants, and reviewer agents audit work before commit. We then audit this evidence with tool-selected netlist mutants, flaw replays, and formal checks. The flow ported an ML-KEM-768/ML-DSA-65 accelerator to a 300 MHz Zynq UltraScale+ key-custody card: all six operations are byte-exact on the board, and it passes all 85 interface-applicable NIST ACVP cases (of 285) and 484 ML-DSA known answers. However, passing module tests overstate the verification strength. Across three RTL modules, existing simulation gates kill only 76% of 399 tool-selected netlist mutants; specification-derived tests guided by the survivors raise the fresh-draw decapsulation kill rate from 48% to 74%. In pre-registered single-model replays on flaws invisible to known-answer tests, undirected whole-design review catches two verifier flaws in only 3/6 and 1/6 sessions, whereas module-scoped review catches every known flaw in all 54 sessions. Even scoped review detects planted key-custody flaws in only 3/20 sessions, whereas formal checks (one added for the replay) flag every one. In this study, trustworthy agentic hardware needed author-independent mutants for test gates, module-level scoping for LLM review, and formal checks for custody invariants.
Figures & tables
| Work | Level | HW | Std. vectors | Sec. | Cost |
|---|---|---|---|---|---|
| ContractHIL-HLS [ 5 ] | HLS, PQC | board | – | – | tokens |
| FPGAgent [ 28 ] | HLS kernels | board | – | – | – |
| LLM4PQC [ 4 ] | HLS, kernels | – | KAT, sim. | – | – |
| Design Conductor 2 [ 25 ] | RTL | mapped | KAT, sim. a | – | rel. tokens |
| MARVEL [ 35 ] | RTL review | – | – | yes | $/run |
| Park et al. [ 6 ] | RTL, PQC | board | KAT, board | – | tokens |
| Asked for | Target A | Target B | Gate on B |
|---|---|---|---|
| Core clock | 100 MHz | 300 MHz | timing; 6/6 byte-exact |
| Host link | PCIe Gen2 8 | PCIe Gen3 8 | lane-order gate; link on the board |
| Conform. | known answers | NIST ACVP and known answers, on the device | 85/85 (of 285); 484/484 |
| Zeroization | on command | two tiers (operational, full) | 49/49 |
| Entropy | external + on-chip | on-chip; health tests fail-closed | 7/7 on the generator output; fail-closed in simulation |
| Host reset | function reset | bus reset, no power cycle | 13/13 a |
| What got past the agent | Caught by | Delay | Lasting change | |
|---|---|---|---|---|
| E1 | A constant ROM was zeroed (init file not a build source); every build closed timing | first run over PCIe: 4 of 6 byte-exact | 3.5 weeks | post-synthesis gate on ROM loads |
| E2 † | Verify lacked reject checks that only malformed signatures reach; all known answers passed | conformance review asking which checks the ACVP vectors reach | since before the port | boundary testbenches and mutants on the added checks |
| E2b † | Two custody flaws from target A let a hostile host recover a wrapped key: generator and key wrap shared a hash core without exclusion; the host chose the wrap nonce | an audit agent with simulation probes (hash core); a redesign, then security reviewers (nonce) | since before the port | grant-exclusion invariant gate; device-chosen nonce |
| E3 | A bitstream that met timing on every clock but returned no DMA output | promotion ladder on the board (vs. the previous bitstream) | 1 day | DMA smoke test plus full ladder before promotion |
| E4 † | A hybrid KEM combiner omitted a required binding; its gates were blind to two regressions | review rounds 1–2, before the board | same day | mutation gates that must fail first |
| E5 | A host-library Verify fell back to a 30-s poll; all 484 correct | pre-committed time prediction (missed) | 7 weeks | completion on the done latch; re-run 75 s |
| Mutants | Gates | Transform | Verifier | Decaps. | All |
|---|---|---|---|---|---|
| 399 tool mutants (first draw) | measured | 92% | 67% | 53% | 76% |
| 150 tool mutants (fresh draw) | measured | 88% | 76% | 48% | 70.7% |
| the same 150 | measured + new | 88% | 82% a | 74% b | 81.3% c |
| Code given, scope | Flaw | Plain request | Lens prompt | All |
|---|---|---|---|---|
| A, part named a | each known flaw | 9/9 | 9/9 | 27/27 |
| A, whole design | shared hash core | 6/6 | 5/6 | 11/12 |
| host-chosen nonce | 6/6 | 6/6 | 12/12 | |
| missing hint checks | 3/6 b | 6/6 | 9/12 | |
| unchecked coefficients | 1/6 b | 6/6 | 7/12 | |
| B, part named | planted, custody c | 1/8 | 2/12 | 3/20 |
LLM Agent Verification
LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.
See all 170 papers