QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
Authors: Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique
Organizations: Ecole Supérieure en Informatique de Sidi Bel Abbès, Algeria · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE · Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE · SDU Microelectronics, Institute of Mechanical and Electrical Engineering, University of Southern Denmark, Odense, Denmark
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.
Figures & tables
Mode
Test checks
Probe scenario
Test
Checker
Too weak DeepSeek V3.2 [ 9 ] syntactic
Entry-point checks only; no execution on official input
Oracle-wrong LLM output
Pass
Wrong
Too strict GPT-5.4 [ 10 ] semantic
Numerical tolerance at 10−4
Human reference output
Fail
Correct
TABLE I : Two failure modes when generated tests are scored against the benchmark checker (Coffee Conundrum challenge).
Fig. 1 : Illustrative generated-test assertions for the Coffee Conundrum challenge (Table I ). Each snippet shows the discriminating checks; harness boilerplate is omitted.
Fig. 2 : QTestBench pipeline. Inputs: task description, official cases, and benchmark checker (oracle). Stage 1 generates syntactic, semantic, and behavioral pytest modules (one LLM call per type, temperature 0.8 , no repair loop). Stage 2 builds a solution cohort, labels probes with the oracle, executes generated tests via runtime solution injection, and reports the metrics in Section III-C under Configuration A (trusted correct only), Configuration B (checker-labeled LLM cohort), and Configuration C (synthetic incorrect calibration; Section III-D ).
Type
Question
What runs
Ground truth used
Syntactic
Valid structure?
Parse/load candidate; no circuit execution
Required entry points only
Semantic
Correct outputs?
Candidate execution on official cases
Embedded official I/O values
Behavioral
Passes harness rules?
run() then candidate check()
Candidate’s embedded check()
TABLE II : Generated test types in Stage 1: scope and oracle interaction.
Fig. 3 : Core assertion patterns for the three generated test types (Table II ), exemplified on the Rainy Days challenge.
Suite
ES
Sc
CS
Sw
BDS
TQS
Syntactic
1
6/6
1.00
2/3
0.67
0.67
Semantic
1
6/6
1.00
3/3
1.00
1.00
Behavioral
1
6/6
1.00
3/3
1.00
1.00
Avg.
–
–
1.00
–
0.89
0.89
TABLE III : Scoring ledger: Chalet Random Gate, Configuration B, Claude Opus 4.6 (test generator). Probe counts then CS, BDS, and TQS (avg. row over test types).
Config A
Config B
Config C
Challenge
ES
CS
TQS
Cin
CQ
ES
CS
BDS
TQS
Cin
CQ
ES
CS
BDS
TQS
Cin
CQ
QSP swamp
0.93
0.73
0.73
0.50
0.47
0.93
0.73
0.73
0.53
0.50
0.47
0.93
0.73
0.73
0.53
0.50
0.47
Chalet gate
0.87
0.80
0.80
0.60
0.53
0.87
0.80
0.64
0.58
0.60
0.53
0.87
0.80
0.67
0.60
0.60
0.53
Coffee
1.00
0.47
0.47
0.97
0.45
1.00
0.47
0.67
0.13
0.97
0.13
1.00
0.47
0.67
0.13
0.97
0.13
Contextuality
0.93
0.80
0.80
0.47
0.40
0.93
0.80
0.60
0.47
0.47
0.40
0.93
0.80
0.60
0.47
0.47
0.40
Mach–Zehnder
0.93
0.80
0.80
0.53
0.47
0.93
0.80
0.60
0.47
0.53
0.47
0.93
0.80
0.60
0.47
0.53
0.47
TABLE IV : QHack benchmark: mean per-challenge scores (averaged over five test generators). Configs A–C appear as column groups; see legend below for TQS/CQ shading.
Config A
Config B
Config C
Model
ES
CS
TQS
Cin
CQ
ES
CS
BDS
TQS
Cin
CQ
ES
CS
BDS
TQS
Cin
CQ
Claude Opus 4.6
1.00
0.88
0.88
0.69
0.57
1.00
0.88
0.80
0.69
0.69
0.55
1.00
0.88
0.83
0.71
0.69
0.53
DeepSeek V3.2
0.96
0.42
0.42
0.35
0.28
0.96
0.47
0.75
0.26
0.35
0.27
0.96
0.42
0.75
0.25
0.35
0.25
Gemini 3 Pro
1.00
0.96
0.96
0.71
0.67
1.00
0.96
0.67
0.63
0.71
0.63
1.00
0.96
0.67
0.63
0.71
0.63
GPT-5.4
1.00
0.83
0.83
0.71
0.58
1.00
0.84
0.71
0.55
0.71
0.56
1.00
0.83
0.71
0.54
0.71
0.54
Qwen3-Coder
0.79
0.37
0.37
0.44
0.12
0.79
0.37
0.50
0.08
0.44
0.08
0.79
0.37
0.50
0.08
0.44
0.08
TABLE V : QHack benchmark: mean scores per test-generator model (Configs A–C as column groups; best TQS per configuration in bold). Layout matches Table IV .
Config A
Config B
Config C
Task
ES
CS
TQS
Cin
CQ
ES
CS
BDS
Cin
CQ
ES
CS
BDS
TQS
Cin
CQ
create_quantum circuit
1.00
0.73
0.73
0.00
0.00
1.00
0.70
n/a
0.00
n/a
1.00
0.73
1.00
0.73
0.00
0.00
run_bell_state simulator
1.00
0.87
0.87
0.93
0.83
1.00
0.87
0.73
0.93
0.60
1.00
0.93
0.62
0.56
0.93
0.59
create_bell statevector
1.00
0.73
0.73
0.93
0.71
1.00
0.73
0.73
0.93
0.47
1.00
0.73
0.00
0.00
0.93
0.00
create_ghz
0.93
0.13
0.13
0.78
0.11
0.93
0.80
n/a
0.78
n/a
0.93
0.13
0.83
0.03
0.78
0.04
transpile circuit_maxopt
1.00
0.40
0.40
0.00
0.00
1.00
0.45
n/a
0.00
n/a
1.00
0.40
0.00
0.00
0.00
0.00
TABLE VI : Qiskit HumanEval: mean per-task scores (ten-task scope; averaged over five test generators). Configuration A: ES, CS, TQS ( TQS=ES×CS ), Cin , CQ. Configuration B: ES, CS, BDS ( n/a if no oracle-wrong probes), Cin , CQ. Configuration C: full metric set. Rows list short HumanEval task identifiers.
Config A
Config B
Config C
Model
ES
CS
TQS
Cin
CQ
ES
CS
BDS
Cin
CQ
ES
CS
BDS
TQS
Cin
CQ
Claude Opus 4.6
1.00
0.77
0.77
0.36
0.28
1.00
0.84
0.65
0.36
0.27
1.00
0.77
0.46
0.29
0.36
0.07
DeepSeek V3.2
1.00
0.30
0.30
0.37
0.13
1.00
0.42
0.73
0.37
0.07
1.00
0.33
0.52
0.06
0.37
0.03
Gemini 3 Pro
1.00
0.70
0.70
0.33
0.24
1.00
0.82
0.75
0.33
0.19
1.00
0.70
0.54
0.34
0.33
0.09
GPT-5.4
1.00
0.47
0.47
0.33
0.16
1.00
0.57
0.72
0.33
0.14
1.00
0.47
0.48
0.14
0.33
0.05
Qwen3-Coder
0.93
0.30
0.30
0.26
0.16
0.93
0.37
0.85
0.26
0.18
0.93
0.30
0.56
0.16
0.26
0.08
TABLE VII : Qiskit HumanEval: mean scores per test-generator model over the scoped ten-task runs. Same column layout as Table VI ; best TQS per configuration in bold (Config A and Config C).
Fig. 4 : Three case studies on challenges other than the motivational example, connecting QTestBench metrics to concrete oracle vs. test outcomes.
We adapt Microsoft's QuantumKatas -- a well-established quantum computing curriculum -- from Q# to Qiskit, the most widely-adopted quantum computing framework, and package it with an evaluation framework for systematic LLM assessment. The resulting benchmark comprises 350 tasks across 26 categories, spanning fundamental gates through advanced algorithms (Grover's, Simon's, Deutsch-Jozsa), error correction, key distribution, and quantum games. Each task includes a natural language prompt, canonical solution, and deterministic test verification via classical circuit simulation. By building on the QuantumKatas' proven pedagogical design rather than creating tasks from scratch, we inherit a principled difficulty progression and comprehensive concept coverage while contributing the framework adaptation, evaluation infrastructure, and empirical analysis. We evaluate 16 LLMs across 7 prompting configurations -- a total of 39,200 model runs -- to demonstrate the benchmark's utility. Three key findings emerge: (1) the benchmark effectively differentiates model capabilities, with best-configuration pass rates ranging from 32.3% to 83.1% and a 26.1 pp average gap between frontier and open-source models; (2) models perform well at implementing known algorithms (SimonsAlgorithm 82.1%, BasicGates 81.6%) but struggle with problem encoding (SolveSATWithGrover 34.4%, DistinguishUnitaries 40.0%); and (3) chain-of-thought prompting shows a modestly bimodal effect -- it is the best strategy for three models (two of them explicitly reasoning-tuned per vendor documentation) but degrades performance for the rest, leaving it mid-pack in aggregate (56.3% mean) behind few-shot-5 (57.8%). We release the benchmark, evaluation framework, and baseline results to support research on LLM capabilities in quantum computing.
We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models × 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.
Pranav Gupta
Cisco, 2901 3rd Ave, Suite 600, Seattle, WA 98121, USA
Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user. We study this problem as API drift and introduce quantum-api-drift, a benchmark for measuring version fidelity, defined here as execution success on the requested SDK version, cross-version compatibility, failure modes, and documentation-guided repair in LLM-generated quantum SDK code. We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0. We evaluate 17 models on 50 tasks with 3 samples per prompt, yielding 450 generated samples and 1,350 executions per model. Sixteen models are tested in a matched REST API setting with a 1024-token output cap, while GPT-5.4 (Codex CLI) is reported separately as a reference configuration. Across the 16 matched REST models, diagonal Pass@1 ranges from 0.02 to 0.85. Claude Opus 4.7 is strongest on v0.43 and v2.0, while Grok 4.20 is strongest on v1.3 at 0.513. Error profiles differ systematically by model strength: weaker models fail mainly with broken imports, while stronger models more often reach deprecation-level failures. Documentation-guided repair succeeds for 0.19 to 0.59 of repair attempts overall and is consistently much more effective for migration to v2.0 than to v1.3. The benchmark artifacts are publicly available at https://github.com/arasyi/quantum-api-drift. These results show that version alignment is a distinct evaluation axis for quantum code generation and that API drift remains only partly recoverable even with migration guidance.
Mohammad Arif Rasyidi, Syahirul Faiz
Department of Computer Science Khalifa University Abu Dhabi, United Arab Emirates