QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

Meriem Terki; Nouhaila Innan; Minghao Shao; Muhammad Kashif; Alberto Marchisio; Muhammad Shafique

cs.LGOct 9, 2026

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

Authors: Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique

Organizations: Ecole Supérieure en Informatique de Sidi Bel Abbès, Algeria · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE · Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE · SDU Microelectronics, Institute of Mechanical and Electrical Engineering, University of Southern Denmark, Odense, Denmark

Abstract

Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.

Figures & tables

Mode	Test checks	Probe scenario	Test	Checker
Too weak DeepSeek V3.2 [ 9 ] syntactic	Entry-point checks only; no execution on official input	Oracle-wrong LLM output	Pass	Wrong
Too strict GPT-5.4 [ 10 ] semantic	Numerical tolerance at $10^{-4}$	Human reference output	Fail	Correct

TABLE I : Two failure modes when generated tests are scored against the benchmark checker (Coffee Conundrum challenge).

Panel of Figure 2 — Fig. 1 : Illustrative generated-test assertions for the Coffee Conundrum challenge (Table I ). Each snippet shows the discriminating checks; harness boilerplate is omitted.

Fig. 2 : QTestBench pipeline. Inputs: task description, official cases, and benchmark checker (oracle). Stage 1 generates syntactic, semantic, and behavioral pytest modules (one LLM call per type, temperature $0.8$ , no repair loop). Stage 2 builds a solution cohort, labels probes with the oracle, executes generated tests via runtime solution injection, and reports the metrics in Section III-C under Configuration A (trusted correct only), Configuration B (checker-labeled LLM cohort), and Configuration C (synthetic incorrect calibration; Section III-D ).

Type	Question	What runs	Ground truth used
Syntactic	Valid structure?	Parse/load candidate; no circuit execution	Required entry points only
Semantic	Correct outputs?	Candidate execution on official cases	Embedded official I/O values
Behavioral	Passes harness rules?	run() then candidate check()	Candidate’s embedded check()

TABLE II : Generated test types in Stage 1: scope and oracle interaction.

Panel of Figure 5 — Fig. 3 : Core assertion patterns for the three generated test types (Table II ), exemplified on the Rainy Days challenge.

Suite	ES	$S_{c}$	CS	$S_{w}$	BDS	TQS
Syntactic	1	6/6	1.00	2/3	0.67	0.67
Semantic	1	6/6	1.00	3/3	1.00	1.00
Behavioral	1	6/6	1.00	3/3	1.00	1.00
Avg.	–	–	1.00	–	0.89	0.89

TABLE III : Scoring ledger: Chalet Random Gate, Configuration B, Claude Opus 4.6 (test generator). Probe counts then CS, BDS, and TQS (avg. row over test types).

	Config A					Config B						Config C
Challenge	ES	CS	TQS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	TQS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	TQS	$C_{\mathrm{in}}$	CQ
QSP swamp	0.93	0.73	0.73	0.50	0.47	0.93	0.73	0.73	0.53	0.50	0.47	0.93	0.73	0.73	0.53	0.50	0.47
Chalet gate	0.87	0.80	0.80	0.60	0.53	0.87	0.80	0.64	0.58	0.60	0.53	0.87	0.80	0.67	0.60	0.60	0.53
Coffee	1.00	0.47	0.47	0.97	0.45	1.00	0.47	0.67	0.13	0.97	0.13	1.00	0.47	0.67	0.13	0.97	0.13
Contextuality	0.93	0.80	0.80	0.47	0.40	0.93	0.80	0.60	0.47	0.47	0.40	0.93	0.80	0.60	0.47	0.47	0.40
Mach–Zehnder	0.93	0.80	0.80	0.53	0.47	0.93	0.80	0.60	0.47	0.53	0.47	0.93	0.80	0.60	0.47	0.53	0.47

TABLE IV : QHack benchmark: mean per-challenge scores (averaged over five test generators). Configs A–C appear as column groups; see legend below for TQS/CQ shading.

	Config A					Config B						Config C
Model	ES	CS	TQS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	TQS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	TQS	$C_{\mathrm{in}}$	CQ
Claude Opus 4.6	1.00	0.88	0.88	0.69	0.57	1.00	0.88	0.80	0.69	0.69	0.55	1.00	0.88	0.83	0.71	0.69	0.53
DeepSeek V3.2	0.96	0.42	0.42	0.35	0.28	0.96	0.47	0.75	0.26	0.35	0.27	0.96	0.42	0.75	0.25	0.35	0.25
Gemini 3 Pro	1.00	0.96	0.96	0.71	0.67	1.00	0.96	0.67	0.63	0.71	0.63	1.00	0.96	0.67	0.63	0.71	0.63
GPT-5.4	1.00	0.83	0.83	0.71	0.58	1.00	0.84	0.71	0.55	0.71	0.56	1.00	0.83	0.71	0.54	0.71	0.54
Qwen3-Coder	0.79	0.37	0.37	0.44	0.12	0.79	0.37	0.50	0.08	0.44	0.08	0.79	0.37	0.50	0.08	0.44	0.08

TABLE V : QHack benchmark: mean scores per test-generator model (Configs A–C as column groups; best TQS per configuration in bold). Layout matches Table IV .

	Config A					Config B					Config C
Task	ES	CS	TQS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	TQS	$C_{\mathrm{in}}$	CQ
create_quantum circuit	1.00	0.73	0.73	0.00	0.00	1.00	0.70	n/a	0.00	n/a	1.00	0.73	1.00	0.73	0.00	0.00
run_bell_state simulator	1.00	0.87	0.87	0.93	0.83	1.00	0.87	0.73	0.93	0.60	1.00	0.93	0.62	0.56	0.93	0.59
create_bell statevector	1.00	0.73	0.73	0.93	0.71	1.00	0.73	0.73	0.93	0.47	1.00	0.73	0.00	0.00	0.93	0.00
create_ghz	0.93	0.13	0.13	0.78	0.11	0.93	0.80	n/a	0.78	n/a	0.93	0.13	0.83	0.03	0.78	0.04
transpile circuit_maxopt	1.00	0.40	0.40	0.00	0.00	1.00	0.45	n/a	0.00	n/a	1.00	0.40	0.00	0.00	0.00	0.00

TABLE VI : Qiskit HumanEval: mean per-task scores (ten-task scope; averaged over five test generators). Configuration A: ES, CS, TQS (

\mathrm{TQS}=\mathrm{ES}\times\mathrm{CS}

C_{\mathrm{in}}

, CQ. Configuration B: ES, CS, BDS ( n/a if no oracle-wrong probes),

C_{\mathrm{in}}

, CQ. Configuration C: full metric set. Rows list short HumanEval task identifiers.

	Config A					Config B					Config C
Model	ES	CS	TQS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	$C_{\mathrm{in}}$	CQ	ES	CS	BDS	TQS	$C_{\mathrm{in}}$	CQ
Claude Opus 4.6	1.00	0.77	0.77	0.36	0.28	1.00	0.84	0.65	0.36	0.27	1.00	0.77	0.46	0.29	0.36	0.07
DeepSeek V3.2	1.00	0.30	0.30	0.37	0.13	1.00	0.42	0.73	0.37	0.07	1.00	0.33	0.52	0.06	0.37	0.03
Gemini 3 Pro	1.00	0.70	0.70	0.33	0.24	1.00	0.82	0.75	0.33	0.19	1.00	0.70	0.54	0.34	0.33	0.09
GPT-5.4	1.00	0.47	0.47	0.33	0.16	1.00	0.57	0.72	0.33	0.14	1.00	0.47	0.48	0.14	0.33	0.05
Qwen3-Coder	0.93	0.30	0.30	0.26	0.16	0.93	0.37	0.85	0.26	0.18	0.93	0.30	0.56	0.16	0.26	0.08

TABLE VII : Qiskit HumanEval: mean scores per test-generator model over the scoped ten-task runs. Same column layout as Table VI ; best TQS per configuration in bold (Config A and Config C).

Panel of Figure 11 — Fig. 4 : Three case studies on challenges other than the motivational example, connecting QTestBench metrics to concrete oracle vs. test outcomes.

Explore similar work

Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation
May 26, 2026Juan Cruz-Benito, Ismael FaroLLM Evaluation
IBM Research
QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks
Sep 28, 2026Pranav GuptaLLM Evaluation Quantum Circuit Simulation
Cisco, 2901 3rd Ave, Suite 600, Seattle, WA 98121, USA
Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions
Jul 5, 2026Mohammad Arif Rasyidi, Syahirul FaizCode Generation Benchmarks LLM Code Generation
Department of Computer Science Khalifa University Abu Dhabi, United Arab Emirates

May 26, 2026, quant-phJ/K move · Enter open · S save

Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation

Juan Cruz-Benito, Ismael Faro

IBM Research

LLM Evaluation

Open paper AskPDF arXiv

We adapt Microsoft's QuantumKatas -- a well-established quantum computing curriculum -- from Q# to Qiskit, the most widely-adopted quantum computing framework, and package it with an evaluation framework for systematic LLM assessment. The resulting benchmark comprises 350 tasks across 26 categories, spanning fundamental gates through advanced algorithms (Grover's, Simon's, Deutsch-Jozsa), error correction, key distribution, and quantum games. Each task includes a natural language prompt, canonical solution, and deterministic test verification via classical circuit simulation. By building on the QuantumKatas' proven pedagogical design rather than creating tasks from scratch, we inherit a principled difficulty progression and comprehensive concept coverage while contributing the framework adaptation, evaluation infrastructure, and empirical analysis. We evaluate 16 LLMs across 7 prompting configurations -- a total of 39,200 model runs -- to demonstrate the benchmark's utility. Three key findings emerge: (1) the benchmark effectively differentiates model capabilities, with best-configuration pass rates ranging from 32.3% to 83.1% and a 26.1 pp average gap between frontier and open-source models; (2) models perform well at implementing known algorithms (SimonsAlgorithm 82.1%, BasicGates 81.6%) but struggle with problem encoding (SolveSATWithGrover 34.4%, DistinguishUnitaries 40.0%); and (3) chain-of-thought prompting shows a modestly bimodal effect -- it is the best strategy for three models (two of them explicitly reasoning-tuned per vendor documentation) but degrades performance for the rest, leaving it mid-pack in aggregate (56.3% mean) behind few-shot-5 (57.8%). We release the benchmark, evaluation framework, and baseline results to support research on LLM capabilities in quantum computing.

Abstract

Explore similar work

Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation

QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions