cs.LGMar 25, 2026
SaveQuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
Organizations: American University of Beirut · King Abdullah University of Science and Technology
Abstract
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.
Figures & tables
Figure 1: The benchmark holds task intent and execution conditions fixed across frameworks. Our workflow standardizes prompts, grading, and runtime settings before comparing models on Qiskit, PennyLane, and Cirq.
| Task | Edit | Rationale |
|---|---|---|
| 5 | Task removed | Did not output a clear probabilistic answer, making quantitative evaluation unreliable. |
| 25 | Prompt modified | The old version measures all qubits, while the new version measures the first 3 qubits. |
| 28 | Prompt modified | The prompt did not clearly instruct measuring all qubits, leading to ambiguous output distributions. |
| 38 | Task removed | The testing procedure for the machine learning task was not clearly specified, preventing consistent evaluation. |
| 41 | Prompt modified | The randomized input library was replaced by a pre-decided randomly generated input. |
Table 1: Only a small subset of tasks required benchmark-level edits. These prompt changes and removals were needed to make grading consistent across Qiskit, PennyLane, and Cirq.
Figure 2: Qiskit is the easiest target and PennyLane the hardest under one-shot generation. Models are ordered by average Pass@1 across frameworks, revealing both a stable ranking and a persistent framework gap.
Figure 3: Feedback repair lifts accuracy across all three frameworks. The gains are broad rather than model-specific, but no framework becomes fully reliable after repair.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Reference |
|---|---|
| Claude-3.7-Sonnet | Anthropic (2025) |
| DeepSeek-Chat | DeepSeek-AI (2024) |
| DeepSeek-R1 | DeepSeek-AI (2024) |
| Gemini-2.5-Flash | Google (2025) |
| Gemini-3-Pro | DeepMind (2025) |
| GPT-4.1 | Achiam et al. (2023) |
Table 2: The benchmark spans both frontier proprietary and open-weight systems. This table lists the evaluated models and the release reference used for each one.
| Qiskit | Cirq | PennyLane | ||||
|---|---|---|---|---|---|---|
| Model | Pass@1 | Pass@1 (FB) | Pass@1 | Pass@1 (FB) | Pass@1 | Pass@1 (FB) |
| Gemini-3-Pro | 59.5 | 73.8 | 54.8 | 76.2 | 40.5 | 66.7 |
| GPT-5.1 | 57.1 | 83.3 | 52.4 | 73.8 | 42.9 | 66.7 |
| DeepSeek-R1 | 50.0 | 71.4 | 45.2 | 61.9 | 33.3 | 52.4 |
| MoonshotAI-Kimi-K2-Thinking | 47.6 | 69.0 | 38.1 | 57.1 | 26.2 | 38.1 |
| Claude-3.7-Sonnet | 45.2 | 57.1 | 35.7 | 59.5 | 26.2 | 47.6 |
Table 3: Feedback repair lifts scores across all three frameworks. Exact Pass@1 and Pass@1 (FB) values reported in the main paper.
Figure 4: The null KL distribution supports the global acceptance threshold. The pooled canonical-repeat ECDF places the 99.7th percentile at 0.048, motivating the paper-wide threshold .
| Category | Number of Tasks |
|---|---|
| Quantum Algorithms | 31 |
| State Preparation | 6 |
| Decomposition | 5 |
| Total | 42 |
Table 4: Quantum algorithms dominate the benchmark mix. QuanBench+ contains 42 tasks, with most concentrated in algorithmic reasoning.
Figure 5: Multiple samples recover additional Qiskit solutions. The gap between Pass@1 and Pass@5 identifies tasks where one-shot decoding leaves recoverable performance on the table.
Figure 6: Cirq also benefits meaningfully from multi-sample generation. The gains are especially visible among the middle of the model ranking.
Figure 7: PennyLane retains large recoverable gaps for weaker models. Multi-sample decoding helps, but it does not close the framework-level difficulty gap.
Figure 8: One-shot success in Qiskit is concentrated in a broad but incomplete task band. Each row corresponds to a model and each column to a task.
Figure 9: PennyLane exposes a noticeably sparser one-shot success map. Each row corresponds to a model and each column to a task.
Figure 10: Cirq sits between Qiskit and PennyLane in first-attempt density. The overall pattern is stronger than PennyLane but less complete than Qiskit.
Figure 11: Pass@5 broadens Qiskit coverage substantially. Multi-sample decoding turns many partial one-shot failures into recoverable successes.
Figure 12: Pass@5 helps in PennyLane, but hard tasks remain visibly persistent. Multi-sample decoding broadens coverage without removing the framework gap.
Figure 13: Cirq gains a wider solvable region under Pass@5. The additional coverage confirms that many one-shot failures are unstable rather than absolute.
| Model | Qiskit | Cirq | PennyLane |
|---|---|---|---|
| Gemini-3-Pro | 61.9 | 59.5 | 40.5 |
| GPT-5.1 | 76.2 | 64.3 | 57.1 |
| DeepSeek-R1 | 69.0 | 61.9 | 59.5 |
| MoonshotAI-Kimi-K2-Thinking | 66.7 | 52.4 | 52.4 |
| Claude-3.7-Sonnet | 50.0 | 57.1 | 38.1 |
| GPT-4.1 | 52.4 | 54.8 | 35.7 |
Table 5: Pass@5 narrows but does not remove framework gaps. Accuracy (%) over benchmark tasks for each framework.
Figure 14: Prefill helps most when PennyLane boilerplate is easy to miss. The ranking changes confirm that setup friction still matters for several mid-tier models.
Figure 15: Cirq also shows meaningful sensitivity to prompt scaffolding. Prefill changes both average accuracy and several mid-tier rankings.
Figure 16: Qiskit benefits from prefill, but less uniformly than weaker frameworks. The effect is real, though not consistent across the full model range.
Figure 17: Most first-attempt failures are semantic, not syntactic. Wrong answers and logic errors dominate the Pass@1 error budget across frameworks.
Figure 18: Feedback densifies the Qiskit success map. Stronger models in particular convert many previously sparse regions into solved tasks.
Figure 19: Feedback improves PennyLane coverage, but the map remains visibly harder. The gains are substantial without fully closing the framework gap.
Figure 20: Feedback broadens Cirq success across much of the ranking. The densification is clear, especially among stronger and mid-tier models.
Figure 21: Most Qiskit feedback gains arrive early. The curves rise quickly in the first repair rounds and then flatten.
Figure 22: PennyLane improves steadily, but not indefinitely, with additional repair attempts. Most of the lift still arrives in the early rounds.
Figure 23: Cirq follows the same early-gain, late-plateau pattern. Additional repair attempts help most in the first few rounds.
Figure 24: Feedback compresses the spread between models, but does not erase it. Aggregate success rates after up to 5 repair attempts across all frameworks.
Figure 25: After repair, the remaining failures are mostly semantic. Residual post-feedback errors become more concentrated in deeper reasoning mistakes.
More in this topic
Code Generation Benchmarks
Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.
See all 148 papers