Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software

Janakan Sivaloganathan; Ainaz Jamshidi; Andriy Miranskyy; Lei Zhang

cs.SEMar 9, 2026

Automating Detection and Root-Cause Analysis of Flaky Tests in Quantum Software

Authors: Janakan Sivaloganathan, Ainaz Jamshidi, Andriy Miranskyy, Lei Zhang

Organizations: Department of Computer Science, Toronto Metropolitan University, Toronto, Canada · Department of Information Systems, University of Maryland, Baltimore County, Baltimore County, United States of America

Abstract

Like classical software, quantum software systems rely on automated testing. However, their inherently probabilistic outputs make them susceptible to quantum flakiness -- tests that pass or fail inconsistently without code changes. Such quantum flaky tests can mask real defects and reduce developer productivity, yet systematic tooling for their detection and diagnosis remains limited. This paper presents an automated pipeline to detect flaky-test-related issues and pull requests in quantum software repositories and to support the identification of their root causes. We aim to expand an existing quantum flaky test dataset and evaluate the capability of Large Language Models (LLMs) for flakiness classification and root-cause identification. Building on a prior manual analysis of 14 quantum software repositories, we automate the discovery of additional flaky test cases using LLMs and cosine similarity. We further evaluate a variety of LLMs from OpenAI GPT, Meta LLaMA, Google Gemini, and Anthropic Claude suites for classifying flakiness and identifying root causes from issue descriptions and code context. Classification performance is assessed using standard performance metrics, including F1-score. Using our pipeline, we identify 25 previously unknown flaky tests, increasing the original dataset size by 54%. The best-performing model, Google Gemini, achieves an F1-score of 0.9420 for flakiness detection and 0.9643 for root-cause identification, demonstrating that LLMs can provide practical support for triaging flaky reports and understanding their underlying causes in quantum software. The expanded dataset and automated pipeline provide reusable artifacts for the quantum software engineering community. Future work will focus on improving detection robustness and exploring automated repair of quantum flaky tests.

Figures & tables

Figure 1: A flaky test can generate random test results without any changes in code.

Platform	Repository	Language	$T$	$F$	$P$
Qiskit	qiskit	Python	4,533	29	0.55%
Qiskit	qiskit-aer	Python	766	3	0.39%
Qiskit	qiskit-ibm-runtime	Python	849	3	0.35%
Qiskit	qiskit-ibm-provider	Python	341	8	0.23%
Qiskit Community	qiskit-nature	Python	385	1	0.26%
Qiskit Community	qiskit-experiments	Python	389	3	0.77%

Table 1: Statistics of quantum software repositories with flaky tests. Three right-most columns are as follows: count of closed issue reports (Column

T

), count of closed flaky test reports (Column

F

), and percentage of reports related to flaky tests (Column

P

) computed as

F/T\times 100\%

	Fix pattern
Cause category	Fix Seed	Alter Software Env.	Make Single Thread	Adjust Tolerance	Add Exception Handler	Synchronize	Use Keys for Order	Others	Grand Total	Percentage
Randomness	12							2	14	19.2%
Software Env.		5						3	8	11.0%
Multi-Threading			4					6	10	13.7%
Floating Point Ops.				5				2	7	9.6%
Visualization								4	4	5.5%

Table 2: Count of cause categories and fix patterns based on flaky test reports.

Figure 2: Automated quantum flakiness detection and root cause analysis pipeline architecture.

Model	Context	F1		MCC		Recall		Total Observations
Model	Context	RQ3	RQ5	RQ3	RQ5	RQ3	RQ4	RQ3	RQ4	RQ5
gpt-4o-2024-11-20	$\{R_{p},C_{p}\}$	0.8592	0.5069	0.7183	0.5187	0.8592	0.8182	142	44	44
	$\{R_{p},C_{f}\}$	0.8592	0.4586	0.7183	0.4386	0.8592	0.6406		64	64
	$\{R_{f},C_{p}\}$	0.8649	0.4968	0.7209	0.5734	0.9014	0.9211		38	38
	$\{R_{f},C_{f}\}$	0.8649	0.5481	0.7209	0.5802	0.9014	0.8393		56	56
	$E_{p}\{R_{p},C_{p}\}$	0.8182	0.4137	0.6145	0.4682	0.8732	0.8182		44	44

Table 3: Model Performances

Root Cause Analysis

Root cause analysis identifies the initiating faults behind observed failures by tracing symptoms through causal and system dependencies.

See all 80 papers

Oct 9, 2026

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.

Meriem Terki, Nouhaila Innan, Minghao Shao +3

Ecole Supérieure en Informatique de Sidi Bel Abbès, Algeria · eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE · Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, NYUAD, Abu Dhabi, UAE +1

Automated Test Generation LLM Evaluation

Ask PDF

Oct 7, 2026

QuSema: Detecting Silent Bugs in Quantum Libraries via Quantum-knowledge-enhanced Agents

Quantum libraries are now critical infrastructure for quantum algorithm development, yet their correctness remains difficult to test. Existing testing techniques mainly rely on failure-based or comparison-based oracles, exposing bugs only when executions fail, violate runtime checks, or disagree with another implementation. Their applicability is limited when suitable execution-based oracles are unavailable, leaving some silent bugs undetected. Such missed bugs can produce incorrect results that propagate into experimental conclusions, simulation studies, and algorithmic designs. Here we present QuSema, an autonomous testing agent for finding silent bugs in quantum libraries. QuSema uses constraints from quantum semantics and documentation as a source-level semantic oracle to assess whether implementation logic can produce invalid outputs from valid inputs. It operates through an agentic loop that repeatedly inspects library API documentation and source code, reasons about the intended behavior of quantum operations, identifies potential semantic deviations, and validates them by generating executable tests through library APIs. Guided by quantum-domain reasoning, QuSema turns high-level behavioral mismatches into concrete, user-triggerable bug reports, enabling it to uncover non-crash defects. We implement QuSema for Qiskit and PennyLane. On a benchmark of 20 historical silent bugs, QuSema achieves higher mean bug relocation counts than Claude Code and Codex, with the DeepSeek configuration costing less than Claude Code. QuSema also discovers 40 previously unknown bugs confirmed by the developers, including 30 silent bugs.

Yujin Song, Kaining Zhang, Qixin Zhang +3

Nanyang Technological University, Singapore · Hong Kong University of Science and Technology, Hong Kong, China · Zhejiang University of Technology, Hangzhou, China

Fault Detection Automated Software Testing Automated Test Generation

Ask PDF

Jul 28, 2026

KQFuzz: Knowledge-Guided Fuzzing for Quantum Libraries via Large Language Models

As quantum computing continually improves, ensuring the reliability and correctness of quantum libraries has become increasingly critical. To this end, many LLM-based fuzzing approaches towards quantum libraries have been proposed to uncover potential bugs. However, these methods still suffer from limitations such as insufficient flexibility and low efficiency, which hinder the progress of the quantum computing field. To address these challenges, we propose KQFuzz, a novel knowledge-guided fuzzer for quantum libraries. It leverages comprehensive codebase knowledge to ground LLM-based test generation, synergizing this with fitness-guided evaluation and two-level mutations to explore complex execution paths and trigger potential bugs. Firstly, KQFuzz introduces a novel prompting scheme tailored to quantum programs, which strategically incorporates knowledge of the codebase to efficiently generate high-quality quantum seed programs. Moreover, we develop evaluation and mutation strategies to handle the generated seed programs, facilitating efficient fuzzing execution while further enriching the diversity of the resulting test cases. We implement KQFuzz and conduct fuzzing on three popular quantum libraries, including Qiskit, PennyLane, and Cirq. Experimental results demonstrate that our approach significantly outperforms other state-of-the-art methods, with coverage improved by up to 18.44%. During the development of KQFuzz, we discovered 13 bugs, all of which have been confirmed and 12 have already been fixed by the developers.

Fuyuan Xia, Qixin Zhang, Chenhao Ying +5

1Shanghai Jiao Tong University · 2Nanyang Technological University · 3Hong Kong University of Science and Technology +1

Automated Software Testing LLMs for Cybersecurity Automated Test Generation

Ask PDF

Abstract

Root Cause Analysis

Explore similar work

QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

QuSema: Detecting Silent Bugs in Quantum Libraries via Quantum-knowledge-enhanced Agents

KQFuzz: Knowledge-Guided Fuzzing for Quantum Libraries via Large Language Models