AG-CoT: Verified Algorithmic Traces for LLM Program Synthesis on Clifford Circuits
Authors: Lu Wei, Yufeng Wang, Chenfeng Cao, Lu Pang, Haibin Ling
Organizations: Data Science Department, Stony Brook University · Department of Computer Science, Stony Brook University · Dahlem Center for Complex Quantum Systems, Freie Universität Berlin · Westlake University
Scientific code generation can produce executable programs that fail to compute the intended scientific object. We study this problem in language-model synthesis of Clifford circuits, which prepare the stabilizer states used in quantum error correction and admit exact classical verification. In our target-conditioned framework, each target is given as compact signed stabilizer generators, and an exact verifier checks the generated OpenQASM circuits. We supervise models with Aaronson-Gottesman chain-of-thought (AG-CoT) traces checked by the verifier, and continue training on model generations that the verifier accepts. Across two independently trained model families (3B and 7B), AG-CoT supervision multiplies greedy-decode state-equivalence accuracy by four to six times over circuit-only baselines, and verifier-filtered continuation training adds a further consistent gain atop both. A complementary 32B study shows that supervised models achieve near-perfect syntax and Clifford validity while the strongest direct model reaches 6.14% state equivalence per target, rising to over 10% under verifier-guided selection with multiple candidates. These results show that algorithmic trace supervision gives a large, statistically significant gain in both model families and that verifier-filtered continuation adds a further repeated gain. The persistent gap between Clifford validity and state equivalence confirms that exact verification is necessary: a circuit can be syntactically and physically valid yet prepare the wrong quantum state.
Figures & tables
Figure 1: Running verifier example. Both candidates parse and remain Clifford-valid, but Candidate B is rejected because it prepares +Z rather than the requested +X target.
Figure 2: Target-conditioned circuit synthesis with algorithmic trace supervision and verifier-filtered continuation. Panel a constructs a verified teacher circuit from signed target generators; b pairs circuit-only and AG-CoT supervision with the same prompt and final circuit. The two-qubit example is schematic, with the blue reduction operations inverted to obtain the preparation circuit. Panel c forms a fixed dataset of accepted AG-CoT responses for supervised continuation (RFT). Acceptance checks the emitted circuit and task requirements. Panel d shows correct single outputs on the same 3,661 test targets for each trained system in Table 1 . The three Qwen RFT versions remain separate.
Figure 3: A target-conditioned AG-CoT training example. The reformatted input and trace excerpt retain all four reduction operations and the corresponding auxiliary and stabilizer rows for qubit 0. Inverting the reduction gives the complete program shown on the right. Circuit labels identify the reduction steps in reverse order. The adjacent final Hadamard gates remain as emitted by the teacher.
System
Correct/total
Suc(%)
Qwen2.5-3B-Instruct
Base model
0 / 3,661
0.00
Direct SFT
33 / 3,661
0.90
AG-CoT SFT
201 / 3,661
5.49
RFT ver. 1
213 / 3,661
5.82
RFT ver. 2
215 / 3,661
5.87
Table 1: Single-output training comparison on the same 3,661 target prompts. Each row uses one greedy completion per target and no program repair.
Figure 4: Qwen2.5-3B-Instruct task success by target size, following requirements in Sec. 5 . Each rate includes all targets at that size and a greedy output per target (Appendix Table 13 ).
Earlier
Later
Exact
Holm
Comparison
only
only
p
p
Direct SFT to AG-CoT SFT
2
170
4.97 × 10 -48
–
AG-CoT SFT to RFT ver. 1
7
19
0.029
0.058
AG-CoT SFT to RFT ver. 2
7
21
0.013
0.038
AG-CoT SFT to RFT ver. 3
9
18
0.122
0.122
Table 2: Paired Qwen2.5-3B-Instruct task success comparisons on the 3,661 targets. Earlier only and later only are discordant successes in the order shown. Exact p uses the two sided McNemar test. Holm adjusted p covers the three comparisons from AG-CoT SFT to RFT.
Surface validity
Target match
System
Syntax
Clifford
Exact@1 lab.
StateEq@1
95% CI
Zero-shot
39.9
39.9
0.10
0.33
[0.15, 0.52]
Basic AG-CoT
99.9
99.9
2.62
5.37
[4.67, 6.06]
Step AG-CoT
99.9
99.9
2.31
4.83
[4.15, 5.50]
GRPO-v2
99.9
99.9
2.21
5.26
[4.57, 5.98]
RFT-v1
99.6
99.6
3.13
6.14
[5.37, 6.91]
Table 3: 32B direct results on 3,894 prompts. Rates are percentages. Exact@1 lab. checks saved-label equality, while StateEq@1 checks state equivalence over the same targets. The final column gives the reported 95% confidence interval for StateEq@1.
Model
Criterion
1
4
8
16
32
64
RFT-v1
Label
3.18
3.69
4.00
4.37
4.78
5.19
Basic AG-CoT
Label
2.13
2.93
3.44
3.78
4.01
4.42
Basic AG-CoT
StateEq
4.65
6.40
7.33
8.29
9.24
10.07
Table 4: Pass@ N (%) from 64 candidates per target. Values for N<64 are estimated from this pool, and the N=64 column gives observed coverage. The Basic AG-CoT rows use the same 249,216 candidates, with StateEq-Pass@64 of 10.07% versus 4.42% under strict saved label equality.
System
Errors
Mean deficit
Median deficit
Deficit =1 Count (%)
Deficit ≤2 Count (%)
Direct SFT
3,616
8.22
9
39 (1.08)
148 (4.09)
AG-CoT SFT
3,425
7.66
8
103 (3.01)
266 (7.77)
RFT ver. 1
3,398
7.61
8
121 (3.56)
278 (8.18)
RFT ver. 2
3,409
7.60
8
108 (3.17)
279 (8.18)
RFT ver. 3
3,416
7.57
8
115 (3.37)
274 (8.02)
Table 5: Constraint deficits among valid but state-incorrect outputs of adapted Qwen2.5-3B models in Table 1 . The deficit counts missing independent target constraints. Each model has its own error set. Means, medians and percentages use the corresponding error count, not a common set of failed targets. Outputs with deficit one also contribute to the final column.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Object
Verifier role
Evaluation source
QCircuitBench Clifford random-circuit instances
Provides targets with exact stabilizer-tableau verification.
Prompt target
Signed Pauli generators Tx
Specifies the target state without exposing a reference circuit.
Candidate output
OpenQASM 3.0 program y
Defines an executable state-preparation circuit from ∣0⟩⊗n .
Strict 32B criterion
P(y) , K(y) , and L(Ty)=L(Tx)
Measures saved-label Exact@1.
State-equivalence criterion
P(y) , K(y) , and Equiv(Ty,Tx)=1
Removes generator-basis sensitivity.
Controlled task criterion
StateEq plus gate, qubit, header, and completion requirements
Measures task success in Table 1 .
Appendix
Table 6: Verifier protocol for the Clifford state-preparation task.
Figure 5: Target representation shift. Signed stabilizer generators replace long numeric state vectors with a compact target interface that still supports exact Clifford-state verification. Here n is the qubit count, Mx stores binary X/Z support, and rx stores generator signs.
Representation
Model-facing object
Role in this study
State vector
2n complex amplitudes
General but long text.
Stabilizer-state tableau
Signed Pauli generators; 2n2+n bits before formatting
Compact exact Clifford target.
OpenQASM output
Clifford circuit program
Executable candidate.
Appendix
Table 7: Representations used in the preprocessed Clifford state-preparation task.
Split or check
Count
Notes
Accepted AG-CoT examples
38,940
Before train/test split.
Generation failures
0
During AG-CoT construction.
Training examples
35,046
90% split.
Held-out test prompts
3,894
10% split.
Conversion check
500
Random tableau/circuit checks.
AG validation
1,000
Random Clifford tests.
Appendix
Table 8: Dataset construction and validation counts.
Qubits
2
3
4
5
6
7
8
9
10
11
12
Prompts
30
51
107
145
219
311
367
475
618
702
869
Appendix
Table 9: Held-out test prompt counts by qubit count. The counts sum to 3,894.
Figure 6: Overview of the 32B construction and verification. The target Tx=(Mx,rx) stores binary Pauli support Mx and generator signs rx . The trace ρx records m reduction steps through full reference Clifford tableaus T0,…,Tm to the identity I , and yx is the emitted OpenQASM program. For a generated program y , P(y) and K(y) indicate parsing and Clifford validity, Ty is its prepared state tableau, and L extracts saved stabilizer labels. The rightmost panel reports 32B direct results and basic AG-CoT search with 64 candidates per target.
Figure 7: AG-CoT data construction for the 32B study. The construction records the row-reduction trace and the synthesized OpenQASM circuit from the same process, then retains completions whose circuits pass the strict saved-label verifier.
Component
Required record
Reproducibility role
Held-out prompts
3,894 targets plus metadata
Replay held-out evaluation inputs.
Verifier/evaluator
Extraction, Clifford conversion, Exact@1 , StateEq@1 , Pass@ N
Table 10: Components required to reproduce the 32B study.
Package
32B study
3B eval.
Python
3.11
3.12.3
PyTorch
2.9.1
2.5.1
Transformers
4.57.6
4.52.4
TRL
0.27.2
–
PEFT
0.19.1
–
vLLM
0.15.0
0.6.4.post1
Appendix
Table 11: Software versions for the 32B study and the 3B single-output evaluations. A dash marks a package version not specified for the 3B evaluations.
System
A6000
H200 NVL
Shared successes
AG-CoT SFT
201
202
201
RFT version 1
213
213
213
Appendix
Table 12: Exact task success across two inference environments on the same 3,661 targets. Shared successes count targets solved in both environments. The A6000 counts are those reported in Table 1 .
Qubits
Targets
Direct SFT
AG-CoT SFT
3
30
0
3
4
95
1
8
5
135
3
12
6
201
2
10
7
282
4
16
8
355
7
29
Appendix
Table 13: Task success counts by qubit count for the 3B comparison. Both models use the same targets and one greedy completion per target. The target count is the denominator for both success rates in each row.
Family
Checkpoint
Correct / total
Task success (%)
Qwen
Base
0 / 3,661
0.00
Qwen
Direct SFT
41 / 3,661
1.12
Mistral
Base
0 / 3,661
0.00
Mistral
Direct SFT
78 / 3,661
2.13
Appendix
Table 14: Matched cross-family Direct-SFT replication on a common 3,661-row target-disjoint development partition. Each row uses one greedy FP16 completion with a 2,048-token generation cap and strict no-repair evaluation. These results are separate from the 8,000-token comparisons in Table 1 .
Qubits
Prompts
Pass@1 (%)
Pass@64 (%)
2
30
21.15
86.67
3
51
7.20
21.57
4
107
6.57
13.08
5
145
4.70
7.59
6
219
3.30
4.57
7
311
7.11
8.04
Appendix
Table 15: Search results by qubit count for RFT. Pass@1 averages the fraction of correct candidates among 64 samples per target, and Pass@64 gives observed target coverage.
Run
Samples
Metric
Value (%)
Zero-shot
1
Exact / StateEq
0.10 / 0.33
Basic AG-CoT
1
Exact / StateEq
2.62 / 5.37
Step AG-CoT
1
Exact / StateEq
2.31 / 4.83
GRPO-v2
1
Exact / StateEq
2.21 / 5.26
RFT-v1
1
Exact / StateEq
3.13 / 6.14
RFT-v1 Best-of-64
64
Pass@64
5.19
Appendix
Table 16: Evaluation budgets for the 32B results over 3,894 held-out prompts.
Run
Max tok.
Rate (%)
Syntax
Exact@1
StateEq@1
Trunc.
QASM-only SFT
2048
24.91
5.50
7.68
83.00
Appendix
Table 17: Exploratory QASM-only format stress test under a 2,048-token generation cap.
Event
Errors
Observed
Expected
Std. dev.
Deficit at most two
961
112
8.84
2.45
Sign conflict
2,581
1,335
1,639.2
16.5
Appendix
Table 18: Random reference comparisons for 32B RFT-v1 errors. The program reference preserves gate types and order on five to eight qubit targets. The sign reference preserves each output’s unsigned overlap on all valid errors. The final column is the reference standard deviation, not a confidence interval.
(a) Mean constraint deficit
n
N
Cap (%)
Observed (SE)
Empty
Length
Uniform
2
19
0.0
1.47 (0.12)
1.84
1.58
1.75
3
46
0.0
2.09 (0.12)
2.24
2.51
2.65
4
95
0.0
3.11 (0.10)
3.29
3.51
3.61
5
131
0.0
3.79 (0.11)
4.08
4.39
4.59
6
207
0.0
4.68 (0.10)
4.84
5.37
5.58
Appendix
Table 19: Complete reference comparison for the 2,581 valid, state-incorrect 32B RFT-v1 outputs. Here n is the qubit count, N is the number of analyzed errors, and r counts missing independent signed target constraints. Empty denotes the zero-gate circuit, length denotes random operands with the observed gate sequence, and uniform denotes a uniformly random stabilizer state. Both random references exclude exact target matches. Cap is the percentage of all outputs at that qubit count excluded by the 2,000-character limit, not a percentage of N . Panel (a) reports mean deficits, with standard errors (SE) across observed outputs in parentheses. Panel (b) gives actual counts for observed and empty candidates and expected counts for the random references.
Deficit r
Nonzero F
F=0 count
F>0 count
Total
1
2−1
70
72
142
2
2−2
100
92
192
3
2−3
106
72
178
4
2−4
143
92
235
5
2−5
116
88
204
6
2−6
153
147
300
Appendix
Table 20: Complete constraint-deficit distribution of the 2,581 valid, state-incorrect 32B RFT-v1 outputs with saved QASM shorter than 2,000 characters, pooled over two to twelve qubits. Deficit r counts missing independent signed target constraints and F is squared overlap with the target state. The second column gives the fidelity value only for the nonzero-overlap branch. The two count columns partition each row, and none of these outputs prepares the target exactly.
System
Exact
Syntax fail
Non-Cliff.
Label mismatch
Basic AG-CoT
102
5
0
3787
GRPO-v2
86
5
0
3803
RFT-v1
122
17
0
3755
Appendix
Table 21: 32B direct-verifier outcomes over 3,894 held-out prompts. Counts use the strict saved-label decision and separate exact matches from syntax failures, non-Clifford parses, and valid Clifford outputs with a saved-label mismatch.
Layer bucket
Correct / total
Exact@1 (%)
Test share (%)
1–2
35 / 47
74.5
1.2
3–5
27 / 64
42.2
1.6
6–10
17 / 132
12.9
3.4
11–14
5 / 102
4.9
2.6
15–20
1 / 142
0.7
3.6
21–50
0 / 664
0.0
17.1
Appendix
Table 22: GRPO-v2 direct evaluation by target circuit-layer bucket under strict saved-label Exact@1. Bucket percentages are rounded to one decimal place. The last bucket contains one correct output, whose rate of approximately 0.0365% rounds to 0.0%.
Direct target match
System
Exact@1 lab. (%)
StateEq@1 (%)
95% CI
Zero-shot
0.10
0.33
[0.15, 0.52]
Basic AG-CoT
2.62
5.37
[4.67, 6.06]
Step AG-CoT
2.31
4.83
[4.15, 5.50]
GRPO-v2
2.21
5.26
[4.57, 5.98]
RFT-v1
3.13
6.14
[5.37, 6.91]
Appendix
Table 23: State-equivalence confidence intervals over 3,894 held-out prompts; lab. denotes the saved-label verifier convention.
Run
Exact successes
Exact@1 (%)
95% CI
Basic AG-CoT
102 / 3894
2.62
[2.13, 3.13]
GRPO-v2
86 / 3894
2.21
[1.77, 2.70]
RFT-v1
122 / 3894
3.13
[2.59, 3.70]
Appendix
Table 24: Bootstrap confidence intervals for the direct Exact@1 metric (%) computed over 10,000 resamples of the 3,894 held-out prompts.
We review thirteen generative systems and five supporting datasets for quantum circuit and quantum code generation, identified through a structured scoping review of Hugging Face, arXiv, and provenance tracing (January-February 2026). We organize the field along two axes: artifact type (Qiskit code, OpenQASM programs, circuit graphs); crossed with training regime (supervised fine-tuning, verifier-in-the-loop RL, diffusion/graph generation, agentic optimization); and systematically apply a three-layer evaluation framework covering syntactic validity, semantic correctness, and hardware executability. The central finding is that while all reviewed systems address syntax and most address semantics to some degree, none reports end-to-end evaluation on quantum hardware (Layer 3b), leaving a significant gap between generated circuits and practical deployment. Scope note: quantum code refers throughout to quantum program artifacts (QASM, Qiskit); we do not cover generation of quantum error-correcting codes (QEC).
Quantum circuit optimization for fault-tolerant computing requires exact functional equivalence while minimizing expensive non-Clifford resources such as T gates. We study this problem using a compact 44.8M-parameter encoder-decoder transformer with structured circuit tokenization, evaluating on parameterized circuits (2-6 qubits) and Clifford+T circuits (3-6 qubits). On parameterized circuits, a hybrid approach -- structure from the transformer, angles from classical optimization -- achieves median fidelity 1.000 on 3-6 qubit circuits. On Clifford+T circuits, where all gates are discrete and no post-processing is possible, the model learns valid syntax and accurate T-Count statistics, yet exact equivalence degrades sharply with target length -- from 88% on circuits with <=9 gates to near zero beyond 26 gates. We trace this failure to autoregressive drift: early-token divergence cascading irrecoverably through left-to-right decoding. Two levers partially mitigate the drift: inference-time strategies that generate multiple candidates and select via equivalence verification raise exact-match rates from 7% to 22.5%, while scaling training data by 2.5x pushes them to 39.5%. Yet the degradation with target length persists -- even with more data, exact equivalence drops from 94% on short circuits to under 4% beyond 26 gates. The contrast between settings is our central finding: when approximate outputs can be rescued by post-processing, the transformer succeeds; when exact discrete correctness is required, autoregressive drift limits reliability, with both inference-time search and data scaling as effective levers while training-side fine-tuning and model-level diversification are not.
Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.
Ali Slim, Haydar Hamieh, Jawad Kotaich +5
American University of Beirut · King Abdullah University of Science and Technology