Many quantum software development kits provide a suite of circuit optimisation passes. These passes have been highly optimised and tested in isolation. However, the order in which they are applied is left to the user, or else defined in general-purpose default pass sequences. While general-purpose sequences miss opportunities for optimisation which are particular to individual circuits, designing pass sequences bespoke to particular circuits requires exceptional knowledge about quantum circuit design and optimisation. Here we propose and demonstrate training a reinforcement learning agent to compose optimisation-pass sequences. In particular the agent's action space consists of passes for two-qubit gate count reduction used in default PyTKET pass sequences. For the circuits in our diverse test set, the (mean, median) fraction of two-qubit gates removed by the agent is (57.7%,56.7%), compared to (41.8%,50.0%) for the next best default pass sequence. In doing so our model demonstrates the ability to forgo reward to improve the final two-qubit gate count. Our approach clearly separates the problem of designing circuit optimisation passes from the problem of using them optimally in practice. We expect this separation of concerns to facilitate innovations in both areas.
Figures & tables
Figure 1: Training and deploying an RL agent for optimisation-pass selection as described in this work. 1(a) : A quantum circuit is loaded into the environment and transformed into a graph representation, as discussed in Section 3.1 . This graph (the RL “observation” [ 36 ] ) is processed by a pair of GNNs, G , as discussed in Section 3.3 . The actor network outputs a vector of logits, a , for selecting the next optimisation pass; the critic head estimates the current circuit state value, V . An optimisation pass, as discussed in Section 3.2 , is selected via softmax sampling of a and applied to the circuit. The reward, along with V and a , is used to compute gradients, ∇ , and update G via PPO. The environment is reset with a fresh circuit from the training dataset (discussed in Section 3.4 ) whenever either: the DoNothing pass is selected; the two-qubit gate count is reduced to 0; or a preset number of passes are applied with no observed improvement. 1(b) : An input circuit is converted to a graph and repeatedly ingested by the actor network of G . Optimisation passes are selected via argmax sampling of a . The cycle terminates when the DoNothing pass is selected, and the optimised circuit is returned to the user.
Figure 2: Example circuit encoded as a graph.
Parameter
Value
PPO
Learning rate
3.36×10−4
Rollout length per environment
128
Batch size
64
Number of epochs
3
Discount factor ( γ )
0.952
Table 1: Optimised hyperparameters for the PPO and GNN architecture.
Figure 3: Cumulative reward (fraction of two-qubit gates removed) for in-distribution circuits. 40,412 circuits are optimised with nine trained models and a selection of PyTKET optimisation passes. Box plots give the median and interquartile range. Whiskers extend to the lowest (highest) data point still within 1.5 times the interquartile range of the lower (upper) quartile. ‘Full test dataset’ accumulates all of the circuits.
Figure 4: Difference in cumulative reward (fraction of two-qubit gates removed) between the RL model and baseline optimisation passes for in-distribution circuits. Positive values indicate improved performance of the RL model. 40,412 circuits are optimised with nine trained models and a selection of PyTKET optimisation methods. Box plots give the median and interquartile range. Whiskers extend to the lowest (highest) data point still within 1.5 times the interquartile range of the lower (upper) quartile. ‘Full test dataset’ accumulates all of the circuits. ‘Best Baseline’ corresponds to the highest cumulative reward generated by any baseline approach for each circuit.
Figure 5: Mean number of optimisation passes selected per circuit for each circuit class. Bars show the mean of the per-run averages across the nine PPO training runs, and error bars indicate ±1 sample standard deviation across those averages. Counts include all optimisation passes in each recorded evaluation sequence, excluding the DoNothing action.
Figure 6: Cumulative reward (fraction of two-qubit gates removed) for out-of-distribution circuits. 8,115 circuits are optimised with one trained model and a selection of PyTKET optimisation passes. Box plots give the median and interquartile range. Whiskers extend to the lowest (highest) data point still within 1.5 times the interquartile range of the lower (upper) quartile. ‘Full test dataset’ accumulates all of the circuits.
Figure 7: Cumulative reward (fraction of two-qubit gates removed) for very large out-of-distribution circuits. 1,000 circuits acting on 100 qubits, equally distributed between all circuit types in Section 3.4 , are optimised with one trained model, QuantinuumDefaultTwo , and QuantinuumDefaultThree . Box plots give the median and interquartile range across all circuit types. Whiskers extend to the lowest (highest) data point still within 1.5 times the interquartile range of the lower (upper) quartile.
Figure 8: Comparison to search-based optimisation techniques. A total of 930 circuits from the Clifford-SU4, Ordered-Clifford-SU4, and Clifford-SU4-SU8 circuit classes are optimised for each of the optimisation methods.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Mean validation learning curves on in-distribution circuits for the 9 PPO training runs. The top panel shows the mean validation cumulative reward at each evaluation; the bottom panel shows the highest mean validation cumulative reward observed so far. Colours identify the same runs in both panels. In the bottom panel, dots mark the last improvement and crosses mark the final evaluation. Environment transitions are summed across the 6 parallel training environments.