Trainability-Simulability Tradeoff in Parameterized IQP-based Reinforcement Learning Policies
Organizations: Quantum Software Engineering, HASLab INESC TEC, Braga, Portugal
Abstract
Parameterized instantaneous quantum polynomial-time (IQP) circuits generate distributions conjectured to be hard to sample classically. As reinforcement learning (RL) policies, their measurement outcomes are typically mapped to a smaller action space by a classical decoder. This aggregation can mitigate initialization concentration while making the policy classically simulable. We characterize this tradeoff through the decoder's Fourier--Pauli expansion. Under independent uniform initialization, we express policy variance as an exact sum over groups of nonconstant Pauli components with identical active-generator signatures. Each group contributes its squared aggregate coefficient times , where is the binary rank of its active generators. An inverse-polynomial aggregate coefficient with logarithmic rank prevents exponential policy concentration. Sparse Pauli expansions and contiguous action bins permit efficient classical additive estimation of action probabilities. We introduce a majority decoder combining protected low-weight components with exponentially large exact Fourier support, preserving initialization nonconcentration while excluding polynomial-size enumeration of the exact expansion. Numerical experiments corroborate the variance laws. In ten-qubit CartPole experiments, majority achieves higher returns than recursive parity and the tested local readouts, while higher-weight Pauli sectors contribute an increasing relative share of gradient magnitude during training. A complementary 21-qubit block-chain policy outperforms an extensive pool of fully connected classical neural networks in episode efficiency, even though its bounded-width structure permits efficient classical simulation. These results guide decoder design for quantum and quantum-inspired RL, distinguishing policy nonconcentration, learning performance, and classical simulability.
Figures & tables
| Decoder or model | Five-qubit ring, | Ten-qubit cubic graph, |
|---|---|---|
| Majority–modulo | ||
| Recursive parity | ||
| Fixed | ||
| Local-linear POVM | ||
| Contiguous | ||
| Matched classical MLP |
| Model | Parameters | Learning rate | AUC | Episodes to criterion | Reached | Evaluation return |
|---|---|---|---|---|---|---|
| MLP ( ) | 9 | 0.10 | 0.7229 | 374.45 | 15/20 | 191.25 |
| MLP ( ) | 16 | 0.10 | 0.7985 | 277.70 | 18/20 | 197.10 |
| MLP ( ) | 30 | 0.10 | 0.8469 | 263.65 | 17/20 | 191.58 |
| MLP ( ) | 44 | 0.10 | 0.8631 | 261.90 | 17/20 | 193.86 |
| MLP ( ) | 65 | 0.10 | 0.8428 | 309.75 | 16/20 | 187.60 |
| MLP ( ) | 86 | 0.03 | 0.8413 | 274.70 | 18/20 | 198.24 |
| Condition | Initial return | AUC | Episodes |
|---|---|---|---|
| IQP, original | 73.63 | 0.9319 | 171.45 |
| IQP, uniform | 22.13 | 0.7615 | 315.90 |
| MLP, 135 parameters | 20.22 | 0.8680 | 236.50 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Depth | Parity | Majority–modulo |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 |
| Depth | Parity | Majority–modulo |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 |
| Circuit | Training condition | Parity | Majority–modulo | Paired difference (95% CI) |
|---|---|---|---|---|
| 5 qubits, | Exact + analytic gradient | |||
| 1,024 shots + SPSA ( ) | ||||
| 4,096 shots + SPSA ( ) | ||||
| 4,096 shots + SPSA ( ) | ||||
| 10 qubits, | Exact + analytic gradient | |||
| 1,024 shots + SPSA ( ) |
| Implementation | Mean seconds |
|---|---|
| IQP, reference contraction | 43.58 |
| IQP, compiled rollout | 38.45 |
| IQP, compiled and post-freeze optimized | 34.03 |
| MLP, reference | 8.98 |
| MLP, post-freeze optimized | 7.90 |
Quantum RL
Quantum reinforcement learning studies how quantum circuits and algorithms can improve learning, planning, and control in classical or quantum environments.
See all 39 papers