quant-phMar 11, 2026
SaveQuantum entanglement provides a competitive advantage in adversarial games
Organizations: Technology, CSIRO Clayton, Research Way, Clayton, VIC 3168, Australia.
Abstract
Whether uniquely quantum resources confer advantages in fully classical, competitive environments remains an open question. Competitive zero-sum reinforcement learning is particularly challenging, as success requires modelling dynamic interactions between opposing agents rather than static state-action mappings. Here, we conduct a controlled study isolating the role of quantum entanglement in a quantum-classical hybrid agent trained on Pong, a competitive Markov game. An 8-qubit parameterised quantum circuit serves as a feature extractor within a proximal policy optimisation framework, allowing direct comparison between separable circuits and architectures incorporating fixed (CZ) or trainable (IsingZZ) entangling gates. Entangled circuits consistently outperform separable counterparts with comparable parameter counts and, in low-capacity regimes, match or exceed classical multilayer perceptron baselines. Representation similarity analysis further shows that entangled circuits learn structurally distinct features, consistent with improved modelling of interacting state variables. These findings establish entanglement as a function resource for representation learning in competitive reinforcement learning.
Figures & tables
Figure 1: The overall architecture of our quantum-classical hybrid agent. The observation of the environment is an -element vector , where and denote the position of the left and right paddle, respectively; and are the coordinates of the ball, and are the velocity of the ball on the and direction, respectively; and are the scores for the left and the right paddle. A backbone network, whether classical or quantum, takes the observation vector as input and produces an 8-dimensional feature vector. A classical actor network and a classical critic network share the feature vector. Based on the features provided by the backbone network, the actor network proposes an action based on the input representations from the PQC feature extractor; the critic network provides a scalar evaluation of the current state. The four different kinds of backbone network structure studied in this paper are shown in (a) to (d). (a): classical multi-layer perception (MLP); (b) -qubit separable parameterised quantum circuit; (c) -qubit parameterised quantum circuit with fixed entanglement gates (controlled-Z gates), and (d) -qubit parameterised quantum circuit with trainable entanglement gates (the IsingZZ gate ). At the end of the parameterised quantum circuits, all qubits are measured with the Pauli X observable.
| Backbone Type | Param. Num. | Final Return Mean | Final Return Std. | Final Return Max. |
|---|---|---|---|---|
| Separable | -17.9 | 4.72 | -8 | |
| Separable | -17.9 | 4.68 | -8 | |
| Separable | -19.5 | 4.50 | -6 | |
| Separable | -21.0 | 0.00 | -21 | |
| Separable | -20.4 | 1.80 | -15 | |
| Separable | -20.8 | 0.60 | -19 |
Table 1: Averaged Performance for our hybrid agent with different parameter configurations. For each configuration, the statistics (mean and standard deviation) of agent performance are calculated from 10 random initialisations of the trainable parameters. For the Pong game, the maximum achievable return is 21, and the minimum is -21. And for single runs before smoothing with an exponential moving average, the return values are always integers. For quantum backbones, the number of parameters is shown as . For classical backbones, since they all have three layers, the number of parameters is shown as . The best performance of all different configurations comes from the classical backbone with parameters, which is supposed to have access to the same “hidden dimension size” as an 8-qubit quantum circuit, which also has a -dimensional state vector. For all other configurations besides the classical backbone with parameters, the minimum reward at the final step during the training process is , while the classical -parameter MLP backbone has a minimum final reward of across different random initialisations. The data in the “Final Return Max.” column is from the raw training statistics before being smoothed with a weighted exponential moving average, as those in Figure 2 and Figure 5 .
Figure 2: The return of different quantum backbone configurations with respect to the total training steps. The curves for separable, CZ-entangled, and IsingZZ-entangled are averaged over 10 runs with different random initialisations. The averaged return curve and the max-min shade for returns are calculated and plotted from the smoothed raw data using a weighted exponential moving average. We can see that, although entangled quantum backbones could, on average, outperform separable ones, it is unclear, based on the current results, whether trainable entanglement gates (IsingZZ) could achieve better performance than fixed entanglement gates (CZ). Even though, on average, the backbone with IsingZZ gates can outperform that with CZ gates (as in (a)), the best performance of quantum circuits with CZ gates is better than that of backbones with IsingZZ gates.
Figure 3: Comparing the (averaged) episodic performance of the three different quantum backbones (separable, CZ-entangled and IsingZZ-entangled) w.r.t. classical MLP with parameters. (a): The performance of the IsingZZ-entangled backbone, compared with the classical MLP backbone with parameters. Each layer of the IsingZZ-entangled backbone has parameters. In this configuration, only the -layer one could outperform the classical baseline, but with fewer parameters ( vs ). In this case, increasing the number of layers (and hence the number of parameters) does not guarantee improved performance either. (b): The performance of the CZ-entangled backbone, compared with the classical MLP backbone with parameters. Among the different layer configurations, those with and layers can outperform the classical baseline after averaging results across random initialisations. Since CZ-entangled quantum circuits have parameters per layer, the CZ-entangled backbones that outperform the classical baseline have and parameters, respectively. However, when the number of layers is increased beyond , the performance of the quantum-classical hybrid agent drops by half on average at the final iteration. (c): The performance of the separable backbone, compared with the classical MLP backbone. We can see that there is little difference in performance across different layer configurations of the separable backbone, and they all have worse average returns than the -parameter MLP backbone.
Figure 4: The similarities of representations generated by different backbone configurations, calculated with centred kernel alignment (CKA) [ 14 ] . The heatmap is divided into approximately sixteen blocks, representing intragroup (diagonal blocks) and intergroup (off-diagonal blocks) similarities. For intra-group similarities, we see that the classical backbone generally produces similar representations, even though the hidden dimension differs. The CZ-entangled backbones produce representations with the most variability between different initialisations (intragroup block) and different backbone topologies (intergroup). Generally, for classical neural networks, CKA would yield very high similarity scores for representations produced by networks with the same structure, trained on the same dataset, but with different random initialisations, as shown in the bottom-right corner of the heatmap. However, such intra-group similarity is much harder to find among the representations produced by the quantum backbones. Separable and IsingZZ-entangled PQC backbones have an overall slightly higher intra-group similarity score compared to the CZ-entangled ones. However, all three types of PQC-based backbone networks produce representations that share little similarity with those from the classical backbone networks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone Type | Params | Episodic Length Mean | Episodic Length Std. | Episodic Length Max. | Episodic Length Min. |
|---|---|---|---|---|---|
| Separable | 274.5 | 106.21 | 417 | 132 | |
| Separable | 331.5 | 115.96 | 607 | 227 | |
| Separable | 274.5 | 114.40 | 417 | 132 | |
| Separable | 227.0 | 73.59 | 417 | 132 | |
| Separable | 236.5 | 78.91 | 417 | 132 | |
| Separable | 303.0 | 254.20 | 987 | 132 |
Table 2: Averaged episodic length for our hybrid agent with different parameter configurations. For each configuration, the statistics (mean and standard deviation) of the lengths of episodes are calculated from 10 random initialisations of the trainable parameters. And for single runs before smoothing with an exponential moving average, the number of steps in a single episode is always an integer. It should be noted that the episodic length is a less informative indicator for the agents’ performance compared to the actual return the agent obtains from the environment. When the number of steps in an episode is low, the agent could perform well or poorly, since in Pong, the agent needs to play with a computer-controlled paddle to win the game. Small episodic length could indicate that the agent finishes the opponent quickly, or gets finished by the opponent quickly.
Figure 5: Comparing the (averaged) episodic performance of the three different quantum backbones (separable, CZ-entangled and IsingZZ-entangled) w.r.t. classical MLP with , , and parameters.
More in this topic
Quantum RL
Quantum reinforcement learning studies how quantum circuits and algorithms can improve learning, planning, and control in classical or quantum environments.
See all 34 papers