Whether uniquely quantum resources confer advantages in fully classical, competitive environments remains an open question. Competitive zero-sum reinforcement learning is particularly challenging, as success requires modelling dynamic interactions between opposing agents rather than static state-action mappings. Here, we conduct a controlled study isolating the role of quantum entanglement in a quantum-classical hybrid agent trained on Pong, a competitive Markov game. An 8-qubit parameterised quantum circuit serves as a feature extractor within a proximal policy optimisation framework, allowing direct comparison between separable circuits and architectures incorporating fixed (CZ) or trainable (IsingZZ) entangling gates. Entangled circuits consistently outperform separable counterparts with comparable parameter counts and, in low-capacity regimes, match or exceed classical multilayer perceptron baselines. Representation similarity analysis further shows that entangled circuits learn structurally distinct features, consistent with improved modelling of interacting state variables. These findings establish entanglement as a function resource for representation learning in competitive reinforcement learning.
Figures & tables
Figure 1: The overall architecture of our quantum-classical hybrid agent. The observation of the environment is an 8 -element vector [pl,pr,bx,by,vb,x,vb,y,sl,sr] , where pl and pr denote the position of the left and right paddle, respectively; bx and by are the coordinates of the ball, vb,x and vb,y are the velocity of the ball on the x− and y− direction, respectively; sl and sr are the scores for the left and the right paddle. A backbone network, whether classical or quantum, takes the observation vector as input and produces an 8-dimensional feature vector. A classical actor network and a classical critic network share the feature vector. Based on the features provided by the backbone network, the actor network proposes an action based on the input representations from the PQC feature extractor; the critic network provides a scalar evaluation of the current state. The four different kinds of backbone network structure studied in this paper are shown in (a) to (d). (a): classical multi-layer perception (MLP); (b) 8 -qubit separable parameterised quantum circuit; (c) 8 -qubit parameterised quantum circuit with fixed entanglement gates (controlled-Z gates), and (d) 8 -qubit parameterised quantum circuit with trainable entanglement gates (the IsingZZ gate exp(−i2θZi⊗Zj) ). At the end of the parameterised quantum circuits, all qubits are measured with the Pauli X observable.
Backbone Type
Param. Num.
Final Return Mean
Final Return Std.
Final Return Max.
Separable
48=48×1
-17.9
4.72
-8
Separable
96=48×2
-17.9
4.68
-8
Separable
144=48×3
-19.5
4.50
-6
Separable
192=48×4
-21.0
0.00
-21
Separable
240=48×5
-20.4
1.80
-15
Separable
288=48×6
-20.8
0.60
-19
Table 1: Averaged Performance for our hybrid agent with different parameter configurations. For each configuration, the statistics (mean and standard deviation) of agent performance are calculated from 10 random initialisations of the trainable parameters. For the Pong game, the maximum achievable return is 21, and the minimum is -21. And for single runs before smoothing with an exponential moving average, the return values are always integers. For quantum backbones, the number of parameters is shown as number of parameters in a single layer×number of layers . For classical backbones, since they all have three layers, the number of parameters is shown as input layer dimension×hidden layer dimension+hidden layer dimension×output layer dimension . The best performance of all different configurations comes from the classical backbone with 4096 parameters, which is supposed to have access to the same “hidden dimension size” as an 8-qubit quantum circuit, which also has a 28 -dimensional state vector. For all other configurations besides the classical backbone with 4096 parameters, the minimum reward at the final step during the training process is −21 , while the classical 4096 -parameter MLP backbone has a minimum final reward of −5 across 10 different random initialisations. The data in the “Final Return Max.” column is from the raw training statistics before being smoothed with a weighted exponential moving average, as those in Figure 2 and Figure 5 .
Figure 2: The return of different quantum backbone configurations with respect to the total training steps. The curves for separable, CZ-entangled, and IsingZZ-entangled are averaged over 10 runs with different random initialisations. The averaged return curve and the max-min shade for returns are calculated and plotted from the smoothed raw data using a weighted exponential moving average. We can see that, although entangled quantum backbones could, on average, outperform separable ones, it is unclear, based on the current results, whether trainable entanglement gates (IsingZZ) could achieve better performance than fixed entanglement gates (CZ). Even though, on average, the backbone with IsingZZ gates can outperform that with CZ gates (as in (a)), the best performance of quantum circuits with CZ gates is better than that of backbones with IsingZZ gates.
Figure 3: Comparing the (averaged) episodic performance of the three different quantum backbones (separable, CZ-entangled and IsingZZ-entangled) w.r.t. classical MLP with 64 parameters. (a): The performance of the IsingZZ-entangled backbone, compared with the classical MLP backbone with 64 parameters. Each layer of the IsingZZ-entangled backbone has 56 parameters. In this configuration, only the 1 -layer one could outperform the classical baseline, but with fewer parameters ( 56 vs 64 ). In this case, increasing the number of layers (and hence the number of parameters) does not guarantee improved performance either. (b): The performance of the CZ-entangled backbone, compared with the classical MLP backbone with 64 parameters. Among the different layer configurations, those with 2 and 3 layers can outperform the classical baseline after averaging results across 10 random initialisations. Since CZ-entangled quantum circuits have 48 parameters per layer, the CZ-entangled backbones that outperform the classical baseline have 96 and 144 parameters, respectively. However, when the number of layers is increased beyond 3 , the performance of the quantum-classical hybrid agent drops by half on average at the final iteration. (c): The performance of the separable backbone, compared with the classical MLP backbone. We can see that there is little difference in performance across different layer configurations of the separable backbone, and they all have worse average returns than the 64 -parameter MLP backbone.
Figure 4: The similarities of representations generated by different backbone configurations, calculated with centred kernel alignment (CKA) [ 14 ] . The heatmap is divided into approximately sixteen blocks, representing intragroup (diagonal blocks) and intergroup (off-diagonal blocks) similarities. For intra-group similarities, we see that the classical backbone generally produces similar representations, even though the hidden dimension differs. The CZ-entangled backbones produce representations with the most variability between different initialisations (intragroup block) and different backbone topologies (intergroup). Generally, for classical neural networks, CKA would yield very high similarity scores for representations produced by networks with the same structure, trained on the same dataset, but with different random initialisations, as shown in the bottom-right corner of the heatmap. However, such intra-group similarity is much harder to find among the representations produced by the quantum backbones. Separable and IsingZZ-entangled PQC backbones have an overall slightly higher intra-group similarity score compared to the CZ-entangled ones. However, all three types of PQC-based backbone networks produce representations that share little similarity with those from the classical backbone networks.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone Type
Params
Episodic Length Mean
Episodic Length Std.
Episodic Length Max.
Episodic Length Min.
Separable
48
274.5
106.21
417
132
Separable
96
331.5
115.96
607
227
Separable
144
274.5
114.40
417
132
Separable
192
227.0
73.59
417
132
Separable
240
236.5
78.91
417
132
Separable
288
303.0
254.20
987
132
Appendix
Table 2: Averaged episodic length for our hybrid agent with different parameter configurations. For each configuration, the statistics (mean and standard deviation) of the lengths of episodes are calculated from 10 random initialisations of the trainable parameters. And for single runs before smoothing with an exponential moving average, the number of steps in a single episode is always an integer. It should be noted that the episodic length is a less informative indicator for the agents’ performance compared to the actual return the agent obtains from the environment. When the number of steps in an episode is low, the agent could perform well or poorly, since in Pong, the agent needs to play with a computer-controlled paddle to win the game. Small episodic length could indicate that the agent finishes the opponent quickly, or gets finished by the opponent quickly.
Figure 5: Comparing the (averaged) episodic performance of the three different quantum backbones (separable, CZ-entangled and IsingZZ-entangled) w.r.t. classical MLP with 128 , 256 , 336 and 4096 parameters.
Quantum reinforcement learning studies how quantum circuits and algorithms can improve learning, planning, and control in classical or quantum environments.
We present an empirical evaluation of quantum entanglement in agent coordination within quantum multi agent reinforcement learning (QMARL). While QMARL has attracted growing interest recently, most prior work evaluates quantum policies without provable baselines, making it impossible to rigorously distinguish quantum advantage from algorithmic coincidence. We address this directly by evaluating a decentralized QMARL framework with variational quantum circuit (VQC) actors with shared entangled states. In the CHSH game, which has a mathematically proven classical performance ceiling of 0.75 win rate, we show that entangled QMARL agents approach the Tsirelson limit of 0.854, providing clear evidence of their quantum advantage. We show that unentangled quantum circuits match the classical baseline, confirming that entanglement and not the quantum circuit itself is the active coordination mechanism. We also explore the effect of specific entanglement structures, as some Bell states enable coordination gains while others actively harm performance. On cooperative navigation (CoopNav), QMARL without entanglement achieves ∼2× improvement in success rate over classical MAA2C (∼0.85 versus ∼0.40), with the hybrid configuration, quantum actor paired with a classical centralised critic, outperforming both fully classical and fully quantum solutions. We present our experimental analysis and discuss future work.
Parameterized quantum circuits (PQCs) are increasingly used as policies and value functions in quantum reinforcement learning, yet it remains unclear when and why quantum policies generalize. We give a PAC-Bayesian account in which generalization is governed not by the raw number of circuit parameters, but by the effective dimension of the Fisher geometry induced by the circuit. This quantity is inflated by entanglement, making entangling connectivity an independent axis of complexity.In controlled experiments that fix the number of trainable rotations and vary only entanglement, we find that circuits with larger Fisher effective dimension exhibit larger train-test gaps, while parameter count is a weak predictor. The resulting bound acts primarily as a ranking certificate: it correctly orders circuits with identical parameter count, which parameter-counting bounds cannot do. We validate this mechanism across supervised classification, quantum contextual bandits, and value-function generalization, where entangled circuits consistently generalize worse than non-entangled circuits of equal parameter count, with gaps shrinking as sample size increases.Our strongest evidence comes from low-variance decision models, including single-observable classifiers, value heads, and one-step policies. In end-to-end multi-step policy learning, entanglement effects remain statistically significant but high return variance leaves the full ordering only partially resolved. Partial-correlation analysis shows that Fisher effective dimension screens off entangling pattern, and controls for training accuracy, readout, and optimizer rule out major optimization confounders. The effect also persists on an IBM Heron quantum processor under real noise. Overall, our results reframe quantum policy design around an entanglement--generalization trade-off rather than expressivity alone.
Jian Xu, Delu Zeng, John Paisley +1
1RIKEN iTHEMS · 2RIKEN AIP · 3South China University of Technology +1
Future quantum networks will make use of entanglement to perform numerous tasks, such as sending quantum information over long distances, distributed quantum computing, and quantum sensing. In general, these tasks will need to be performed simultaneously in various regions of a network, while minimizing resources and latency. We will thus require policies for scheduling link-level entanglement resources, and using the link-level entanglement to create various forms of multipartite entanglement required for every task. In this work, we address this problem using reinforcement learning. We formulate a Markov Decision Process for the problem and use double deep Q-networks (DQN) with Message Passing Neural Networks (MPNNs), experience replay buffers, and curriculum training to obtain policies. The key physical parameter is the probability of link-level entanglement generation, i.e., the link activation probability. We show that our policies maintain 100% success for up to 71% lower link activation probability than the baseline heuristics for a set of physically relevant network topologies. We then examine an additional constraint where experiment (task) placements are restricted to specific hardware types and demonstrate a similar advantage in performance over heuristics, with our policy maintaining at least an 80% success rate for up to a 59% lower link activation probability. Finally, we explore methods to interpret the learned policy by defining metrics enabling conclusions to be drawn about the model's behavior and by tasking a large language model (LLM) to derive a novel heuristic given example actions taken by the DQN-trained policy. We find that the LLM heuristic performs similarly to the DQN-trained policy in performance, indicating a promising method for interpretable policy extraction for large quantum networks, where direct training becomes computationally expensive.
Leon Rode, Sumeet Khatri, Supartha Podder
Department of Computer Science, Stony Brook University · Department of Computer Science and Center for Quantum Information Science and Engineering, Virginia Tech