MADQRL: Distributed Quantum Reinforcement Learning Framework for Multi-Agent Environments
Authors: Abhishek Sawaika, Samuel Yen-Chi Chen, Udaya Parampalli, Rajkumar Buyya
Organizations: Quantum Cloud Computing and Distributed Systems Lab (qCLOUDS), School of Computing and Information Systems, The University of Melbourne, Australia · Brookhaven National Laboratory, Upton, NY, USA
Reinforcement learning (RL) is one of the most practical ways to learn from real-life use-cases. Motivated from the cognitive methods used by humans makes it a widely acceptable strategy in the field of artificial intelligence. Most of the environments used for RL are often high-dimensional, and traditional RL algorithms becomes computationally expensive and challenging to effectively learn from such systems. Recent advancements in practical demonstration of quantum computing (QC) theories, such as compact encoding, enhanced representation and learning algorithms, random sampling, or the inherent stochastic nature of quantum systems, have opened up new directions to tackle these challenges. Quantum reinforcement learning (QRL) is seeking significant traction over the past few years. However, the current state of quantum hardware is not enough to cater for such high-dimensional environments with complex multi-agent setup. To tackle this issue, we propose a distributed framework for QRL where multiple agents learn independently, distributing the load of joint training from individual machines. Our method works well for environments with disjoint sets of action and observation spaces, but can also be extended to other systems with reasonable approximations. We analyze the proposed method on cooperative-pong environment and our results indicate ~10% improvement from other distribution strategies, and ~5% improvement from classical models of policy representation.
Figures & tables
Fig. 1 : Architecture of MADQRL framework. Agents learn independently to optimize for joint objective defined by the environment. This independent training happens on local observation, reward and action spaces, such that the joint policy can be approximated by the product of local policies. Here, the MDP representation of the environment is just an example for completeness of the system.
Flatten
Linear_Layer
VQC
RELU
Linear_layer
RELU
Table 2
Fig. 2 : A sample quantum circuit with 4 qubits, having angle encoding using RX gates and L=3 layers of strongly entangled variational layers, followed by full measurement in Z -Basis. Here we use three different entanglement layers.
Fig. 3 : A snapshot of the cooperative-pong game environment. The entire black box is the ”playing arena”, the small white dot at the center is the ”ball”, which moves freely based on the laws of physics, and the two elongated white area on the left and the right edges are the paddles, representing agents. Here, the right one is a cake shaped paddle, while the one on the left side is a standard block paddle.
Model - Distribution
Total Runtime (s)
Test Episodes
Trainable Weights
Sampling Saturation (%iterations)
Classical-Shared
377
193
40074
40
Classical-Joint
291
254
20560
35.8
Classical-Independent
391
264
18752
45
Quantum-Shared
2753
176
20140
37.9
Quantum-Joint SE
1529
259
10889
37.8
Quantum-Joint BE
1795
245
10893
37.5
TABLE I : Performance analysis on two agent cooperative pong game for different combinations of policy model and distribution strategies. A CNN and a QNN based architectures are used for classical and quantum models, respectively. There are 1170 tunable parameters in the VQCs used in quantum models.
Fig. 4 : Comparing learning curves for different distributed learning frameworks between (a) classical, (b) quantum–classical, and (c) quantum models. Results are reported for a 2-agent cooperative Pong game with the observation space reduced to 64×64 . Curves show the moving average of the mean reward with a window size of 5 , while the shaded regions indicate the corresponding standard deviation.
Quantum reinforcement learning studies how quantum circuits and algorithms can improve learning, planning, and control in classical or quantum environments.
High-Assurance Software Laboratory (HASLab), INESC TEC, Portugal · Department of Computer Science, University of Minho, Portugal · International Iberian Nanotechnology Laboratory (INL), Portugal