cs.LGMar 7, 2025

Partially Decentralized Multi-Agent Q-Learning via Digital Cousins for Wireless Networks

Authors: Talha Bozkus, Urbashi Mitra

Organizations: Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, USA.

Abstract

Q-learning is a widely used reinforcement learning (RL) algorithm for optimizing wireless networks, but faces challenges with large state-spaces. Recently proposed multi-environment mixed Q-learning (MEMQ) algorithm addresses these challenges by employing multiple Q-learning algorithms across multiple synthetically generated, distinct but structurally related environments, so-called digital cousins. In this paper, we propose a novel multi-agent MEMQ (M-MEMQ) for cooperative decentralized wireless networks with multiple networked transmitters (TXs) and base stations (BSs). TXs do not have access to global information (joint state and actions). The new concept of coordinated and uncoordinated states is introduced. In uncoordinated states, TXs act independently to minimize their individual costs and update local Q-functions. In coordinated states, TXs use a Bayesian approach to estimate the joint state and update the joint Q-functions. The cost of information-sharing scales linearly with the number of TXs and is independent of the joint state-action space size. Several theoretical guarantees, including deterministic and probabilistic convergence, bounds on estimation error variance, and the probability of misdetecting the joint states, are given. Numerical simulations show that M-MEMQ outperforms several decentralized and centralized training with decentralized execution (CTDE) multi-agent RL algorithms by achieving 60% lower average policy error (APE), 40% faster convergence, 45% reduced runtime complexity, and 40% less sample complexity. Furthermore, M-MEMQ achieves comparable APE with significantly lower complexity than centralized methods. Simulations validate the theoretical analyses.

Figures & tables

Explore similar work

CardsList
  1. Variance-Reduced Q-Learning over Static and Time-Varying Networks

    Jul 24, 2026Sreejeet Maity, Feng Zhu, Aritra Mitra +1Reinforcement LearningQ-Learning

  2. Distributed scalable coupled policy algorithm for networked multi-agent reinforcement learning

    Dec 5, 2025Pengcheng Dai, Dongming Wang, Wenwu Yu +1Cooperative MARLDecentralized MARL

  3. Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry

    Aug 13, 2026Larissa Xu, King Bi, William ChangEpisodic RLRegret Minimization in RL