cs.LGMar 7, 2025

Partially Decentralized Multi-Agent Q-Learning via Digital Cousins for Wireless Networks

Authors: Talha Bozkus, Urbashi Mitra

Organizations: Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, USA.

Abstract

Q-learning is a widely used reinforcement learning (RL) algorithm for optimizing wireless networks, but faces challenges with large state-spaces. Recently proposed multi-environment mixed Q-learning (MEMQ) algorithm addresses these challenges by employing multiple Q-learning algorithms across multiple synthetically generated, distinct but structurally related environments, so-called digital cousins. In this paper, we propose a novel multi-agent MEMQ (M-MEMQ) for cooperative decentralized wireless networks with multiple networked transmitters (TXs) and base stations (BSs). TXs do not have access to global information (joint state and actions). The new concept of coordinated and uncoordinated states is introduced. In uncoordinated states, TXs act independently to minimize their individual costs and update local Q-functions. In coordinated states, TXs use a Bayesian approach to estimate the joint state and update the joint Q-functions. The cost of information-sharing scales linearly with the number of TXs and is independent of the joint state-action space size. Several theoretical guarantees, including deterministic and probabilistic convergence, bounds on estimation error variance, and the probability of misdetecting the joint states, are given. Numerical simulations show that M-MEMQ outperforms several decentralized and centralized training with decentralized execution (CTDE) multi-agent RL algorithms by achieving 60% lower average policy error (APE), 40% faster convergence, 45% reduced runtime complexity, and 40% less sample complexity. Furthermore, M-MEMQ achieves comparable APE with significantly lower complexity than centralized methods. Simulations validate the theoretical analyses.

Figures & tables

Explore similar work

Jul 24, 2026

Variance-Reduced Q-Learning over Static and Time-Varying Networks

We investigate a decentralized reinforcement learning problem involving multiple agents that interact with the same Markov Decision Process (MDP). The agents can exchange information over a network to collectively learn the optimal state-action value function. For this setting, we introduce a novel epoch-based distributed QQ-learning algorithm called VRDQ, where within each epoch, agents locally estimate the Bellman optimality operator and diffuse information using a consensus-based protocol. For both static and time-varying networks, we establish high-probability finite-time convergence rates for VRDQ that enjoy linear speedups from collaboration. Crucially, we prove that such speedups in sample-complexity require only O~(1)\tilde{O}(1) communication, substantially improving upon the communication costs in prior work.
Dec 5, 2025

Distributed scalable coupled policy algorithm for networked multi-agent reinforcement learning

This paper studies networked multi-agent reinforcement learning (NMARL) with interdependent rewards and coupled policies. In this setting, each agent's reward depends on its own state-action pair as well as those of its direct neighbors, and each agent's policy is parameterized by its local parameters together with those of its κpκ_{p}-hop neighbors, with κp≥1κ_{p}\geq 1 denoting the coupled radius. To address the challenge of interdependent policies in collaborative optimization, we introduce a novel concept termed the ``neighbors' averaged QQ-function'' and derive a new expression for the coupled-policy gradient. Based on these theoretical foundations, we develop a distributed scalable coupled policy (DSCP) algorithm, where each agent relies only on the state-action pairs of its κpκ_{p}-hop neighbors and the rewards of its (κp+1)(κ_{p}+1)-hop neighbors. Specifically, in the DSCP algorithm, we employ a geometric 2-horizon sampling method that does not require storing a full QQ-table to obtain an unbiased estimate of the surrogate coupled-policy gradient under the executed policy. Moreover, each agent interacts exclusively with its direct neighbors to obtain accurate policy parameters, while maintaining local estimates of other agents' parameters to execute its local policy and collect samples for optimization. These estimates and policy parameters are updated in a distributed manner through local information exchange among direct neighbors, enabling coordinated policy updates across the network. We prove that the gradient norm of the objective function along the joint policy sequence produced by the proposed algorithm converges to zero almost surely. Finally, the effectiveness of the DSCP algorithm is demonstrated through simulations in a robot path planning environment, showing clear improvement over the compared algorithms in the tested setting.
Aug 13, 2026

Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry

We study decentralized multi-player reinforcement learning in episodic tabular Markov decision processes (MDPs) under three forms of information asymmetry: (A) unobserved actions with common rewards, (B) observed actions with independent rewards, and (C) unobserved actions with independent rewards. Players cannot communicate during learning but may agree on a protocol a priori. For Problems A and B we propose mQ-learning and mQ-learning-intervals, achieving O~(H4SAjoint T)\tilde{O}(\sqrt{H^4 S A_{\text{joint}}\, T}) regret, where HH is the horizon, SS the state count, T=KHT = KH the total steps, and Ajoint=∏i=1M∣Ai∣A_{\text{joint}} = \prod_{i=1}^M |\mathcal{A}_i| the joint action space across MM players. For Problem C we give mEXC and mEXC-Bellman, two-phase explore-then-commit algorithms with regret O~(H(SAjoint)1/3T2/3)\tilde{O}(H (S A_{\text{joint}})^{1/3} T^{2/3}). Against the centralized joint-action benchmark, decentralized learning under information asymmetry matches the single-agent Q-learning rate of \cite{jin2018q} up to logarithmic factors. Because AjointA_{\text{joint}} grows exponentially in MM, the bounds are most meaningful for small MM or small per-player action sets.