Dec 5, 2025
This paper studies networked multi-agent reinforcement learning (NMARL) with interdependent rewards and coupled policies. In this setting, each agent's reward depends on its own state-action pair as well as those of its direct neighbors, and each agent's policy is parameterized by its local parameters together with those of its
κp-hop neighbors, with
κp≥1 denoting the coupled radius. To address the challenge of interdependent policies in collaborative optimization, we introduce a novel concept termed the ``neighbors' averaged
Q-function'' and derive a new expression for the coupled-policy gradient. Based on these theoretical foundations, we develop a distributed scalable coupled policy (DSCP) algorithm, where each agent relies only on the state-action pairs of its
κp-hop neighbors and the rewards of its
(κp+1)-hop neighbors. Specifically, in the DSCP algorithm, we employ a geometric 2-horizon sampling method that does not require storing a full
Q-table to obtain an unbiased estimate of the surrogate coupled-policy gradient under the executed policy. Moreover, each agent interacts exclusively with its direct neighbors to obtain accurate policy parameters, while maintaining local estimates of other agents' parameters to execute its local policy and collect samples for optimization. These estimates and policy parameters are updated in a distributed manner through local information exchange among direct neighbors, enabling coordinated policy updates across the network. We prove that the gradient norm of the objective function along the joint policy sequence produced by the proposed algorithm converges to zero almost surely. Finally, the effectiveness of the DSCP algorithm is demonstrated through simulations in a robot path planning environment, showing clear improvement over the compared algorithms in the tested setting.
Pengcheng Dai, Dongming Wang, Wenwu Yu +1
Engineering Systems and Design Pillar, Singapore University of Technology and Design, Singapore 487372 · Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA · Frontiers Science Center for Mobile Information Communication and Security, School of Mathematics, Southeast University, Nanjing 210096, China, and also with Purple Mountain Laboratories, Nanjing 211102, China