Enhancing the MADDPG Algorithm for Multi-Agent Learning via Action Inference and Importance Sampling

Marc Walden; Jason Liu; Shaashwath Sivakumar; Ryan Liu; Hamza Khan

cs.LGJun 3, 2026

Enhancing the MADDPG Algorithm for Multi-Agent Learning via Action Inference and Importance Sampling

Authors: Marc Walden, Jason Liu, Shaashwath Sivakumar, Ryan Liu, Hamza Khan

Organizations: Department of Mathematics, University of California Los Angeles, Los Angeles, CA, USA

Abstract

We investigate multi-agent deep reinforcement learning and propose two enhancements to the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm. First, we introduce a novel Action Inference mechanism that enables each agent to predict other agents' intended actions, thereby improving the accuracy and stability of its own policy. Second, we apply an importance sampling strategy, using geometric distribution, in the replay buffer to prioritize more recent and informative experiences, which helps mitigate the non-stationarity inherent in multi-agent environments. We evaluate both modifications on the discrete-action Predator-Prey task provided by the PettingZoo library, a flexible Python interface for general multi-agent reinforcement learning benchmarks. Our results indicate that Action Inference is effective in improving learning stability and inter-agent cooperation and that importance sampling using geometric distribution can lead to significant improvements in exploration efficiency over standard MADDPG. Code available at https://github.com/shaashwathsivakumar/MARL_Proj

Explore similar work

Scalable Neighborhood-Based Multi-Agent Actor-Critic
Apr 20, 2026Tim Goppelsroeder, Rasmus JensenMulti-Agent Reinforcement Learning Critic-Free Reinforcement Learning
Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning
May 8, 2026Yuyang Zhang, Haldun Balim, Na LiMulti-Agent Reinforcement Learning Diffusion Policies
MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning
May 3, 2026Haohan Yu, Jinmiao Cong, Shengzhi Wang +2Multi-Agent Reinforcement Learning Team Formation

Apr 20, 2026 · cs.LGJ/K move · Enter open · S save

Scalable Neighborhood-Based Multi-Agent Actor-Critic

Tim Goppelsroeder, Rasmus Jensen

Department of Computer Science, ETH Zürich.

We propose MADDPG-K, a scalable extension to Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that addresses the computational limitations of centralized critic approaches. Centralized critics, which condition on the observations and actions of all agents, have demonstrated significant performance gains in cooperative and competitive multi-agent settings. However, their critic networks grow linearly in input size with the number of agents, making them increasingly expensive to train at scale. MADDPG-K mitigates this by restricting each agent's critic to the

k

closest agents under a chosen metric which in our case is Euclidean distance. This ensures a constant-size critic input regardless of the total agent count. We analyze the complexity of this approach, showing that the quadratic cost it retains arises from cheap scalar distance computations rather than the expensive neural network matrix multiplications that bottleneck standard MADDPG. We validate our method empirically across cooperative and adversarial environments from the Multi-Particle Environment suite, demonstrating competitive or superior performance compared to MADDPG, faster convergence in cooperative settings, and better runtime scaling as the number of agents grows. Our code is available at https://github.com/TimGop/MADDPG-K .

Multi-Agent Reinforcement Learning Critic-Free Reinforcement Learning Adversaries

Open paper Ask about it PDF

Abstract

Explore similar work

Scalable Neighborhood-Based Multi-Agent Actor-Critic

Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning