cs.LGMar 7, 2026

NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning

Authors: Addison Kalanther, Sanika Bharvirkar, Shankar Sastry, Chinmay Maheshwari

Organizations: Electrical Engineering and Computer Sciences, UC Berkeley · Electrical and Computer Engineering, Data Science & Artificial Intelligence Institute, Johns Hopkins University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-agent reinforcement learning (MARL) is increasingly used to design learning-enabled agents that interact in shared environments. However, training MARL algorithms in general-sum games remains challenging: learning dynamics can become unstable, and convergence guarantees typically hold only in restricted settings such as two-player zero-sum or fully cooperative games. Moreover, when agents have heterogeneous and potentially conflicting preferences, it is unclear what system-level objective should guide learning. In this paper, we propose a new MARL pipeline called Near-Potential Policy Optimization (NePPO) for computing approximate Nash equilibria in mixed cooperative--competitive environments. The core idea is to learn a player-independent potential function such that the Nash equilibrium of a cooperative game with this potential as the common utility approximates a Nash equilibrium of the original game. To this end, we introduce a novel MARL objective such that minimizing this objective yields the best possible potential function candidate and consequently an approximate Nash equilibrium of the original game. We develop an algorithmic pipeline that minimizes this objective using zeroth-order gradient descent and returns an approximate Nash equilibrium policy. We empirically show the superior performance of this approach compared to popular baselines such as IPPO and MAPPO.

Figures & tables

Figure 1: Toy-example results for the game in ( 4.1 ). Panel (a) shows the evolution of ww in Algorithm 1 . Panel (b) shows the evolution of Fi(Φw)F_{i}(\Phi_{w}) . Panel (c) shows the change in value function and potential function due to unilateral best responses. Panel (d) compares MAPPO with Algorithm 1 .
α\alphaNePPOIPPOMAPPO
0.00.0000.0000.500
0.20.0000.0000.800
0.40.036DNC1.100
0.60.052DNC1.400
0.80.034DNC1.700
1.00.0000.011N/A
Table 1: Max-Regret for Varying α\alpha by Algorithm
Figure 2: Maximum player regret over training at α∈{0.4,0.6,0.8}\alpha\in\{0.4,0.6,0.8\} . NePPO runs for 500 iterations, whereas IPPO runs for 5000 iterations.
 NePPOIPPOMAPPO
Simple World Comm11.1423.9051.78
Table 2: Max Regret for Environment by Algorithm
Figure 3: Evolution of maximum regret across players over training steps.
More in this topic

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 362 papers

Explore similar work

CardsList
Mar 4, 2026

Learning Approximate Nash Equilibria in Cooperative Multi-Agent Reinforcement Learning via Mean-Field Subsampling

Many large-scale platforms and networked control systems have a centralized decision maker interacting with a massive population of agents under strict observability constraints. Motivated by such applications, we study a cooperative Markov game with a global agent and nn homogeneous local agents in a communication-constrained regime, where the global agent only observes a subset of kk local agent states per time step. We propose an alternating learning framework (ALTERNATING-MARL)(\texttt{ALTERNATING-MARL}), where the global agent performs subsampled mean-field QQ-learning against a fixed local policy, and local agents update by optimizing in an induced MDP. We prove that these approximate best-response dynamics converge to an O~(1/k)\widetilde{O}(1/\sqrt{k})-approximate Nash Equilibrium, while separating the sample complexities between the joint state and action spaces. Finally, we validate our results in numerical simulations for multi-robot control.

Emile Anand, Ishani Karmarkar

Georgia Institute of Technology, Atlanta, GA · Stanford University, Palo Alto, CA

Reinforcement LearningCooperative MARLMulti-Agent CoordinationPartially Observable RL
AskPDF
May 7, 2026

Independent Learning of Nash Equilibria in Partially Observable Markov Potential Games with Decoupled Dynamics

We study Nash equilibrium learning in partially observable Markov games (POMGs), a multi-agent reinforcement learning framework in which agents cannot fully observe the underlying state. Prior work in this setting relies on centralization or information sharing, and suffers from sample and computational complexity that scales exponentially in the number of players. We focus on a subclass of POMGs with independent state transitions, where agents remain coupled through their rewards, and assume that the underlying fully observed Markov game is a Markov potential game. For this class, we present an independent learning algorithm in which players, observing only their own actions and observations and without communication, jointly converge to an approximate Nash equilibrium. Due to partial observability, optimal policies may in general depend on the full action-observation history. Under a filter stability assumption, we show that policies based on finite history windows provide sufficient approximation guarantees. This enables us to approximate the POMG by a surrogate Markov game that is near-potential, leading to quasi-polynomial sample and computational complexity for independent Nash equilibrium learning in the underlying POMG.

Philip Jordan, Maryam Kamgarpour

SYCAMORE, EPFL

Nash EquilibriumReinforcement LearningImperfect-Information GamesPartially Observable RLGame TheoryDecentralized MARL+1
AskPDF
Aug 9, 2026

Multi-Agent Reinforcement Learning via Agent-Specific Preference

Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized preferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical.

Ni Mu, Yao Luan, Yiqin Yang +1

CFINS, Department of Automation, BNRist, Beijing Key Laboratory of Embodied Intelligence Systems, Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Decentralized MARLCooperative MARLReward Design for RLMulti-Agent Reinforcement LearningPreference-Based RL
AskPDF