Action-Graph Policies: Learning Action Co-dependencies in Multi-Agent Reinforcement Learning
Authors: Nikunj Gupta, James Zachary Hare, Jesse Milzman, Rajgopal Kannan, Viktor Prasanna
Organizations: University of Southern California, Los Angeles, CA, USA · DEVCOM Army Research Laboratory, Adelphi, MD, USA · DEVCOM ARL Army Research Office, Los Angeles, CA, USA.
Coordinating actions is the most fundamental form of cooperation in multi-agent reinforcement learning (MARL). Successful decentralized decision-making often depends not only on good individual actions, but on selecting compatible actions across agents to synchronize behavior, avoid conflicts, and satisfy global constraints. In this paper, we propose Action Graph Policies (AGP), that model dependencies among agents' available action choices. It constructs, what we call, coordination contexts, that enable agents to condition their decisions on global action dependencies. Theoretically, we show that AGPs induce a strictly more expressive joint policy compared to fully independent policies and can realize coordinated joint actions that are provably more optimal than greedy execution even from centralized value-decomposition methods. Empirically, we show that AGP achieves 80-95% success on canonical coordination tasks with partial observability and anti-coordination penalties, where other MARL methods reach only 10-25%. We further demonstrate that AGP consistently outperforms these baselines in diverse multi-agent environments.
Figures & tables
Figure 1: Illustration of Action-Graph Policies (AGP). The global action graph GA contains one node uai for each available action a∈Ai of every agent i , with edges representing learned coordination dependencies across action choices. Coordination contexts κi are constructed by aggregating information from this action graph prior to decentralized action selection.
Figure 2: Coordination matrix games. (a) Top- K Selection ( N=6 , K=2 ): baselines cluster near the independent-execution bound, while AGP achieves near-optimal coordination. (b) Top- K with anti-coordination penalty: baselines suffer from relative overgeneralization, whereas AGP remains robust.
Figure 3: Ablations. Removing cross-agent action dependencies or the action graph drastically collapses the performance, while a policy-gradient AGP variant recovers similar performance.
Figure 4: Performance on multi-agent particle environments. AGP achieves the highest final test return across all six tasks, outperforming all MARL baselines. Shaded regions indicate ± one standard deviation over five seeds.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Top- K Selection: attention over the action graph. We visualize all attention heads and their mean at the final checkpoint. Attention exhibits structured cross-agent coupling (off-diagonal mass) rather than purely local/self-attention, supporting the interpretation that AGP constructs coordination contexts through action-level dependencies.
Figure 6: Top- K with anti-coordination penalty: attention over the action graph. Compared to Top- K , attention exhibits stronger structured coupling between act1 and act0 nodes across agents, consistent with resolving near-ties and avoiding joint act1 selections that incur the penalty.