Offline MARL

MARL: Multi-Agent Reinforcement Learning

Overview

Offline MARL studies how multiple agents can learn to cooperate or compete from logged trajectories, where their actions affect one another and new mistakes cannot be tested freely. Approaches include behavior cloning, conservative value learning, sequence models, diffusion and flow policies, and world models; theoretical work also studies equilibrium guarantees under dataset coverage assumptions. Methods are evaluated in settings such as SMAC and MPE coordination tasks, continuous-control benchmarks, traffic management, and wireless resource allocation, often with decentralized execution. A central challenge is that policies can take joint actions unlike those in the data, making value estimates unreliable and coordination brittle; current work explores support-aware learning, test-time planning, and stronger generalization while balancing performance, safety, and reliable guarantees.

Latest papers

32 papers2 papers in the last four weeks, down 60% on the four weeks before. 0.0% of all new papers.

Mar 9, 2026

A Recipe for Stable Offline Multi-agent Reinforcement Learning

Despite remarkable achievements in single-agent offline reinforcement learning (RL), multi-agent RL (MARL) has struggled to adopt this paradigm, largely persisting with on-policy training and self-play from scratch. One reason for this gap comes from the instability of non-linear value decomposition, leading prior works to avoid complex mixing networks in favor of linear value decomposition (e.g., VDN) with value regularization used in single-agent setups. In this work, we analyze the source of instability in non-linear value decomposition within the offline MARL setting. Our observations confirm that they induce value-scale amplification and unstable optimization. To alleviate this, we propose a simple technique, scale-invariant value normalization (SVN), that stabilizes actor-critic training without altering the Bellman fixed point. Empirically, we examine the interaction among key components of offline MARL (e.g., value decomposition, value learning, and policy extraction) and derive a practical recipe that unlocks its full potential.
Feb 24, 2026

Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning

Multi-agent imitation learning (MA-IL) aims to learn optimal policies from expert demonstrations of interactions in multi-agent interactive domains. Despite existing guarantees on the performance of the resulting learned policies, characterizations of how far the learned polices are from a Nash equilibrium are missing for offline MA-IL. In this paper, we demonstrate impossibility and hardness results of learning low-exploitable policies in general nn-player Markov Games. We do so by providing examples where even exact measure matching fails, and demonstrating a new hardness result on characterizing the Nash gap given a fixed measure matching error. We then show how these challenges can be overcome using strategic dominance assumptions on the expert equilibrium. Specifically, for the case of dominant strategy expert equilibria, assuming Behavioral Cloning error εBCε_{\text{BC}}, this provides a Nash imitation gap of O(nεBC/(1−γ)2)\mathcal{O}\left(nε_{\text{BC}}/(1-γ)^2\right) for a discount factor γγ. We generalize this result with a new notion of best-response continuity, and argue that this is implicitly encouraged by standard regularization techniques.