cs.AINov 13, 2026

Imitation Bootstrapped Multi-Agent Reinforcement Learning

Authors: Tony H. Zhang, Connor Mattson, Daniel S. Brown

Organizations: University of Utah SLC, UT, USA

Abstract

Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations. We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives one action proposal from its imitation policy and one from its RL policy, and an ensemble of centralized critics selects among the per-agent combinations of the two. A gated imitation penalty, a soft Q-filtered behavior cloning loss, regularizes each RL policy toward its imitation policy only where the critics value the imitation action more, so that after training each agent executes its RL policy alone, without the centralized critics. On four cooperative multi-agent RL benchmarks, IBMARL surpasses its imitation policies 2.62.6-4.2×4.2\times faster than demonstration-based baselines on two tasks, is the only method to surpass it on the other two, and exceeds the demonstrator itself on two of the four. With real human demonstrations IBMARL still surpasses imitation learning alone, and does so more than twice as fast as the best baseline.

Figures & tables

Explore similar work

CardsList
  1. Coordinated Diffusion: Generating Multi-Agent Behavior Without Multi-Agent Demonstrations

    May 12, 2026Lasse Peters, Laura Ferranti, Andrea Bajcsy +1Multi-Agent CoordinationDiffusion Policy

  2. Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

    Feb 25, 2026David Eckel, Henri MeeßHierarchical MARLMulti-Agent Reinforcement Learning

  3. MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies

    Sep 17, 2025Dayi Dong, Maulik Bhatt, Seoyeon Choi +1Multi-Agent CoordinationDiffusion Policy