Optimizing Team Behavior via Extremum-seeking Control in Multi-agent Reinforcement Learning
Authors: Svar Rajankumar Patel, Kshitij Jerath
Organizations: Department of Computer Science, University of Massachusetts Lowell, Lowell, Massachusetts, USA · Department of Mechanical and Industrial Engineering, University of Massachusetts Lowell, Lowell, Massachusetts, USA
In cooperative Multi-Agent Reinforcement Learning (MARL), agents must learn to coordinate their actions to achieve shared objectives. A key indicator of effective coordination is behavioral diversity, i.e., the agents' potential to enact distinct yet complementary control policies given the same set of observations. Although behavioral diversity can enable role specialization and richer joint behaviors, its appropriate level is highly context- and task-dependent and often costly to tune. We propose Adaptive Diversity Control (ADiCo), a two-timescale framework that adapts behavioral diversity online during training. On a fast timescale, Diversity Control (DiCo) regulates system diversity toward a target value. On a slower timescale, an Extremum Seeking Controller (ESC) adjusts this target using only mean episodic returns, without requiring an explicit model or analytical gradient of the diversity-performance landscape. We evaluate ADiCo on six configurations across two cooperative MARL tasks, where it maintains task-appropriate diversity levels: low diversity when homogeneous policies are beneficial and higher diversity when heterogeneous policies are required. Through this adaptation, ADiCo matches or exceeds the mean performance of DiCo (best grid-searched diversity target) and Unconstrained DiCo (diversity allowed to emerge without target regulation). These results demonstrate that incorporating classical feedback control into MARL training enables effective coordination by providing a systematic alternative to task-specific manual selection of the diversity target.
Figures & tables
Fig. 1: ADiCo system architecture. The DiCo policy decomposition and rescaling pipeline (top) is retained from [ 4 ] , where policies are rescaled to match the desired diversity SNDdes , with the current diversity estimated via exponential moving average. Note the illustrative rescaling of Agent 1’s policy π1(o) in the pipeline (top). The ESC signal chain (bottom) processes the mean reward to estimate the gradient and update the diversity setpoint. The complete procedure is described in Algorithm 1 .
Fig. 2: Navigation (left) and Balance (right). In Navigation, agents (colored circles) apply continuous 2D forces to reach goal locations (smaller circles). In Balance, agents (blue) coordinate forces on a shared platform (grey) to move the packet (red) toward the goal (green). The background shows behavioral diversity measured using SND [ 9 ] , with yellow indicating high and purple indicating low diversity.
Fig. 3: Training curves across Navigation (top) and Balance (bottom) tasks comparing ADiCo (solid), Grid-Searched DiCo (dash-dot), and Unconstrained DiCo (dashed). Colors denote task configurations; the left panel shows DiCo’s manually tuned SND target for each. (a) System Neural Diversity. (b) Mean episode reward. (c) Episode length.
Fig. 4: Convergence analysis on Navigation 3A-3G comparing ADiCo (solid lines) against DiCo (dashed lines) at SND levels 0, 0.5, and 1.0. (a) SND trajectories. (b) Mean episode reward. (c) Episode length.
Fig. 5: Comparison across three Navigation configurations between ADiCo with continued ESC adaptation (Closed-Loop ADiCo) and a variant that disables perturbations and freezes the diversity target when its stopping criterion is met (Open-Loop ADiCo). (a) SND trajectories. (b) Episode length.
Effective multi-agent cooperation requires agents to adopt diverse behaviors as task conditions evolve-and to do so at the right moment. Yet, current Multi-Agent Reinforcement Learning (MARL) frameworks that facilitate this diversity are still limited by the fact that they bind fixed behaviors to fixed agent identities. Consequently, they are ill-equipped for tasks where agents need to take on different roles at very specific moments in time. We argue that, to define these behavioral transitions, the missing ingredient is events. Events are changes in the state of the system that induce qualitative changes in the task. Based on this view, we introduce a framework that decouples agent identity from behavior, capturing a continuous manifold from which agents instantiate their behaviors in response to events. This framework is based on two elements. First, to build an expressive behavior manifold, we introduce Neural Manifold Diversity (NMD), a formal distance metric that remains well-defined when behaviors are transient and agent-agnostic. Second, we use an event-based hypernetwork that generates Low-Rank Adaptation (LoRA) modules over a shared team policy, enabling on-the-fly agent-policy reconfiguration in response to events. We prove that this construction ensures that diversity does not interfere with reward maximization by design. Empirical results demonstrate that our framework outperforms established baselines across benchmarks while exhibiting zero-shot generalization, and being the only method that solves tasks requiring sequential behavior reassignment.
Hannes Büchi, Manon Flageat, Eduardo Sebastián +1
Department of Computer Science and Technology, University of Cambridge, UK
Cooperative multi-agent reinforcement learning (MARL) benchmarks commonly emphasize aggregate outcomes such as return, success rate, or completion time. While essential, these metrics can obscure how agents coordinate and fail to distinguish policies that achieve similar outcomes through different interaction processes. We propose a process-aware evaluation framework that complements aggregate performance with diagnostics of interference, coordinated task completion, participation, and coordination efficiency. We evaluate the framework in STAT, a controlled spatial task-allocation testbed, and examine its broader applicability in two established cooperative MARL benchmarks. Across diverse value-based and actor-critic methods, we find policies with nearly identical return can exhibit substantial differences in coordination behavior. Controlled scaling in STAT reveals distinct process signatures for spatial, task, and agent scaling, while a reward-shaping ablation shows that explicitly penalizing assignment conflicts can alter coordination behavior with little change in task return. Results in the established benchmarks further show that the framework remains informative under different coordination mechanisms. These findings support process-aware evaluation as a complement to outcome-based benchmarking. The framework provides a common structure for selecting and interpreting environment-specific diagnostics that characterize not only what cooperative policies achieve, but how they coordinate to achieve it.
In this work, we propose a framework that combines multi-agent reinforcement learning (MARL) with model-based control to achieve safe, dynamically feasible actions in cooperative multi-agent tasks. Multi-agent reinforcement learning provides the advantage of learning cooperative policies for multi-agent teams from discrete non-differentiable rewards in a long planning horizon. Model-predictive control is robust and offers safe, dynamically feasible actions in a fast replanning framework for short horizons. We propose an algorithm that extends actor-critic model predictive control for MARL which we refer to as multi-agent actor-critic model predictive control (MA-AC-MPC). We demonstrate the capabilities of this algorithm by applying it to a multi-agent pursuit-evasion scenario. Specifically, we compare the evader team's strategy using the MA-AC-MPC model and a multi-layer perceptron model (MA-AC-MLP). The pursuer team uses augmented proportional navigation as it is accepted as an advanced adversarial control law. We also provide an example with a heterogeneous environment where a drone and omni-wheeled rover cooperate to achieve repeatable and successful landing with 100% success rate in hardware for MA-AC-MPC compared to 60% for MA-AC-MLP. We demonstrate the robustness of the proposed MA-AC-MPC algorithm in hardware for both environments.
Christian Llanes, Spencer W. Jensen, Samuel Coogan
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA. · Sandia National Laboratories, Albuquerque, NM 87123, USA.