Optimizing Team Behavior via Extremum-seeking Control in Multi-agent Reinforcement Learning
Authors: Svar Rajankumar Patel, Kshitij Jerath
Organizations: Department of Computer Science, University of Massachusetts Lowell, Lowell, Massachusetts, USA · Department of Mechanical and Industrial Engineering, University of Massachusetts Lowell, Lowell, Massachusetts, USA
In cooperative Multi-Agent Reinforcement Learning (MARL), agents must learn to coordinate their actions to achieve shared objectives. A key indicator of effective coordination is behavioral diversity, i.e., the agents' potential to enact distinct yet complementary control policies given the same set of observations. Although behavioral diversity can enable role specialization and richer joint behaviors, its appropriate level is highly context- and task-dependent and often costly to tune. We propose Adaptive Diversity Control (ADiCo), a two-timescale framework that adapts behavioral diversity online during training. On a fast timescale, Diversity Control (DiCo) regulates system diversity toward a target value. On a slower timescale, an Extremum Seeking Controller (ESC) adjusts this target using only mean episodic returns, without requiring an explicit model or analytical gradient of the diversity-performance landscape. We evaluate ADiCo on six configurations across two cooperative MARL tasks, where it maintains task-appropriate diversity levels: low diversity when homogeneous policies are beneficial and higher diversity when heterogeneous policies are required. Through this adaptation, ADiCo matches or exceeds the mean performance of DiCo (best grid-searched diversity target) and Unconstrained DiCo (diversity allowed to emerge without target regulation). These results demonstrate that incorporating classical feedback control into MARL training enables effective coordination by providing a systematic alternative to task-specific manual selection of the diversity target.
Figures & tables
Fig. 1: ADiCo system architecture. The DiCo policy decomposition and rescaling pipeline (top) is retained from [ 4 ] , where policies are rescaled to match the desired diversity SNDdes , with the current diversity estimated via exponential moving average. Note the illustrative rescaling of Agent 1’s policy π1(o) in the pipeline (top). The ESC signal chain (bottom) processes the mean reward to estimate the gradient and update the diversity setpoint. The complete procedure is described in Algorithm 1 .
Fig. 2: Navigation (left) and Balance (right). In Navigation, agents (colored circles) apply continuous 2D forces to reach goal locations (smaller circles). In Balance, agents (blue) coordinate forces on a shared platform (grey) to move the packet (red) toward the goal (green). The background shows behavioral diversity measured using SND [ 9 ] , with yellow indicating high and purple indicating low diversity.
Fig. 3: Training curves across Navigation (top) and Balance (bottom) tasks comparing ADiCo (solid), Grid-Searched DiCo (dash-dot), and Unconstrained DiCo (dashed). Colors denote task configurations; the left panel shows DiCo’s manually tuned SND target for each. (a) System Neural Diversity. (b) Mean episode reward. (c) Episode length.
Fig. 4: Convergence analysis on Navigation 3A-3G comparing ADiCo (solid lines) against DiCo (dashed lines) at SND levels 0, 0.5, and 1.0. (a) SND trajectories. (b) Mean episode reward. (c) Episode length.
Fig. 5: Comparison across three Navigation configurations between ADiCo with continued ESC adaptation (Closed-Loop ADiCo) and a variant that disables perturbations and freezes the diversity target when its stopping criterion is met (Open-Loop ADiCo). (a) SND trajectories. (b) Episode length.
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA. · Sandia National Laboratories, Albuquerque, NM 87123, USA.