Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

David Eckel; Henri Meeß

cs.LGFeb 25, 2026

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

Authors: David Eckel, Henri Meeß

Organizations: Fraunhofer Institute IVI, Ingolstadt, Germany

Abstract

Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.

Figures & tables

Figure 1: Hierarchical Lead Critic (HLC) structure. Agents are evaluated by all related critics (local critics and Lead Critics) sequentially.

Figure 2: HLC sequential updating scheme for a selected agent with a local critic and a Lead Critic.

Figure 3: HLC actor. The observation is processed through multiple paths that are combined through cross-attention and concatenation.

Panel of Figure 4 — Figure 4: SimpleSpread ( Terry et al., 2021 ) , Momaland Escort , and Surveillance environments used in our evaluations. In SimpleSpread, black circles are landmarks and purple particles are agents. In Escort8, agent drones (red) escort a target (yellow), while keeping an implicit formation. In Surveillance4, agents observe a target with fixed relative altitude and distribute themselves evenly around the target.

Task	Termination reward
Escort3	-200
Escort	-200
Escort8	-500
Surveillance4	-200

Table 1: Termination rewards in Escort & Surveillance tasks

Panel of Figure 5 — Figure 5: Performance of HLC compared to HASAC and ISAC on the Escort tasks with episode lengths, Surveillance task and SimpleSpread.

Panel of Figure 6 — Figure 6: Ablation Study on HLC components and Escort 8 variants.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Transformer-Encoder based Lead Critic grouping two agents

Panel of Figure 8 — Figure 8: HLC compared to HLC-Average

Panel of Figure 9 — Figure 9: Additional results for HASAC variants on Escort3, Escort5 and increasing gradient steps for HASAC and ISAC on Escort8.

Hyperparameter	Value
Parallel environments $P$	10
Replay buffer size	1e6
Discount factor	0.99
Polyak coefficient	0.005
Adam $\epsilon$	1e-5
Batch size	$256\cdot P$

Appendix

Table 2: Common hyperparameters in all environments for all algorithms

Hyperparameter	EscortX	SimpleSpread
Train interval	20	50
Total env steps	5e5	2e6
Warmup steps	5e3	1e4
Observation history	2	1
n-step	5	2
Actor lr	3e-4	5e-4

Appendix

Table 3: Environment specific hyperparameters for all algorithms. With EscortX for all Escort tasks.

Hyperparameter	Value
Actor network type	HLC Actor
Actor optimizer	RAdam ( Liu et al., 2020 )
Actor optimizer weight decay	1e-5
Subnetworks	2
Subnetwork hidden sizes	[256, 256]
Subnetwork activation	ReLU

Appendix

Table 4: HLC specific hyperparameters in all environments

Hyperparameter	Value
Actor network type	MLP
Actor optimizer	Adam ( Kingma and Ba, 2017 )
Actor hidden sizes	[256, 256]
Lead critic network type	Transformer-Encoder
Lead critic optimizer	RAdam ( Liu et al., 2020 )
Lead critic optimizer weight decay	1e-5

Appendix

Table 5: Common hyperparameters for HASAC

Hyperparameter	Value
Actor network type	MLP
Actor optimizer	Adam ( Kingma and Ba, 2017 )
Actor hidden sizes	[256, 256]
Critic network type	MLP
Critic hidden sizes	[256, 256]
Critic activations	ReLU

Appendix

Table 6: Common hyperparameters for ISAC

Figure 10: Single frame of video provided in the supplementary material

Explore similar work

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning
Jul 31, 2025Tommaso Marzi, Cesare Alippi, Andrea CiniMulti-Agent Coordination Hierarchical RL
IDSIA USI-SUPSI, Università della Svizzera italiana, Lugano, Switzerland · Politecnico di Milano, Milan, Italy · EPFL, Lausanne, Switzerland
HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning
Jun 28, 2026Runze Zhao, Dongruo Zhou, Sumit Kumar Jha +2Multi-Agent Communication Cooperative MARL
Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996 +1
CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning
Jul 21, 2026Dongming Wang, Jie Xu, Yanyu Zhang +1Multi-Agent Coordination Asynchronous RL
Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA

Jul 31, 2025J/K move · Enter open · S save

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

Tommaso Marzi, Cesare Alippi, Andrea Cini

IDSIA USI-SUPSI, Università della Svizzera italiana, Lugano, Switzerland · Politecnico di Milano, Milan, Italy · EPFL, Lausanne, Switzerland

Multi-Agent Coordination Hierarchical RL Decentralized MARL Multi-Agent Communication Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Decentralized Multi-Agent Reinforcement Learning (MARL) methods allow for learning scalable multi-agent policies, but suffer from partial observability and induced non-stationarity. These challenges can be addressed by introducing mechanisms that facilitate coordination and high-level planning. Specifically, coordination and temporal abstraction can be achieved through communication (e.g., message passing) and Hierarchical Reinforcement Learning (HRL) approaches to decision-making. However, optimization issues limit the applicability of hierarchical policies to multi-agent systems. As such, the combination of these approaches has not been fully explored. To fill this void, we propose a novel and effective methodology for learning multi-agent hierarchies of message-passing policies. We adopt the feudal HRL framework and rely on a hierarchical graph structure for planning and coordination among agents. Agents at lower levels in the hierarchy receive goals from the upper levels and exchange messages with neighboring agents at the same level. To learn hierarchical multi-agent policies, we design a novel reward-assignment method based on training the lower-level policies to maximize the advantage function associated with the upper levels. Results on relevant benchmarks show that our method performs favorably compared to the state of the art.

Abstract

Explore similar work

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

CHMAS: A Coupled Hierarchical Framework for Multi-Agent Reinforcement Learning