H^2MARL: Hierarchical Heterogeneous Multi-agent Reinforcement Learning for Humanoid Robot Control
Authors: Xuemin Hu, Kaixin Li, Qingqing Xia, Yuxin Wang, Wei Xie, Bo Tang, Long Chen
Organizations: School of Artificial Intelligence, Hubei University, Wuhan, China. · Worcester Polytechnic Institute, Worcester, MA, USA. · Institute of Automation, Chinese Academy of Sciences, Beijing, China.
Coordinated whole-body control of humanoid robots remains challenging due to high-dimensional coupled dynamics and substantial physical heterogeneity across joint degrees of freedom (JDoFs). Existing multi-agent reinforcement learning methods either employ coarse agent partitions or directly apply generic heterogeneous-agent algorithms, while hierarchical controllers commonly optimize high-level planning and low-level execution independently, leading to uncoordinated motion. In this paper, we propose H^2MARL, a hierarchical heterogeneous multi-agent reinforcement learning framework with two key mechanisms. We first formulate humanoid whole-body control using a JDoF-level physical-heterogeneity-aware policy factorization, where each torque-actuated JDoF is conditioned on an explicit descriptor of its actuation, kinematic, and dynamic properties. Moreover, we develop a closed-loop policy-executor co-optimization mechanism that jointly adapts the high-level pose planner and the low-level joint executor using shared interaction trajectories, thereby reducing the distribution mismatch between planning and execution. Extensive experiments in MuJoCo and Isaac Gym demonstrate that H^2MARL consistently improves locomotion stability, gait coordination, and motion quality over homogeneous, coarse-grained, and independently optimized alternatives.
Figures & tables
Fig. 1: Limb- and JDoF-level multi-agent formulations for the humanoid robot. (a) Traditional limb-level formulation, where each limb is formulated as an independent homogeneous agent. (b) JDoF-level formulation, where each torque-actuated JDoF is formulated as an independent heterogeneous agent with physical attributes and functional roles.
Fig. 2: H 2 MARL overview. The high-level planner generates JDoF-level pose increments and converts them into absolute target poses. The low-level joint executor receives the absolute target poses combined with historical proprioception and outputs executable joint positions which are converted to torques by a PD controller. Both the high-level policy and the low-level executor are optimized by the coupling trajectories in a closed-loop.
Method
FS ↓
TS ↓
GS ↑
MS ↓
HAPPO
0.9827
0.5091
0.000150
7.3819
MAPPO
0.5790
0.3873
0.000905
0.2329
HATD3
0.8220
0.4739
0.000050
2.3818
H 2 MARL(ours)
0.5367
0.2516
0.001039
0.3514
TABLE I: Comparative Results with SOTA methods.
Method
VE ↓
CoT ↓
MJ ↓
CAM ↓
TS ↓
H 2 MARL
0.4807
1.5213
0.7448
1.0324
0.0140
w/o HA
1.0360
5.8804
2.9680
3.8187
0.0417
w/o CPECO
0.9908
11.9586
6.4986
6.0132
0.0106
w/o Rew
1.0580
14.2529
2.6578
12.9126
0.1691
w/o All (HAPPO)
1.3667
14.7716
19.5962
9.0912
0.0333
TABLE II: Ablation study of different mechanisms on Isaac Gym.
Method
VE ↓
CoT ↓
MJ ↓
CAM ↓
TS ↓
Single-agent
1.4003
10.0510
2.8654
0.6105
0.0307
BP-level
0.9695
3.5571
2.1089
0.5330
0.0539
Limb-level
1.6811
7.3547
6.3729
1.4076
0.0495
JDoF-level
0.4807
1.5213
0.7448
1.0324
0.0140
TABLE III: Comparison of different agent formulation strategies.
Method
VE ↓
CoT ↓
MJ ↓
CAM ↓
TS ↓
Complete
0.4807
1.5213
0.7448
1.0324
0.0140
w/o Slip
0.9632
10.5866
7.9879
6.7192
0.0263
w/o Alt
0.7981
9.1799
7.9996
5.3886
0.0176
w/o Both
1.0580
14.2529
2.6578
12.9126
0.1691
TABLE IV: Ablation study of different reward terms.
Method
P50
P99
Mean
Std
HAPPO
7.589
8.896
7.677
0.315
MAPPO
4.958
6.209
5.005
0.426
HATD3
7.533
8.353
7.592
0.381
H 2 MARL (High-level)
8.241
9.609
8.332
0.330
H 2 MARL (Low-level)
0.910
1.100
0.927
0.225
TABLE V: Single-step inference time (ms) for comparative methods.
Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.
To improve generalization and resilience in human-robot collaboration (HRC), robots must contend with diverse combinations of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG), where decentralized policy updates deviate from cooperative joint optimization. The resulting learning problem is a general-sum differentiable game, so independent policy-gradient updates can oscillate or diverge without added structure. We propose heterogeneous-agent Lyapunov policy optimization (HALO), a framework that stabilizes decentralized MARL by enforcing Lyapunov-based contraction in policy-parameter space. Unlike Lyapunov-based safe RL, which targets state/trajectory constraints in constrained Markov decision processes, HALO uses Lyapunov certification to stabilize decentralized policy learning. HALO rectifies decentralized gradients via optimal quadratic projections, ensuring monotonic contraction of RG and enabling effective exploration of open-ended interaction spaces. Extensive simulations and real-world humanoid-robot experiments show that this certified stability improves generalization and robustness in collaborative corner cases. Our project website is available at https://HaoZhang-THU.github.io/HALO/.
Hao Zhang, Yaru Niu, Yikai Wang +2
ETAIC Lab, Department of Electrical Engineering, University of Texas at Arlington, USA · Safe AI Lab, Department of Mechanical Engineering, Carnegie Mellon University, USA.
Multi-agent systems are widely used in safety-critical applications that require coordinated behavior under strict safety constraints. Existing approaches face a fundamental trade-off: learning-based methods achieve strong empirical performance but lack theoretical safety guarantees, while control-theoretic methods enforce safety but often lead to overly conservative and inefficient behaviors. We propose a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints under mild assumptions at low level via a constraint manifold, while enabling effective coordination through high-level policy learning. Our approach provides theoretical safety guarantees in the multi-agent setting and yields stationary learning dynamics, thereby enabling stable and efficient training. Empirically, our method achieves competitive performance while maintaining nearly perfect safety rates, and generalizes effectively to varying numbers of agents and obstacles.
Zihao Guo, Jianing Zhao, Ling Li +3
1King’s College London · University of California, Berkeley · 3The Alan Turing Institute