HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Hao Zhang; Yaru Niu; Yikai Wang; Ding Zhao; H. Eric Tseng

cs.ROMar 4, 2026

HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Authors: Hao Zhang, Yaru Niu, Yikai Wang, Ding Zhao, H. Eric Tseng

Organizations: ETAIC Lab, Department of Electrical Engineering, University of Texas at Arlington, USA · Safe AI Lab, Department of Mechanical Engineering, Carnegie Mellon University, USA.

Abstract

To improve generalization and resilience in human-robot collaboration (HRC), robots must contend with diverse combinations of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG), where decentralized policy updates deviate from cooperative joint optimization. The resulting learning problem is a general-sum differentiable game, so independent policy-gradient updates can oscillate or diverge without added structure. We propose heterogeneous-agent Lyapunov policy optimization (HALO), a framework that stabilizes decentralized MARL by enforcing Lyapunov-based contraction in policy-parameter space. Unlike Lyapunov-based safe RL, which targets state/trajectory constraints in constrained Markov decision processes, HALO uses Lyapunov certification to stabilize decentralized policy learning. HALO rectifies decentralized gradients via optimal quadratic projections, ensuring monotonic contraction of RG and enabling effective exploration of open-ended interaction spaces. Extensive simulations and real-world humanoid-robot experiments show that this certified stability improves generalization and robustness in collaborative corner cases. Our project website is available at https://HaoZhang-THU.github.io/HALO/.

Figures & tables

Figure 1: The HALO framework architecture combining the transition from standard decentralized learning to Lyapunov policy optimization for real-world HRC. Key components include the computation of the rationality gap $V(\theta)$ and the stability normal vector $h$ to derive the final analytic closed-form projection $d^{*}$ .

Panel of Figure 2 — Figure 2: Simulation benchmark and learning dynamics: (a) massively parallelized training infrastructure in Isaac Lab, where the arrows indicate the emergent synergy collaboration; (b) performance comparison across nine scenarios, where HALO demonstrates significantly faster convergence, reaching its performance plateau at approximately 1.3B steps.

Figure 3: Comparison of HALO and baseline MARL algorithms across the nine scenarios in OSP, SCT and SLH tasks.

Task category	Scenario name	HAPPO	HATRPO	PCGrad	HALO (Ours)	Improvement
OSP	Alignment	83.6 $\pm$ 5.3	87.9 $\pm$ 4.5	87.7 $\pm$ 3.8	92.8 $\pm$ 3.0	+5.6%
	Turnaround	77.6 $\pm$ 6.3	81.5 $\pm$ 5.5	83.6 $\pm$ 4.8	86.7 $\pm$ 3.5	+3.7%
	Corner entry	72.7 $\pm$ 7.0	75.3 $\pm$ 6.0	77.8 $\pm$ 5.3	82.2 $\pm$ 4.0	+5.7%
	OSP average	78.0 $\pm$ 6.3	81.6 $\pm$ 5.3	83.0 $\pm$ 4.5	87.2 $\pm$ 3.5	+5.1%
SCT	Narrow gate	80.1 $\pm$ 4.8	83.4 $\pm$ 4.3	88.6 $\pm$ 3.5	91.1 $\pm$ 2.8	+2.8%
	S-shaped path	76.3 $\pm$ 5.8	82.1 $\pm$ 5.0	80.6 $\pm$ 4.5	84.9 $\pm$ 3.8	+3.4%

Table 1: Comprehensive performance matrix and global optimization analysis of heterogeneous coordination: the upper part evaluates the scenario-specific success rate across nine representative coordination challenges in OSP, SCT and SLH tasks, reported as mean

\pm

std. The lower part provides a synchronized mechanism analysis at the 2B-step steady state, correlating overall task proficiency with fundamental optimization metrics including Overall success rate, convergence, final return, gradient alignment (

\cos\phi

), rationality gap (

V

) and gradient conflict rate. Bold and underlined indicate first and second best, respectively.

Panel of Figure 4 — Figure 4: Optimization dynamics analysis: (a) monotonic dissipation of $V(\bm{\theta})$ under the Lyapunov stability certificate; (b) rapid convergence of gradient alignment. HALO eliminates solenoidal components to stabilize the joint parameter manifold.

Figure 5: Scalability and algorithm metrics analysis: (a) convergence steps required to reach performance plateau; (b) steady-state rationality gap

V

; (c) final gradient alignment

\cos\phi

. (d) gradient conflict rate across algorithms.

Panel of Figure 6 — Figure 6: Detailed mechanism evolution: (a) standard deviation of rationality gap; (b) standard deviation of alignment; (c) temporal decay rate of the gap; (d) instantaneous change rate of alignment.

Panel of Figure 7 — Figure 7: Micro-view analysis of coordination resilience: (a) reactive height modulation during unscripted partner motion; (b) stability maintenance during 20s obstructions through stationary stepping and velocity re-synchronization.

Figure 8: Sim-to-real deployment across embodied tasks: macro-view of deployment on OSP (top), SCT (Middle) and SLH (Bottom). Temporal progression is indicated by color gradients; arrows trace the stable trajectories maintained by HALO despite complex physical coupling and human-induced perturbations.

Variant	$\mathcal{P}$	$\eta$	$\cos\phi$	SR (%) $\uparrow$	Gap $V$ $\downarrow$
HAPPO baseline	$\times$	$\times$	$\times$	74.9 $\pm$ 6.0	4.89
Soft- $V$ penalty	$\times$	$\times$	$\times$	76.5 $\pm$ 5.8	3.21
Static projection	✓	$\times$	$\times$	79.5 $\pm$ 4.8	0.85
HALO w/o align	✓	✓	$\times$	80.8 $\pm$ 4.2	0.24
HALO (full)	✓	✓	✓	82.3 $\pm$ 3.8	0.09

Table 2: Ablation matrix on SLH-extreme (pivoting) task. Results report mean

\pm

std.

\mathcal{P}

: Lyapunov projection;

\eta

: adaptive scheduling;

\cos\phi

: alignment rectification.

	Task 1: OSP		Task 2: SCT
Method	T $\downarrow$	SR $\uparrow$	T $\downarrow$	DR $\downarrow$	Ang $\downarrow$
Robot-Script	74.2	100%	101.6	60%	4.3
PCGrad	65.2	100%	81.5	0%	2.4
HALO	61.7	100%	76.2	0%	2.2
	Task 3: SLH (Stability Under Halting)
Method	Wait	T $\downarrow$	DR $\downarrow$	Ang $\downarrow$	$v_{d}$ $\downarrow$

Table 3: Real-world performance: metrics are mean values over 5 trials. T: time to destination (s); SR: success rate; DR: object drop rate (%); Ang: absolute value of tilt rate (

{}^{\circ}/s

);

v_{d}

: post-halt drift (cm/s); Wait: proactive waiting for partner synchrony.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Layer	Frequency	Update schedule	Core functional scope
Top-level	N/A	Episode initialization	Global path planning (A* or VLM) for object CoM
Mid-level	2 Hz	Tactical MARL policy	Motion command generation for collaboration
Bottom-level	50 Hz	WBC execution	Body balancing and motion command tracking

Appendix

Table 4: Temporal and functional specification of the control hierarchy.

Feature domain	Dim	Description (Egocentric coordinates)
Look-ahead guidance	10	2D XY waypoints sampled via a sliding window along the global path
Self-proprioception	13	XY pos/vel, yaw, CoM height, torso pitch, wrist-to-shoulder XYZ
Partner observation	13	Relative XY pos/vel, yaw, CoM height, torso pitch, wrist-to-shoulder XYZ
Object geometry	18	4 Top-surface corners XYZ, CoM XYZ pos, CoM XYZ vel
Contact feedback	4	Contact signals (left/right end-effector contact for both of the agents)
Temporal features	174	Snapshot accumulation (58D per frame $\times$ 3 snapshots)

Appendix

Table 5: Composition of the 210-dimensional observation space.

Command set	Dim	Formulation	Physical meaning
Locomotion base	3	Absolute	Target velocities $[v_{x},v_{y}]$ and orientation (yaw angle)
Postural setpoints	2	Absolute	Target CoM height ( $H_{\text{CoM}}$ ) and torso pitch ( $\alpha_{\text{torso}}$ )
Wrist modulation	6	Relative delta	3D XYZ offset added to task-specific base poses for both Wrist

Appendix

Table 6: MARL command space specification (11D).

Optimization parameter	Value	Architecture parameter	Value
Learning rate ( $\alpha$ )	$1.0\times 10^{-4}$	Hidden layers (actor/critic)	[256, 256, 128]
Epochs per update	10	Activation function	ReLU
Minibatch count	16	Layer initialization	Orthogonal
Entropy coefficient	0.01	Optimizer	Adam
Discount factor ( $\gamma$ )	0.99	Weight decay ( $L_{2}$ )	$1.0\times 10^{-4}$
GAE parameter ( $\lambda$ )	0.95	Gradient clipping ( $\\|g\\|_{2}$ )	10.0

Appendix

Table 7: Optimization and topology hyperparameters.

Noise type	Distribution	Magnitude	Target simulation phenomenon
Vertical jitter	Gaussian	2.0 cm	Gait-induced oscillations and lifting shifts
Velocity noise	Uniform	0.1 m/s	Non-uniform walking speed and intent changes
Yaw perturbation	Gaussian	3.0 deg	Subtle directional adjustments during transport

Appendix

Table 8: Stochastic perturbations for simulating human-like dynamical loads.

Optimization parameter	Value	Architecture parameter	Value
Learning rate ( $\alpha$ )	$1.0\times 10^{-4}$	Hidden layers (actor/critic)	[256, 256, 128]
Epochs per update	10	Activation function	ReLU
Minibatch count	16	Layer initialization	Orthogonal
Entropy coefficient	0.01	Optimizer	Adam
Discount factor ( $\gamma$ )	0.99	Weight decay ( $L_{2}$ )	$1.0\times 10^{-4}$
Learning rate schedule	Cosine annealing	Total RL action steps	$2.0\times 10^{9}$

Appendix

Table 9: Robot-script training hyperparameters.

Explore similar work

Mar 3, 2026

Heterogeneous Agent Collaborative Reinforcement Learning

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.

Zhixia Zhang, Zixuan Huang, Gongxun Li +10

Beihang University · Bytedance China · Apple +2

Multi-Agent Collaboration Heterogeneous MARL Cooperative MARL Reinforcement Learning with Verifiable Rewards

Ask PDF

May 12, 2026

Metric-Gradient Projection for Stable Multi-Agent Policy Learning

General-sum multi-agent learning is often governed by a stacked update field in which each agent's policy update changes the optimization landscape faced by the others. This coupling can entangle an integrable component of collective improvement with cyclic interaction dynamics, leading to slow or unstable multi-agent learning. Existing approaches, such as regularization, credit assignment, and consensus methods, stabilize MARL through local or algorithmic modifications; HPML complements them by projecting the joint update field onto a metric-gradient component. We introduce HPML (Hodge-Projected Multi-agent Learning), which views the joint update field of a multi-agent system as an element of an

L^2

space of vector fields and computes a Hodge-type projection onto the closest metric-gradient potential flow. HPML follows the projected component as the update direction, yielding the closest metric-gradient field under the chosen metric and sampling measure. The projection is defined variationally, characterized by a Poisson-type equation, and implemented through graph-based and amortized neural realizations that recover projected directions from samples. We show that the projected dynamics admit a Lyapunov potential and yield equilibrium-gap bounds with an explicit additive non-potentiality term. Controlled experiments validate the geometric mechanism, and CTDE benchmarks show improved stability and normalized return when HPML is used as a plug-in projection layer in MARL pipelines.

Zuyuan Zhang, Sizhe Tang, Mahdi Imani +1

The George Washington University · Northeastern University

Hodge Decomposition Multi-Agent Systems Policy Learning Multi-Agent Reinforcement Learning Gradient Descent Dynamics

Ask PDF

Feb 25, 2026

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning

Cooperative Multi-Agent Reinforcement Learning (MARL) solves complex tasks that require coordination from multiple agents, but is often limited to either local (independent learning) or global (centralized learning) perspectives. In this paper, we introduce a novel sequential training scheme and MARL architecture, which learns from multiple perspectives on different hierarchy levels. We propose the Hierarchical Lead Critic (HLC) - inspired by natural emerging distributions in team structures, where following high-level objectives combines with low-level execution. HLC demonstrates that introducing multiple hierarchies, leveraging local and global perspectives, can lead to improved performance with high sample efficiency and robust policies. Experimental results conducted on cooperative, non-communicative, and partially observable MARL benchmarks demonstrate that HLC outperforms single hierarchy baselines and scales robustly with increasing amounts of agents and difficulty.

David Eckel, Henri Meeß

Fraunhofer Institute IVI, Ingolstadt, Germany

Multi-Agent Reinforcement Learning Cooperative MARL Hierarchical RL Sequential MARL

Ask PDF