HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Hao Zhang; Yaru Niu; Yikai Wang; Ding Zhao; H. Eric Tseng

cs.ROMar 4, 2026

HALO: Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Authors: Hao Zhang, Yaru Niu, Yikai Wang, Ding Zhao, H. Eric Tseng

Organizations: ETAIC Lab, Department of Electrical Engineering, University of Texas at Arlington, USA · Safe AI Lab, Department of Mechanical Engineering, Carnegie Mellon University, USA.

Abstract

To improve generalization and resilience in human-robot collaboration (HRC), robots must contend with diverse combinations of human behaviors and contexts, motivating multi-agent reinforcement learning (MARL). However, inherent heterogeneity between robots and humans creates a rationality gap (RG), where decentralized policy updates deviate from cooperative joint optimization. The resulting learning problem is a general-sum differentiable game, so independent policy-gradient updates can oscillate or diverge without added structure. We propose heterogeneous-agent Lyapunov policy optimization (HALO), a framework that stabilizes decentralized MARL by enforcing Lyapunov-based contraction in policy-parameter space. Unlike Lyapunov-based safe RL, which targets state/trajectory constraints in constrained Markov decision processes, HALO uses Lyapunov certification to stabilize decentralized policy learning. HALO rectifies decentralized gradients via optimal quadratic projections, ensuring monotonic contraction of RG and enabling effective exploration of open-ended interaction spaces. Extensive simulations and real-world humanoid-robot experiments show that this certified stability improves generalization and robustness in collaborative corner cases. Our project website is available at https://HaoZhang-THU.github.io/HALO/.

Figures & tables

Figure 1: The HALO framework architecture combining the transition from standard decentralized learning to Lyapunov policy optimization for real-world HRC. Key components include the computation of the rationality gap $V(\theta)$ and the stability normal vector $h$ to derive the final analytic closed-form projection $d^{*}$ .

Panel of Figure 2 — Figure 2: Simulation benchmark and learning dynamics: (a) massively parallelized training infrastructure in Isaac Lab, where the arrows indicate the emergent synergy collaboration; (b) performance comparison across nine scenarios, where HALO demonstrates significantly faster convergence, reaching its performance plateau at approximately 1.3B steps.

Figure 3: Comparison of HALO and baseline MARL algorithms across the nine scenarios in OSP, SCT and SLH tasks.

Task category	Scenario name	HAPPO	HATRPO	PCGrad	HALO (Ours)	Improvement
OSP	Alignment	83.6 $\pm$ 5.3	87.9 $\pm$ 4.5	87.7 $\pm$ 3.8	92.8 $\pm$ 3.0	+5.6%
	Turnaround	77.6 $\pm$ 6.3	81.5 $\pm$ 5.5	83.6 $\pm$ 4.8	86.7 $\pm$ 3.5	+3.7%
	Corner entry	72.7 $\pm$ 7.0	75.3 $\pm$ 6.0	77.8 $\pm$ 5.3	82.2 $\pm$ 4.0	+5.7%
	OSP average	78.0 $\pm$ 6.3	81.6 $\pm$ 5.3	83.0 $\pm$ 4.5	87.2 $\pm$ 3.5	+5.1%
SCT	Narrow gate	80.1 $\pm$ 4.8	83.4 $\pm$ 4.3	88.6 $\pm$ 3.5	91.1 $\pm$ 2.8	+2.8%
	S-shaped path	76.3 $\pm$ 5.8	82.1 $\pm$ 5.0	80.6 $\pm$ 4.5	84.9 $\pm$ 3.8	+3.4%

Table 1: Comprehensive performance matrix and global optimization analysis of heterogeneous coordination: the upper part evaluates the scenario-specific success rate across nine representative coordination challenges in OSP, SCT and SLH tasks, reported as mean

\pm

std. The lower part provides a synchronized mechanism analysis at the 2B-step steady state, correlating overall task proficiency with fundamental optimization metrics including Overall success rate, convergence, final return, gradient alignment (

\cos\phi

), rationality gap (

V

) and gradient conflict rate. Bold and underlined indicate first and second best, respectively.

Panel of Figure 4 — Figure 4: Optimization dynamics analysis: (a) monotonic dissipation of $V(\bm{\theta})$ under the Lyapunov stability certificate; (b) rapid convergence of gradient alignment. HALO eliminates solenoidal components to stabilize the joint parameter manifold.

Figure 5: Scalability and algorithm metrics analysis: (a) convergence steps required to reach performance plateau; (b) steady-state rationality gap

V

; (c) final gradient alignment

\cos\phi

. (d) gradient conflict rate across algorithms.

Panel of Figure 6 — Figure 6: Detailed mechanism evolution: (a) standard deviation of rationality gap; (b) standard deviation of alignment; (c) temporal decay rate of the gap; (d) instantaneous change rate of alignment.

Panel of Figure 7 — Figure 7: Micro-view analysis of coordination resilience: (a) reactive height modulation during unscripted partner motion; (b) stability maintenance during 20s obstructions through stationary stepping and velocity re-synchronization.

Figure 8: Sim-to-real deployment across embodied tasks: macro-view of deployment on OSP (top), SCT (Middle) and SLH (Bottom). Temporal progression is indicated by color gradients; arrows trace the stable trajectories maintained by HALO despite complex physical coupling and human-induced perturbations.

Variant	$\mathcal{P}$	$\eta$	$\cos\phi$	SR (%) $\uparrow$	Gap $V$ $\downarrow$
HAPPO baseline	$\times$	$\times$	$\times$	74.9 $\pm$ 6.0	4.89
Soft- $V$ penalty	$\times$	$\times$	$\times$	76.5 $\pm$ 5.8	3.21
Static projection	✓	$\times$	$\times$	79.5 $\pm$ 4.8	0.85
HALO w/o align	✓	✓	$\times$	80.8 $\pm$ 4.2	0.24
HALO (full)	✓	✓	✓	82.3 $\pm$ 3.8	0.09

Table 2: Ablation matrix on SLH-extreme (pivoting) task. Results report mean

\pm

std.

\mathcal{P}

: Lyapunov projection;

\eta

: adaptive scheduling;

\cos\phi

: alignment rectification.

	Task 1: OSP		Task 2: SCT
Method	T $\downarrow$	SR $\uparrow$	T $\downarrow$	DR $\downarrow$	Ang $\downarrow$
Robot-Script	74.2	100%	101.6	60%	4.3
PCGrad	65.2	100%	81.5	0%	2.4
HALO	61.7	100%	76.2	0%	2.2
	Task 3: SLH (Stability Under Halting)
Method	Wait	T $\downarrow$	DR $\downarrow$	Ang $\downarrow$	$v_{d}$ $\downarrow$

Table 3: Real-world performance: metrics are mean values over 5 trials. T: time to destination (s); SR: success rate; DR: object drop rate (%); Ang: absolute value of tilt rate (

{}^{\circ}/s

);

v_{d}

: post-halt drift (cm/s); Wait: proactive waiting for partner synchrony.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Layer	Frequency	Update schedule	Core functional scope
Top-level	N/A	Episode initialization	Global path planning (A* or VLM) for object CoM
Mid-level	2 Hz	Tactical MARL policy	Motion command generation for collaboration
Bottom-level	50 Hz	WBC execution	Body balancing and motion command tracking

Appendix

Table 4: Temporal and functional specification of the control hierarchy.

Feature domain	Dim	Description (Egocentric coordinates)
Look-ahead guidance	10	2D XY waypoints sampled via a sliding window along the global path
Self-proprioception	13	XY pos/vel, yaw, CoM height, torso pitch, wrist-to-shoulder XYZ
Partner observation	13	Relative XY pos/vel, yaw, CoM height, torso pitch, wrist-to-shoulder XYZ
Object geometry	18	4 Top-surface corners XYZ, CoM XYZ pos, CoM XYZ vel
Contact feedback	4	Contact signals (left/right end-effector contact for both of the agents)
Temporal features	174	Snapshot accumulation (58D per frame $\times$ 3 snapshots)

Appendix

Table 5: Composition of the 210-dimensional observation space.

Command set	Dim	Formulation	Physical meaning
Locomotion base	3	Absolute	Target velocities $[v_{x},v_{y}]$ and orientation (yaw angle)
Postural setpoints	2	Absolute	Target CoM height ( $H_{\text{CoM}}$ ) and torso pitch ( $\alpha_{\text{torso}}$ )
Wrist modulation	6	Relative delta	3D XYZ offset added to task-specific base poses for both Wrist

Appendix

Table 6: MARL command space specification (11D).

Optimization parameter	Value	Architecture parameter	Value
Learning rate ( $\alpha$ )	$1.0\times 10^{-4}$	Hidden layers (actor/critic)	[256, 256, 128]
Epochs per update	10	Activation function	ReLU
Minibatch count	16	Layer initialization	Orthogonal
Entropy coefficient	0.01	Optimizer	Adam
Discount factor ( $\gamma$ )	0.99	Weight decay ( $L_{2}$ )	$1.0\times 10^{-4}$
GAE parameter ( $\lambda$ )	0.95	Gradient clipping ( $\\|g\\|_{2}$ )	10.0

Appendix

Table 7: Optimization and topology hyperparameters.

Noise type	Distribution	Magnitude	Target simulation phenomenon
Vertical jitter	Gaussian	2.0 cm	Gait-induced oscillations and lifting shifts
Velocity noise	Uniform	0.1 m/s	Non-uniform walking speed and intent changes
Yaw perturbation	Gaussian	3.0 deg	Subtle directional adjustments during transport

Appendix

Table 8: Stochastic perturbations for simulating human-like dynamical loads.

Optimization parameter	Value	Architecture parameter	Value
Learning rate ( $\alpha$ )	$1.0\times 10^{-4}$	Hidden layers (actor/critic)	[256, 256, 128]
Epochs per update	10	Activation function	ReLU
Minibatch count	16	Layer initialization	Orthogonal
Entropy coefficient	0.01	Optimizer	Adam
Discount factor ( $\gamma$ )	0.99	Weight decay ( $L_{2}$ )	$1.0\times 10^{-4}$
Learning rate schedule	Cosine annealing	Total RL action steps	$2.0\times 10^{9}$

Appendix

Table 9: Robot-script training hyperparameters.

Explore similar work

Heterogeneous Agent Collaborative Reinforcement Learning
Mar 3, 2026Zhixia Zhang, Zixuan Huang, Gongxun Li +10Multi-Agent Collaboration Heterogeneous MARL
Beihang University · Bytedance China · Apple +2
Metric-Gradient Projection for Stable Multi-Agent Policy Learning
May 12, 2026Zuyuan Zhang, Sizhe Tang, Mahdi Imani +1Hodge Decomposition Multi-Agent Systems
The George Washington University · Northeastern University
Hierarchical Lead Critic based Multi-Agent Reinforcement Learning
Feb 25, 2026David Eckel, Henri MeeßMulti-Agent Reinforcement Learning Cooperative MARL
Fraunhofer Institute IVI, Ingolstadt, Germany

Mar 3, 2026J/K move · Enter open · S save

Heterogeneous Agent Collaborative Reinforcement Learning

Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang+9

Beihang University · Bytedance China · Apple+2

Multi-Agent Collaboration Heterogeneous MARL Cooperative MARL Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem that addresses the inefficiencies of isolated multi-agent on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional homogeneous teacher-to-student transfer. Building on this problem, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO with double rollouts by an average of 3.6% while using only half the rollout cost.

Abstract

Explore similar work

Heterogeneous Agent Collaborative Reinforcement Learning

Metric-Gradient Projection for Stable Multi-Agent Policy Learning

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning