Cognition to Control - Multi-Agent Learning for Human-Humanoid Collaborative Transport

Hao Zhang; Yisen Li; Ruize Geng; Yves Tseng; Yaru Niu; Ding Zhao; H. Eric Tseng

cs.ROMar 4, 2026

Cognition to Control - Multi-Agent Learning for Human-Humanoid Collaborative Transport

Authors: Hao Zhang, Yisen Li, Ruize Geng, Yves Tseng, Yaru Niu, Ding Zhao, H. Eric Tseng

Organizations: University of Texas at Arlington. · Carnegie Mellon University

Abstract

Full-stack human-robot collaboration (HRC) can become brittle when replacing a planner, partner model, coordination policy, or controller changes the physical meaning of cross-layer signals. We introduce C2C, an object-centric cognition-to-control architecture that preserves these meanings through physical contracts. Rather than standardizing individual modules, C2C standardizes the physical semantics exchanged between them: task intent is represented by a geometrically verified payload path, partner information by a source-independent physical state, learned coordination by bounded 11-D task-space commands, and robot-specific feasibility remains inside whole-body control (WBC). The reference system combines a vision-language model with deterministic geometric verification and multi-agent reinforcement learning (MARL) for partner-aware coordination. Across nine transport scenarios, adaptive MARL variants achieve 77.1-82.1% mean success under unchanged interfaces, compared with 56.5% for a scripted-partner reference. The same contracts further support neural and interpretable hard-tree actors, two- and three-carrier teams, and simulated-to-human partner substitution. Physical Unitree G1-human tests achieve 100% success in spatially confined transport and 80% in super-long-object handling, while a three-carrier system with two G1 humanoids and one human validates the same interface structure on hardware. Together, these results show that C2C converts a tightly coupled HRC stack into a plug-compatible system in which key components can change without redefining the physical collaboration task.

Figures & tables

Fig. 1: Cognition-to-control (C2C) resolves interface entanglement around the shared payload. Cognition outputs a verified object path, partner sensing is mapped to a source-independent physical state, coordination outputs bounded task-space adaptation, and robot-specific feasibility remains inside whole-body control. A module is plug compatible when it can change without changing these interface semantics.

Fig. 2: Reference realization of C2C. The two VLM views produce candidate paths for the same payload, not separate robot paths. A deterministic verifier fuses them into one object reference $P_{t}$ . The coordination policy receives this reference together with self, partner, object, contact, and range states and outputs the same bounded 11-D task-space command $u_{i,t}$ for every tested policy. The execution layer converts that command to constrained joint torque. During hardware deployment, measured human motion fills the same partner-state fields used by the simulated teammate during training.

Component	Size	Description
Observation space ( $o\in\mathbb{R}^{210}$ )
Global guidance	10	5 anchors (relative planar)
Self state	13	Pose/velocity, CoM, torso, wrists
Partner state	13	Measured/policy partner
Object geometry	18	4 corners, CoM pose/velocity
Contact	4	4 binary contact indicators

TABLE I: Default two-carrier policy input and output.

Fig. 3: (a) Episode return during training for IPPO with a scripted partner, HAPPO, HATRPO, and PPO+PCGrad over

2.0\times 10^{9}

steps, with shading denoting one standard deviation across five seeds. (b) Mean success rate by task category, comparing the robot-script baseline with the same learning-based tactical variants. (c) Physical deployment: success rate, completion time (s), and mean object tilt rate ( ∘ /s) for robot-script IPPO and PPO+PCGrad.

Category	Scenario	Robot-script	HAPPO	HATRPO	PPO+PCGrad	Mean gain $\Delta$
OSP	$S_{11}$ : Alignment	65.4 $\pm$ 7.2	83.6 $\pm$ 5.3	87.9 $\pm$ 4.5	87.7 $\pm$ 3.8	+32.1%
	$S_{12}$ : Turnaround	60.1 $\pm$ 8.5	77.6 $\pm$ 6.3	81.5 $\pm$ 5.5	83.6 $\pm$ 4.8	+34.6%
	$S_{13}$ : Corner entry	58.2 $\pm$ 9.1	72.7 $\pm$ 7.0	75.3 $\pm$ 6.0	77.8 $\pm$ 5.3	+29.3%
SCT	$S_{21}$ : Narrow gate	59.2 $\pm$ 9.0	80.1 $\pm$ 4.8	83.4 $\pm$ 4.3	88.6 $\pm$ 3.5	+42.0%
	$S_{22}$ : S-shaped path	57.5 $\pm$ 8.8	76.3 $\pm$ 5.8	82.1 $\pm$ 5.0	80.6 $\pm$ 4.5	+38.6%
	$S_{23}$ : U-shaped path	55.6 $\pm$ 9.5	75.7 $\pm$ 6.5	79.2 $\pm$ 5.3	81.3 $\pm$ 4.8	+41.6%

TABLE II: System-level success rate under the shared C2C interfaces across nine scenarios (mean

\pm

standard deviation over five seeds, 100 episodes per seed).

\Delta

: relative difference of the mean adaptive-MARL success rate from the robot-script reference.

Parameter	Value	Parameter	Value
Learning rate	$10^{-4}$	Hidden layers	[256,256,128]
Epochs/update	10	Activation	ReLU
Minibatches	16	Initialization	Orthogonal
Entropy weight	0.01	Optimizer	Adam
Discount factor	0.99	Weight decay	$10^{-4}$
GAE factor	0.95	Gradient clip	10.0

TABLE III: Shared training settings.

Fig. 4: VLM guidance and system stress tasks. Panels (a)–(c) show two carrier-centered views. Blue and orange traces are payload-path proposals from the VLM adapter. Deterministic common-frame collision and clearance checks select the verified path, with the active anchor shown in green and the 36-D range observations in cyan. Panels (d)–(f) show the OSP, SCT, and SLH task families used under the same C2C contracts.

Axis	Substitution	Invariant interface	Evidence
Cognition	VLM $\rightarrow$ deterministic fallback	Verified payload path $P_{t}$	Same downstream path contract
Coordination	HAPPO / HATRPO / PPO+PCGrad	Observation, 11-D action, reward, WBC	SR: 77.1 / 80.8 / 82.1%
Partner	Simulated carrier $\rightarrow$ human	Partner-state fields and robot action	Physical SR: 100 / 100 / 80%
Policy	MLP $\rightarrow$ hard tree	N=3 adapter, action, reward, WBC	SR: 74.7 $\rightarrow$ 70.3%
Team size	Two $\rightarrow$ three carriers	Guidance, action semantics, reward, WBC	SR: 77.1 $\rightarrow$ 74.7%

TABLE IV: Summary of contract-preserving substitutions under the C2C interfaces.

Fig. 5: Physical human–single robot collaboration demonstrations (a) online heading adjustment, (b) following-to-leading transition, (c) constrained gate traversal, and (d) super-long-payload corridor transport.

Fig. 6: Task-family success rate for two-carrier HAPPO-MLP, three-carrier HAPPO-MLP, and three-carrier HAPPO-Tree. Parentheses give carrier count.

Fig. 7: Physical human–multiple robots collaborative transport with two Unitree G1 humanoids and one human: (a) payload reorientation and (b) passage traversal around laboratory partitions, shown at 20 s intervals.

Task	Robot-script IPPO			PPO+PCGrad
Task	SR	Tilt ( ∘ /s)	Time (s)	SR	Tilt ( ∘ /s)	Time (s)
OSP	100%	–	74.2	100%	–	65.2
SCT	40%	3.2	101.6	100%	2.4	81.5
SLH	40%	3.6	113.6	80%	2.7	86.9

TABLE V: Physical results over five trials per task.

Explore similar work

Learning from Humans for Proactive Assistance in Human-Robot Collaborative Transport
Sep 21, 2026Elvin Yang, Christoforos MavrogiannisRobotic Control Human-Robot Collaboration
Department of Robotics, University of Michigan, Ann Arbor, United States
Learning Multi-Humanoid Pickup and Transport via Decentralized Object-Centric Control
Sep 15, 2026Bikram Pandit, Mohitvishnu S. Gadde, Aayam Kumar Shrestha +1Robotic Control Multi-Robot Systems
Dynamics Robotics and AI Lab (DRAIL), Oregon State University, Corvallis, OR, USA
TATIC: Task-Aware Temporal Learning for Human Intent Inference from Physical Corrections in Human-Robot Collaboration
Mar 10, 2026Jiurun Song, Xiao Liang, Minghui ZhengHuman-Robot Collaboration Human-Robot Interaction
Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843, USA

Sep 21, 2026J/K move · Enter open · S save

Learning from Humans for Proactive Assistance in Human-Robot Collaborative Transport

Elvin Yang, Christoforos Mavrogiannis

Department of Robotics, University of Michigan, Ann Arbor, United States

Robotic Control Human-Robot Collaboration Human Behavior Prediction

Open paper AskPDF arXiv

We focus on human-robot collaborative transport, a challenging task of broad relevance spanning logistics, manufacturing, and the home, in which a user and a robot work together to relocate a large or heavy object. To act as an effective partner, the robot should reduce the user's effort by contributing to efficient relocation of the object while remaining physically responsive to them. Prior work often addresses these capabilities separately, producing robots that may move the object efficiently but resist user input, or accommodate the user but depend on continuous guidance. Our key insight is that obstacle-constrained collaborative transport requires integrating predictions of human collaborative behavior with compliant robot control. To this end, we introduce PROACT, a framework for human-robot collaborative transport that incorporates anticipation into compliant whole-body control through a learned model of human collaborative behavior. Trained on a large-scale, real-world dataset of dyadic human transport demonstrations, our transformer architecture distills collaborative behavior into predictions of future object motion. Across 108 real-world trials with a 9-DoF mobile manipulator, PROACT reduces mean interaction work by 59.2% and 20.4%, and mean completion time by 12.9% and 6.9%, relative to compliance-only and MPC baselines, respectively. Footage from our experiments can be found at https://youtu.be/qAGvQfVPjbk.

Abstract

Explore similar work

Learning from Humans for Proactive Assistance in Human-Robot Collaborative Transport

Learning Multi-Humanoid Pickup and Transport via Decentralized Object-Centric Control

TATIC: Task-Aware Temporal Learning for Human Intent Inference from Physical Corrections in Human-Robot Collaboration