From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction

Zekai Deng; Kangyi Chen; Ye Shi; Jingya Wang

cs.ROOct 8, 2026

From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction

Authors: Zekai Deng, Kangyi Chen, Ye Shi, Jingya Wang

Organizations: ShanghaiTech University · InstAdapt

Abstract

Physics-based human-object interaction has achieved robust single-agent manipulation skills, yet extending them to multi-agent cooperative tasks remains challenging. Existing approaches typically adapt interaction policies through task-specific fine-tuning, which entangles low-level contact-rich execution with high-level coordination and limits reuse across object geometries, interaction types, and team sizes. We propose a hierarchical framework that converts a single-agent HOI policy into a reusable Object-oriented Motion Skill. Specifically, we reinterpret teacher rollouts as object-oriented action supervision by extracting short-horizon object-proxy motions from executed trajectories, and distill task-specific teachers into a low-level skill operating in an Object-oriented Action Space. For downstream tasks, the distilled skill is frozen as a reusable executor, while a high-level policy coordinates multiple agents by generating region-wise object-oriented actions conditioned on the shared object, task goal, agent states, and local manipulation regions. This formulation shifts multi-agent HOI learning from direct contact-rich full-body control to compact object-level proxy-motion coordination. Experiments on diverse HOI tasks show that the distilled Object-oriented Motion Skill supports robust proxy-motion execution and enables composable policy learning across different interaction types, object geometries, and team sizes.

Figures & tables

Figure 1: Our method composes object-oriented motion skills and cooperative planning to support shared object manipulation across team sizes, object geometries, and interaction types.

Figure 2: Overview of our hierarchical framework. We distill and post-train a reusable Object-oriented Motion Skill from task-specific single-human HOI rollouts re-labeled as proxy-motion supervision. The frozen skill then executes local proxy motions planned by high-level policies, enabling composable single- and multi-human HOI.

Task	Object	#Agents	CooHOI		Ours
Task	Object	#Agents	Succ. $\uparrow$	Precision $\downarrow$	Succ. $\uparrow$	Precision $\downarrow$
Carrying	Armchair	1	97.26	5.05	97.81	4.99
	Table	1	97.07	5.23	97.19	5.11
	Sofa	2	84.17	10.12	88.21	8.70
	H-box	4	80.96	17.50	85.63	14.24
	Cross-box	4	82.38	15.13	86.14	13.81

Table 1: Quantitative evaluation of downstream compositional HOI tasks. The Object-oriented Motion Skill is frozen across all tasks, and only task-specific high-level policies are trained to generate object-oriented actions for local proxies.

Method	Teacher-Rollout Targets					Random-Motion Targets
Method	$E_{\mathrm{pos}}$ $\downarrow$	$E_{\mathrm{vel}}$ $\downarrow$	$E_{\mathrm{acc}}$ $\downarrow$	TTR $\uparrow$	Succ traj $\uparrow$	$E_{\mathrm{pos}}$ $\downarrow$	$E_{\mathrm{vel}}$ $\downarrow$	$E_{\mathrm{acc}}$ $\downarrow$	TTR $\uparrow$	Succ traj $\uparrow$
InterPhys [ 8 ]	106.5	47.8	50.9	82.6	68.2	112.2	63.9	54.1	76.4	60.8
SkillMimic [ 63 ]	91.0	40.3	44.2	90.8	81.2	99.3	54.6	49.7	84.9	68.3
Ours	79.8	38.2	31.3	100.0	98.9	95.8	45.2	39.2	96.3	91.8
Ours w/o Online Distill.	117.9	52.3	50.8	86.7	84.8	148.2	63.1	55.9	58.4	23.8
Ours w/o Random-Motion RL	88.1	41.3	32.8	97.2	98.1	156.1	65.5	57.2	63.1	38.7

Table 2: Quantitative evaluation of Object-oriented Motion Skill learning. We report proxy-motion tracking under teacher-rollout targets and procedurally generated random proxy-motion targets.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}_{\mathrm{BC}}=\mathbb{E}_{t}\left[\left\|\mu_{\mathrm{low}}(\mathbf{s}_{t},\mathbf{o}_{t},\mathbf{m}^{\mathrm{ref}}_{t})-\mu_{\mathrm{tea}}(\mathbf{s}_{t},\mathbf{o}_{t},\mathbf{g}^{\mathrm{tea}}_{t})\right\|_{2}^{2}\right].

Appendix

Algorithm 1 Training the Object-oriented Motion Skill

\mathbf{P}_{1:K}=\mathbf{p}_{t}+\mathrm{cumsum}(\Delta_{t-1}).

Appendix

Algorithm 2 Procedural Random Proxy-Motion Generation

Item	Value
Simulator	Isaac Gym
Number of environments	$4096$
Humanoid model	AMP humanoid
Control mode	PD target control
Control frequency	$30$ Hz
Physics substeps	$2$

Appendix

Table A.1: Implementation details and main hyperparameters.

Figure A.1: A deadlock failure of CooHOI [ 5 ] . After failed grasping attempts, the agents may stay near the object but stop making progress. Our method is less likely to get stuck because it follows explicit short-horizon trajectory plans.

Explore similar work

SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy
Apr 20, 2026Wei Yao, Haohan Ma, Hongwen Zhang +6Robotic Control HOI Generation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +2
Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllers
May 12, 2026Sanghyeok Nam, Byoungjun Kim, Daehyung Park +1HOI Generation Human Motion Generation
KAIST
Co ${}^{2}$ Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction
Oct 7, 2026Jeonghwan Kim, Hyeonwoo Kim, Hanbyul JooRobot Skill Learning Whole-Body Robotic Manipulation
Department of Computer Science Seoul National University

Apr 20, 2026, cs.CVJ/K move · Enter open · S save

SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy

Wei Yao, Haohan Ma, Hongwen Zhang, Yunlian Sun+5

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China+2

Robotic Control HOI Generation Generalization in Robotic Manipulation Robot Imitation Learning Cooperative MARL Humanoid Robotics+1

Open paper AskPDF arXiv

Controllable cooperative humanoid manipulation is a fundamental yet challenging problem for embodied intelligence, due to severe data scarcity, complexities in multi-agent coordination, and limited generalization across objects. In this paper, we present SynAgent, a unified framework that enables scalable and physically plausible cooperative manipulation by leveraging Solo-to-Cooperative Agent Synergy to transfer skills from single-agent human-object interaction to multi-agent human-object-human scenarios. To maintain semantic integrity during motion transfer, we introduce an interaction-preserving retargeting method based on an Interact Mesh constructed via Delaunay tetrahedralization, which faithfully maintains spatial relationships among humans and objects. Building upon this refined data, we propose a single-agent pretraining and adaptation paradigm that bootstraps synergistic collaborative behaviors from abundant single-human data through decentralized training and multi-agent PPO. Finally, we develop a trajectory-conditioned generative policy using a conditional VAE, trained via multi-teacher distillation from motion imitation priors to achieve stable and controllable object-level trajectory execution. Extensive experiments demonstrate that SynAgent significantly outperforms existing baselines in both cooperative imitation and trajectory-conditioned control, while generalizing across diverse object geometries. Codes and data will be available after publication. Project Page: http://yw0208.github.io/synagent

Abstract

Explore similar work

SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy

Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllers

Co2{}^{2}2Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction

Co ${}^{2}$ Skill: Whole-Body Control via Skill Composition for Long-Horizon Human-Environment Interaction