TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size

Stefan Lionar; Gim Hee Lee

cs.CVMar 9, 2026

TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Size

Authors: Stefan Lionar, Gim Hee Lee

Organizations: Garena · Sea AI Lab · National University of Singapore

Abstract

Physics-based humanoid control has achieved remarkable progress in enabling realistic and high-performing single-agent behaviors, yet extending these capabilities to cooperative human-object interaction (HOI) remains challenging. We present TeamHOI, a framework that enables a single decentralized policy to handle cooperative HOIs across any number of cooperating agents. Each agent operates using local observations while attending to other teammates through a Transformer-based policy network with teammate tokens, allowing scalable coordination across variable team sizes. To enforce motion realism while addressing the scarcity of cooperative HOI data, we further introduce a masked Adversarial Motion Prior (AMP) strategy that uses single-human reference motions while masking object-interacting body parts during training. The masked regions are then guided through task rewards to produce diverse and physically plausible cooperative behaviors. We evaluate TeamHOI on a challenging cooperative carrying task involving two to eight humanoid agents and varied object geometries. Finally, to promote stable carrying, we design a team-size- and shape-agnostic formation reward. TeamHOI achieves high success rates and demonstrates coherent cooperation across diverse configurations with a single policy.

Figures & tables

Figure 2: Overview of TeamHOI framework. A transformer-based policy network enables coordination between the observing agent (green humanoid) and its teammates (grey humanoids) through alternating self- and cross-attention layers. By training across diverse team-size environments, the framework learns a unified policy that works across different team configurations. To maintain motion realism and enhance skill diversity, a masked AMP strategy blends full-body and masked discriminators based on object interaction.

Figure 3: Illustration of our principal-axes coverage reward.

Model	Agents Formation	SR (%) $\uparrow$ / $d$ ( m ) $\downarrow$			$t_{\text{coop}}$ (%) $\uparrow$			$\|J\|$ ( $\text{m/s}^{3}$ ) $\downarrow$			SR 5× (%) $\uparrow$ / $d_{5\times}$ ( m ) $\downarrow$
Model	Agents Formation	2A	4A	8A	2A	4A	8A	2A	4A	8A	4A	8A
CooHOI*-2	Pre-defined	97.5 / 0.19	73.2 / 1.49	10.1 / 4.25	90.3	54.6	1.0	48.3	85.0	189.9	0.0 / 6.38	0.4 / 6.00
CooHOI*-4	Pre-defined	95.5 / 0.18	94.5 / 0.36	61.5 / 1.83	96.0	92.1	27.2	40.7	38.6	96.7	1.2 / 5.34	14.2 / 4.26
CooHOI*-8	Pre-defined	29.4 / 3.60	52.4 / 2.68	42.2 / 3.83	93.8	93.6	81.6	39.5	36.5	45.2	0.1 / 6.22	4.1 / 5.73
Ours	Learned coop.	99.1 / 0.06	99.2 / 0.08	97.5 / 0.18	95.2	96.1	90.1	51.0	44.7	34.2	3.5 / 4.78	81.1 / 0.49

Table 1: Quantitative comparison across team sizes (2A, 4A, 8A). Our method achieves consistently high success rates, collective cooperation, and motion smoothness across all settings using a single unified policy. Unlike CooHOI* baselines, where agent formations are pre-defined, our agents must infer cooperation to establish stable formations autonomously, making the coordination requirement more demanding. Under the heavy-load setting (5× table weights), only our method demonstrates effective cooperation among eight agents. All results are averaged over 10,000 simulation episodes.

Figure 4: Qualitative comparison across 4-agent (top) and 8-agent (bottom) configurations. Our method produces synchronized and stable teamwork across both cases, whereas the CooHOI* baselines exhibit limited or ineffective cooperation. Red line indicates the table’s movement trajectory, and the black dot marks its final position at the end of each episode.

Figure 5: Ablation on the masked AMP strategy. Comparison between models trained with and without masked AMP, showing improved task rewards and successful hand-object interactions when masking is applied.

Figure 6: Formation reward comparison. Adding principal-axes coverage reward produces stable formations aligned with the object’s principal axes, facilitating learned natural locomotion.

Figure 7: Comparison of task reward curves for models trained with team-size and global advantage normalizations.

Figure 8: Environment setup in IsaacGym using a dummy ceiling to support flexible team-size training. Extra agents are moved to the ceiling and excluded from observations, rewards, and gradient updates.

Hyperparameter	Value
Horizon length	32
Optimizer	Adam [ 20 ]
Learning rate	$2\times 10^{-5}$
Task reward weight	0.5
Style reward weight	0.5
PPO clip threshold $(\epsilon)$	0.2

Table 2: Key training hyperparameters in our experiment.

Normal weight (1 $\times$ )
Team size	SR (%) $\uparrow$	$d$ (m) $\downarrow$	$t_{\text{coop}}$ (%) $\uparrow$	$\|J\|$ (m/s 3 ) $\downarrow$
2	99.1	0.06	95.2	51.0
3	99.4	0.06	98.3	50.5
4	99.2	0.08	96.1	44.7
5	99.5	0.06	97.3	40.6
6	99.3	0.07	95.9	38.0

Table 3: Performance of our unified policy across team sizes under normal (1

\times

) and heavy (5

\times

) table weights.

Small tables
Team size	SR (%) $\uparrow$	$d$ (m) $\downarrow$	$t_{\text{coop}}$ (%) $\uparrow$	$\|J\|$ (m/s 3 ) $\downarrow$
2	93.1	0.37	94.8	63.2
3	97.5	0.14	97.0	64.7
4	98.4	0.12	97.1	55.5
8	96.4	0.23	85.4	45.0
Large tables

Table 4: Performance of our unified policy across team sizes for small and large tables. All results are averaged over 10,000 simulation episodes.

Figure 9: Qualitative visualization of the zero-shot generalization under unseen table geometries and team sizes. Red line indicates the table’s movement trajectory, and the black dot marks its final position at the end of each episode.

Panel of Figure 10 — Figure 10: Examples of multiple affordance behaviors learned by adapting the task reward. The agents are able to adapt to side-holding or edge-lifting while being able to walk toward diverse directions. The policy is trained using the same single-human reference motions as our main experiments.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction
Oct 8, 2026Zekai Deng, Kangyi Chen, Ye Shi +1Skill Distillation HOI Generation
ShanghaiTech University · InstAdapt
SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy
Apr 20, 2026Wei Yao, Haohan Ma, Hongwen Zhang +6Robotic Control HOI Generation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +2
Learning Multi-Humanoid Pickup and Transport via Decentralized Object-Centric Control
Sep 15, 2026Bikram Pandit, Mohitvishnu S. Gadde, Aayam Kumar Shrestha +1Robotic Control Multi-Robot Systems
Dynamics Robotics and AI Lab (DRAIL), Oregon State University, Corvallis, OR, USA

Oct 8, 2026J/K move · Enter open · S save

From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction

Zekai Deng, Kangyi Chen, Ye Shi, Jingya Wang

ShanghaiTech University · InstAdapt

Skill Distillation HOI Generation Multi-Agent Coordination Hierarchical RL

Open paper AskPDF arXiv

Physics-based human-object interaction has achieved robust single-agent manipulation skills, yet extending them to multi-agent cooperative tasks remains challenging. Existing approaches typically adapt interaction policies through task-specific fine-tuning, which entangles low-level contact-rich execution with high-level coordination and limits reuse across object geometries, interaction types, and team sizes. We propose a hierarchical framework that converts a single-agent HOI policy into a reusable Object-oriented Motion Skill. Specifically, we reinterpret teacher rollouts as object-oriented action supervision by extracting short-horizon object-proxy motions from executed trajectories, and distill task-specific teachers into a low-level skill operating in an Object-oriented Action Space. For downstream tasks, the distilled skill is frozen as a reusable executor, while a high-level policy coordinates multiple agents by generating region-wise object-oriented actions conditioned on the shared object, task goal, agent states, and local manipulation regions. This formulation shifts multi-agent HOI learning from direct contact-rich full-body control to compact object-level proxy-motion coordination. Experiments on diverse HOI tasks show that the distilled Object-oriented Motion Skill supports robust proxy-motion execution and enables composable policy learning across different interaction types, object geometries, and team sizes.

Abstract

Human-Object Interaction

Explore similar work

From Solo to Ensemble: A Hierarchical Framework for Composable Multi-Agent Human-Object Interaction

SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy

Learning Multi-Humanoid Pickup and Transport via Decentralized Object-Centric Control