OmniInteract: Towards Unified Whole-Body Interaction Control for Dual Humanoids

Yinghao Li; Tianyu Zhang; Zhuohao Li; Yi-Lin Wei; Fa-Ting Hong; Qing Zhang; Wei-Shi Zheng

cs.ROOct 19, 2026

OmniInteract: Towards Unified Whole-Body Interaction Control for Dual Humanoids

Authors: Yinghao Li, Tianyu Zhang, Zhuohao Li, Yi-Lin Wei, Fa-Ting Hong, Qing Zhang, Wei-Shi Zheng

Organizations: School of Computer Science and Engineering, Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Institute of Technology · The Hong Kong University of Science and Technology

Abstract

Physical interaction is a key capability for humanoid collaboration, requiring coordinated whole-body motion and precise contact. However, learning diverse dual-humanoid interactions faces two major challenges: (1) human-robot morphological differences disrupt contact interfaces and spatial relationships during retargeting, and (2) jointly learning motor skills and partner coordination makes it difficult to retain control fidelity as interaction diversity grows. In this paper, we propose OmniInteract to address these challenges through surface-aware modeling and scalable residual policy learning. First, we introduce Surface Contact-Aware Retargeting, which leverages dense human-robot surface correspondences to jointly retarget both humanoids, preserving explicit interaction contact interfaces and their near-field geometric context. Second, we propose a Scalable Flow-based Interaction Policy for partner-conditioned residual generation. It factorizes dual-humanoid control into a reusable motor foundation and interaction-specific adaptations: partner-aware experts learn corrections over a frozen whole-body motion prior, while conditional flow matching unifies them in a shared residual action space. This enables scalable skill consolidation while retaining pretrained motor capabilities. Extensive experiments demonstrate that OmniInteract improves retargeting fidelity and dual-humanoid coordination while better retaining interaction performance as skill diversity grows. Real-world deployment further demonstrates successful contact establishment, stable whole-body coordination, and recovery from external perturbations.

Figures & tables

Figure 1: Top center: OmniInteract enables diverse dual-humanoid interactions from human demonstrations. Left: SCAR (Surface Contact-Aware Retargeting) preserves fine-grained surface interaction geometry during retargeting. Right: Specialist-to-unified residual learning consolidates diverse interaction skills into a unified flow policy. Bottom (Real, a–c): Deployment on dual Unitree G1 robots performing High-Five, Hug, and Fencing whole-body interactions.

Figure 2: Overview of the OmniInteract pipeline. Left: Starting from human-human demonstrations, SCAR jointly retargets the two humanoids while preserving individual motion, surface contacts, and near-field interaction context, yielding synchronized inetraction motions. Center: A frozen SONIC tracker provides the base actions, while task-specific residual experts learn partner-aware interaction corrections. Right: These diverse interaction experts are then distilled into a unified flow matching policy through DAgger. During deployment, the unified policy predicts residual actions on top of the frozen SONIC for coordinated dual-humanoid control.

Dataset	Method	Feasibility	Interaction Fidelity			Utility
Dataset	Method	MPD ${}_{(\mathrm{cm})}\downarrow$	CIE ${}_{(\%)}\downarrow$	NGE ${}_{(\%)}\downarrow$	C-F1 ${}_{(\%)}\uparrow$	DSR ${}_{(\%)}\uparrow$
InterHuman	PHC	2.59	10.07	10.89	47.32	40.6
	GMR	4.76	13.94	13.03	31.98	36.4
	OmniRetarget	3.54	6.71	6.91	58.69	58.7
	UMR	4.04	7.83	7.73	51.14	56.9
	SCAR (Ours)	0.93	4.96	6.49	65.18	75.4

Table 1: Quantitative Results of Retargeting. Metrics include physical feasibility (MPD), interaction fidelity (CIE, NGE, C-F1), and utility (DSR). Best results are shown in bold .

Figure 3: Qualitative visualization comparison of interaction retargeting. ( Top : Touch Head, Bottom : Pat on Back)

Single-Agent Trackers
Method	InterHuman † + Inter-X †			Unseen Motions
Method	${E_{\mathit{mpjpe}}}\downarrow$	${E_{\mathit{inter\text{-}geo}}}\downarrow$	Succ(%) $\uparrow$	${E_{\mathit{mpjpe}}}\downarrow$	${E_{\mathit{inter\text{-}geo}}}\downarrow$	Succ(%) $\uparrow$
TWIST2	86.56	11.44	41.36	113.35	14.72	28.58
BeyondMimic	64.08	8.91	69.27	–	–	–
SONIC	65.31	9.56	63.32	66.31	9.51	62.65
Peer-Aware Controllers

Table 2: Comparison of controllers on dual-humanoid interaction benchmarks. InterHuman † and Inter-X † denote the motion subsets selected from each dataset for control evaluation.

Figure 4: Shoulder Hold in simulation (Issac Lab). Compared with SONIC (bottom), OmniInteract (top) improves hand–shoulder alignment through partner-aware residual corrections.

Figure 6: Real-world deployment of contact interaction tasks and robustness to disturbances.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of contact interface and near-field context.

Parameter	Value
Contact thresholds $\tau_{\mathit{in}}/\tau_{\mathit{out}}$	$0.02/0.03$ m
Contact-pair budget	$128$ per frame
Context-center budget $B_{\mathit{n}}$	$128$ total across both actors
Context radius $r_{\mathit{n}}$ / Gaussian width $\sigma_{\mathit{n}}$	$0.15/0.06$ m
Maximum neighbors $K$	$6$ per center
Pose slots per region / sampling seeds	$12$ / $0,1$ for actors $1,2$

Appendix

Table 4: SCAR interaction and optimization settings. Context centers share one budget across the two humanoids.

Region	Position $w^{\mathit{pos}}$	Normal $w^{\mathit{nor}}$
Torso	20	5
Hips	20	0
Head	1	1
Hand, Foot	10	1
Arm, ForeArm, UpLeg, Leg	10	5
Shoulder	0	0.5

Appendix

Table 5: Region-dependent pose weights inherited from UMR. Left and right regions share weights. Arm and UpperArm denote distinct regions in the 19-region configuration.

Architecture
Hyperparameter	Value
History / future window length	$20$ / $20$
Input projection, history / future	$239\rightarrow 60$ / $93\rightarrow 60$
Temporal CNN channels	$60\rightarrow 40\rightarrow 20$
CNN kernel sizes / strides	$(6,4)$ / $(2,2)$
Temporal pooling / output projection	Average / $20\rightarrow 64$

Appendix

Table 6: Network architecture and MAPPO hyperparameters for residual interaction expert learning.

Category	Reward term	Equation	Weight
Interaction	Relative anchor position	$\exp\!\left(-\frac{\\|\Delta\mathbf{p}^{\mathit{sim}}-\Delta\mathbf{p}^{\mathit{ref}}\\|_{2}^{2}}{0.5^{2}}\right)$	$1.5$
	Relative anchor orientation	$\exp\!\left(-\frac{d_{Q}(\Delta Q^{\mathit{sim}},\Delta Q^{\mathit{ref}})^{2}}{0.75^{2}}\right)$	$0.5$
	Interaction topology	$\exp\!\left(-\frac{\sum_{(u,v)\in\mathcal{P}_{\mathit{topo}}}\rho_{uv}(\ell_{uv}^{\mathit{sim}}-\ell_{uv}^{\mathit{ref}})^{2}}{0.35^{2}\,\|\mathcal{P}_{\mathit{topo}}\|}\right)$	$1.5$
	Contact/support timing	$\exp\!\left(\begin{aligned} &\textstyle-\frac{\sum_{(u,v)\in\mathcal{P}_{\mathit{cont}}}\|c_{uv}^{\mathit{sim}}-c_{uv}^{\mathit{ref}}\|}{\|\mathcal{P}_{\mathit{cont}}\|}\\[-1.0pt] &\textstyle-\frac{\sum_{a=1}^{2}\sum_{f\in\mathcal{F}}\|s_{a,f}^{\mathit{sim}}-s_{a,f}^{\mathit{ref}}\|}{2\|\mathcal{F}\|}\end{aligned}\right)$	$1.0$
Motion tracking	Anchor position	$\exp\!\left(-\frac{\\|\mathbf{p}_{a,\mathit{anc}}^{\mathit{sim}}-\mathbf{p}_{a,\mathit{anc}}^{\mathit{ref}}\\|_{2}^{2}}{0.3^{2}}\right)$	$0.25$
	Anchor orientation	$\exp\!\left(-\frac{d_{Q}(Q_{a,\mathit{anc}}^{\mathit{sim}},Q_{a,\mathit{anc}}^{\mathit{ref}})^{2}}{0.4^{2}}\right)$	$0.25$

Appendix

Table 7: Reward terms and weights for residual interaction expert learning.

Category	Parameter	Range / setting
Dynamics and contact	Static friction	$[0.3,1.6]$
	Dynamic friction	$[0.3,1.2]$
	Restitution	$[0,0.5]$
	Torso CoM offset	$x:\ \pm 0.025$ m; $y,z:\ \pm 0.05$ m
	Default joint offset	$\pm 0.01$ rad
Initial state	Root position offset	$x,y:\ \pm 0.05$ m; $z:\ \pm 0.01$ m

Appendix

Table 8: Domain randomization for residual interaction expert learning.

Component	Specification
Architecture
History preprocessing MLP	$239\rightarrow 256\rightarrow 256$ ; ReLU
Future preprocessing MLP	$93\rightarrow 256\rightarrow 256$ ; ReLU
Preprocessing weight sharing	Across roles; separate history/future weights
Token sequence per policy evaluation	$2$ roles $\times 2$ streams $\times 20$ entries
Token width / count	$256$ / $80$

Appendix

Table 9: Architecture and learning specification of the shared per-humanoid residual flow policy.

Motion ID	Motion Source	Duration	Category	Motion Description
InterHuman
1	InterHuman-2312	8.0 s	Dance	Synchronized dance with shoulder and waist contact.
2	InterHuman-2853	58.4 s	Push	Repeated right-hand pushing, blocking.
3	InterHuman-6436	29.6 s	Dance	Two people dance together.
4	InterHuman-6494	23.9 s	Embrace	Two people dance while embracing.
5	InterHuman-6774	32.3 s	Dance	Face-to-face ballroom dance.

Appendix

Table 10: Summary of interaction motions in the control benchmark.

Variant	Inter-X			InterHuman
Variant	CIE $\downarrow$	NGE $\downarrow$	DSR $\uparrow$	CIE $\downarrow$	NGE $\downarrow$	DSR $\uparrow$
w/o CIO	6.78	5.43	64.2	7.45	7.21	68.8
w/o NCO	4.21	6.63	65.8	5.23	8.21	69.2
SCAR (full)	3.98	4.99	72.1	4.96	6.49	75.4

Appendix

Table 11: SCAR component ablation on Inter-X and InterHuman. All metrics are reported in %; arrows indicate the preferred direction, and the best results are in bold.

Figure 8: Qualitative comparison of interaction retargeting on Inter-X and InterHuman. (Top: High-five, Middle: Push, Bottom: Holding Hands.) Columns show the raw motion and results from GMR, OmniRetarget, UMR, PHC, and SCAR (ours). Zoomed-in views highlight contact details.

Figure 9: Sensitivity to the residual action scale

\alpha_{\mathrm{res}}

. A scale of 0.3 achieves the lowest tracking and interaction geometry errors and the highest success rate.

Figure 10: Qualitative results of OmniInteract for interaction control in Isaac simulation. From top to bottom: dancing, fighting, and walking with an arm around a partner’s shoulder.

Figure 11: Qualitative visualization of real-world humanoid interactions. (Top: Boxing, Middle: Dancing, Bottom: Meeting.) Each row presents five keyframes arranged chronologically from left to right, illustrating coordinated movements and close-range interactions between two humanoid robots.)

Humanoid Robot Control

Humanoid robot control develops methods for coordinating bipedal balance, locomotion, and whole-body interaction with people and objects.

See all 201 papers

Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids
Mar 3, 2026Hongjin Chen, Wei Zhang, Pengfei Li +10RL for Robotics Humanoid Robot Control
Fudan University · TARS Robotics · Tsinghua University +3
CoCarry: Contact-Support-Structured Retargeting for Dual-Humanoid Cooperative Carrying
Oct 23, 2026Ziyu Lin, Shihang Du, Tianhang Wang +6Motion Retargeting Multi-Robot Cargo Transportation
Physical Intelligence Lab, Tongji University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · AGIBOT, Shanghai, China +1
HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction
Sep 28, 2026Jihwan Shin, Adrià López Escoriza, Junzhe He +2Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics
Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Mar 3, 2026J/K move · Enter open · S save

Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids

Hongjin Chen, Wei Zhang, Pengfei Li, Shihao Ma+9

Fudan University · TARS Robotics · Tsinghua University+3

RL for Robotics Humanoid Robot Control Whole-Body Control Motion Retargeting

Open paper AskPDF arXiv

Realizing interactive whole-body control for multi-humanoid systems is critical for unlocking complex collaborative capabilities in shared environments. Although recent advancements have significantly enhanced the agility of individual robots, bridging the gap to physically coupled multi-humanoid interaction remains challenging, primarily due to severe kinematic mismatches and complex contact dynamics. To address this, we introduce Rhythm, the first unified framework enabling real-world deployment of dual-humanoid systems for complex, physically plausible interactions. Our framework integrates three core components: (1) an Interaction-Aware Motion Retargeting (IAMR) module that generates feasible humanoid interaction references from human data; (2) an Interaction-Guided Reinforcement Learning (IGRL) policy that masters coupled dynamics via graph-based rewards; and (3) a real-world deployment system that enables robust transfer of dual-humanoid interaction. Extensive experiments on physical Unitree G1 robots demonstrate that our framework achieves robust interactive whole-body control, successfully transferring diverse behaviors such as hugging and dancing from simulation to reality.

Abstract

Humanoid Robot Control

Explore similar work

Rhythm: Learning Interactive Whole-Body Control for Dual Humanoids

CoCarry: Contact-Support-Structured Retargeting for Dual-Humanoid Cooperative Carrying

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction