HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

Jihwan Shin; Adrià López Escoriza; Junzhe He; Matthias Heyrman; Marco Hutter

cs.ROSep 28, 2026

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

Authors: Jihwan Shin, Adrià López Escoriza, Junzhe He, Matthias Heyrman, Marco Hutter

Organizations: Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Abstract

Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object. We publicly release the code and the retargeted motion dataset.

Figures & tables

Fig. 2: Overview of HOI-Retarget . (a) The source is a captured or video-reconstructed HOI clip, providing human motion, an object trajectory and contact labels. (b) IK retargeting maps the human onto the robot, while the object mesh and trajectory are scaled by the robot-to-human height ratio, carrying the contact targets with them. (c) A windowed trajectory optimization with tracking, contact and smoothness costs recovers those contacts under the robot’s kinematic limits. (d) The result drives downstream policies directly, or after dynamic refinement in simulation.

Method	Source Human Motion	Object Interaction	Contact Location Preservation	Data Augmentation	Multi-Agent Source	Temporal Coupling	Dynamics Enforcement	Optimization Method
GMR [ 11 ]	✓	✗	✗	✗	✗	✗	✗	per-frame IK
OmniRetarget [ 13 ]	✓	✓	✗	✓	✗	✗	✗	per-frame SOCP
DynaRetarget [ 14 ]	✗	✓	✗	✓	✗	✓	✓	progressive SBTO
SPIDER [ 15 ]	✗	✓	✗	✓	✗	✓	✓	receding-horizon sampling
HOI-Retarget (ours)	✓	✓	✓	✓	✓	✓	✗	windowed NLP

TABLE I: Capabilities of representative retargeting methods.

Cost term	Symbol	Definition	$w$
Tracking	$E^{\mathrm{T}}$	$E^{\theta}+E^{\mathrm{b}}+E^{\mathrm{h}}+E^{\mathrm{f}}+E^{\mathrm{r}}$
Joint position	$E^{\theta}$	$\lVert\boldsymbol{\theta}-\boldsymbol{\theta}^{*}\rVert^{2}$	$1$
Base position	$E^{\mathrm{b}}$	$\lVert\boldsymbol{p}^{w}_{b}-\boldsymbol{p}^{w*}_{b}\rVert^{2}$	$1$
Head position	$E^{\mathrm{h}}$	$\lVert\boldsymbol{p}^{w}_{\mathrm{head}}-\boldsymbol{p}^{w*}_{\mathrm{head}}\rVert^{2}$	$1$
Feet position	$E^{\mathrm{f}}$	$\sum_{i\in\mathcal{F}}\lVert\boldsymbol{p}^{w}_{c,i}-\boldsymbol{p}^{w*}_{c,i}\rVert^{2}$	$100$
Torso orientation	$E^{\mathrm{r}}$	$\lVert\boldsymbol{R}^{b}_{\mathrm{torso}}-\boldsymbol{R}^{b*}_{\mathrm{torso}}\rVert_{F}^{2}$	$1$

TABLE II: Cost terms of ( 2 ), grouped as in ( 2a ).

Fig. 3: Representative retargets across HOI sources and embodiments. Rows show the source SMPL-X motion, Unitree G1, and Unitree H2.

Metric	GMR	OmniRetarget	Ours
Body pose deviation ( ∘ ) $\downarrow$	20.0	25.0	21.0
Link vel. direction ( ∘ ) $\downarrow$	19.4	28.9	25.2
Contact-point gap (m) $\downarrow$	0.368	0.183	0.005
Rel. hand-orient. change ( ∘ ) $\downarrow$	2.3	37.6	6.1
Body jerk (m/s 3 ) $\downarrow$	66.7	193.5	49.9
Compute time (s/clip) $\downarrow$	9.1	156.2	33.7

TABLE III: Kinematic retargeting on the common OMOMO subset (3,997 clips; 3,604 for contact metrics).

Fig. 4: Qualitative comparison with OmniRetarget. HOI-Retarget recovers the labeled source contact regions on the coat rack and chair.

Fig. 5: Contact-preserving object-scale augmentation over the tested range $\times 0.25$ – $\times 1.50$ on the G1 and H2.

	RL Tracker		SBTO
Metric	Omni.	Ours	Omni.	Ours
Body pose deviation ( ∘ ) $\downarrow$	26.5	23.5	26.0	23.1
Link vel. direction ( ∘ ) $\downarrow$	47.9	46.7	51.2	$50.8^{\ast}$
Object path err. (m) $\downarrow$	0.285	0.213	0.269	0.194
Contact-point gap (m) $\downarrow$	0.096	0.086	0.141	0.099
Rel. hand-orient. change ( ∘ ) $\downarrow$	31.5	25.3	38.4	28.1

TABLE IV: Dynamic-refinement quality on the common subset (68 clips; 63 for contact metrics).

Fig. 6: Example of using HOI-Retarget from video reconstruction using CARI4D. Contact refinement tool has been used to account for monocular reconstruction artifacts.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$(\cdot)^{*}$	Reference quantity
$(\cdot)^{\mathrm{h}}$	Human source quantity
$(\cdot)^{w},(\cdot)^{b},(\cdot)^{o}$	World, base, object frame
$\delta(\cdot)$	Backward difference,
	$\delta(\cdot)_{t}=(\cdot)_{t}-(\cdot)_{t-1}$
$\boldsymbol{q}$	Configuration $(\boldsymbol{p}^{w}_{b},\boldsymbol{R}^{w}_{b},\boldsymbol{\theta})$

Appendix

TABLE V: Symbols used for HOI-Retarget.

Metric	Definition
Body pose deviation ( ∘ )	Mean over frames of the mean angle between the robot’s and the human’s $16$ bone directions, taken in the pelvis frame for the legs and the trunk frame for the arms.
Link vel. direction ( ∘ )	Mean angle between the robot’s and the human’s object-relative link velocity $\boldsymbol{R}^{w\top}_{o}(\dot{\boldsymbol{p}}^{w}-\dot{\boldsymbol{p}}^{w}_{o})$ , over the palms, ankles, head and pelvis, where both exceed $2$ cm/s.
Contact-point gap (m)	Mean distance on the object between the robot’s palm and the human’s contact point ( 1 ), over frames in contact. This is the residual $E^{\mathrm{c}}$ minimizes.
Rel. hand-orient. change ( ∘ )	Mean angle between the robot’s and the human’s change in palm orientation in the object frame since the first frame of the contact segment.
Object path err. (m)	Mean distance between the robot-side and the human-side object translation, each taken relative to its own first frame.
Body jerk (m/s 3 )	Mean magnitude of the third difference of world link position, over links and frames.

Appendix

TABLE VI: Evaluation metrics reported in Tables III and IV .

Horizon	Coll.	Solved	Time (s)	RAM (MB)	Link vel. dir. ( ∘ )	Joint jerk (rad/s 3 )
Per-frame	✗	130/130	37.7	1440	40.3	588
Windowed	✗	130/130	39.4	1763	24.4	251
Windowed	✓	130/130	159.9	3876	24.5	257
Full traj.	✗	130/130	45.8	2400	23.5	185
Full traj.	✓	87/130	210.5	8671	24.4	186

Appendix

TABLE VII: Optimization horizon, with and without the collision cost.

Explore similar work

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions
Sep 15, 2026Liu Cao, Xingze Wu, Jingzhi Cui +4Contact-Rich Dexterous Manipulation Humanoid Loco-Manipulation
1Tsinghua IIIS · 3Dalian University of Technology · 4The Chinese University of Hong Kong +1
OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport
Sep 29, 2026Guillaume Besset, Erwann Carn, Timothée Carecchio +5Cross-Embodiment Retargeting Motion Model
Inria, Département d’Informatique de l’École Normale Supérieure, PSL Research University, Paris, France
GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training
Jun 11, 2026Zhihai Bi, Qiang Zhang, Guoyang Zhao +8Hand-Object Interaction Detection Long Video Generation
The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1

Sep 15, 2026, cs.ROJ/K move · Enter open · S save

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu+3

1Tsinghua IIIS · 3Dalian University of Technology · 4The Chinese University of Hong Kong+1

Contact-Rich Dexterous Manipulation Humanoid Loco-Manipulation Articulated Objects Dexterous Manipulation Whole-Body Weave

Open paper AskPDF arXiv

Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: https://xiaohu-art.github.io/Weave/

Abstract

Explore similar work

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

OTRetarget: Joint Robot and Object Motion Retargeting via Optimal Transport

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training