VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction

Yuto Shibata; Kotaro Amaya; Kashu Yamazaki; Lalit Jayanti; Yoshimitsu Aoki; Mariko Isogawa; Katerina Fragkiadaki

cs.ROOct 20, 2026

VLM-Guided Contact Retargeting and Generative Partner Modeling for Humanoid-Human Interaction

Authors: Yuto Shibata, Kotaro Amaya, Kashu Yamazaki, Lalit Jayanti, Yoshimitsu Aoki, Mariko Isogawa, Katerina Fragkiadaki

Organizations: Carnegie Mellon University · Keio AI Research Center · Keio University

Abstract

Humanoid-human interaction requires robots to operate under severe partial observability: during close physical contact, the human partner is occluded, truncated, or outside the robot's field of view, causing pose estimation to fail and often necessitating external tracking or privileged human state. At the same time, heavy mutual occlusions introduce errors into the human-human motion-capture data used for learning. We present REACT, a framework for learning deployable humanoid-human interaction policies from human-human demonstrations. REACT treats semantic contact structure as the key supervisory signal connecting demonstration processing, interaction learning, and onboard control. First, a VLM-guided contact modeling and retargeting pipeline identifies semantically meaningful contacts and transfers them onto humanoid-human geometry while preserving interaction structure. Using these annotations, privileged interaction policies learn physically grounded behaviors through interaction-aware contact rewards and are then distilled into deployable egocentric depth policies trained entirely from onboard sensing using photorealistic human depth rendering. To address severe partial observability, we introduce generative partner-state modeling objectives that predict partner motion from occluded egocentric views. We evaluate REACT in simulation and on a real Unitree G1 humanoid across five contact-rich interactions. Semantic contact modeling substantially improves preservation of demonstrated interactions, while generative partner-state modeling improves contact recall on four of five interactions, most under heavy occlusion. On real hardware, REACT performs all five interactions using onboard sensing alone.

Figures & tables

Figure 2: Overview of REACT. REACT first annotates semantic contacts in human–human demonstrations with a VLM and retargets them onto the humanoid while preserving them. A privileged teacher then learns the interaction with contact rewards and is finally distilled into an egocentric depth student that predicts the partner’s motion and runs on the real robot.

Method	Jitter $\downarrow$	Penetration $\downarrow$	Recall $\uparrow$	Precision $\uparrow$	User Study $\downarrow$
	(m/s 3 )	(cm)	(%)	(%)	(rank 1–4)
GMR	$\mathbf{134.1}_{\pm 96}$	$2.14_{\pm 3.05}$	$0.42_{\pm 3.70}$	$2.26_{\pm 9.72}$	$3.61_{\pm 0.04}$
OmniRetarget	$560.4_{\pm 341}$	$1.31_{\pm 1.85}$	$22.13_{\pm 34.24}$	$12.51_{\pm 18.47}$	$2.57_{\pm 0.04}$
Ours w/o Stage 2 co-opt	$153.7_{\pm 71}$	$1.31_{\pm 1.73}$	$13.39_{\pm 26.56}$	$11.79_{\pm 19.58}$	—
Ours w/o VLM filter	$161.7_{\pm 76}$	$1.04_{\pm 1.11}$	$58.02_{\pm 41.71}$	$11.60_{\pm 14.39}$	$2.06_{\pm 0.03}$
Ours	$155.4_{\pm 71}$	$\mathbf{0.96}_{\pm 1.10}$	$\mathbf{59.43}_{\pm 38.91}$	$\mathbf{14.17}_{\pm 15.46}$	$\mathbf{1.76}_{\pm 0.03}$

Table 1: Contact-aware retargeting

	High-five		Hug		Link-arms
Method	MPJPE $\downarrow$	Contact $\uparrow$	MPJPE $\downarrow$	Contact $\uparrow$	MPJPE $\downarrow$	Contact $\uparrow$
	(cm)	(%)	(cm)	(%)	(cm)	(%)
Ours	6.5	$\mathbf{70.9\pm 0.3}$	8.3	$\mathbf{53.1\pm 0.5}$	7.2	$\mathbf{74.8\pm 0.1}$
(-) mesh contact reward	6.5	$39.5\pm 0.3$	9.5	$31.4\pm 0.1$	8.6	$40.5\pm 0.4$
(-) VLM contact filtering	6.6	$67.7\pm 0.2$	8.1	$46.1\pm 0.2$	7.7	$73.4\pm 0.4$

Table 2: Teacher tracking results. Contact recall at 15 cm, mean

\pm

std over 3 seeds.

Method	Hug	High-five	Hand-shake	Link-arms	Pat-on-back
Ours	34.5	13.1	82.0	67.8	51.1
(-) future generation	43.2	14.0	75.2	42.5	44.2
generation $\to$ regression	31.3	33.9	75.2	44.5	44.4
(-) pose prediction	20.4	15.0	64.3	46.9	38.4

Table 3: Student policy results. Contact recall (%) at 30 cm on the held-out evaluation split, mean over 5 room layouts.

Figure 3: Within each interaction, frames are split into moments when the partner is hard to see (blue), either too close to fit in the view or largely out of view, and the remaining frames (gray). (Appendix C.4 ).

Panel of Figure 4 — Figure 4: Student rollouts on held-out motions in simulation (inset: egocentric depth input)

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Generalization across partner distance and orientation. REACT performs pat-on-back with the partner at $70\,\text{cm}$ and $130\,\text{cm}$ and with the partner back-on or side-on to the robot; the per-interaction student reaches and holds the back contact in every condition from onboard depth alone.

Figure 6: Long-horizon behavior: three consecutive hugs in a single episode. A continuous ${\sim}30\,\text{s}$ rollout in which the policy completes the hug interaction three times in a row, with no reset or re-initialisation between repetitions. The ten snapshots are read in boustrophedon order, following the arrows. After each embrace the robot releases the partner, re-approaches, and re-establishes contact, so the three hug cycles appear as three separate close-contact phases across the panels. The policy sustains repeated contact-rich interaction over a long horizon rather than a single one-shot contact.

High-five	Hug	Hand-shake	Link-arms	Pat-on-back	Total
5/10	8/10	5/10	5/10	8/10	31/50

Appendix

Table 4: Real-world success rate. Successful / total trials per interaction on the physical Unitree G1, ten trials per interaction.

Figure 7: VLM skill-selection state machine. Solid boxes are states and dashed boxes are the policies they invoke: the HOMIE locomotion controller for Search, Approach, and Retreat, and one of the five per-interaction students for Interact.

Figure 8: One continuous rollout with VLM skill selection. No interaction is specified in advance. From left to right: the partner opens their arms, and the VLM selects the hug student; the partner is then displaced mid-rollout, and the robot re-finds them, approaches, and switches to the hand-shake student, all without a reset.

Figure 9: In-loop human depth rendering closes the sim-to-real gap. Egocentric depth of a hug frame. Capsule : the partner as collision primitives, blocky and non-human. SMPL-X Mesh : the partner rendered as a realistic body mesh. SMPL-X Mesh + Scene (Ours) : our full rendering with the randomised scene. This last variant closely resembles the real-sensor depth shown in Figure 1 of the main paper.

Method	Hug	Hand-shake	Pat-on-back	Link-arms	High-five
Ext. human pose (HMR2.0 [ 10 ] )	5.7	8.2	1.3	3.9	5.4
Ours (E2E)	38.0	78.6	23.8	30.1	15.3

Appendix

Table 5: End-to-end vs. external human-pose estimation. Mesh-contact recall at a

30\,\text{cm}

threshold, in percent, on the held-out evaluation split. Ext. human pose (HMR2.0) drives the policy from partner state produced by an external human-pose estimator while Ours (E2E) outputs actions end-to-end from egocentric depth and proprioception. Both rows use earlier model settings than Table 3 and are evaluated on the same held-out motions.

Figure 10: Why the external pose estimation-based method fails at contact range. Temporally ordered snapshots of an external-pose rollout, read left to right, with the ground-truth partner ( green , Human (Sim) ) and the HMR2.0 reconstruction ( red , HMR2.0 Prediction ) overlaid against the robot (black). Left three: high-five; as the partner’s hand leaves the robot’s egocentric field of view, the reconstructed hand position deviates sharply from the ground truth at the moment of contact. Right three: hug; at the instant of embrace the partner’s torso occludes the camera, and the reconstruction collapses into a large error behind the true mesh. The estimator degrades precisely in the close-contact, self-occluded regime that REACT’s end-to-end student handles directly from depth.

Interaction	Regime	Share	Seeds 1 / 2 / 3	Mean
Hug	PO (occupancy $\geq 0.8$ )	26%	$+7.4$ / $+5.7$ / $+7.7$	$\mathbf{+6.9}$
Hug	otherwise	74%	$+2.3$ / $+4.2$ / $+1.6$	$+2.7$
Pat-on-back	PO (occupancy $\geq 0.8$ )	22%	$+12.4$ / $+12.1$ / $+6.2$	$\mathbf{+10.2}$
Pat-on-back	otherwise	78%	$+0.1$ / $-1.6$ / $+4.6$	$+1.0$
Link-arms	PO (occupancy $<0.4$ )	70%	$+35.9$ / $+34.1$ / $+31.4$	$\mathbf{+33.8}$
Link-arms	otherwise	30%	$+10.9$ / $-6.1$ / $+7.6$	$+4.1$

Appendix

Table 6: Contact recall gain of Ours over the deterministic variant (pt at 30 cm) by occupancy. Scene randomisation disabled, held-out split. PO: partially observable regime. Share: fraction of contact frames in the regime.

	15 cm			30 cm
Interaction	T	S	Rec.	T	S	Rec.
Hug	52.3	18.1	35%	80.2	49.9	62%
High-five	69.5	23.9	34%	87.7	62.1	71%
Hand-shake	93.9	78.6	84%	96.7	91.1	94%
Link-arms	72.5	44.1	61%	89.9	74.1	82%
Pat-on-back	87.0	38.4	44%	93.9	61.4	65%

Appendix

Table 7: Teacher vs. student contact recall (%) on matched motions. T: privileged teacher, S: egocentric depth student, Rec.: S/T. The mean recovery averages the per-interaction ratios.

Figure 11: VLM contact filtering, qualitative examples. Distance candidates and their VLM verdicts on the five contact-rich Inter-X actions (rows: hug, hand-shake, pat-on-back, high-five, link-arms), sampled at $1\,\text{s}$ intervals over a $4\,\text{s}$ window (columns). Green markers are candidates kept after VLM verification, labelled essential or incidental. Red markers are Stage 1 distance candidates the VLM removes as spurious near-contacts. The proximity test fires on many transient arm passes; the VLM stage prunes them while preserving the genuine contacts, which is the precision gain quantified in Tab. 8 .

	Ours w/o VLM filtering (distance)			Ours (VLM-verified)
Action	P $\uparrow$	R $\uparrow$	$F_{1}$ $\uparrow$	P $\uparrow$	R $\uparrow$	$F_{1}$ $\uparrow$
Hug	$0.060_{\pm 0.05}$	$0.241_{\pm 0.22}$	$0.127_{\pm 0.08}$	$0.081_{\pm 0.10}$	$0.201_{\pm 0.21}$	$\mathbf{0.181}_{\pm 0.13}$
Hand-shake	$0.275_{\pm 0.15}$	$0.785_{\pm 0.38}$	$0.495_{\pm 0.10}$	$0.442_{\pm 0.33}$	$0.661_{\pm 0.41}$	$\mathbf{0.674}_{\pm 0.21}$
Pat-on-back	$0.107_{\pm 0.07}$	$0.824_{\pm 0.34}$	$0.212_{\pm 0.09}$	$0.227_{\pm 0.12}$	$0.760_{\pm 0.35}$	$\mathbf{0.387}_{\pm 0.12}$
High-five	$0.260_{\pm 0.14}$	$0.866_{\pm 0.32}$	$0.417_{\pm 0.16}$	$0.336_{\pm 0.27}$	$0.660_{\pm 0.46}$	$\mathbf{0.626}_{\pm 0.17}$
Link-arms	$0.257_{\pm 0.16}$	$0.600_{\pm 0.31}$	$0.374_{\pm 0.18}$	$0.339_{\pm 0.20}$	$0.549_{\pm 0.34}$	$\mathbf{0.453}_{\pm 0.18}$

Appendix

Table 8: VLM contact-label quality on

80

manually-annotated Inter-X clips (

16

per action). Per-action and total precision, recall, and

F_{1}

as across-clip mean

\pm

std under per-frame vertex-distance matching at

\epsilon=10\,\text{cm}

on the SMPL-X rest-pose template, after symmetric-pair deduplication. Ours w/o VLM filtering uses the raw Stage 1 distance candidates; Ours additionally applies VLM verification. Reporting precision and recall separately shows that the VLM gain is almost entirely a precision gain, while recall is essentially preserved (or only mildly reduced). The verifier removes spurious near-contacts rather than acting as a stricter detector.

Figure 12: Body correspondence projects the SMPL-X mesh onto either the G1 or the partner-capsule surface. The offline registration introduced in Sec. D.2 establishes a dense body correspondence that maps every SMPL-X body vertex to a $(\text{link},\,\mathbf{p}_{\text{local}})$ pair, so the same mesh can be projected onto either target surface, the Unitree G1 robot or the partner’s SMPL-X collision capsules. The three columns show one identical set of colour-coded SMPL-X vertices in three configurations, from left to right: projected onto the G1 robot surface, in their SMPL-X template positions, and projected onto the partner capsule surface. Each colour denotes the G1 link a vertex is registered to, and the connecting lines join the same representative vertices across the three configurations, so that any contact vertex can be carried onto the corresponding robot or capsule surface by forward kinematics. The columns are drawn at true relative scale, exposing the proportion mismatch between the human template and the shorter G1 that the registration must bridge. The dense map covers the full template at a mean vertex-to-surface residual of $0.89\,\text{mm}$ , well below the $\epsilon=10\,\text{cm}$ contact threshold.

Figure 13: Qualitative retargeting comparison. Rows: the five contact-rich actions (hug, hand-shake, pat-on-back, high-five, link-arms). Columns: the original human–human demonstration and four retargets (GMR, OmniRetarget, Ours w/o VLM filtering, Ours), each at the action’s peak-contact frame. The red arrow marks the demonstrated contact the retarget must reproduce on the G1. Ours establishes the touch on the robot surface where the baselines reach short or push apart.

Method $\backslash$ $\epsilon$ (cm)	$3$	$5$	$10$	$15$	$20$	$25$	$30$
GMR	0.0	0.0	0.4	0.6	1.3	4.9	10.6
OmniRetarget	2.7	4.9	22.1	43.3	58.4	77.3	86.2
Ours w/o Stage 2 co-opt	1.7	5.0	13.4	33.6	54.4	72.4	79.4
Ours w/o VLM filtering	12.1	23.1	58.0	79.3	87.2	91.3	97.2
Ours	12.1	27.1	59.4	80.0	92.0	94.3	95.9

Appendix

Table 9: Retargeting Recall vs. contact threshold

\epsilon

(cm) (per-event, across-clip mean, %). The

\epsilon{=}10

column reproduces the Recall column of the main paper’s retargeting table. Relaxing

\epsilon

shows that for Ours the contact is in fact established but slightly displaced: Recall rises from

59.4\%

10\,\text{cm}

80.0\%

15\,\text{cm}

and

92.0\%

20\,\text{cm}

	Hug		Hand-shake		Pat-on-back		High-five		Link-arms		Total
Method	R	P	R	P	R	P	R	P	R	P	R	P
GMR	$0.0$	$1.3$	$0.0$	$0.0$	$0.0$	$7.6$	$0.0$	$0.0$	$2.1$	$1.9$	$0.4$	$2.3$
OmniRetarget	$17.4$	$4.5$	$20.8$	$\mathbf{21.8}$	$16.7$	$3.4$	$9.4$	$24.5$	$54.2$	$7.6$	$22.1$	$12.5$
Ours w/o Stage 2 co-opt	$17.5$	$5.2$	$0.0$	$19.6$	$9.4$	$\mathbf{8.2}$	$6.2$	$24.1$	$33.9$	$\mathbf{9.7}$	$13.4$	$11.8$
Ours w/o VLM filtering	$\mathbf{38.1}$	$6.4$	$\mathbf{84.4}$	$15.3$	$42.4$	$5.0$	$49.0$	$27.8$	$\mathbf{75.0}$	$4.5$	$58.0$	$11.6$
Ours	$31.7$	$\mathbf{6.8}$	$81.2$	$19.7$	$\mathbf{42.9}$	$5.9$	$\mathbf{70.8}$	$\mathbf{33.3}$	$68.8$	$5.2$	$\mathbf{59.4}$	$\mathbf{14.2}$

Appendix

Table 10: Per-action retargeting Recall and Precision (

\epsilon{=}10\,\text{cm}

, per-event, across-clip mean

{}_{\pm\text{std}}

, %). For each action the two columns are Recall (R, vertex-pair-targeted) and Precision (P, body-part

\tau_{0}

); the Total columns equal the Recall and Precision columns of the main paper’s retargeting table. Recall uses

16

GT clips per action (

58

in total for OmniRetarget, whose collision solver makes the rest infeasible); Precision averages over clips with at least one predicted event.

Figure 14: User-study web interface (one trial). The original demonstration ( $3$ -view orbit, top) and the four blinded retarget conditions A–D (middle) are played synchronously; the participant gives a strict $1$ – $4$ ranking for each of the three questions (bottom). Full ranking criteria are in Sec. E.5 .

Method	Q1: G1 contact	Q2: Human contact	Q3: Overall
	+ motion $\downarrow$	+ motion $\downarrow$	preference $\downarrow$
GMR	$3.52\pm 0.07$	$3.45\pm 0.07$	$3.61\pm 0.04$
OmniRetarget	$2.64\pm 0.04$	$2.49\pm 0.05$	$2.57\pm 0.04$
Ours w/o VLM filtering	$2.05\pm 0.04$	$2.18\pm 0.05$	$2.06\pm 0.03$
Ours	$\textbf{1.79}\pm 0.04$	$\textbf{1.88}\pm 0.05$	$\textbf{1.76}\pm 0.03$

Appendix

Table 11: User study: per-question mean ranks

\pm

SEM (

n{=}28

participants,

80

sequences each, lower is better). Each entry is the across-participant mean of per-participant per-condition mean ranks. Ranks lie in

\{1,2,3,4\}

with

1{=}\text{best}

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 453 papers

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction
Sep 28, 2026Jihwan Shin, Adrià López Escoriza, Junzhe He +2Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics
Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland
HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization
Sep 18, 2026Lalit Jayanti, Kashu Yamazaki, Yuto Shibata +2Robot Motion Generation Humanoid Robot Control
Carnegie Mellon University · Keio AI Research Center · Keio University
Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions
Sep 15, 2026Liu Cao, Xingze Wu, Jingzhi Cui +4HOI Generation Humanoid Loco-Manipulation
1Tsinghua IIIS · 3Dalian University of Technology · 4The Chinese University of Hong Kong +1

Sep 28, 2026J/K move · Enter open · S save

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

Jihwan Shin, Adrià López Escoriza, Junzhe He, Matthias Heyrman+1

Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics Motion Retargeting Humanoid Robotics Human-Object Interaction

Open paper AskPDF arXiv

Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object. We publicly release the code and the retargeted motion dataset.

Abstract

Human-Robot Interaction

Explore similar work

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction

HIGenNTO: Scalable Humanoid Interaction Generation via Noise-Space Trajectory Optimization

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions