ReContact: Reconstruct and Retarget Human Contacts from Video for Contact-Rich Humanoid Manipulation

Zhongkai Gu; Yang Liu; Wei Zhu; Mitsuhiro Hayashibe

cs.RONov 2, 2026

ReContact: Reconstruct and Retarget Human Contacts from Video for Contact-Rich Humanoid Manipulation

Authors: Zhongkai Gu, Yang Liu, Wei Zhu, Mitsuhiro Hayashibe

Organizations: The authors are with the Department of Robotics, Graduate School of Engineering, Tohoku University, Sendai 980-8579, Japan.

Abstract

Contact transitions allow humanoid robots to redistribute object support across the whole body, enabling manipulation beyond hand-only grasps. Human videos provide a scalable source of such transitions for robots to learn from. However, body occlusion disrupts object pose tracking and corrupts retargeted motion. Moreover, accurately reproducing the object trajectory does not ensure that the robot applies stabilizing forces at the intended contacts, risking slips or drops in later phases. To address these challenges, we propose ReContact, which reconstructs and retargets human-object interactions through an object-centric contact plan. The plan specifies contact phases, contact pairs, anchors, pressing directions, and sticking or sliding modes. Contact priors stabilize object pose reconstruction through occlusion. A residual policy over a frozen whole-body tracker is rewarded for making contact at planned anchors with forces aligned to the specified directions. In simulation, on a thigh-supported regrasp task, ReContact achieves the planned two-palm bottom hold in 83.5 of episodes over three training seeds, versus 0.16% for the best adapted baseline. One policy trained on 247 references re-solved for new placements raises in-workspace tube-transfer success from 20.4% (source reference only) to 97.8%. On a real Unitree G1, policies execute seven contact-rich manipulation and two loco-manipulation tasks.

Figures & tables

Fig. 1: Overview of ReContact. (a) An exocentric video supplies body and object motion; Fig. 2 shows the egocentric route. (b) GPT-6 Astra reviews sparse keyframes against the video, and contact-constrained whole-body optimization turns them into humanoid–object references. (c) The object-centric contact plan, here for the thigh-supported regrasp, lists, per phase, the paired robot–object surfaces with their contact points, contact directions and sticking or sliding modes; adding or removing a pair gives a contact transition. Keyframes of the run shown: contact keyframes where support changes ( $\blacklozenge$ ) and motion keyframes inside phases ( $\lozenge$ ). An objective that does not specify the pressing direction also accepts a squeeze near the lower side edges ( $\times$ ). (d) The plan defines the rewards of a residual controller for contact at the planned point and along the planned direction. (e) Re-solving the placement-dependent contact segment and its approach at new object placements gives a task family for one policy.

Method	Data	Contact representation	Body support	Direction	Stick/ slide	Reference generation
HDMI [ 21 ]	Video, MoCap	Hand and foot points in the object frame; per-frame contact flag	–	–	–	–
HumanX [ 16 ]	Video	Body-anchor–object transform; binary contact labels	–	–	–	✓
PRISM [ 1 ]	Video	Palm anchors in the object frame, fixed within a phase	–	–	–	✓
VICAR [ 20 ]	Video	Annotated hand and foot poses in the world frame	–	–	–	✓
CHORD [ 5 ]	Video, MoCap	Hand contact points and normals as an object-centric wrench space	–	✓	–	–
OmniRetarget [ 19 ]	MoCap	Interaction mesh over body joints, object and terrain points	–	–	–	✓

TABLE I: Comparison of Humanoid Manipulation Methods and How They Specify Contacts and Whether They Generate References

Fig. 2: Egocentric route, shown for a spray bottle and a banana. Recovered hand–object segments are retargeted to the Inspire RH56F1 hand; planning generates the approach and the remaining G1 body motion. Blue paths mark the approach, orange hands the grasp targets. Recovered views use the source camera and crop; the last column shows the composed reference at the same source frame.

Panel of Figure 4 — Fig. 3: Evaluated tasks in simulation; arrows give the temporal order. T5 shows one policy trained on $247$ generated references grasping the tube at four of their positions (tube $xy$ in the world frame).

Method	Complete	SR	Thigh	Bottom L	Bottom R	Normal $\downarrow$	$E_{o}\downarrow$	Mech. work $\downarrow$
	(%)	(%) $\uparrow$	(%)	(%)	(%)	(deg)	(cm)	(J)
HDMI-style	$24.3\pm 20.0$	$0.08\pm 0.07$	$0.2\pm 0.3$	$10.6\pm 16.1$	$3.5\pm 4.2$	$47.8\pm 4.6$	$11.24\pm 1.06$	$565.2\pm 90.1$
ResMimic-style	$65.7\pm 5.8$	$0.01\pm 0.02$	$20.6\pm 5.6$	$0.3\pm 0.4$	$4.5\pm 7.1$	$55.0\pm 3.0$	$9.47\pm 1.29$	$868.0\pm 21.1$
CWS adapter	$52.3\pm 9.2$	$0.16\pm 0.24$	$36.6\pm 25.1$	$0.6\pm 0.5$	$65.7\pm 56.2$	$49.2\pm 2.8$	$9.25\pm 2.08$	$787.1\pm 59.9$
Ours	$\mathbf{89.0\pm 8.6}$	$\mathbf{83.52\pm 5.20}$	$\mathbf{94.4\pm 2.8}$	$\mathbf{80.8\pm 1.8}$	$\mathbf{81.2\pm 7.6}$	$\mathbf{16.3\pm 4.5}$	$\mathbf{5.54\pm 0.64}$	$\mathbf{544.9\pm 20.2}$

TABLE II: Shared-Reference Controller Comparison in Simulation

Fig. 4: Only ReContact moves the T2 box from thigh support to a two-palm bottom hold. Vertical contact force (normal plus friction) exerted on the box by the right thigh and each palm, divided by the box weight

mg

(dashed line), in

48

episodes per controller. Lines are medians and bands interquartile ranges over the episodes alive at each reference time (

0.1

s moving median within each episode); the axis is logarithmic beyond

\pm mg

. Shading marks the palm slide and the terminal bottom hold. Panel titles give two-palm successes.

Fig. 5: The planned contact transition persists across payload variation. (a) Completion of ReContact and the ResMimic-style, HDMI-style and CWS controllers at eight payloads, $3000$ episodes per method and mass; shading marks the mass randomization used in training ( $0.074$ – $0.135$ kg around the $0.1$ kg nominal mass). (b) Completed episodes pooled over the payloads, split by the bottom-hold criterion: met by both palms (the T2 success predicate), by one palm, or by neither (side squeeze). Images: terminal holds of completed $0.5$ kg episodes (ReContact, CWS, ResMimic-style).

Fig. 6: Contact preservation in retargeted humanoid–object references. (a) Comparison of retargeting methods on a medium-box carry using shared body motion and object tracks. (b) Comparison of object tracks for the tissue-pack transfer and thigh-supported regrasp using the same ReContact retargeting procedure.

Fig. 7: Workspace coverage from one reference (T5 tube-transfer family). Top: success of the task-family policy, smoothed with a $1.5$ cm Gaussian kernel, inside the generated workspace (outline) and in the $6$ cm band outside it; white points mark the $247$ references and the star the source reference’s tube position. Bottom: per-position outcomes of ReContact policies of the same configuration trained on the source reference (local, $\pm 2.5$ cm; workspace-wide; no start randomization) and on the family.

References	Start	$0$ – $10$	$10$ – $20$	$20$ – $30$	$>30$	All	Ext.
Source	None	$11.8$	$0.0$	$0.0$	$0.0$	$1.5$	$0.1$
Source	$\pm 2.5$ cm	$95.5$	$25.3$	$0.1$	$0.0$	$20.4$	$8.4$
Source	Workspace	$12.2$	$0.0$	$0.0$	$0.0$	$1.6$	$0.3$
Family ( $247$ )	$\pm 2.5$ cm	$\mathbf{99.3}$	$\mathbf{98.0}$	$\mathbf{96.6}$	$\mathbf{98.6}$	$\mathbf{97.8}$	$\mathbf{19.5}$

TABLE III: T5 Tube-Transfer Success (%) of Single-Reference Controls and the Family Policy by Distance (cm) From the Source Tube Position

Panel of Figure 11 — Fig. 8: Hardware deployment of learned contact skills on a Unitree G1. (a) Walk-then-grasp using locomotion and the bottle-family policy. (b) Table-to-floor box transfer combining forward and reverse thigh-supported regrasps with loaded walking.

Panel of Figure 12 — Fig. 9: Final hardware states of seven contact-rich manipulation tasks on a Unitree G1.

Contact-Rich Robotic Manipulation

Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.

See all 438 papers

SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction
Jun 25, 2026Sirui Chen, Shibo Zhao, Zhen Wu +3Robotic Control Humanoid Robot Locomotion
Stanford, Amazon FAR United States · Amazon FAR United States · CMU, Amazon FAR United States
ContactMimic: Humanoid Object Interaction via Contact Control
Jul 9, 2026Xinyao Li, Xialin He, Runpei Dong +1Robotic Control Robot Skill Learning
University of Illinois Urbana-Champaign
HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction
Sep 28, 2026Jihwan Shin, Adrià López Escoriza, Junzhe He +2Contact-Rich Robotic Manipulation Synthetic Data Generation for Robotics
Robotic Systems Lab, ETH Zürich, 8092 Zürich, Switzerland

Jun 25, 2026J/K move · Enter open · S save

SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li+2

Stanford, Amazon FAR United States · Amazon FAR United States · CMU, Amazon FAR United States

Robotic Control Humanoid Robot Locomotion Contact-Rich Robotic Manipulation Humanoid Robot Control Robotic RL Humanoid Motion Tracking

Open paper AskPDF arXiv

Current humanoid reinforcement-learning policies excel at free-space motions but struggle with contact-rich tasks, as pure kinematic tracking cannot resolve the physical ambiguities of interacting with objects and uneven terrain. To address this, we introduce SceneBot, a unified motion-tracking framework capable of handling freespace locomotion, terrain traversal, and whole-body manipulation. SceneBot conditions a single policy on both reference motions and per-link contact labels, explicitly defining expected environmental interactions. To overcome the lack of annotated interaction data, we propose a hindsight scene reconstruction approach that infers scene-interaction graphs from retargeted human motion. Trained on 7.5 hours of this reconstructed, contact-rich data, SceneBot successfully generalizes to unseen motions and environments. Our results demonstrate that SceneBot is the first general framework to seamlessly unify free-space and contact-rich behaviors executing complex, long-horizon tasks like carrying a box upstairs and establishing contact conditioning as a powerful interface for humanoid control. All code and data will be open-sourced. More demos and information are available at: https://ericcsr.github.io/scenebot/

Abstract

Contact-Rich Robotic Manipulation

Explore similar work

SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

ContactMimic: Humanoid Object Interaction via Contact Control

HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction