Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics

Zetao Du; Weiqing Wang; Hanbing Zhang; Yimiao Zhou; Ye Shi; Liang Wang; Jingya Wang

cs.ROOct 11, 2026

Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics

Authors: Zetao Du, Weiqing Wang, Hanbing Zhang, Yimiao Zhou, Ye Shi, Liang Wang, Jingya Wang

Organizations: ShanghaiTech University · InstAdapt · Institute of Automation, Chinese Academy of Sciences

Abstract

Human demonstrations can capture task-relevant manipulation without using the target robot, but the recorded hand motions cannot directly serve as robot action supervision. Hand reconstruction and motion retargeting provide a direct route to such supervision, yet recovering hand motion alone does not ensure physically consistent hand-object interactions. To address this issue, we introduce Hand2Robot, an object-centric 4D hand reconstruction framework for robot policy post-training. Given videos of humans performing the target task, Hand2Robot extracts interaction clips, reconstructs object geometry, and generates robot action supervision through hand-object interaction optimization and motion retargeting. For object tracking, we combine observation confidence with temporal motion constraints to select a globally consistent object pose trajectory from per-frame pose candidates. For hand refinement, we use object geometry and motion as a reference and combine staged 2D alignment with object-local contact constraints to refine hand trajectories and finger joint configurations. Experiments on DexYCB and HO3D-v3 show improved object pose accuracy and hand-object interaction quality. Experiments on three real-robot dexterous manipulation tasks further show that reconstructed and retargeted human demonstrations can effectively supplement limited robot data for target-task policy post-training.

Figures & tables

Figure 1: Overview of the Hand2Robot pipeline. Given human demonstration videos, we first parse them into several semantic tasks by VLM. Then we apply our hand-object reconstruction and refinement framework for physically realistic hand trajectories. Finally, these hand trajectories are retargeted to the robot action space, enabling human-robot data co-training on a Vision-Language-Action model for real-world deployment.

Figure 2: Overview of our object-centric 4D hand reconstruction pipeline.

Method	Translation error (cm) $\downarrow$	Rotation error ( ∘ ) $\downarrow$	ADD AUC $\uparrow$	ADD-S AUC $\uparrow$
FoundationPose ( Wen et al., 2024 )	6.80	27.14	98.96	99.15
Ours w/o $d,c$	4.88	61.63	99.07	99.44
Ours w/o $d$	3.62	38.20	99.37	99.61
Ours w/o $c$	3.44	19.86	99.47	99.62
Ours	3.44	18.47	99.48	99.62

Table 1: Quantitative object pose results on DexYCB using reconstructed meshes and first-frame alignment. See Appendix C for evaluation details.

Dataset	Method	MPJPE $\downarrow$	MRPE $\downarrow$	IV-mean $\downarrow$	IV-max $\downarrow$	JM $\downarrow$
DexYCB ( Chao et al., 2021 )	WiLoR ( Potamias et al., 2025 )	8.35	97.31	2.57	5.41	2.58
	Ours	7.58	38.49	1.07	3.25	2.49
HO3D-v3 ( Hampali et al., 2020 )	WiLoR	4.22	24.22	20.11	64.82	0.86
	Ours	4.40	16.99	3.42	22.40	0.34

Table 2: Quantitative evaluation for in-scene object manipulation. We evaluate on HO3D-v3 and DexYCB, comparing the WiLoR initialization with our refined hand trajectories. We highlight the best values in bold.

Figure 3: Examples of real-robot manipulation tasks. Each column shows one task, with the three rows corresponding to its initial, intermediate, and successful final states. The tasks are Place Vase, Open Box, and Insert Toast, from left to right.

Training data and method	Place Vase	Open Box	Insert Toast	Mean
20R	45.0	60.0	40.0	48.3
20R + 100H, WiLoR	25.0	20.0	10.0	18.3
20R + 100H, MagicHOI	40.0	25.0	15.0	26.7
20R + 100H, Hand2Robot	80.0	85.0	75.0	80.0

Table 3: Task success rates (%) after policy post-training with different sources of human action supervision. R and H denote the numbers of robot and human demonstrations per task. The mixed-data conditions use the same 20 robot demonstrations and 100 source human videos. Mean is the unweighted average across the three tasks.

Method	Robot R	Human H	Place Vase	Open Box	Insert Toast	Mean
Robot-only	20	0	45.0	60.0	40.0	48.3
Robot-only	40	0	60.0	65.0	55.0	60.0
Robot-only	60	0	80.0	90.0	75.0	81.7
Do As I Do ( Paliwal et al., 2026 )	20	50	50.0	45.0	40.0	45.0
Hand2Robot	20	50	50.0	70.0	45.0	55.0
Hand2Robot	20	100	80.0	85.0	75.0	80.0

Table 4: Success rates (%) under different demonstration budgets and action-supervision sources. R and H denote the numbers of robot and human demonstrations per task, respectively. Do As I Do includes Sharpa-to-xHand adaptation after dynamics-based post-optimization. Mean is the unweighted average across the three tasks. Conditions shared with Table 3 use the same results.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	CD (cm 2 ) $\downarrow$	F@5 (%) $\uparrow$	F@10 (%) $\uparrow$
HOLD ( Fan et al., 2024 )	19.30	33.20	54.94
MagicHOI ( Wang et al., 2025 )	2.05	45.67	67.14
Ours	0.57	49.35	87.54

Appendix

Table 5: Object reconstruction quality on DexYCB. Lower CD and higher F-scores are better. Best values are highlighted in bold.

VLM	Detector	Valid (%) $\uparrow$	mIoU $\uparrow$	[email protected] (%) $\uparrow$	[email protected] (%) $\uparrow$	[email protected] (%) $\uparrow$
$\times$	$\times$	20.2	0.08/0.38	13.2/65.3	7.3/36.1	2.5/12.4
$\times$	✓	84.5	0.56/0.66	76.0/89.9	70.2/83.1	42.3/ 50.1
✓	$\times$	87.0	0.58/0.67	83.5/96.0	71.1/81.7	38.8/44.6
✓	✓	95.8	0.66 / 0.69	93.8 / 97.9	81.1 / 84.7	44.8 /46.8

Appendix

Table 6: We ablate the object grounding module. “Valid” denotes the percentage of sequences with a valid prediction and non-zero overlap with the ground-truth mask. Metrics are reported as full/valid, where full counts detection failures and zero-overlap predictions as IoU

=0

, and valid is computed only over valid sequences.

$\mathcal{L}_{\mathrm{2d}}$	$\mathcal{L}_{\mathrm{match}}$	$\mathcal{L}_{\mathrm{temp}}$	MPJPE $\downarrow$	MRPE $\downarrow$	IV-mean $\downarrow$	IV-max $\downarrow$	JM $\downarrow$
$\checkmark$	$\checkmark$	$\checkmark$	7.58	38.49	1.07	3.25	2.49
$\times$	$\checkmark$	$\checkmark$	8.38	113.17	0.27	1.55	9.93
$\checkmark$	$\times$	$\checkmark$	7.64	48.70	0.53	2.27	2.53
$\checkmark$	$\checkmark$	$\times$	7.58	38.62	1.07	3.12	3.19

Appendix

Table 7: Ablation of the optimization losses on DexYCB. We report MPJPE, MRPE, and physics-based measures. The first row uses the full objective, while each following row removes one loss term to assess the role of each constraint.

Dataset	Method	MPJPE $\downarrow$	MRPE $\downarrow$	IV-mean $\downarrow$	IV-max $\downarrow$	JM $\downarrow$
HO3D-v3	MagicHOI	5.90	22.64	13.24	48.80	1.25
	WiLoR	4.22	24.22	20.11	64.82	0.86
	Ours	4.40	16.99	3.42	22.40	0.34

Appendix

Table 8: We compare additional baseline MagicHOI with WiLoR initialization and our refined hand trajectories on HO3D-v3. Best values are highlighted in bold.

Figure 4: Qualitative results of hand-object reconstruction on HO3D-v3. The reconstruction result is visualized in both the front and side views.

Figure 5: Hardware setup of real-world robot experiments.

Explore similar work

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling
Jul 4, 2026Yuanchuan Lai, Qing Gao, Ziyan Liang +2Contact-Rich Robotic Manipulation Motion Retargeting
School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China. · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK.
Touch2Robot: Robot Touch in the Human Demonstration Loop
Sep 21, 2026Shengcheng Luo, Xiaoyang Cheng, Hong Ying +6Robotic Data Collection Motion Retargeting
ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Jiao Tong University +1
Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Sep 23, 2026Tara Sadjadpour, Siming He, C. K. Wolfe +5Zero-Shot Robotic Manipulation Contact-Rich Robotic Manipulation
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA

Jul 4, 2026J/K move · Enter open · S save

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

Yuanchuan Lai, Qing Gao, Ziyan Liang, Junjie Hu+1

School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen 518172, China. · School of Computing, University of Portsmouth, Portsmouth PO1 3HE, UK.

Contact-Rich Robotic Manipulation Motion Retargeting Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

Learning robot dexterous manipulation from human manipulation videos requires reliably retargeting human intent to executable robot actions while maintaining stable hand-object contact, which remains a key challenge in embodied intelligence. Existing retargeting methods often ignore explicit contact modeling or rely on reinforcement learning, resulting in limited accuracy and generalization. To address this, we propose ObjRetarget, a human-to-robot motion retargeting framework for learning robot dexterous manipulation from human videos, which integrates anthropomorphic arm trajectory constraints with structured hand-object geometric modeling. For arm motion, reference trajectories extracted from human videos are used for initialization, followed by anthropomorphic constraints and redundancy-aware optimization to generate natural and accurate movements. For hand manipulation, ObjRetarget represents multi-finger contacts using polytope clusters and preserves contact structure through geometric invariants to improve stability. Experiments on real robots show that ObjRetarget improves manipulation success rates and contact stability across multiple dexterous tasks, and generalizes well to different demonstrations, object poses, and task settings.

Abstract

Explore similar work

ObjRetarget: An Object-Aware Motion Retargeting Framework with Anthropomorphic Arm Constraints and Polyhedral Hand Modeling

Touch2Robot: Robot Touch in the Human Demonstration Loop

Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy