Humans are dexterous, but traditional data collection paradigms such as teleoperation, kinesthetic teaching, and wearable exoskeletons force humans to act through the robot embodiment, bottlenecking their dexterity. Retargeting can avoid this. However, existing methods purely translate kinematics and ignore the real-world contact dynamics essential for task success. We present CARD: Contact-Anchored Retargeting for Dexterous Manipulation, a simulator and object-free framework that retargets human demonstrations to robot actions by explicitly preserving contact force targets. CARD unifies human data sources including video, motion capture, and tactile gloves via a shared hand--contact representation and generates explicit contact force targets in addition to kinematics. To maintain these contact targets during hardware open-loop replay, CARD also introduces a tactile tracking controller. For video which does not contain an explicit tactile source, we present Vid-HACO, our video-to-contact model that sets a new state of the art. Across our video, glove, motion-capture data and the GRAB dataset, CARD maintains contact on 43--78% of contact frames, while kinematic and object-aware retargeting methods maintain contact on <10% and 22.5% of the frames respectively. Further, integrating our formulation into physics-based retargeting improves strict task success from 12% to 32% on GRAB dataset. Finally, we show that our tactile tracker is critical to reproduce demonstrated forces during hardware replay. These results establish that contact preservation via CARD is a scalable interface for transferring human dexterity to robots.
Figures & tables
Figure 2: CARD overview. Video, glove, and motion capture are converted into a unified hand–contact representation, a MANO mesh with one contact force target per semantic link ( Sec. 3.1 ). Geometric initialization maps human joint angles to the robot skeleton, followed by contact-anchored retargeting that aligns the corresponding robot pads with the contact force targets ( Sec. 3.3 ). Joint and contact force targets are replayed by a tactile tracking controller ( Sec. 3.4 ) on hardware.
Figure 3: Demonstration sources. We show some demonstration exmaples from monocular video (Vid-HACO contact), tactile glove (taxel pressure), and motion capture with the tactile glove (taxel pressure).
Figure 4: Contact-anchored vs. kinematic retargeting on two video clips. Per clip: the human source, CARD , DexPilot, AnyTeleop-K, AnyTeleop-P, and CARD ’s geometric initialization (init only), the columns of Tab. 1 . The scanned object is for visualization only, and orange spheres are the contact force target . Kinematic baselines reach the object but do not track the contact.
Figure 5: Tactile tracker. Each finger uses fingertip force feedback to track the desired contact force through an admittance loop whose curls form the joint offset Δq , while feedforward torque directly applies the target force through the joint impedance controller ( Appendix F ).
CARD
DexPilot
AnyTeleop-K
AnyTeleop-P
ReGrind
CARD
[ 22 ]
[ 56 ]
[ 56 ]
[ 18 ] (w/ Object)
(init only)
Our data: monocular video (20 clips)
Contact rate (%) ↑
61.4
9.7
8.3
7.6
–
20.5
Correct region (%) ↑
60.1
9.5
8.0
7.5
–
19.2
In-cone (%) ↑
35.2
4.2
4.7
5.0
–
–
Gap (mm) ↓
2.40
9.99
10.80
13.19
–
5.76
Table 1: Contact fidelity on our data (video, glove, motion capture) and on GRAB. Rates in percent, pooled over anchor-frames. Millimeter columns are medians. Bold : best per bed and metric. Underline : best published baseline. On GRAB we additionally compare against ReGrind ported to SHARPA Wave, which takes the GRAB mesh as input. CARD uses only the contact-reconstructed pseudo-object ( Fig. 12 ), and the GRAB mesh only to score penetration. The glove + motion capture gap combines the two beds’ medians weighted by anchor-frames.
Figure 6: Tactile tracker ablation. We show the same trajectories replayed with (top) and without (bottom) tactile tracking. Tactile tracking maintains the desired contact forces, enabling stable grasps and successful task execution.
Figure 7: Replay on hardware. We show a series of snapshots of CARD trajectories replayed on the SHARPA Wave hand, Wuji Hand 2, and Unitree Dex3-1. The articulated Rubik’s cube and the bending deck of cards use the same pipeline as the rigid objects.
Figure 8: Cross-embodiment transfer. Two GRAB demonstrations (rows) retargeted to SHARPA Wave, Wuji Hand 2, Unitree Dex3-1, and a parallel-jaw gripper (columns, after the human source). Only the semantic map changes. Orange spheres are the contact force targets.
Vertex
Frame
F1 ↑
Prec. ↑
Rec. ↑
AP ↑
F1 ↑
12-dataset validation
HACO
57.1
44.4
80.0
51.7
86.1
HACO-R
74.5
63.4
90.3
70.4
86.1
Vid-HACO
84.3
80.4
88.5
92.4
98.3
HRDexDB (held out)
Table 2: Contact from monocular video. Vid-HACO versus released HACO [ 30 ] and HACO-R (retrained on our data), on 12-dataset validation and held-out HRDexDB [ 37 ] . Percent, threshold 0.52 except for the threshold-free AP. Vertex metrics are pooled over all vertices and frames. Frame F1 measures whether any vertex is in contact.
Motion capture (48)
GRAB (50)
SPIDER [ 71 ]
40%
12%
CARD -physics
52%
32%
Table 3: CARD -physics. We compare CARD -physics, our physics retargeting variant, against SPIDER’s physics stage [ 71 ] . Strict task success in percent on 48 motion-capture demonstrations (2 seeds) and 50 GRAB clips (1 seed). A task counts as solved when at least one seed succeeds ( Appendix I ).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Initializer vs. solution on six video clips (hanger, can, cup, cards, marker, bowl): the geometric initializer qinit (left of each pair) seats the hand near the contact force target . The contact solve (right) closes the last millimeters. Orange spheres are the contact force target .
Dataset
Contact
View
Train
Val.
HOT3D [ 3 ]
object
ego
643,980
71,792
ARCTIC [ 16 ] , allocentric
object
exo
459,867
51,186
ARCTIC [ 16 ] , egocentric
object
ego
168,343
18,707
HOI4D [ 41 ]
object
ego
106,278
11,995
DECAF [ 65 ]
face
exo
58,829
6,493
Hi4D [ 85 ]
body
exo
49,902
5,306
Appendix
Table 4: Vid-HACO training data. Contact: what the hand touches. View: egocentric, third-person (exo, including ARCTIC’s eight allocentric cameras), or web images.
Figure 10: Semantic maps. The human hand (left of each pair) and the robot it is mapped to, colored by semantic region: the same color marks a human region and the robot link that receives its contacts. Skin marks human regions the embodiment has no link for, and grey marks unmapped robot bodies. SHARPA Wave and Wuji Hand 2 take the 15 finger regions and the palm one-to-one, and SHARPA also the pinky base. The Unitree Dex3-1 folds middle, ring, and pinky onto its third finger, and the parallel-jaw gripper assigns the thumb to one jaw and the four fingers to the other.
Figure 11: Semantic regions. The 18 regions shared by every source and robot, three per finger, the palm, the upper palm, and the pinky base, each with its region center of pressure.
Figure 12: Pseudo-object for video. Left to right: the contact force target of every frame of a video clip, mapped into the wrist frame and accumulated (registered contact cloud); their convex hull, the pseudo-object that serves as the collision field; the MANO hand at the peak-contact frame grasping the translucent hull; and the input video at the same frame. The first three panels are drawn in the camera frame of that frame, so their orientation matches the video.
Group
Term
Symbol
Value
Contact
reach and cone, Eq. 3
wc
40
direction price
λ,r
2, 10 mm
correspondence blend
αf
0.15 (distal, middle), 0 (proximal)
contact stability
wfix
2.5
force weighting
ρf
1+21Ff/Fmax∈[1,1.5]
Style
fingertip apex
wtip
3
Appendix
Table 5: Solver settings shared by the video, glove, and motion-capture sources. Weights multiply the residual before squaring, as written in Eqs. 3 , 10 , 11 and 12 . Residual units differ across groups, so weights are not comparable by magnitude.
Variant
Contact ↑
Region ∣ contact ↑
In-cone ↑
Gap (mm) ↓
Pen. (mm) ↓
CARD (full)
64.7
97.4
82.2
2.0
4.2
− geometric initializer
52.3
98.8
79.0
2.5
6.0
previous initializer (gain 0.7)
51.7
97.2
81.4
2.3
3.4
− penetration barrier
59.2
97.6
81.5
2.0
6.0
Appendix
Table 6: Initializer and penetration barrier on the 20-clip video bed. Rates in percent, Gap median and Pen. maximum in mm. Best in bold.
Variant
Setting
Contact ↑
Region ∣ contact ↑
In-cone ↑
Gap ↓
Pen. ↓
CARD (full)
αf=0.15,λ=2
64.7
97.4
82.2
2.0
4.2
Reach only
αf=0,λ=0
62.7
98.1
57.5
1.7
4.4
No correspondence
αf=0,λ=2
58.4
98.5
81.3
2.1
4.5
Correspondence only
αf=1,λ=0
59.1
97.6
53.3
1.9
4.2
Correspondence + cone
αf=1,λ=2
50.4
97.8
80.3
2.5
4.0
No cone
αf=0.15,λ=0
69.1
97.3
54.6
1.5
4.2
Appendix
Table 7: Contact-term ablation on the 20-clip video bed. αf is the correspondence blend and λ the friction-cone weight of Eq. 3 , and Estyle is the style group Eq. 10 . Gap median and Pen. maximum in mm. The contact point is the nearest pad sample throughout. Best in bold.
Figure 13: Force tracking on the SHARPA Wave. (a) The index finger follows a 0 – 4 N periodic contact force target against a fixed object: the measured force reaches the peaks within a few tenths of a newton and lags the target by about 0.2 s. The floor near 0.5 N is the preload the state machine keeps while in contact. (b) A three-finger grasp with step targets during mustard bottle pick-up replay (thumb 9.7 N, middle 7 N, index 2.6 N): the forces rise to their set-points within a second of the step, the thumb overshoots while the object settles, and release returns every finger to zero within about a second.
Table 8: Contact fidelity by source. The clips of Tab. 1 grouped by contact signal. Rates in percent, pooled over anchor-frames; millimeter rows are the median over clips of the per-clip median. Penetration is reported only where a tracked object mesh exists. Bold : best per source and metric.
Human demonstrations offer a scalable data source for dexterous manipulation, but transferring them to robot actions remains challenging due to the embodiment gap. Today's retargeting is mostly kinematic, yet manipulation is decided by force, which governs how the hand interacts with the object and how the object moves. In this paper, we present ReForce, a Force-aware Retargeting method that turns human motion and forces into robot actions that reproduce the intended contact. ReForce predicts a residual on the kinematically retargeted action to reach the desired force, using a general force tracker trained on large-scale simulation interactions. It supports both online force-aware teleoperation and offline data translation. In simulation and on real hardware, ReForce achieves lower force-tracking error and stronger multi-finger contact engagement on contact-rich tasks such as paper-cup grasping and tongs manipulation.
Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and alignment over all baselines on the ContactPose Dataset, improves Pen-Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.
Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at https://yunhaifeng.com/REGRIND.