Imitation learning increasingly relies on novice operators, whose demonstrations fall short of experts' in number and inter-trial consistency. Operators act on what they are shown, yet how the rendering of the robot affects their demonstrations, and the policies trained on them, has not been examined. We study this question with PointHand, a teleoperation interface that renders a multi-fingered robot hand as sparse keypoints at its fingertips and joints, the same keypoints a policy can observe. In a simulated mug-in-box task, 18 novices and 4 experts collected 2,200 trials under three renderings: the real shape of the hand, all keypoints emphasized, or the thumb and index finger alone emphasized. The novices shown the thumb and index finger emphasized collected 441 successful demonstrations from 600 trials, against 320 for those shown the real shape. We then trained diffusion policies on these demonstrations, replacing the scene point cloud and proprioceptive state they observe with the hand keypoints and two object keypoints. Within 400 steps these demonstrations went from the weakest of the three novice conditions under the point-cloud policy (46.3%) to the strongest under the keypoint policy (57.3%), a gain of +11.0 percentage points in all five seeds. Those collected with all keypoints emphasized changed by at most +0.7 percentage points, and in no more than three seeds. The value of a demonstration therefore depends on the rendering shown during collection and on what the policy observes together. A shared representation can pay off on both sides, in the number of successful demonstrations and in the success rate of the policy trained on them. Project page: https://omron-sinicx.github.io/point-hand/
Figures & tables
Fig. 2: Robot hand presented to the operator: real shape (left, Baseline ), all keypoints emphasized (center, KP-Standard ), and only those of the thumb and index finger emphasized (right, KP-Salient ). Fingertip keypoints are spheres (radius 0.007 m) and intermediate joint keypoints cubes (edge 0.012 m). Emphasized keypoints have a per-finger color that brightens on contact. The others and the wrist keypoint, which the policy does not receive, are gray.
Fig. 3: Observation encoding of the two policies. Top: DP3 ( π∅ ) uses the DP3 encoder [ 7 ] , which encodes the dense scene point cloud with a PointNet-based MLP of widths 64, 128, 256 and max-pooling, projected to a 64-dimensional feature and concatenated with a 64-dimensional state feature. Bottom: DP-KP ( πP ) uses the Point Policy encoder [ 16 ] , which embeds each hand and object keypoint with a shared layer of width 512 projected to 256, adds a learnable per-keypoint positional embedding (PE), and passes each as one token to a four-layer, two-head Transformer, so the correspondence between tokens and robot parts is preserved. Both condition a diffusion head of the same architecture and share the training procedure.
Condition
Hand rendering
Emphasized keypoints
Contact cue
Baseline
Real shape
—
None
KP-Standard
11 keypoints
All fingers (11)
Brightness
KP-Salient
11 keypoints
Thumb and index (5)
Brightness
TABLE I: Visual representation conditions assigned to novices in Study 1. The experts used Baseline .
Fig. 4: Success rate during collection, one value per participant ( n=6/6/6/4 ). Boxes show the median as a dark line and the mean as a cross, and the hatch pattern identifies the condition.
Fig. 5: Levels of the five main metrics (top) and their within-participant coefficients of variation (bottom). Each box summarizes one value per participant ( n=6/6/6/4 ), with the median as a dark line and the mean as a cross. The hatch pattern identifies the condition.
Fig. 6: Overall NASA-TLX score (weighted) and six subscales (Mental, Physical, and Temporal Demand, Performance, Effort, Frustration) per condition. The first author, one of the experts, knew the purpose of the experiment and did not answer ( n=6/6/6/3 ). Boxes show the median as a dark line and the mean as a cross. Hatch patterns identify the condition.
Fig. 7: Post hoc analysis of finger joint targets in successful trials: motion ratio of the middle, ring, and little fingers to the thumb and index finger (left) and four-finger against thumb–four-finger correlation (right, one marker per participant, n=6/6/6/4 , dashed line y=x ). Boxes show the median as a dark line and the mean as a cross. Hatch patterns and marker shapes (circle Baseline , square KP-Standard , diamond KP-Salient , triangle Expert ) identify the condition.
Fig. 8: Simulated success rate within 200, 400, and 600 steps, one panel per observation representation and one bar per collection condition. Bars show the mean over five seeds and error bars the seed-to-seed standard deviation, and the hatch pattern identifies the collection condition.
Training data
100 demonstrations per condition and seed
Seeds
5 (sampled demonstrations)
Scene point cloud points N
512
Epochs, batch size
500, 128
Optimizer
AdamW, 10−4 , weight decay 10−6 ,
5 warmup epochs then cosine, clip 1.0
Evaluated weights
EMA, lowest validation loss
TABLE II: Policy training and evaluation settings in Study 2.
Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots. A common remedy is to fine-tune these models on robot-specific demonstrations. However, robot data collection can be prohibitively expensive and time-consuming, which is particularly acute in dexterous manipulation, e.g., teleoperating a multi-fingered hand for even a single atomic task can take days. To address this, we introduce Dexterous Point Policy, a framework that learns dexterous manipulation policies directly from human videos and requires no robot demonstrations. Our core insight is that a unified 3D keypoint representation can bridge human and robot embodiments when used for both observations and actions. Specifically, we extract 3D keypoints of task-relevant objects and human hands from raw videos, and train an autoregressive transformer over these keypoints. We observe that at the keypoint level, specifically the wrist and fingertips, human and robot behaviors closely align, enabling direct policy transfer. On a suite of real-robot tasks spanning pick-and-place and tool use, Dexterous Point Policy attains 75.0% success, whereas a state-of-the-art VLA baseline reaches only 1.0%. Furthermore, our method generalizes strongly to unseen scenarios, including multi-object environments and novel object categories.
Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO outperforms five baselines in contact F1, improving on the strongest ones by 8 to 28 points, while improving the success rate of downstream dynamic retargeting by as much as 35 points. On a Sharpa hand, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects spanning 10 categories. Project page: https://morphometricimitation.github.io
Tara Sadjadpour, Siming He, C. K. Wolfe +5
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA
A dexterous hand can change an object's pose without setting it down. This survey covers 113 learning-based methods that attempt it, the hands they run on, the simulators they train in, and how they report results. The field has largely converged on one recipe: reinforcement learning on privileged state in a GPU simulator, distilled into a vision-only student for transfer to hardware. Almost half the corpus trains that way, nearly all of it in one simulator, whose own successors are barely used. A second and faster-growing branch learns from teleoperated demonstration instead, through vision-language-action models, diffusion policies, and systems whose contribution is the collection rig itself. Published work has tripled since 2022. Grasping remains the largest task family and bimanual coordination the fastest-rising, and almost every experiment runs on one of four hands. What the survey adds comes from reading released code and vendor specifications beside the papers. Eight hands that can be bought or built from published designs appear in no method paper here. Where the depth a hand sinks into the object it holds is reported at all, it is not measured in the physics that produced the motion. Section VII proposes a protocol for the unreported quantities; the body gives the counts behind every claim above.