The fidelity of teleoperated demonstrations used for imitation learning is determined by how faithfully a robot hand can replicate the operator's posture. The posture retargeting law that the robot will execute is typically selected only once the hardware has been finalized, and conventional hand designs often neglect human-like grasping postures. We introduce PACT, an optimization based on human grasping postures that jointly optimizes the mechanical parameters of a 17-servo robot hand (link lengths, joint offsets, mounting angles, and base placements) and its nonlinear posture retargeter (a linear human-to-robot joint map, per-joint posture-IK weights, and damping). The optimization enforces full-finger correspondence (MCP, PIP, DIP, and tip of each finger) to human poses from the DexYCB dataset, and is constrained by servo interference, assembly clearances, and flat-hand protrusion, so the resulting design can be physically assembled. We fabricate the PACT Hand and validate it through teleoperation experiments and an imitation-learning demonstration. The hand reproduces all human hand keypoints with a kinematic RMSE of 3.9 mm, compared to 12.8 mm for the LEAP Hand baseline, and keeps all fingertips within 10 mm on every grasping posture, while the baseline achieves this for only 21.6% of the poses. Under matched per-joint torque limits, it lifts twice the baseline payload, 2 kg against 1 kg, and deforms a held paper cup seven times less, reaching a deformation level comparable to a human hand. It also achieves an 86% grasping success rate on YCB daily objects, compared to 64% for the baseline. Finally, as a demonstration, an imitation-learning policy trained on demonstrations collected with the PACT Hand achieved a 75% success rate at held-out placements in a can pick-and-place task. Project page: https://pact-hand-2027.github.io
Figures & tables
Fig. 1: The PACT Hand (left), a 17-servo anthropomorphic hand co-designed with its posture-IK retargeter. At runtime (right), the same retargeting law that served as the design objective maps a bare-hand posture from a single RGB camera onto the hand, shown grasping a mustard bottle.
Fig. 2: Fabrication constraints reshaping the hand, stage by stage. Optimized joint chains and as-built servo volumes are overlaid on the human skeleton at the flat-hand pose after stages S0, S1, S3, and S5 (blue: flexion, salmon: abduction; gray: human skeleton at 1.56× ; annotations list constraints added since the previous panel). Right: the final design.
Hand
Set
n
All-kp RMSE
Tip RMSE
MCP/ PIP
IP/ DIP
HPCR
PACT (17)
train
1788
3.90
1.84
4.39
4.80
100.0%
PACT (17)
test
277
3.88
1.66
—
—
100.0%
LEAP Hand
train
1788
12.82
13.45
14.13
10.60
21.6%
LEAP Hand
test
277
13.09
13.59
—
—
7.9%
Tip-only (16)
train
1788
23.66
2.46
26.33
31.31
99.5%
Tip-only (16)
test
277
22.79
< 0.01
—
—
100.0%
TABLE I: Retargeting accuracy, in human-scale mm, on the training and test sets.
Fig. 3: Payload under a per-joint torque cap. Left: the replayed grasp and lift of the instrumented cylinder. Right: the largest mass held reliably at each cap for each robot hand. The two baseline curves are offset by 20g so that they remain distinguishable. The LEAP Hand’s occasional 1.3kg holds, at most three of five trials, are not counted as reliable.
Fig. 4: Rim deformation holding the 400g cup, trials at 0.167Nm . The detected band is shown in green, with pre-frame normalized circularity at top left.
Fig. 5: Everyday-object comparison. Left: the ten YCB objects. Right: the teleoperation setup, operator with a GELLO leader arm, one RGB camera, and the UR5e-mounted hand.
Fig. 6: Qualitative demonstrations. Top: in-hand stashing of four objects, and two tennis balls rolled on the palm. Bottom left: a 2L bottle grasped, lifted, and placed. Bottom right: the ACT policy autonomously picking the can and placing it on the target.
Object
PACT
LEAP
Tip-only
master chef can
5/5
3/5
2/5
cracker box
4/5
2/5
0/5
sugar box
5/5
5/5
4/5
tomato soup can
2/5
4/5
0/5
pudding box
5/5
4/5
5/5
banana ‡
5/5
3/5
0/5
TABLE II: Everyday-object grasping, successes out of five scored trials at a cap of 0.167Nm . † marks objects unseen during design. ‡ marks objects using grip tape.
Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO outperforms five baselines in contact F1, improving on the strongest ones by 8 to 28 points, while improving the success rate of downstream dynamic retargeting by as much as 35 points. On a Sharpa hand, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects spanning 10 categories. Project page: https://morphometricimitation.github.io
Tara Sadjadpour, Siming He, C. K. Wolfe +5
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA
Robot learning has advanced rapidly in learning control, but learning the physical body of a robot remains much more difficult because jointly searching over design and control creates a very large combinatorial problem. Here, we present a data-driven framework for generating robot hands from human demonstrations. Instead of learning a complex controller together with each candidate design, we generate robot hand designs using the same simple control policy used after fabrication: matching fingertip positions through inverse kinematics. Using more than 4 million frames of human fingertip motion from everyday manipulation, our algorithm optimizes tree-structured robot hands to reproduce desired target motions. The framework produced both a 6-degree-of-freedom (DoF) general-purpose hand and lower-DoF task-specific hands with spatial four-bar mimic joints. To accelerate the search over designs, we trained a reinforcement-learning (RL) actor to propose good hand designs and joint angles, reducing search time from hours to minutes. We fabricated the mechanisms directly as one-piece articulated structures with print-in-place joints. In real-world experiments, the 6-DoF hand achieved highly accurate teleoperated fingertip tracking better than available commercial robot hands, whereas the specialized 3-DoF hands reproduced structured human and synthetic trajectories with reduced mechanical complexity. These results showed that large-scale human motion data can be used not only to train robot controllers but also as a reference for optimizing and generating the physical embodiment of robots.
Sha Yi, Nicklas Hansen, Xueqian Bai +3
University of California San Diego · Amazon Frontier AI & Robotics
Human hand-object demonstrations provide dense reference motions for training dexterous manipulation reinforcement learning (RL) policies through reference tracking. However, to use such demonstrations for RL policy learning, retargeting must preserve hand pose and task-relevant hand-object contact structure. Otherwise, contact and feasibility artifacts can degrade downstream RL policy performance. We introduce TopoRetarget, an interaction-preserving retargeting framework that uses a single set of parameters across diverse retargeting conditions while maintaining task-relevant hand-object interaction and adapting human demonstrations to dexterous robot hands. The method constructs a sparse interaction graph over hand and object keypoints and optimizes distance-weighted Laplacian deformation with directional consistency, kinematic constraints, and penetration handling. Evaluations show that the generated references improve both interaction fidelity and policy learning: TopoRetarget achieves the best contact precision and alignment over all baselines on the ContactPose Dataset, improves Pen-Spin training success by 40.6 percentage points over the existing baseline methods, and enables zero-shot transfer to Wuji Hand hardware on cube reorientation and pen spinning.