We introduce SoftAct, a framework for teaching soft robot hands to perform human-like manipulation skills by explicitly reasoning about contact forces. Leveraging immersive virtual reality, our system captures rich human demonstrations, including hand kinematics, object motion, dense contact patches, and detailed contact force information. Unlike conventional approaches that retarget human joint trajectories, SoftAct employs a two-stage, force-aware retargeting algorithm. The first stage attributes demonstrated contact forces to individual human fingers and allocates robot fingers proportionally, establishing a force-balanced mapping between human and robot hands. The second stage performs online retargeting by combining baseline end-effector pose tracking with geodesic-weighted contact refinements, using contact geometry and force magnitude to adjust robot fingertip targets in real time. This formulation enables soft robotic hands to reproduce the functional intent of human demonstrations while naturally accommodating extreme embodiment mismatch and nonlinear compliance. We evaluate SoftAct on a suite of contact-rich manipulation tasks using a custom non-anthropomorphic pneumatic soft robot hand. SoftAct's controller reduces fingertip trajectory tracking RMSE by up to 55 percent and reduces tracking variance by up to 69 percent compared to kinematic and learning-based baselines. At the policy level, SoftAct achieves consistently higher success in zero-shot real-world deployment and in simulation. These results demonstrate that explicitly modeling contact geometry and force distribution is essential for effective skill transfer to soft robotic hands, and cannot be recovered through kinematic imitation alone. Project videos and additional details are available at https://soft-act.github.io/.
Figures & tables
Fig. 2: Design of the pneumatic soft manipulator. The manipulator features a rigid–soft hybrid architecture, with each soft finger actuated by three radially arranged pneumatic chambers. The elastomeric finger body is reinforced with internal rigid structures to enhance controllability and repeatability. Differential pressurization enables planar and out-of-plane bending, while the fingers are mounted on a rigid base attached to a 7-DoF robotic arm.
Fig. 3: Simulation Setup. We approximate pneumatic pressure–induced bending using internal torque-based actuation applied to a rigid spine, which deforms the surrounding soft finger through distributed virtual spring constraints.
Fig. 4: Contact-Rich Manipulation Tasks and Demonstration Force Profiles. We evaluate SoftAct on six manipulation tasks (rows): light bulb insertion , light bulb twisting , cup pouring , marker grabbing , bottle unscrewing , and box reorienting . For each task, the left column visualizes the average demonstrated contact-force distribution over the human hand, highlighting task-dependent asymmetries (e.g., thumb-dominant vs. distributed contact). The remaining columns show representative time-lapse frames from a single VR demonstration for the task.
Fig. 5: Low-level Control Performance. We evaluate trajectory tracking accuracy for a single soft finger executing planar reference trajectories, including square, circular, triangular, and rectangular motions. Each plot shows the desired fingertip trajectory in the xy plane (black curve) and the executed fingertip trajectories produced by different controllers. Colored curves correspond to controller rollouts. Tracking error is computed as the Euclidean distance between the executed position and the reference trajectory at each timestep. The proposed controller produces trajectories that closely follow the reference paths with low bias and variance, whereas baseline controllers exhibit noticeable drift, distortion, and accumulated error.
Fig. 6: Retargeting Stages. Overview of the two-stage force-aware retargeting pipeline. Stage 1 performs offline force-balanced finger assignment, allocating robot fingers based on demonstrated contact force distribution. Stage 2 performs contact-informed refinement, adjusting fingertip targets using geodesic-weighted contact to align the contact surfaces between the human hand mesh and the soft robot.
Fig. 7: Real-World Manipulation Results. Sequential frames showing zero-shot real-world execution of contact-rich manipulation tasks with SoftAct , including paper cup pouring, box reorientation, and light bulb screwing. These tasks are challenging as they require compliant manipulation for contact-rich interaction. The robot successfully maintains stable contact and executes task-relevant object motion despite severe embodiment mismatch between the human demonstrator and the non-anthropomorphic soft robot hand.
Controller
RMSE
Mean ± Std
Max
(mm)
(mm)
(mm)
Direct KNN
8.74
5.33 ± 6.93
34.34
Direct MLP
6.11
4.78 ± 3.80
13.85
Direct Linear
5.05
3.97 ± 3.13
10.89
SoftAct
2.28
2.06 ± 0.98
5.47
TABLE I: Fingertip trajectory tracking performance for all controllers. Lower is better.
Task
Method
Pos. (cm)
Rot. (deg)
Light Bulb Insertion
Kinematic
1.52±0.61
11.8±3.4
SoftAct -stage 2
0.97±0.42
25.1±7.9
SoftAct
0.48±0.21
12.4±8.1
Light Bulb Twisting
Kinematic
0.05±0.07
18.2±6.5
SoftAct -stage 2
0.07±0.03
9.4±3.4
SoftAct
0.02±0.005
2.9±1.3
TABLE II: Object trajectory tracking error in simulation experiments. Lower is better.
Task
Domain
SoftAct
Kinematic Baseline
Paper Cup Pouring
Simulation
90%
40%
Light Bulb Insertion
Simulation
47%
0%
Marker Grasping
Simulation
97%
73%
Bottle Unscrewing
Simulation
80%
47%
Box Reorienting
Simulation
90%
30%
Light Bulb Screwing
Simulation
100%
77%
TABLE III: Task success rates in simulation and the real world for SoftAct compared to a kinematic-only baseline. For each baseline per task, we do 30 rollouts in simulation, and 20 in the real world. Higher is better.
Human demonstrations offer a scalable data source for dexterous manipulation, but transferring them to robot actions remains challenging due to the embodiment gap. Today's retargeting is mostly kinematic, yet manipulation is decided by force, which governs how the hand interacts with the object and how the object moves. In this paper, we present ReForce, a Force-aware Retargeting method that turns human motion and forces into robot actions that reproduce the intended contact. ReForce predicts a residual on the kinematically retargeted action to reach the desired force, using a general force tracker trained on large-scale simulation interactions. It supports both online force-aware teleoperation and offline data translation. In simulation and on real hardware, ReForce achieves lower force-tracking error and stronger multi-finger contact engagement on contact-rich tasks such as paper-cup grasping and tongs manipulation.
Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO outperforms five baselines in contact F1, improving on the strongest ones by 8 to 28 points, while improving the success rate of downstream dynamic retargeting by as much as 35 points. On a Sharpa hand, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects spanning 10 categories. Project page: https://morphometricimitation.github.io
Tara Sadjadpour, Siming He, C. K. Wolfe +5
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA
Learning dexterous manipulation from demonstrations is bottlenecked by data: the contact forces that determine whether a grasp succeeds are absent from every scalable source of human demonstrations. This paper builds on two observations. First, what survives the change from a human hand to a robot hand is the contact structure of a demonstration - which finger regions touch which object locations, and in what order - rather than its joint motion. Second, physical consistency need not be engineered per task: a single residual reinforcement learning (RL) policy, trained once across diverse demonstrations, can repair kinematic recordings into physically consistent, contact-annotated trajectories, and the same residual formulation restores dynamic feasibility after retargeting. These observations yield a three-stage pipeline that converts human motion-capture recordings into dexterous robot policies with no real-robot training data: physics refinement with a simulated MANO hand recovers contacts and forces, contact-anchored retargeting transfers the demonstrated contact structure through an objective independent of hand morphology, and residual policy learning adapts the result to robot actuation. The pipeline reconstructs 25,454 single-hand trajectories (success 7.3% -> 59.3%) and 25 dual-hand tasks (16.0% -> 62.4%) with one shared policy per setting, transfers one human dataset to four morphologically distinct robot hands (+62.4 pp), and executes four contact-rich bimanual tasks on physical hardware with zero real-robot training data.
Zihao Yang, Chengyuan Liu, Yu Zhou +6
DexGEM Lab · Shanghai Jiao Tong University · Tongji University +1