We introduce SoftAct, a framework for teaching soft robot hands to perform human-like manipulation skills by explicitly reasoning about contact forces. Leveraging immersive virtual reality, our system captures rich human demonstrations, including hand kinematics, object motion, dense contact patches, and detailed contact force information. Unlike conventional approaches that retarget human joint trajectories, SoftAct employs a two-stage, force-aware retargeting algorithm. The first stage attributes demonstrated contact forces to individual human fingers and allocates robot fingers proportionally, establishing a force-balanced mapping between human and robot hands. The second stage performs online retargeting by combining baseline end-effector pose tracking with geodesic-weighted contact refinements, using contact geometry and force magnitude to adjust robot fingertip targets in real time. This formulation enables soft robotic hands to reproduce the functional intent of human demonstrations while naturally accommodating extreme embodiment mismatch and nonlinear compliance. We evaluate SoftAct on a suite of contact-rich manipulation tasks using a custom non-anthropomorphic pneumatic soft robot hand. SoftAct's controller reduces fingertip trajectory tracking RMSE by up to 55 percent and reduces tracking variance by up to 69 percent compared to kinematic and learning-based baselines. At the policy level, SoftAct achieves consistently higher success in zero-shot real-world deployment and in simulation. These results demonstrate that explicitly modeling contact geometry and force distribution is essential for effective skill transfer to soft robotic hands, and cannot be recovered through kinematic imitation alone. Project videos and additional details are available at https://soft-act.github.io/.
Figures & tables
Fig. 2: Design of the pneumatic soft manipulator. The manipulator features a rigid–soft hybrid architecture, with each soft finger actuated by three radially arranged pneumatic chambers. The elastomeric finger body is reinforced with internal rigid structures to enhance controllability and repeatability. Differential pressurization enables planar and out-of-plane bending, while the fingers are mounted on a rigid base attached to a 7-DoF robotic arm.
Fig. 3: Simulation Setup. We approximate pneumatic pressure–induced bending using internal torque-based actuation applied to a rigid spine, which deforms the surrounding soft finger through distributed virtual spring constraints.
Fig. 4: Contact-Rich Manipulation Tasks and Demonstration Force Profiles. We evaluate SoftAct on six manipulation tasks (rows): light bulb insertion , light bulb twisting , cup pouring , marker grabbing , bottle unscrewing , and box reorienting . For each task, the left column visualizes the average demonstrated contact-force distribution over the human hand, highlighting task-dependent asymmetries (e.g., thumb-dominant vs. distributed contact). The remaining columns show representative time-lapse frames from a single VR demonstration for the task.
Fig. 5: Low-level Control Performance. We evaluate trajectory tracking accuracy for a single soft finger executing planar reference trajectories, including square, circular, triangular, and rectangular motions. Each plot shows the desired fingertip trajectory in the xy plane (black curve) and the executed fingertip trajectories produced by different controllers. Colored curves correspond to controller rollouts. Tracking error is computed as the Euclidean distance between the executed position and the reference trajectory at each timestep. The proposed controller produces trajectories that closely follow the reference paths with low bias and variance, whereas baseline controllers exhibit noticeable drift, distortion, and accumulated error.
Fig. 6: Retargeting Stages. Overview of the two-stage force-aware retargeting pipeline. Stage 1 performs offline force-balanced finger assignment, allocating robot fingers based on demonstrated contact force distribution. Stage 2 performs contact-informed refinement, adjusting fingertip targets using geodesic-weighted contact to align the contact surfaces between the human hand mesh and the soft robot.
Fig. 7: Real-World Manipulation Results. Sequential frames showing zero-shot real-world execution of contact-rich manipulation tasks with SoftAct , including paper cup pouring, box reorientation, and light bulb screwing. These tasks are challenging as they require compliant manipulation for contact-rich interaction. The robot successfully maintains stable contact and executes task-relevant object motion despite severe embodiment mismatch between the human demonstrator and the non-anthropomorphic soft robot hand.
Controller
RMSE
Mean ± Std
Max
(mm)
(mm)
(mm)
Direct KNN
8.74
5.33 ± 6.93
34.34
Direct MLP
6.11
4.78 ± 3.80
13.85
Direct Linear
5.05
3.97 ± 3.13
10.89
SoftAct
2.28
2.06 ± 0.98
5.47
TABLE I: Fingertip trajectory tracking performance for all controllers. Lower is better.
Task
Method
Pos. (cm)
Rot. (deg)
Light Bulb Insertion
Kinematic
1.52±0.61
11.8±3.4
SoftAct -stage 2
0.97±0.42
25.1±7.9
SoftAct
0.48±0.21
12.4±8.1
Light Bulb Twisting
Kinematic
0.05±0.07
18.2±6.5
SoftAct -stage 2
0.07±0.03
9.4±3.4
SoftAct
0.02±0.005
2.9±1.3
TABLE II: Object trajectory tracking error in simulation experiments. Lower is better.
Task
Domain
SoftAct
Kinematic Baseline
Paper Cup Pouring
Simulation
90%
40%
Light Bulb Insertion
Simulation
47%
0%
Marker Grasping
Simulation
97%
73%
Bottle Unscrewing
Simulation
80%
47%
Box Reorienting
Simulation
90%
30%
Light Bulb Screwing
Simulation
100%
77%
TABLE III: Task success rates in simulation and the real world for SoftAct compared to a kinematic-only baseline. For each baseline per task, we do 30 rollouts in simulation, and 20 in the real world. Higher is better.