SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior
Authors: Ziyong Ma, Uksang Yoo, Jonathan Francis, Weiming Zhi, Jeffrey Ichnowski, Jean Oh
Organizations: Robotics Institute, Carnegie Mellon University, USA · Bosch Center for Artificial Intelligence, USA · School of Computer Science, The University of Sydney, Australia · Australian Centre for Robotics, The University of Sydney, Australia · College of Connected Computing, Vanderbilt University, USA
While soft robot manipulators offer compelling advantages over rigid counterparts, including inherent compliance, safe human-robot interaction, and the ability to conform to complex geometries, accurate forward modeling from low-dimensional actuation commands remains an open challenge due to nonlinear material phenomena such as hysteresis and manufacturing variability. We present SOFTMAP, a sim-to-real learning framework for real-time 3D forward modeling of tendon-actuated soft finger manipulators. SOFTMAP combines four components: (1) As-Rigid-As-Possible (ARAP)-based topological alignment that projects simulated and real point clouds into a shared, topologically consistent vertex space; (2) a lightweight MLP forward model pretrained on simulation data to map servo commands to full 3D finger geometry; (3) a residual correction network trained on a small set of real observations to predict per-vertex displacement fields that compensate for sim-to-real discrepancies; and (4) a closed-form linear actuation calibration layer enabling real-time inference at 30 FPS. We evaluate SOFTMAP on both simulated and physical hardware, achieving state-of-the-art shape prediction accuracy with a Chamfer distance of 0.389 mm in simulation and 3.786 mm on hardware, millimeter-level fingertip trajectory tracking across multiple target paths, and a 36.5% improvement in teleoperation task success over the baseline. Our results show that SOFTMAP provides a data-efficient approach for 3D forward modeling and control of soft manipulators.
Figures & tables
Fig. 1 : SOFTMAP overview. Simulation and real multi-view data are aligned via ARAP into a shared topology, enabling a lightweight learned forward model to predict full 3D soft finger geometry from servo commands for real-time control and teleoperation.
Fig. 2 : SOFTMAP pipeline. Simulation data and real multi-view images are independently processed into 3D representations, then encoded via ARAP into a shared topologically consistent vertex space to form an aligned dataset. A learned MLP maps servo commands to 3D shape predictions, which are subsequently refined by a residual correction network. The resulting forward model enables downstream applications, including trajectory generation and real-time teleoperation.
Fig. 3 : Simulation environment. The soft finger is modeled in the SOFA Framework using a Neo-Hookean hyperelastic material with four embedded tendons. Actuation commands usim(i)∈R2 are swept across a dense grid to generate shape observations Ysim(i)∈R548×3 , forming the simulation dataset used to pretrain SOFTMAP’s model.
Fig. 4 : Real-world data collection setup. Two RGB cameras are mounted at distinct viewpoints around a soft finger actuated by an xArm 7 robot. For each servo command, synchronized multi-view images are captured and fed into our reconstruction pipeline to produce 3D point clouds used for real world deployment.
Fig. 5 : Evaluation Comparison. Left: Point Cloud Comparisons in Simulation Data; Right: Point Cloud Comparisons in Real Data
Fig. 6 : Vision-based teleoperation demonstration. Top: human motion capture with hand landmarks. Middle: real-time 3D shape prediction. Bottom: corresponding physical soft-finger motion.
Model
Chamfer ↓ (mm)
Mean vertex ↓ (mm)
Simulation
Laplacian-Encoded Model
8.739±3.67
8.757±4.811
DeepSoRo
4.520±2.437
7.970±4.127
Linear Model
2.726±1.043
2.620±1.143
XGBoost Model
1.28±0.58
0.868±0.563
ARAP-Encoded Model
0.867±0.53
0.438±0.274
TABLE I : Shape prediction evaluation in simulation and real settings (lower is better). Mean vertex error is not reported for real data as point cloud observations lack vertex-level correspondence.
Fig. 7 : Qualitative trajectory tracking results. Each column shows snapshots at three timesteps for trajectories S, H, O, and E, comparing SOFTMAP against the DeepSoRo baseline and simulation ground truth. SOFTMAP consistently produces tighter fingertip paths with less drift and distortion, particularly on asymmetric strokes.
Trajectory
Method
Sim
Real
MSE [mm]
Max [mm]
MSE [mm]
Max [mm]
Letter S
DeepSoRo
2.685±1.109
3.646
5.917±5.12
23.154
SOFTMAP (Proposed)
1.124±0.617
1.95
3.458±2.976
15.38
Letter H
DeepSoRo
2.766±0.999
4.319
5.854±3.816
14.7
SOFTMAP (Proposed)
1.214±0.652
2.55
3.569±2.343
10.3
Letter O
DeepSoRo
1.336±0.918
1.582
3.164±1.683
6.887
TABLE II : Trajectory tracking error comparison (lower is better). Each trajectory is evaluated in simulation (Sim) and on real hardware (Real) using Mean Squared Error (MSE) ± standard deviation and maximum error, all in mm.
Fig. 8 : Teleoperation task progression. From left to right, the soft finger pushes the T, Cube and a triangle prism from its starting pose toward the marked target region.
Task
Method
IoU ↑
Overlap ↑ (%)
Push-T
DeepSoRo
0.38±0.26
46±27
SOFTMAP (Proposed)
0.72±0.13
80±11
Cube
DeepSoRo
0.63±0.16
71±15
SOFTMAP (Proposed)
0.80±0.10
87±8
Triangular Prism
DeepSoRo
0.55±0.10
67±6.5
SOFTMAP (Proposed)
0.61±0.18
72±15
TABLE III : Teleoperation push-task comparisons (higher is better).
Deformable-object manipulation is essential for robotic tasks such as folding laundry and handling food, where robots must control shape changes as well as object motion. Predictive soft-body simulation supports these tasks by anticipating deformation under external interactions. However, spatial neighborhoods can misrepresent deformation dependencies, introducing local errors that accumulate over successive predictions. Models fitted to individual scenes must also accommodate changes in object geometry and manipulation conditions. In this work, we propose AIM, an Adaptive Interaction Modeling framework that treats real-to-sim soft-body simulation as a local-global interaction modeling problem. AIM uses motion history and geometry to adapt particle relations over current spatial neighbors and retained connections, while geometry-conditioned global communication coordinates object-wide responses. A unified kinematic control-point interface represents different manipulation configurations, and multi-step autoregressive supervision trains the model on its own predicted trajectories. Experiments on PhysTwin and PGND demonstrate improved motion accuracy and visual fidelity, with a 20.0% reduction in future-prediction tracking error relative to PhysTwin and a 22.8% reduction in mean long-horizon particle error across six object categories relative to PGND. The framework further supports transfer across actions, object instances, and scenes, including zero-shot transfer from robot interactions to human manipulation without target-domain dynamics fitting.
Tiancheng Yang, Dingshuo Chen, Tianle Chen +2
NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1
Soft robots and tactile sensors have demonstrated great potential in delicate manipulation tasks. Soft pneumatic robots enable safe contact through compliance, and vision-based tactile sensors offer high-resolution touch perception. However, learning tactile manipulation with compliant robots has been challenging, bottlenecked by the lack of efficient simulation. Existing simulators typically model them in isolation, and exhibit large calibration gaps that are difficult to overcome efficiently. We present PneuTac, a unified framework for tactile-feedback manipulation with soft pneumatic robots. We leverage the material point method (MPM) for modelling the dynamics of the soft robot and the deformable tactile membrane, and 3D Gaussian splatting (3DGS) for rendering. Real-to-sim modelling is done with a simple vision-based method, to then train action and perception networks for efficient simulation with surrogate models. We use the framework to drive a tactile-guided pipeline to collect demonstrations in simulation. Through experiments on a custom-designed pneumatic soft finger with a tactile sensing tip, together with additional cross-device evaluations, we show that PneuTac is capable of accurately modelling soft robots with tactile sensors, and that policies trained with simulation-augmented demonstrations outperform baselines trained on the same real data on three real-world contact-rich compliant manipulation tasks, making it a practical framework for tactile manipulation on compliant hardware.
Shaohong Zhong, Marco Pontin, Joe Watson +2
Oxford Robotics Institute, Department of Engineering Science, University of Oxford, OX1 3PJ Oxford, U.K.
We introduce SoftAct, a framework for teaching soft robot hands to perform human-like manipulation skills by explicitly reasoning about contact forces. Leveraging immersive virtual reality, our system captures rich human demonstrations, including hand kinematics, object motion, dense contact patches, and detailed contact force information. Unlike conventional approaches that retarget human joint trajectories, SoftAct employs a two-stage, force-aware retargeting algorithm. The first stage attributes demonstrated contact forces to individual human fingers and allocates robot fingers proportionally, establishing a force-balanced mapping between human and robot hands. The second stage performs online retargeting by combining baseline end-effector pose tracking with geodesic-weighted contact refinements, using contact geometry and force magnitude to adjust robot fingertip targets in real time. This formulation enables soft robotic hands to reproduce the functional intent of human demonstrations while naturally accommodating extreme embodiment mismatch and nonlinear compliance. We evaluate SoftAct on a suite of contact-rich manipulation tasks using a custom non-anthropomorphic pneumatic soft robot hand. SoftAct's controller reduces fingertip trajectory tracking RMSE by up to 55 percent and reduces tracking variance by up to 69 percent compared to kinematic and learning-based baselines. At the policy level, SoftAct achieves consistently higher success in zero-shot real-world deployment and in simulation. These results demonstrate that explicitly modeling contact geometry and force distribution is essential for effective skill transfer to soft robotic hands, and cannot be recovered through kinematic imitation alone. Project videos and additional details are available at https://soft-act.github.io/.
Uksang Yoo, Mengjia Zhu, Evan Pezent +8
The Robotics Institute, Carnegie Mellon University · Meta Reality Labs