SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior

Ziyong Ma; Uksang Yoo; Jonathan Francis; Weiming Zhi; Jeffrey Ichnowski; Jean Oh

cs.ROMar 19, 2026

SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior

Authors: Ziyong Ma, Uksang Yoo, Jonathan Francis, Weiming Zhi, Jeffrey Ichnowski, Jean Oh

Organizations: Robotics Institute, Carnegie Mellon University, USA · Bosch Center for Artificial Intelligence, USA · School of Computer Science, The University of Sydney, Australia · Australian Centre for Robotics, The University of Sydney, Australia · College of Connected Computing, Vanderbilt University, USA

Abstract

While soft robot manipulators offer compelling advantages over rigid counterparts, including inherent compliance, safe human-robot interaction, and the ability to conform to complex geometries, accurate forward modeling from low-dimensional actuation commands remains an open challenge due to nonlinear material phenomena such as hysteresis and manufacturing variability. We present SOFTMAP, a sim-to-real learning framework for real-time 3D forward modeling of tendon-actuated soft finger manipulators. SOFTMAP combines four components: (1) As-Rigid-As-Possible (ARAP)-based topological alignment that projects simulated and real point clouds into a shared, topologically consistent vertex space; (2) a lightweight MLP forward model pretrained on simulation data to map servo commands to full 3D finger geometry; (3) a residual correction network trained on a small set of real observations to predict per-vertex displacement fields that compensate for sim-to-real discrepancies; and (4) a closed-form linear actuation calibration layer enabling real-time inference at 30 FPS. We evaluate SOFTMAP on both simulated and physical hardware, achieving state-of-the-art shape prediction accuracy with a Chamfer distance of 0.389 mm in simulation and 3.786 mm on hardware, millimeter-level fingertip trajectory tracking across multiple target paths, and a 36.5% improvement in teleoperation task success over the baseline. Our results show that SOFTMAP provides a data-efficient approach for 3D forward modeling and control of soft manipulators.

Figures & tables

Fig. 1 : SOFTMAP overview. Simulation and real multi-view data are aligned via ARAP into a shared topology, enabling a lightweight learned forward model to predict full 3D soft finger geometry from servo commands for real-time control and teleoperation.

Fig. 2 : SOFTMAP pipeline. Simulation data and real multi-view images are independently processed into 3D representations, then encoded via ARAP into a shared topologically consistent vertex space to form an aligned dataset. A learned MLP maps servo commands to 3D shape predictions, which are subsequently refined by a residual correction network. The resulting forward model enables downstream applications, including trajectory generation and real-time teleoperation.

Fig. 3 : Simulation environment. The soft finger is modeled in the SOFA Framework using a Neo-Hookean hyperelastic material with four embedded tendons. Actuation commands $u_{\text{sim}}^{(i)}\in\mathbb{R}^{2}$ are swept across a dense grid to generate shape observations $Y_{\text{sim}}^{(i)}\in\mathbb{R}^{548\times 3}$ , forming the simulation dataset used to pretrain SOFTMAP’s model.

Fig. 4 : Real-world data collection setup. Two RGB cameras are mounted at distinct viewpoints around a soft finger actuated by an xArm 7 robot. For each servo command, synchronized multi-view images are captured and fed into our reconstruction pipeline to produce 3D point clouds used for real world deployment.

Fig. 5 : Evaluation Comparison. Left: Point Cloud Comparisons in Simulation Data; Right: Point Cloud Comparisons in Real Data

Fig. 6 : Vision-based teleoperation demonstration. Top: human motion capture with hand landmarks. Middle: real-time 3D shape prediction. Bottom: corresponding physical soft-finger motion.

Model	Chamfer $\downarrow$ (mm)	Mean vertex $\downarrow$ (mm)
Simulation
Laplacian-Encoded Model	$8.739\pm 3.67$	$8.757\pm 4.811$
DeepSoRo	$4.520\pm 2.437$	$7.970\pm 4.127$
Linear Model	$2.726\pm 1.043$	$2.620\pm 1.143$
XGBoost Model	$1.28\pm 0.58$	$0.868\pm 0.563$
ARAP-Encoded Model	$0.867\pm 0.53$	$0.438\pm 0.274$

TABLE I : Shape prediction evaluation in simulation and real settings (lower is better). Mean vertex error is not reported for real data as point cloud observations lack vertex-level correspondence.

Fig. 7 : Qualitative trajectory tracking results. Each column shows snapshots at three timesteps for trajectories S, H, O, and E, comparing SOFTMAP against the DeepSoRo baseline and simulation ground truth. SOFTMAP consistently produces tighter fingertip paths with less drift and distortion, particularly on asymmetric strokes.

Trajectory	Method	Sim		Real
Trajectory	Method	MSE [mm]	Max [mm]	MSE [mm]	Max [mm]
Letter S	DeepSoRo	$2.685\pm 1.109$	3.646	$5.917\pm 5.12$	23.154
Letter S	SOFTMAP (Proposed)	$\mathbf{1.124\pm 0.617}$	$\mathbf{1.95}$	$\mathbf{3.458\pm 2.976}$	$\mathbf{15.38}$
Letter H	DeepSoRo	$2.766\pm 0.999$	4.319	$5.854\pm 3.816$	14.7
Letter H	SOFTMAP (Proposed)	$\mathbf{1.214\pm 0.652}$	$\mathbf{2.55}$	$\mathbf{3.569\pm 2.343}$	$\mathbf{10.3}$
Letter O	DeepSoRo	$1.336\pm 0.918$	1.582	$3.164\pm 1.683$	6.887

TABLE II : Trajectory tracking error comparison (lower is better). Each trajectory is evaluated in simulation (Sim) and on real hardware (Real) using Mean Squared Error (MSE)

\pm

standard deviation and maximum error, all in mm.

Fig. 8 : Teleoperation task progression. From left to right, the soft finger pushes the T, Cube and a triangle prism from its starting pose toward the marked target region.

Task	Method	IoU $\uparrow$	Overlap $\uparrow$ (%)
Push-T	DeepSoRo	$0.38\pm 0.26$	$46\pm 27$
	SOFTMAP (Proposed)	$\mathbf{0.72\pm 0.13}$	$\mathbf{80\pm 11}$
Cube	DeepSoRo	$0.63\pm 0.16$	$71\pm 15$
	SOFTMAP (Proposed)	$\mathbf{0.80\pm 0.10}$	$\mathbf{87\pm 8}$
Triangular Prism	DeepSoRo	$0.55\pm 0.10$	$67\pm 6.5$
	SOFTMAP (Proposed)	$\mathbf{0.61\pm 0.18}$	$\mathbf{72\pm 15}$

TABLE III : Teleoperation push-task comparisons (higher is better).

Explore similar work

AIM: Adaptive Interaction Modeling Networks for Real-to-Sim Soft-Body Simulation
Oct 5, 2026Tiancheng Yang, Dingshuo Chen, Tianle Chen +2Robotics Simulation Robot Dynamics Learning
NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences +1
PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation
Sep 29, 2026Shaohong Zhong, Marco Pontin, Joe Watson +2Robotics Simulation Synthetic Data Generation for Robotics
Oxford Robotics Institute, Department of Engineering Science, University of Oxford, OX1 3PJ Oxford, U.K.
Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies
Apr 1, 2026Uksang Yoo, Mengjia Zhu, Evan Pezent +8Motion Retargeting Robot Imitation Learning
The Robotics Institute, Carnegie Mellon University · Meta Reality Labs

Oct 5, 2026J/K move · Enter open · S save

AIM: Adaptive Interaction Modeling Networks for Real-to-Sim Soft-Body Simulation

Tiancheng Yang, Dingshuo Chen, Tianle Chen, Zhaocheng Liu+1

NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences+1

Robotics Simulation Robot Dynamics Learning Deformable Object Simulation

Open paper AskPDF arXiv

Deformable-object manipulation is essential for robotic tasks such as folding laundry and handling food, where robots must control shape changes as well as object motion. Predictive soft-body simulation supports these tasks by anticipating deformation under external interactions. However, spatial neighborhoods can misrepresent deformation dependencies, introducing local errors that accumulate over successive predictions. Models fitted to individual scenes must also accommodate changes in object geometry and manipulation conditions. In this work, we propose AIM, an Adaptive Interaction Modeling framework that treats real-to-sim soft-body simulation as a local-global interaction modeling problem. AIM uses motion history and geometry to adapt particle relations over current spatial neighbors and retained connections, while geometry-conditioned global communication coordinates object-wide responses. A unified kinematic control-point interface represents different manipulation configurations, and multi-step autoregressive supervision trains the model on its own predicted trajectories. Experiments on PhysTwin and PGND demonstrate improved motion accuracy and visual fidelity, with a 20.0% reduction in future-prediction tracking error relative to PhysTwin and a 22.8% reduction in mean long-horizon particle error across six object categories relative to PGND. The framework further supports transfer across actions, object instances, and scenes, including zero-shot transfer from robot interactions to human manipulation without target-domain dynamics fitting.

Abstract

Explore similar work

AIM: Adaptive Interaction Modeling Networks for Real-to-Sim Soft-Body Simulation

PneuTac: Tactile Manipulation with Soft Pneumatic Robots via Unified MPM-Gaussian Splatting Simulation

Functional Force-Aware Retargeting from Virtual Human Demos to Soft Robot Policies