RAFL: Generalizable Sim-to-Real of Soft Robots with Residual Acceleration Field Learning
Authors: Dong Heon Cho, Boyuan Chen
Organizations: Department of Computer Science, Duke University, 308 Research Dr., Durham, 27705, NC, USA · Department of Mechanical Engineering and Material Science, 121 Hudson Hall, Durham, 27708, NC, USA · Department of Electrical and Computer Engineering, 100 Science Dr., Durham, 27708, NC, USA
Differentiable simulators enable gradient-based optimization of soft robots over material parameters, control, and morphology, but accurately modeling real systems remains challenging due to the sim-to-real gap. This issue becomes more pronounced when geometry is itself a design variable. System identification reduces discrepancies by fitting global material parameters to data; however, when constitutive models are misspecified or observations are sparse, identified parameters often absorb geometry-dependent effects rather than reflect intrinsic material behavior. More expressive constitutive models can improve accuracy but substantially increase computational cost, limiting practicality. We propose a residual acceleration field learning (RAFL) framework that augments a base simulator with a transferable, element-level corrective dynamics field. Operating on shared local features, the model is agnostic to global mesh topology and discretization. Trained end-to-end through a differentiable simulator using sparse marker observations, the learned residual generalizes across shapes. In both sim-to-sim and sim-to-real experiments, our method achieves consistent zero-shot improvements on unseen morphologies, while system identification frequently exhibits negative transfer. The framework also supports continual refinement, enabling simulation accuracy to accumulate during morphology optimization.
Figures & tables
Fig. 1: RAFL overview. Our generalizable residual acceleration field framework is trained end-to-end to reduce error on recorded sparse marker trajectories, and is agnostic to mesh topology, allowing zero-shot transfer to unseen morphologies.
Fig. 2: RAFL pipeline overview. The network predicts element-level residual accelerations from local deformation/rate features, which are mass-scaled and accumulated via FEM quadrature to form residual forces. Trained end-to-end from sparse marker trajectories, the model transfers zero-shot across morphologies and discretizations.
Feature
Dimension
Description
\textscdiag(Σe)−1
3
Principal Stretches centered by 1
\textscflattened(FeTFe−I)
9
Right Cauchy Green Tensor centered by identity matrix
det(Fe)−1
1
Determinant of Fe centered by 1
\textscdiag(ReTDeRe)
3
Strain Rates along principal axes (co-rotated to rest frame)
Fig. 3: Example generalization trajectories for non-scaled geometries. SysID suffered from negative transfer, where the tuned simulation (i.e. blue curve) showed worse performance compared to the original simulation (i.e. red curve). Our framework showed consistent improvement in all generalization cases, and showed more qualitatively accurate oscillatory behavior within the trajectories.
Fig. 4: Sim-to-real trajectory visualization across beam geometries. The leftmost frame in each row shows the initial state from the recorded real-world video, with surface markers highlighted by a red bounding box. Subsequent panels show simulated frames over the first 0.4 seconds (sampled every 5 frames). Rows compare SysID (both) and our RAFL model. Blue spheres indicate measured marker positions from the real system, while red spheres indicate simulated marker positions.
Fig. 5: Expanded cross-shape generalization results. SysID frequently failed to transfer between morphologies, leading to negative transfer. In contrast, RAFL successfully demonstrated zero-shot accuracy improvement between meshes in most cases. RAFL also achieved better accuracy than SysID in most settings.
Fig. 6: Mean residual force norm per vertex. Residual forces collected for SRL are highly dependent on vertex position compared to those learned implicitly through RAFL.
Fig. A1: Mean marker errors of the generalization trajectories for non-scaled geometries. SysID suffers from negative transfer with greater mean marker error than the untuned simulation in the shorter and thicker beams.
Fig. A2: Final frames of representative generalization trajectories for non-scaled shorter and thicker geometries. Black arrows indicate the equilibrium configuration produced by the untuned simulator. Red arrows highlight negative transfer in SysID, where the equilibrium configuration deviates further from the real trajectory than the untuned simulation. Green arrows indicate successful zero-shot transfer with RAFL.
With the explosive growth of rigid-body simulators, policy learning in simulation has become the de facto standard for most rigid morphologies. In contrast, soft robotic simulation frameworks remain scarce and are seldom adopted by the soft robotics community. This gap stems partly from the lack of easy-to-use, general-purpose frameworks and partly from the high computational cost of accurately simulating continuum mechanics, which often renders policy learning infeasible. In this work, we demonstrate that rapid soft robot policy learning is indeed achievable via implicit time-stepping. Our simulator of choice, DisMech, is a general-purpose, fully implicit soft-body simulator capable of handling both soft dynamics and frictional contact. We further introduce delta natural curvature control, a method analogous to delta joint position control in rigid manipulators, providing an intuitive and effective means of enacting control for soft robot learning. To highlight the benefits of implicit time-stepping and delta curvature control, we conduct extensive comparisons across four diverse soft manipulator tasks against one of the most widely used soft-body frameworks, Elastica. With implicit time-stepping, parallel stepping of 500 environments achieves up to 6x faster speeds for non-contact cases and up to 40x faster for contact-rich scenarios. Finally, a comprehensive sim-to-sim gap evaluation--training policies in one simulator and evaluating them in another--demonstrates that implicit time-stepping provides a rare free lunch: dramatic speedups achieved without sacrificing accuracy.
Andrew Choi, Dezhong Tong, Xiaonan Huang
Horizon Robotics, Cupertino, CA, USA · Robotics Department at the University of Michigan, Ann Arbor, MI, USA
Accurate physics simulation is essential for robotic learning and control, yet analytical simulators often fail to capture complex contact dynamics, while learning-based simulators typically require large amounts of costly real-world data. To bridge this gap, we propose a few-shot real-to-sim approach that combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)-based models. Using only a small amount of real-world data, our method calibrates analytical simulators to generate large-scale synthetic datasets that capture diverse contact interactions. On this foundation, we introduce a mesh-based GNN that implicitly models rigid-body forward dynamics and derive surrogate gradients for collision detection, achieving full differentiability. Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories. In addition, the differentiable design supports gradient-based optimization, which we validate through simulation-based policy learning in multi-object interaction scenarios. Extensive experiments show that our framework not only improves simulation fidelity with minimal supervision but also increases the efficiency of policy learning. Taken together, these findings suggest that differentiable simulation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.
Zhenhao Huang, Siyuan Luo, Bingyang Zhou +3
National University of Singapore, Singapore · Prana Lab, San Francisco, USA
Calibrating a robot simulator's physics parameters (friction, damping, material stiffness) to match real hardware is often done by hand or with black-box optimizers that reduce error but cannot explain which physical discrepancies drive the error. When sensing is limited to external cameras, the problem is further compounded by perception noise and the absence of direct force or state measurements. We present Vid2Sid, a video-driven system identification pipeline that couples foundation-model perception with a VLM-in-the-loop optimizer that analyzes paired sim-real videos, diagnoses concrete mismatches, and proposes physics parameter updates with natural language rationales. We evaluate our approach on a tendon-actuated finger (rigid-body dynamics in MuJoCo) and a deformable continuum tentacle (soft-body dynamics in PyElastica). On sim2real holdout controls unseen during training, Vid2Sid achieves the best average rank across all settings, matching or exceeding black-box optimizers while uniquely providing interpretable reasoning at each iteration. Sim2sim validation confirms that Vid2Sid recovers ground-truth parameters most accurately (mean relative error under 13% vs. 28--98%), and ablation analysis reveals three calibration regimes. VLM-guided optimization excels when perception is clean and the simulator is expressive, while model-class limitations bound performance in more challenging settings.