Accurate physics simulation is essential for robotic learning and control, yet analytical simulators often fail to capture complex contact dynamics, while learning-based simulators typically require large amounts of costly real-world data. To bridge this gap, we propose a few-shot real-to-sim approach that combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)-based models. Using only a small amount of real-world data, our method calibrates analytical simulators to generate large-scale synthetic datasets that capture diverse contact interactions. On this foundation, we introduce a mesh-based GNN that implicitly models rigid-body forward dynamics and derive surrogate gradients for collision detection, achieving full differentiability. Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories. In addition, the differentiable design supports gradient-based optimization, which we validate through simulation-based policy learning in multi-object interaction scenarios. Extensive experiments show that our framework not only improves simulation fidelity with minimal supervision but also increases the efficiency of policy learning. Taken together, these findings suggest that differentiable simulation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.
Figures & tables
Fig. 1 : Top : A blue cube is pushed to collide with a green cube, with the objective of stopping it at the red target area. The initial pushing velocity of the blue cube is optimized using stochastic gradient descent in our differentiable simulator, successfully achieving the goal. Bottom : Convergence of optimization across multiple runs, showing the loss curve (left) and the magnitude of the initial pushing velocity (right).
Simulator
Contact Model
Contact Accuracy
Scalability
Differentiable
IsaacLab
PBD [ 20 ]
Medium
High
✗
MuJoCo
CCP
High
High
✗
Brax
LCP
Low
Medium
✓
ContactNets
DNN+LCP
Medium
Low
✗
FIGNet
GNN
Medium
Medium
✗
Ours
GNN
High
High
✓
TABLE I : Comparison of our simulator against existing analytical and learned simulators.
Fig. 2 : Illustration of our framework. (1) We utilize sampling-based identification of contact parameters to minimize the trajectory loss between real world and simulation, followed by (2) data scaling in terms of quantity, geometry, etc. (3) The GNN is trained on the scaled dataset, equipped with differential collision detection and shape matching to achieve differentiability. The resulting simulator is capable of accurately modeling complex contact interactions.
Fig. 3 : (a) We collect real-world data on a tabletop setup with two cubes, considering a quasi-planar frictional contact scenario where one cube is pushed towards another cube at rest. (b) The poses of the cubes are estimated using TagSLAM.
Simulator
Parameter
Range
All Simulators
μ
[0,1]
d0 ( solimp )
[0.9,0.95]
dwidth ( solimp )
[0.95,0.99]
MuJoCo
Width ( solimp )
[0.0001,0.01]
&
Midpoint ( solimp )
[0.001,0.1]
Brax (Generalized)
Power ( solimp )
[1,5]
TABLE II : Contact Parameters Optimization Range
Fig. 4 : Comparison of trajectory errors (see Eq. ( 1 )) in MuJoCo before and after contact parameter identification. The identified parameters significantly enhance the simulation’s accuracy in replicating real-world contact dynamics.
Fig. 5 : Illustration of a scenario where the moving cube (green) collides with the static cube (red) before and after contact parameter identification.
Fig. 6 : A rollout example from our GNN-based simulator, illustrating a cube is pushed and collides with a bowling-like array of ten cubes. The ground truth is the scaled data generated by the identified MuJoCo. The output trajectory of our simulator is close to the ground truth, showing that our simulator accurately captures the dynamics and interactions between objects.
Fig. 7 : The positional and angular errors of our GNN-based simulator against analytical and learned baselines. Our simulator achieves comparable performance to identified MuJoCo and outperforms all pipelines of identified Brax, while our data scaling method further enhances the accuracy of our simulator.
Robotics simulators have improved significantly in computational speed and scalability, enabling them to generate years of simulated data for complex systems in minutes or hours. Despite these advances, efficiently and accurately computing simulation derivatives remains an open challenge. Addressing this would accelerate the convergence of reinforcement learning and trajectory optimization algorithms, particularly for contact-rich problems. This paper introduces a unifying framework for robotic simulation that accounts for all factors, including dynamics, collisions, and friction. The resulting algorithm computes analytical derivatives of the simulation by implicit differentiation, explicitly handling the intrinsic non-smoothness of the collision and frictional stages while exploiting the sparsity induced by the multi-body structure. Benchmark results demonstrate state-of-the-art performance, with timings ranging from 5μs for a 7-dof manipulator to 95μs for a 36-dof humanoid, an improvement of at least two orders of magnitude over alternative methods. Implemented in C++, the code will be open-sourced after the review process to support applications such as simulation-driven learning and real-time control.
Quentin Le Lidec, Louis Montaut, Yann de Mont-Marin +3
Inria, Ecole normale sup´erieure CNRS, PSL Research University 75005 Paris, France
Differentiable simulation provides analytic gradients of robot dynamics, enabling fast and sample-efficient first-order policy optimization. However, obtaining smooth and informative gradients through rigid-body contact typically requires softened contact models, often at the expense of physical fidelity and thereby limiting learned policies largely to simulation. This trade-off becomes particularly consequential for dynamic humanoid motions, where accurate contact dynamics are critical for transferring policies to the real world. Increasing contact stiffness in rigid-body simulation improves the fidelity of interactions, but also makes the dynamics increasingly sensitive to small state perturbations, producing high-variance gradients that can destabilize first-order policy learning. To address this, we propose Bundled Contact Gradients (BCG), a contact-local randomized smoothing framework for differentiable policy learning. When stiff contact is detected, our method evaluates a local bundle of randomized perturbation rollouts around the stiff contact configuration and aggregates their gradient signal thereby reducing gradient variance. We demonstrate the effectiveness of our method by successfully training and transferring dynamic motions zero-shot onto a real-world Unitree G1 humanoid platform. Videos and supplementary information can be found at https://bundledcontactgradients.github.io/
Dyuman Aditya, Jin Cheng, Clemens Schwarke +4
University of Southern California, LA, USA. · ETH Zurich, Switzerland. · NVIDIA. +1
Robotic manipulation with deformable objects represents a data-intensive regime in embodied learning, where shape, contact, and topology co-evolve in ways that far exceed the variability of rigids. Although simulation promises relief from the cost of real-world data acquisition, prevailing sim-to-real pipelines remain rooted in rigid-body abstractions, producing mismatched geometry, fragile soft dynamics, and motion primitives poorly suited for cloth interaction. We posit that simulation fails not for being synthetic, but for being ungrounded. To address this, we introduce SIM1, a physics-aligned real-to-sim-to-real data engine that grounds simulation in the physical world. Given limited demonstrations, the system digitizes scenes into metric-consistent twins, calibrates deformable dynamics through elastic modeling, and expands behaviors via diffusion-based trajectory generation with quality filtering. This pipeline transforms sparse observations into scaled synthetic supervision with near-demonstration fidelity. Experiments show that policies trained on purely synthetic data achieve parity with real-data baselines at a 1:15 equivalence ratio, while delivering 90% zero-shot success and 50% generalization gains in real-world deployment. These results validate physics-aligned simulation as scalable supervision for deformable manipulation and a practical pathway for data-efficient policy learning.
Yunsong Zhou, Hangxu Liu, Xuekun Jiang +12
Shanghai AI Lab · Fudan University · Shanghai Jiao Tong University +1