Organizations: Physical Superintelligence (PSI) Lab, University of Southern California · Viterbi School of Engineering, University of Southern California · Department of EECS, University of California, Berkeley
Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially in physical parameter identification. In this work, we introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine, enabling object mass identification from real-world visual observations and robot control signals, while enabling grasping policy learning simultaneously. Through optimizing the mass of the manipulated object, our method automatically builds high-fidelity and physically plausible digital twins. Additionally, we propose a novel approach to train force-aware grasping policies from limited data by transferring feasible human demonstrations into simulated robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values. Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping, effectively reducing the sim-to-real gap.
Figures & tables
Figure 1: We present D-REX, a differentiable real-to-sim-to-real engine that enables 4D photorealistic rendering and physical simulation by identifying object mass from real-world visual observations and robot interaction data. D-REX reconstructs object geometry using Gaussian Splat representations and leverages a differentiable physics engine for end-to-end mass identification. The identified mass is then used to enable force-aware policy learning from human demonstrations, supporting robust grasping and sim-to-real transfer in dexterous grasping tasks.
Figure 2: Overview of our method. Our approach consists of four components: (1) Real-to-Sim, (2) Mass Identification, (3) Learning from Human Demonstrations, and (4) Policy Learning. We begin by capturing videos of the scene and human demonstrations. Robotic actions are then executed in both simulation and the real world to identify object mass via our differentiable physics engine. Lastly, a manipulation policy is trained using the demonstrations and identified mass.
Figure 3: Objects for Mass Identification. We conduct experiments on mass identification across diverse object geometries and identical geometries with varying densities. Our method accurately estimates mass in both settings, demonstrating robustness to shape and density variations.
Table 4
Figure 4: Quantitative Results of Mass Identification. We show the real-world object pushing (top) and render object trajectories using Gaussian Splats: simulated with optimized mass (middle), and simulated with a lighter mass (bottom), all using the same robot actions. The optimized mass closely reproduces real-world dynamics, reducing the sim-and-real gap with high visual fidelity.
Figure 6
Figure 6: Qualitative Results. Left to right: policies trained on medium, light, and heavy objects. Only the mass-matched policy achieves stable grasps, while mismatched ones fail due to excessive or insufficient force, causing bounce-off for lighter objects or slippage for heavier ones.
Figure 7: Quantitative Results of Grasping Policies. Grasp success rates across eight objects with varying geometries and mass values, with the average and standard deviation of each method.
Figure 8: Qualitative Results of Our Policy. We evaluate our force-aware grasping policy across various objects. The first row illustrates the approach to the pre-grasp pose, while the second row shows two post-grasp positions, demonstrating that the policy achieves stable, secure grasps.
Object
# Images
COLMAP+3DGS+2DGS runtime
Mesh vertices
Lego
306
30 min
47,266
Ketchup
337
35 min
13,487
Domino
320
30 min
35,445
Cookie
309
30 min
25,682
A
306
30 min
67,895
U
327
33 min
15,322
Table 4: Offline reconstruction statistics for representative objects, including the number of input images, total runtime, and the number of vertices in the extracted mesh.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Object
Ground truth (g)
Semi-implicit (g)
Explicit (g)
Semi-implicit (s)
Explicit (s)
Lego
59
51
34
1.38
1.21
Ketchup
726
667
685
1.43
1.19
Cookie
210
200
189
1.41
1.18
Domino
106
117
135
1.36
1.17
U
125
110
98
1.39
1.20
A
134
145
120
1.42
1.22
Appendix
Table 5: Ablation study comparing semi-implicit and explicit Euler schemes for mass identification. We report the ground-truth mass, the identified mass under each integrator, and the average optimization time per iteration (in seconds). Semi-implicit Euler yields more accurate mass estimates at a modest runtime cost.
Component
Operation
Output Dim.
Details
Input
Positional Encoding
N×3
N object vertices (XYZ)
Linear Layer 1
Fully Connected
256
Input: 3→256
Activation 1
ReLU
256
Non-linearity
Linear Layer 2
Fully Connected
256
256→256
Activation 2
ReLU
256
Non-linearity
Linear Layer 3
Fully Connected
256
256→256
Appendix
Table 6: Architecture of the proposed GraspMLP network. The input consists of per-vertex 3D coordinates. The shared backbone maps the input into a latent feature space, which is subsequently decoded into separate heads for predicting joint actions, contact-based reward signals, and grasping force.
Method
Opt. runtime
PSNR ↑
SSIM ↑
LPIPS ↓
4DGS Wu et al. [2024a]
30 min
17.14
0.632
0.411
D-REX (3DGS + opt. traj.)
20 min
23.54
0.783
0.237
Appendix
Table 7: Real-to-sim reconstruction quality on the real push–ketchup sequence. Our D-REX pipeline improves photometric metrics over vanilla 4DGS while reducing optimization time.
Training Object
Test Object
Success Rate
10×3×3cm,600g
7×2×2cm,500g
90%→70%
Appendix
Table 8: Cross-object generalization from a larger to a smaller electric screwdriver. The policy is trained on five human demonstrations and conditioned on object mass and reconstructed mesh.
Method
117 g (Light)
206 g (Medium)
324 g (Heavy)
CrossDex
4/10
7/10
9/10
Ours
9/10
10/10
9/10
Appendix
Table 9: Real-world grasp success across mass variants of a single object geometry ( Symbol Y ); 10 trials per condition. CrossDex was trained with mass randomization in [0.5,1.5]kg ; all test masses are below this range. Higher is better.
Figure 9: Scaling performance of our force-aware grasping policy with increasing number of demonstrations.
Figure 10: Quantitative Results of Object Grasping trained on the heavy one. Only the mass-matched policy achieves stable grasps, while mismatched ones fail due to excessive or insufficient force, causing bounce-off for lighter objects or slippage for heavier ones.
Figure 11: Mass-Loss curves. We present three examples of our system applied to mass identification. The blue curves represent the estimated masses, all of which converge reliably to the ground-truth values, demonstrating the accuracy of our approach.
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom