Organizations: Physical Superintelligence (PSI) Lab, University of Southern California · Viterbi School of Engineering, University of Southern California · Department of EECS, University of California, Berkeley
Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially in physical parameter identification. In this work, we introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine, enabling object mass identification from real-world visual observations and robot control signals, while enabling grasping policy learning simultaneously. Through optimizing the mass of the manipulated object, our method automatically builds high-fidelity and physically plausible digital twins. Additionally, we propose a novel approach to train force-aware grasping policies from limited data by transferring feasible human demonstrations into simulated robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values. Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping, effectively reducing the sim-to-real gap.
Figures & tables
Figure 1: We present D-REX, a differentiable real-to-sim-to-real engine that enables 4D photorealistic rendering and physical simulation by identifying object mass from real-world visual observations and robot interaction data. D-REX reconstructs object geometry using Gaussian Splat representations and leverages a differentiable physics engine for end-to-end mass identification. The identified mass is then used to enable force-aware policy learning from human demonstrations, supporting robust grasping and sim-to-real transfer in dexterous grasping tasks.
Figure 2: Overview of our method. Our approach consists of four components: (1) Real-to-Sim, (2) Mass Identification, (3) Learning from Human Demonstrations, and (4) Policy Learning. We begin by capturing videos of the scene and human demonstrations. Robotic actions are then executed in both simulation and the real world to identify object mass via our differentiable physics engine. Lastly, a manipulation policy is trained using the demonstrations and identified mass.
Figure 3: Objects for Mass Identification. We conduct experiments on mass identification across diverse object geometries and identical geometries with varying densities. Our method accurately estimates mass in both settings, demonstrating robustness to shape and density variations.
Table 4
Figure 4: Quantitative Results of Mass Identification. We show the real-world object pushing (top) and render object trajectories using Gaussian Splats: simulated with optimized mass (middle), and simulated with a lighter mass (bottom), all using the same robot actions. The optimized mass closely reproduces real-world dynamics, reducing the sim-and-real gap with high visual fidelity.
Figure 6
Figure 6: Qualitative Results. Left to right: policies trained on medium, light, and heavy objects. Only the mass-matched policy achieves stable grasps, while mismatched ones fail due to excessive or insufficient force, causing bounce-off for lighter objects or slippage for heavier ones.
Figure 7: Quantitative Results of Grasping Policies. Grasp success rates across eight objects with varying geometries and mass values, with the average and standard deviation of each method.
Figure 8: Qualitative Results of Our Policy. We evaluate our force-aware grasping policy across various objects. The first row illustrates the approach to the pre-grasp pose, while the second row shows two post-grasp positions, demonstrating that the policy achieves stable, secure grasps.
Object
# Images
COLMAP+3DGS+2DGS runtime
Mesh vertices
Lego
306
30 min
47,266
Ketchup
337
35 min
13,487
Domino
320
30 min
35,445
Cookie
309
30 min
25,682
A
306
30 min
67,895
U
327
33 min
15,322
Table 4: Offline reconstruction statistics for representative objects, including the number of input images, total runtime, and the number of vertices in the extracted mesh.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Object
Ground truth (g)
Semi-implicit (g)
Explicit (g)
Semi-implicit (s)
Explicit (s)
Lego
59
51
34
1.38
1.21
Ketchup
726
667
685
1.43
1.19
Cookie
210
200
189
1.41
1.18
Domino
106
117
135
1.36
1.17
U
125
110
98
1.39
1.20
A
134
145
120
1.42
1.22
Appendix
Table 5: Ablation study comparing semi-implicit and explicit Euler schemes for mass identification. We report the ground-truth mass, the identified mass under each integrator, and the average optimization time per iteration (in seconds). Semi-implicit Euler yields more accurate mass estimates at a modest runtime cost.
Component
Operation
Output Dim.
Details
Input
Positional Encoding
N×3
N object vertices (XYZ)
Linear Layer 1
Fully Connected
256
Input: 3→256
Activation 1
ReLU
256
Non-linearity
Linear Layer 2
Fully Connected
256
256→256
Activation 2
ReLU
256
Non-linearity
Linear Layer 3
Fully Connected
256
256→256
Appendix
Table 6: Architecture of the proposed GraspMLP network. The input consists of per-vertex 3D coordinates. The shared backbone maps the input into a latent feature space, which is subsequently decoded into separate heads for predicting joint actions, contact-based reward signals, and grasping force.
Method
Opt. runtime
PSNR ↑
SSIM ↑
LPIPS ↓
4DGS Wu et al. [2024a]
30 min
17.14
0.632
0.411
D-REX (3DGS + opt. traj.)
20 min
23.54
0.783
0.237
Appendix
Table 7: Real-to-sim reconstruction quality on the real push–ketchup sequence. Our D-REX pipeline improves photometric metrics over vanilla 4DGS while reducing optimization time.
Training Object
Test Object
Success Rate
10×3×3cm,600g
7×2×2cm,500g
90%→70%
Appendix
Table 8: Cross-object generalization from a larger to a smaller electric screwdriver. The policy is trained on five human demonstrations and conditioned on object mass and reconstructed mesh.
Method
117 g (Light)
206 g (Medium)
324 g (Heavy)
CrossDex
4/10
7/10
9/10
Ours
9/10
10/10
9/10
Appendix
Table 9: Real-world grasp success across mass variants of a single object geometry ( Symbol Y ); 10 trials per condition. CrossDex was trained with mass randomization in [0.5,1.5]kg ; all test masses are below this range. Higher is better.
Figure 9: Scaling performance of our force-aware grasping policy with increasing number of demonstrations.
Figure 10: Quantitative Results of Object Grasping trained on the heavy one. Only the mass-matched policy achieves stable grasps, while mismatched ones fail due to excessive or insufficient force, causing bounce-off for lighter objects or slippage for heavier ones.
Figure 11: Mass-Loss curves. We present three examples of our system applied to mass identification. The blue curves represent the estimated masses, all of which converge reliably to the ground-truth values, demonstrating the accuracy of our approach.
Human-like dexterous hands with multiple fingers offer human-level manipulation capabilities but remain difficult to train the control policies that can deploy on real hardware due to contact-rich physics and imperfect actuation. We present a sim-to-real reinforcement learning method that leverages dense tactile feedback combined with joint torque sensing to explicitly regulate physical interactions. To enable effective sim-to-real transfer, we introduce (i) a computationally fast tactile simulation that computes distances between dense virtual tactile units and the object via parallel forward kinematics, providing high-rate, high-resolution touch signals needed by RL; (ii) a current-to-torque calibration that eliminates the need for torque sensors on dexterous hands by mapping motor current to joint torque; and (iii) actuator dynamics modeling with randomization to account for non-ideal torque-speed effects and bridge the actuation gaps. Using an asymmetric actor-critic PPO pipeline, we train policies entirely in simulation and deploy them directly to a five-finger hand. The resulting policies demonstrate two essential human-hand skills: (1) command-based controllable grasp force tracking and (2) reorientation of objects in the hand, both of which are robustly executed without fine-tuning on the robot. By combining tactile and torque in the observation space with scalable sensing and actuation modeling, our system provides a practical solution to achieve reliable dexterous manipulation. To our knowledge, this is the first demonstration of controllable grasping on a multi-finger dexterous hand trained entirely in simulation and transferred zero-shot on real hardware.
Zhe Zhao, Zhibin Li, Yilin Ou +1
State Key Laboratory of Networking and Switching Technology Beijing University of Posts and Telecommunications, China · University College London, United Kingdom
Blind grasping with a dexterous hand is a crucial manipulation capability. Nevertheless, learning such tactile-only policies for real robots remains challenging due to the tactile sim-to-real gap and the limited expressiveness of sparse tactile signals. To bridge this gap, we propose a framework for tactile-only blind grasping that is deployable on a physical multi-fingered robotic hand. Our approach combines three key components. First, we introduce a Real2Sim tactile calibration pipeline that constructs a contact-calibrated digital-twin simulator capable of reproducing real tactile signals. Second, we improve the expressiveness of sparse tactile observations using a layout-aware tactile encoder, which incorporates sensor-geometry priors through self-supervised pretraining. Third, to improve generalization to unseen objects, we train object-specific reinforcement-learning experts in the calibrated simulator and aggregate their successful grasp trajectories into a tactile-conditioned Diffusion Policy. We evaluate our method on a physical LEAP Hand equipped with distributed tactile sensing across 10 seen and 10 unseen objects. The deployed policy achieves a 27% real-world grasp success rate across all 20 objects, without real-world grasping demonstrations or visual input. Simulation ablations show that layout-aware tactile pretraining improves grasping performance, while sensing-level evaluations confirm that Real2Sim calibration increases the consistency of tactile contact events between simulation and hardware. Together, these results suggest that contact-event calibration, geometry-aware tactile representation learning, and diffusion-based policy aggregation provide an effective path toward tactile-only blind grasping on real dexterous robotic hands. Project page:Dex-Blind-Grasp.github.io.
Shengcheng Luo, Xiyan Huang, Zhe Xu +3
1ShanghaiTech University · 2Beijing Institute for General Artificial Intelligence
Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/
Jin Wu, Lianjie Yuan, Zeyan Sun +4
The University of Texas at Austin · University of Florida · BrainCo