SyncTwin: Fast Digital Twin Construction and Synchronization for Safe Robotic Manipulation
Organizations: University of California, Riverside · University of Southern California
Abstract
Accurate and safe robotic manipulation under dynamic and visually occluded conditions remains a core challenge in real-world deployment. We introduce SyncTwin, a novel digital twin framework that unifies fast 3D scene reconstruction and real-to-sim synchronization for robust and safety-aware robotic manipulation in such environments. In the offline stage, we employ VGGT to rapidly reconstruct object-level 3D assets from RGB images, forming a reusable geometry library. During execution, SyncTwin continuously synchronizes the digital twin by tracking real-world object states and uses stored assets to complete partially observed geometry. The synchronized twin enables motion planners to compute collision-aware robot trajectories in simulation, with execution on the real robot closing the real-to-sim-to-real loop. Experiments in dynamic and occluded scenes show that SyncTwin improves manipulation performance and motion safety, demonstrating the effectiveness of digital twin synchronization for real-world robotic execution.
Figures & tables
| Method | Motion | Trans. Velocity | Unseen | Seen (Memory Bank) | ||||||||||||||||||||||||
| Box1 | Box2 | Box3 | Box4 | Box1* | Box2* | Box3* | Box4* | |||||||||||||||||||||
| FA | EA | CO | FA | EA | CO | FA | EA | CO | FA | EA | CO | mean SD | FA | EA | CO | FA | EA | CO | FA | EA | CO | FA | EA | CO | mean SD | |||
| NVBlox | SelfRot | 0 m/s | 6 | 4 | 10 | 9 | 5 | 6 | 3 | 4 | 13 | 3 | 11 | 6 | 50.3 15.1 | – | – | – | – | – | – | – | – | – | – | – | – | – |
| EnterTraj | 0.2 m/s | 3 | 9 | 8 | 4 | 9 | 7 | 1 | 6 | 13 | 0 | 3 | 17 | 37.0 20.4 | – | – | – | – | – | – | – | – | – | – | – | – | – | |
| SyncTwin | SelfRot | 0 m/s | 12 | 7 | 1 | 14 | 5 | 1 | 11 | 7 | 2 | 9 | 9 | 2 | 85.5 4.2 | 18 | 1 | 1 | 16 | 4 | 0 | 15 | 5 | 0 | 13 | 6 | 1 | 93.5 3.1 |
| EnterTraj | 0.2 m/s | 8 | 8 | 4 | 11 | 6 | 3 | 8 | 7 | 5 | 7 | 8 | 5 | 71.5 5.4 | 12 | 7 | 1 | 12 | 5 | 3 | 10 | 6 | 4 | 9 | 7 | 4 | 78.8 6.9 | |
| Box | Cup | Bottle | Ketchup | Overall (mean SD) | Four-object tracking | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Condition | Trans. | Rot. | Trans. | Rot. | Trans. | Rot. | Trans. | Rot. | Trans. | Rot. | Rate (FPS) | GPU memory (GiB) |
| RT-ICP | Unoccluded | 95.1 | 78.3 | 96.3 | 88.7 | 94.8 | 90.4 | 94.7 | 82.0 | 95.23 0.74 | 84.85 5.68 | 3.43 | 5.70 0.50 / 6.10 |
| Occluded | 85.2 | 70.4 | 87.7 | 78.2 | 90.3 | 82.6 | 91.5 | 76.3 | 88.68 2.81 | 76.88 5.06 | – | - | |
| FP | Unoccluded | 97.1 | 93.2 | 97.4 | 92.1 | 98.7 | 95.3 | 97.4 | 96.8 | 97.65 0.71 | 94.35 2.10 | 11.57 | 7.70 0.38 / 16.03 |
| Occluded | 91.5 | 89.3 | 94.2 | 90.8 | 96.3 | 92.6 | 95.1 | 93.7 | 94.28 2.04 | 91.60 1.94 | – | – | |
| Component | RT-ICP (mean SD) | FP (mean SD) |
| Camera preprocessing | 1.35 0.71 ms | 1.28 0.41 ms |
| Detection ∗ | 5.83 0.55 ms | 5.90 0.79 ms |
| Segmentation ∗ | 134.36 40.48 ms | 138.51 44.99 ms |
| FP object registration ∗ | N/R | 650.13 254.20 ms |
| Point-cloud extraction | 1.78 0.52 ms | N/R |
| V-HACD generation + | 111.41 20.18 ms | N/R |
Digital Twins
Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.
See all 274 papers