GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins

Yichen Cai; Paul Jansonnie; Cristiana de Farias; Oleg Arenz; Jan Peters

cs.ROMar 5, 2026

GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins

Authors: Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters

Organizations: Intelligent Autonomous Systems Lab, Technical University of Darmstadt, Germany. · NAVER LABS Europe. · Hessian.AI, Germany. · German Research Center for AI (DFKI), SAIROL, Germany. ·

Abstract

Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.

Figures & tables

Fig. 1: Tracking performance of GaussTwin on rigid objects and DLO. The system takes multi-view camera observations as input. First, the objects in the scene are masked, represented by particles, and bonded to 3D Gaussians. The motion of these particles is then predicted at each time step using PBD, and subsequently refined through Gaussian splatting optimization. The bottom two rows illustrate the prediction–correction process carried out by GaussTwin.

Panel of Figure 2 — Fig. 2: On the left, we show the experimental setup, with the (i) Franka Research 3 robot with the custom end effector tool, (ii) the haptic interface for teleoperation, and (iii) the set of Intel RealSense D415 scene mounted cameras. On the right, we show the set of all objects used for real-robot experiments.

Fig. 3: We show qualitative results for tracking both the rope and T-shaped block objects. The ground truth, overlaid on the particle simulation, is shown in green, while the simulated spheres are depicted in blue. The arrow indicates the direction of the applied force. Below the particle representation, we show rendered 3D Gaussians throughout the experiment.

Fig. 4: Error bars showing rope tracking error for five different configurations. The error is measured as the IoU between ground-truth rope pixels and the projected spheres. We compare GaussTwin with two ablations: (i) GaussTwin using only the pose, and (ii) GaussTwin using only the mask.

Method	Push †	Push-Down †	Multi-object
PEGS +M ∗	0.59 $\pm$ 0.61	3.72 $\pm$ 2.43	0.58 $\pm$ 0.81
PEGS +M+P ∗	0.54 $\pm$ 0.57	3.40 $\pm$ 2.12	0.44 $\pm$ 0.62
RBD	0.42 $\pm$ 0.45	1.21 $\pm$ 0.94	0.48 $\pm$ 0.74
GaussTwin	0.34 $\pm$ 0.35	0.86 $\pm$ 0.60	0.38 $\pm$ 0.39

TABLE I: Baseline comparison on the dataset from [ 14 ]

Method	TE ( $\pm$ std)	RE ( $\pm$ std)	Lat.	TE ( $\pm$ std)	RE ( $\pm$ std)	Lat.	TE ( $\pm$ std)	RE ( $\pm$ std)	Lat.
	Single Object Push			Single Object Push Down			Multi-object
SM [ 14 ]	3.39 $\pm$ 2.18	33.64 $\pm$ 23.61	11	1.12 $\pm$ 0.91	6.19 $\pm$ 5.84	11	5.65 $\pm$ 4.77	29.35 $\pm$ 26.43	13
RBD [ 2 ]	3.49 $\pm$ 2.84	17.56 $\pm$ 20.01	10	0.99 $\pm$ 0.43	5.41 $\pm$ 4.77	10	4.06 $\pm$ 3.46	18.86 $\pm$ 18.97	12
GaussTwin (only mask)	0.60 $\pm$ 0.42	4.85 $\pm$ 3.32	35	0.91 $\pm$ 0.45	4.71 $\pm$ 3.50	35	2.08 $\pm$ 1.91	15.32 $\pm$ 15.08	40
GaussTwin (only pose)	3.40 $\pm$ 4.34	18.72 $\pm$ 20.31	11	1.15 $\pm$ 0.49	4.95 $\pm$ 3.43	11	10.18 $\pm$ 9.69	51.63 $\pm$ 42.25	13
GaussTwin (full)	0.43 $\pm$ 0.17	3.32 $\pm$ 1.46	36	0.70 $\pm$ 0.32	3.01 $\pm$ 1.80	36	0.85 $\pm$ 0.45	7.87 $\pm$ 6.11	42

TABLE II: Comparison of Methods on Different Real-World Tasks

	Reward	Pos. error (cm)	Yaw error (rad.)
1st Push	$-41.5\pm 41.2$	$1.4\pm 0.7$	$0.37\pm 0.42$
Last Push	$-3.48\pm 2.92$	$1.2\pm 0.7$	$0.01\pm 0.01$

TABLE III: Mean and standard deviation of reward, position and yaw errors after the first and final push.

Digital Twins

Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.

See all 253 papers

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos
Jun 23, 2026Pranjal Mishra, René Zurbrügg, Max Wilder-Smith +4Articulated Object Reconstruction 3D Gaussian Splatting
1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.
ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction
Sep 18, 2026Tim Engelbracht, René Zurbrügg, Mayank Mittal +4Digital Twins Robotic Manipulation
ETH Zurich · NVIDIA · Microsoft +1
VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning
Sep 27, 2026Zhenghao Xiao, Minting Pan, Nantian He +2Action-Conditioned Video Generation Robotic Manipulation
Carnegie Mellon University · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Jun 23, 2026J/K move · Enter open · S save

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos

Pranjal Mishra, René Zurbrügg, Max Wilder-Smith, Marco Hutter+3

1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.

Articulated Object Reconstruction 3D Gaussian Splatting 3D Reconstruction Digital Twins

Open paper AskPDF arXiv

Deploying robots in unstructured real-world environments needs accurate, interactive models of the objects. Constructing these models at scale remains a critical bottleneck for robotic system integration. We present ArtiTwinSplat, a framework that automatically constructs articulated, photo-realistic digital twins of objects directly from RGB-D videos, requiring no CAD models, simulation assets, or manual annotations. Our method is built on 3D Gaussian Splatting that preserve geometric fidelity and photometric realism, coupled with an unsupervised articulation discovery pipeline that recovers part structure and joint kinematics from observed motion alone. With tracking and optimization stages our method provides stable, queryable digital twins that support real-time rendering, viewpoint control, and interactive manipulation. Unlike prior methods confined to simulation, ArtiTwinSplat operates directly on real-world observations and produces twins that are immediately usable by downstream robot planning and learning systems. This method offers a practical, scalable pathway toward digital twin construction, lowering the integration barrier for articulated object manipulation in embodied AI and human-robot collaboration contexts.

Abstract

Digital Twins

Explore similar work

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos

ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning