Digital twins are increasingly used in robotics to simulate interactions, predict action outcomes, and support planning. However, constructing accurate digital twins requires precise estimates of physical properties such as mass, friction, and center of mass, which are often difficult to obtain in real-world environments. We propose probabilistic digital twins, which represent uncertain physical properties as probability distributions rather than fixed values. By simulating actions across these distributions, the twin predicts a range of possible outcomes and their likelihoods. These probabilistic simulations can guide both human and robot actions by identifying actions that are effective despite physical uncertainty. We demonstrate this framework through human-robot interaction scenarios, including interfaces that communicate simulated outcomes to users and robot planning strategies that explicitly account for uncertainty.
Figures & tables
Fig. 1 : Visualization of scalar consequence-field construction. (a) and (b) show the stability field, (c) and (d) show the impact field. Each sampled point represents an expected consequence obtained by averaging multiple simulated episodes over sampled physical realizations.
Fig. 2 : Visualization of location-valued consequence-field construction. The distribution of locations are sampled and constructed into a field.
Fig. 3 : User interface for probabilistic placement guidance. The live camera view (upper left) shows the physical scene and tracked objects, while the main view displays the corresponding Digital Twin with the reconstructed stability field over candidate support surfaces. The field is visualized from stable (blue) to unstable (red) and is used to provide placement guidance during interaction.
Fig. 4 : Four Isaac twin scenarios at representative sweep settings. Blue dots mark the fixed release candidates shared by all conditions; all cubes are 50mm . (a,b) Seesaws: plate COM 10/6\mathrm{mm} inside the supporting cube’s pivot edge; cube/cylinder riders land $r=10/11/12$ or $4/5/6\,$\mathrm{mm} beyond it, respectively (largest shown). Tipping depends on r relative to the balance point. (c) Kinked ramp: 15∘ upper face, θ2∈{26,29,32}∘ lower face ( 29∘ shown), and release 30mm downslope. Sliding depends on μpair versus tanθ2=0.488/0.554/0.625 (nominal μ=0.60 ). (d) Cube-on-cube: COM offset 20/25\mathrm{mm}$$ ( 25 shown, at the edge); geometry determines stability.
condition (belief)
tipping
sliding
requested pose, uncorrected
38.6[33.3,43.9]
32.9[27.2,38.4]
None reactive, no field
31.9[24.7,39.4]
32.9 —
Geo shape
25.0[18.3,31.9]
32.9 —
Geo+Pose shape + pose
0.3
32.9 —
Det physics, point
25.8[19.7,32.5]
16.2[10.0,23.0]
Prob physics, sampled (ours)
18.3[12.2,25.0]
11.7[6.1,17.8]
TABLE I : Failure rate (%), robot in the loop, K=4 rollouts. Tipping: 360 paired trials; sliding: 179. Brackets: 95% nested-bootstrap intervals; a dash marks a condition identical to None on every trial.
Fig. 5 : User study stacking tasks. (a) Simple stack (T1). (b) Complex stack (T2). (c) Balanced stack (T3).
Manipulating objects requires understanding not only their motion, but also the physical properties that determine it. For articulated objects, these include inertia, friction, and mechanisms such as springs or door closers, whose effects can vary with configuration and velocity. Such properties are not directly observable from appearance: visually identical doors may require very different effort to manipulate. Existing digital-twin pipelines recover primarily kinematics or assign static physical parameters from visual and language priors, which can yield physically implausible estimates. As a result, state-dependent mechanism dynamics remain unidentified and are not represented in standard asset formats. We present ForceTwin, a system for identifying physics-informed digital twins of articulated objects from instrumented human interaction. A person probes an object using a handheld force-sensing gripper, providing synchronized poses and interaction forces from which we estimate the articulation, parametric dynamics including inertia, Coulomb friction, viscous damping, and a structured neural residual capturing state-dependent mechanism forces. ForceTwin nearly halves the inertial-parameter error of a VLM prior. As a feedforward dynamics model for impedance control on a Spot and a Franka FR3, ForceTwin achieves 87% goal completion across nine object-embodiment pairs, compared with 60% using VLM-prior and 57% using kinematics-only twins, with the largest gains on objects whose strong mechanisms cause both baselines to stall. We further use the identified twins to train whole-body door-traversal policies and deploy them in the real world. Project Page: https://timengelbracht.github.io/forcetwin-website/
Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.
Yichen Cai, Paul Jansonnie, Cristiana de Farias +2
Intelligent Autonomous Systems Lab, Technical University of Darmstadt, Germany. · NAVER LABS Europe. · Hessian.AI, Germany. +2
Growing automation makes collaborative robots work in more variable environments, increasing the need for adaptation. We propose a human-in-the-loop online training framework combining a digital twin (DT), reinforcement learning (RL), and human demonstrations. Unlike DTs used mainly to generate synthetic data before task execution, our DT is synchronized with the physical system in real time through camera feeds, allowing the virtual robot to update its observations and policy from real-world feedback. A dual actor framework integrates imitation learning (IL) without adding a direct imitation loss to the RL actor, so demonstrations can guide adaptation instead of manual reprogramming. The proposed framework is demonstrated on the Ufactory Xarm5 collaborative robot, where the robot's end-effector aims to reach the target position while avoiding obstacles. The experiments show that the framework can resume training after a change in the physical workspace and that, with a fixed set of non-optimal demonstrations, the dual actor framework achieves a much higher final success rate than two methods that add an imitation loss to the actor. The same pattern holds with real human demonstrations collected in virtual reality (VR): with demonstrations that never reach the goal, the dual actor framework reached 83-100% mean deterministic evaluation success, against 0-17% for the two imitation-loss methods.
Yuzhu Sun, Mien Van, Nguyen Minh Nhat +2
School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Northern Ireland, UK