ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

Tim Engelbracht; René Zurbrügg; Mayank Mittal; Marco Hutter; Marc Pollefeys; Hermann Blum; Zuria Bauer

cs.ROSep 18, 2026

ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

Authors: Tim Engelbracht, René Zurbrügg, Mayank Mittal, Marco Hutter, Marc Pollefeys, Hermann Blum, Zuria Bauer

Organizations: ETH Zurich · NVIDIA · Microsoft · University of Bonn

Abstract

Manipulating objects requires understanding not only their motion, but also the physical properties that determine it. For articulated objects, these include inertia, friction, and mechanisms such as springs or door closers, whose effects can vary with configuration and velocity. Such properties are not directly observable from appearance: visually identical doors may require very different effort to manipulate. Existing digital-twin pipelines recover primarily kinematics or assign static physical parameters from visual and language priors, which can yield physically implausible estimates. As a result, state-dependent mechanism dynamics remain unidentified and are not represented in standard asset formats. We present ForceTwin, a system for identifying physics-informed digital twins of articulated objects from instrumented human interaction. A person probes an object using a handheld force-sensing gripper, providing synchronized poses and interaction forces from which we estimate the articulation, parametric dynamics including inertia, Coulomb friction, viscous damping, and a structured neural residual capturing state-dependent mechanism forces. ForceTwin nearly halves the inertial-parameter error of a VLM prior. As a feedforward dynamics model for impedance control on a Spot and a Franka FR3, ForceTwin achieves 87% goal completion across nine object-embodiment pairs, compared with 60% using VLM-prior and 57% using kinematics-only twins, with the largest gains on objects whose strong mechanisms cause both baselines to stall. We further use the identified twins to train whole-body door-traversal policies and deploy them in the real world. Project Page: https://timengelbracht.github.io/forcetwin-website/

Figures & tables

Fig. 1: The ForceTwin pipeline. Instrumented human interaction provides synchronized tool poses and contact wrenches from which we identify an articulation model and its generalized-effort dynamics. Geometry is reconstructed from the same recording or supplied independently, and the physical model is registered to it to produce a twin for robot control or simulation.

Fig. 2: Construction of a physics-informed twin. (a) Reconstructed panel instance, associated handle, and registered joint axis. (b) Exported articulated twin. The physical model is transformed into the asset frame, reparameterized to match its joint-coordinate convention, and attached to the corresponding articulated instance.

Fig. 3: Dynamics model fit for the wood door. (a) Measured generalized effort (gray) against the parametric model alone (dashed) and with the structured mechanism added (green); (b) the probing trajectory. The parametric model tracks the shape of the effort but is offset from it by a configuration-dependent amount, which the learned mechanism recovers. Shaded bands mark the probing regimes for inertia (yellow), viscous damping (purple), and the mechanism (blue).

	Prismatic	Revolute		Type acc. [%] $\uparrow$
Method	$\theta_{\mathrm{err}}$ [ ∘ ] $\downarrow$	$\theta_{\mathrm{err}}$ [ ∘ ] $\downarrow$	$d_{L_{2}}$ [mm] $\downarrow$	Pris.	Rev.
ForceTwin (Ours)	1.4 $\pm$ 0.2	1.5 $\pm$ 0.8	23 $\pm$ 26	100.0	100.0
MoMa-SG [ 2 ]	1.6 $\pm$ 0.8	1.7 $\pm$ 0.0	46 $\pm$ 20	100.0	100.0
OPDMulti [ 14 ]	4.8 †	34.9 $\pm$ 29.1	143 $\pm$ 117	100.0	54.1

TABLE I: Articulation Axis Estimation. Evaluated on the real-world test bench against 12 axes annotated in a Leica RTC360 laser scan. Metrics follow ArtiPoint [ 17 ] :

\theta_{\mathrm{err}}

is the angle between axis lines, folded to

[0,90]^{\circ}

, and

d_{L_{2}}

the distance between revolute axis lines along their common perpendicular. Errors accumulate only over predictions with the correct joint type. Values are means with the sample standard deviation in smaller type; † no deviation available. Bold indicates best performance, underline second best; tied entries are all marked best.

Fig. 4: Inertial-parameter fidelity on the four objects for which ground truth is available: estimate against ground truth on logarithmic axes, so an estimate on the diagonal is exact and vertical distance from it is the error. Marker shape distinguishes revolute (circle) from prismatic (square) joints, whose parameters carry the two units given on the axes. Nested shaded bands mark agreement to within

\pm 25\%

(inner) and to within a factor of two (outer). The macro mean of the absolute relative errors is

29.5\%

for ForceTwin and

53.0\%

for the VLM prior.

Table 6

Fig. 5: Real-world experiments of zero-shot model-based articulation ( Table III ) across two embodiments: Spot and Franka FR3. We visualize the initial and final articulation states as blended overlays for both embodiments. The recovered quasi-static mechanism torque for each object is plotted over the opening trajectory, highlighting the variation in mechanism behavior across objects.

Fig. 6: Policy learning with identified door models. Left: We use the identified wood- and metal-door models (real-to-sim) to train whole-body policies, which we then deploy on ANYmal (sim-to-real). Right: Both policies on the real wood door; shaded regions show the average over five trials. The wood-door policy moves the end-effector smoothly, while the metal-door policy stops and starts, pulling hard and then pausing, which gives it higher jerk (bottom) due to the worse sim-to-real calibration.

Digital Twins

Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.

See all 253 papers

Jul 19, 2026

BoxTwin: Learning Elastoplastic Articulated Object Dynamics from Videos

Digital twins enable robots to anticipate and adapt to physical interactions, but existing models struggle with elastoplastic articulated objects (EAOs) that exhibit nonlinear elasticity, plastic yielding, and damage accumulation. We present BoxTwin, an interactive digital twin framework that learns the full dynamics of EAOs from videos. Our pipeline reconstructs the scene, identifies a physics aware constitutive model for each EAO. Experiments on manual folding and dual arm manipulation of EAOs show that BoxTwin accurately tracks joint trajectories and reproduces post contact plastic behavior over long horizons. By integrating video driven reconstruction with elastoplastic damage modeling, BoxTwin advances digital twins toward predictive, adaptive control of deformable articulated objects in unstructured environments.

Heng Zhang, Gehan Zheng, Kaifeng Zhang +6

Heng Zhang, Gehan Zheng, Kaifeng Zhang, Jay Song, Shivansh Patel, Sonny Hu, Yunzhu Li and Changxi Zheng are with SceniX, Inc. · Peter Yichen Chen is with the University of British Columbia.

Articulated Object Reconstruction Deformable Object Manipulation Robotic Manipulation Digital Twins Deformable Object Simulation

Ask PDF

Oct 14, 2026

Probabilistic Physics Digital Twin for Human-Robot Interaction

Digital twins are increasingly used in robotics to simulate interactions, predict action outcomes, and support planning. However, constructing accurate digital twins requires precise estimates of physical properties such as mass, friction, and center of mass, which are often difficult to obtain in real-world environments. We propose probabilistic digital twins, which represent uncertain physical properties as probability distributions rather than fixed values. By simulating actions across these distributions, the twin predicts a range of possible outcomes and their likelihoods. These probabilistic simulations can guide both human and robot actions by identifying actions that are effective despite physical uncertainty. We demonstrate this framework through human-robot interaction scenarios, including interfaces that communicate simulated outcomes to users and robot planning strategies that explicitly account for uncertainty.

Yushen Hu, Harshvardhan Chandirasekar, Apurv Kushwaha +2

University of Minnesota, USA · Howard University, USA

Digital Twins Robotics Simulation Human-Robot Interaction Robot Task Planning

Ask PDF

Jun 23, 2026

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos

Deploying robots in unstructured real-world environments needs accurate, interactive models of the objects. Constructing these models at scale remains a critical bottleneck for robotic system integration. We present ArtiTwinSplat, a framework that automatically constructs articulated, photo-realistic digital twins of objects directly from RGB-D videos, requiring no CAD models, simulation assets, or manual annotations. Our method is built on 3D Gaussian Splatting that preserve geometric fidelity and photometric realism, coupled with an unsupervised articulation discovery pipeline that recovers part structure and joint kinematics from observed motion alone. With tracking and optimization stages our method provides stable, queryable digital twins that support real-time rendering, viewpoint control, and interactive manipulation. Unlike prior methods confined to simulation, ArtiTwinSplat operates directly on real-world observations and produces twins that are immediately usable by downstream robot planning and learning systems. This method offers a practical, scalable pathway toward digital twin construction, lowering the integration barrier for articulated object manipulation in embodied AI and human-robot collaboration contexts.

Pranjal Mishra, René Zurbrügg, Max Wilder-Smith +4

1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.

Articulated Object Reconstruction 3D Gaussian Splatting 3D Reconstruction Digital Twins

Ask PDF

Abstract

Digital Twins

Explore similar work

BoxTwin: Learning Elastoplastic Articulated Object Dynamics from Videos

Probabilistic Physics Digital Twin for Human-Robot Interaction

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos