ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

Tim Engelbracht; René Zurbrügg; Mayank Mittal; Marco Hutter; Marc Pollefeys; Hermann Blum; Zuria Bauer

cs.ROSep 18, 2026

ForceTwin: Physics-informed Digital Twins for Robotic Manipulation from Instrumented Human Interaction

Authors: Tim Engelbracht, René Zurbrügg, Mayank Mittal, Marco Hutter, Marc Pollefeys, Hermann Blum, Zuria Bauer

Organizations: ETH Zurich · NVIDIA · Microsoft · University of Bonn

Abstract

Manipulating objects requires understanding not only their motion, but also the physical properties that determine it. For articulated objects, these include inertia, friction, and mechanisms such as springs or door closers, whose effects can vary with configuration and velocity. Such properties are not directly observable from appearance: visually identical doors may require very different effort to manipulate. Existing digital-twin pipelines recover primarily kinematics or assign static physical parameters from visual and language priors, which can yield physically implausible estimates. As a result, state-dependent mechanism dynamics remain unidentified and are not represented in standard asset formats. We present ForceTwin, a system for identifying physics-informed digital twins of articulated objects from instrumented human interaction. A person probes an object using a handheld force-sensing gripper, providing synchronized poses and interaction forces from which we estimate the articulation, parametric dynamics including inertia, Coulomb friction, viscous damping, and a structured neural residual capturing state-dependent mechanism forces. ForceTwin nearly halves the inertial-parameter error of a VLM prior. As a feedforward dynamics model for impedance control on a Spot and a Franka FR3, ForceTwin achieves 87% goal completion across nine object-embodiment pairs, compared with 60% using VLM-prior and 57% using kinematics-only twins, with the largest gains on objects whose strong mechanisms cause both baselines to stall. We further use the identified twins to train whole-body door-traversal policies and deploy them in the real world. Project Page: https://timengelbracht.github.io/forcetwin-website/

Figures & tables

Fig. 1: The ForceTwin pipeline. Instrumented human interaction provides synchronized tool poses and contact wrenches from which we identify an articulation model and its generalized-effort dynamics. Geometry is reconstructed from the same recording or supplied independently, and the physical model is registered to it to produce a twin for robot control or simulation.

Fig. 2: Construction of a physics-informed twin. (a) Reconstructed panel instance, associated handle, and registered joint axis. (b) Exported articulated twin. The physical model is transformed into the asset frame, reparameterized to match its joint-coordinate convention, and attached to the corresponding articulated instance.

Fig. 3: Dynamics model fit for the wood door. (a) Measured generalized effort (gray) against the parametric model alone (dashed) and with the structured mechanism added (green); (b) the probing trajectory. The parametric model tracks the shape of the effort but is offset from it by a configuration-dependent amount, which the learned mechanism recovers. Shaded bands mark the probing regimes for inertia (yellow), viscous damping (purple), and the mechanism (blue).

	Prismatic	Revolute		Type acc. [%] $\uparrow$
Method	$\theta_{\mathrm{err}}$ [ ∘ ] $\downarrow$	$\theta_{\mathrm{err}}$ [ ∘ ] $\downarrow$	$d_{L_{2}}$ [mm] $\downarrow$	Pris.	Rev.
ForceTwin (Ours)	1.4 $\pm$ 0.2	1.5 $\pm$ 0.8	23 $\pm$ 26	100.0	100.0
MoMa-SG [ 2 ]	1.6 $\pm$ 0.8	1.7 $\pm$ 0.0	46 $\pm$ 20	100.0	100.0
OPDMulti [ 14 ]	4.8 †	34.9 $\pm$ 29.1	143 $\pm$ 117	100.0	54.1

TABLE I: Articulation Axis Estimation. Evaluated on the real-world test bench against 12 axes annotated in a Leica RTC360 laser scan. Metrics follow ArtiPoint [ 17 ] :

\theta_{\mathrm{err}}

is the angle between axis lines, folded to

[0,90]^{\circ}

, and

d_{L_{2}}

the distance between revolute axis lines along their common perpendicular. Errors accumulate only over predictions with the correct joint type. Values are means with the sample standard deviation in smaller type; † no deviation available. Bold indicates best performance, underline second best; tied entries are all marked best.

Fig. 4: Inertial-parameter fidelity on the four objects for which ground truth is available: estimate against ground truth on logarithmic axes, so an estimate on the diagonal is exact and vertical distance from it is the error. Marker shape distinguishes revolute (circle) from prismatic (square) joints, whose parameters carry the two units given on the axes. Nested shaded bands mark agreement to within

\pm 25\%

(inner) and to within a factor of two (outer). The macro mean of the absolute relative errors is

29.5\%

for ForceTwin and

53.0\%

for the VLM prior.

Table 6

Fig. 5: Real-world experiments of zero-shot model-based articulation ( Table III ) across two embodiments: Spot and Franka FR3. We visualize the initial and final articulation states as blended overlays for both embodiments. The recovered quasi-static mechanism torque for each object is plotted over the opening trajectory, highlighting the variation in mechanism behavior across objects.

Fig. 6: Policy learning with identified door models. Left: We use the identified wood- and metal-door models (real-to-sim) to train whole-body policies, which we then deploy on ANYmal (sim-to-real). Right: Both policies on the real wood door; shaded regions show the average over five trials. The wood-door policy moves the end-effector smoothly, while the metal-door policy stops and starts, pulling hard and then pausing, which gives it higher jerk (bottom) due to the worse sim-to-real calibration.

Digital Twins

Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.

See all 253 papers

BoxTwin: Learning Elastoplastic Articulated Object Dynamics from Videos
Jul 19, 2026Heng Zhang, Gehan Zheng, Kaifeng Zhang +6Articulated Object Reconstruction Deformable Object Manipulation
Heng Zhang, Gehan Zheng, Kaifeng Zhang, Jay Song, Shivansh Patel, Sonny Hu, Yunzhu Li and Changxi Zheng are with SceniX, Inc. · Peter Yichen Chen is with the University of British Columbia.
Probabilistic Physics Digital Twin for Human-Robot Interaction
Oct 14, 2026Yushen Hu, Harshvardhan Chandirasekar, Apurv Kushwaha +2Digital Twins Robotics Simulation
University of Minnesota, USA · Howard University, USA
ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos
Jun 23, 2026Pranjal Mishra, René Zurbrügg, Max Wilder-Smith +4Articulated Object Reconstruction 3D Gaussian Splatting
1ETH Zürich, Zürich, Switzerland. · 2Microsoft, Zürich, Switzerland. · University of Bonn, Bonn, Germany.

Jul 19, 2026J/K move · Enter open · S save

BoxTwin: Learning Elastoplastic Articulated Object Dynamics from Videos

Heng Zhang, Gehan Zheng, Kaifeng Zhang, Jay Song+5

Heng Zhang, Gehan Zheng, Kaifeng Zhang, Jay Song, Shivansh Patel, Sonny Hu, Yunzhu Li and Changxi Zheng are with SceniX, Inc. · Peter Yichen Chen is with the University of British Columbia.

Articulated Object Reconstruction Deformable Object Manipulation Robotic Manipulation Digital Twins Deformable Object Simulation

Open paper AskPDF arXiv

Digital twins enable robots to anticipate and adapt to physical interactions, but existing models struggle with elastoplastic articulated objects (EAOs) that exhibit nonlinear elasticity, plastic yielding, and damage accumulation. We present BoxTwin, an interactive digital twin framework that learns the full dynamics of EAOs from videos. Our pipeline reconstructs the scene, identifies a physics aware constitutive model for each EAO. Experiments on manual folding and dual arm manipulation of EAOs show that BoxTwin accurately tracks joint trajectories and reproduces post contact plastic behavior over long horizons. By integrating video driven reconstruction with elastoplastic damage modeling, BoxTwin advances digital twins toward predictive, adaptive control of deformable articulated objects in unstructured environments.

Abstract

Digital Twins

Explore similar work

BoxTwin: Learning Elastoplastic Articulated Object Dynamics from Videos

Probabilistic Physics Digital Twin for Human-Robot Interaction

ArtiTwinSplat: Interactable Digital Twin Reconstruction via Gaussian Splatting from RGB-D videos