MorphTwin: Enriching Deformable Material Responses from Videos for Morphable and Interactive Physical Twins
Authors: Haoqin Hong, Ding Fan, Fubin Dou, Tianyu Huang, Yuzhuo Chen, Zhemeng Zhang, Hanxiao Jiang, Congcong Zhu, Gerhard Petrus Hancke, Jingrun Chen, Rynson WH Lau
Organizations: University of Science and Technology of China · City University of Hong Kong · Multimedia Laboratory, The University of Hong Kong · Columbia University
Reconstructing interactive physical twins from video requires accurately capturing how objects deform while maintaining efficient simulation. We observe that reproducing motion in a video does not guarantee correct responses to new forces when the physical representation restricts how an object can deform. In spring-based twins, coupling between volume and shape responses creates such a limitation. We present MorphTwin, which enriches local deformation modes to capture and independently control these responses while retaining standard spring forces and explicit simulation. To support interpretable material editing and transfer, we use offline calibration to map shared relative response changes to stiffness updates on each reconstructed graph. Under identical inverse settings on synthetic scenes, MorphTwin reduces the mean absolute error of Poisson's ratio by 75.8% and the mean relative error of Young's modulus by 22.1% compared to PhysTwin. We introduce an egocentric stereo and RGB-D benchmark covering interactions with objects of varying stiffness. Real-world experiments demonstrate improved reconstruction accuracy and up to 14.4% lower future tracking error than PhysTwin at interactive speeds.
Figures & tables
Figure 2: Overview of MorphTwin. Part A: Sparse video observations provide geometry and motion cues. Part B: We fit the physical model to observed motion, followed by Gaussian appearance optimization with physics fixed. Part C: Relaxable response carriers enable separate volumetric and shear control. Offline calibration maps shared relative response changes to spring stiffnesses for each graph, enabling material editing and reuse. Online interaction retains explicit spring dynamics.
Figure 3: Response carriers and calibration. Part (A) : Candidate carriers on local four-node cells are jointly selected to meet bulk and shear response targets while minimizing auxiliary mass, then realized using auxiliary nodes and spokes. Under standardized volumetric and deviatoric strains, internal relaxation yields the corresponding responses. Part (B) : Each graph is calibrated at its reference state to relate carrier controls to changes in material response. A shared response space then maps the same material edit to graph-specific spring updates.
Views
Method
PSNR ↑
SSIM ↑
LPIPS ↓
CD ↓
Track ↓
PhysTwin [ jiang2025phystwin ]
28.74
0.9451
0.0598
0.00946
0.0172
PGND [ zhang2025pgnd ]
27.59
0.9403
0.0722
0.01448
0.0192
GIC [ cai2024gic ]
24.58
0.9225
0.1302
0.03457
0.0756
All
MorphTwin
32.36
0.9665
0.0511
0.00925
0.0171
[1pt/1.5pt]
PhysTwin [ jiang2025phystwin ]
28.06
0.9395
0.0782
0.00952
0.0178
PGND [ zhang2025pgnd ]
27.18
0.9342
0.0893
0.01416
0.0196
Table 2: Quantitative evaluation on 15-sequence benchmark using single RGB-D, egocentric stereo, and combined views. CD and Track are in meters. Best / second-best per input setting and task.
Figure 4: Reconstruction and prediction on PhysTwin scenes. PGND can exhibit unstable future predictions, while PhysTwin’s explicit mass-spring model may not fully reproduce material responses. Semi-transparent GT overlays highlight discrepancies.
Figure 5: Material editing on our egocentric interaction benchmark. Four representative sequences from our 15-sequence benchmark show reconstruction, future prediction, and material editing under the same hand interactions. At the same observation time within each row, stiffening the pillow and hat reduces deformation, while softening the mousepad and book enables bending.
Reconstructing objects with mechanical properties from video observations enables physically consistent dynamic prediction, benefiting robotics planning and interaction. Existing spring--mass based physical driven reconstruction approaches offer efficient and differentiable physical reconstruction, but they typically rely on axial springs alone. Such formulations oversimplify the underlying structural mechanics and can become mechanically under-constrained when the physical graph is coarsened, limiting their ability to preserve stable local deformation. We present BendTwin, a bending-aware differentiable spring--mass framework for video-based reconstruction and future prediction of deformable objects. BendTwin introduces bending stiffness and damping over local surface triplets, penalizing deviations from rest angles and regularizing higher-order deformation. These bending constraints improve mechanical stability while preserving the simplicity of spring--mass system. Experiments show that BendTwin consistently outperforms the axial-only PhysTwin baseline. Ablation studies further demonstrate that the bending constraints maintain system stability across different downsampling ratios and consistently improve upon the original PhysTwin formulation. Overall, BendTwin provides an effective approach for constructing mechanically faithful digital twins from sparse-view RGB-D videos.
Yixiong Jing, Qi Wang, Lin Chen +6
University of Cambridge · Institute of Automation, Chinese Academy of Sciences · Northwestern Polytechnical University +1
Reconstructing simulation-ready deformable objects is important for vision, graphics, and robotics. Existing physics-driven methods can recover physical digital twins from videos, but they suffer from two fundamental limitations: they typically assume a homogeneous material across the whole object, and their scene-specific inverse optimization, combined with the inherent ambiguity of monocular observation, yields inconsistent parameters for the same material across different scenes or interactions. We propose MatPhys, a material-aware feed-forward framework that predicts spring-mass parameters from a single-view video, addressing these two issues with two coupled designs. To relax the homogeneous material assumption, we use DINO features to decompose the object into semantically meaningful parts and to query a part-level material prior, assigning each part its own physical behavior. To enforce cross-scene consistency, we introduce a learned material codebook of shared material embeddings as the bridge between appearance and physics, and further use the part-level prior as a reference distribution that constrains the decoder so that the same material yields consistent parameters across scenes and interactions. Together, these designs turn an under-constrained monocular problem into feed-forward inference grounded on shared, reusable material concepts. Experiments show that our method matches per-scene optimization baselines in reconstruction and future prediction, while achieving stronger generalization to unseen interactions and objects with more consistent physical parameters.
Yang Yang, Yiyan Wang, Zheming Liu +1
1The University of Osaka · 3Huawei Technologies Japan K.K. · 2The University of Tokyo
Humans naturally understand object physics through everyday interactions, but faithfully predicting complex deformable dynamics, such as elastic materials and fabrics, remains a major challenge for computer vision and robotics. We present EgoPhys, a framework that constructs deformable physical digital twins from egocentric RGB-only video using generalizable priors. EgoPhys overcomes the limitations of existing methods to enable controllable deformable digital twin generation from egocentric videos by distilling per-object inverse-physics solutions into a compact codebook, enabling prediction of dense spring stiffness fields for unseen objects without per-spring test-time optimization. Trained with generalizable priors from diverse egocentric interactions, EgoPhys outperforms baselines in reconstruction, future prediction, and zero-shot generalization. To support training and evaluation, we curate an egocentric interaction dataset covering diverse deformable objects, scenes, and manipulation styles. We deploy EgoPhys on a real xArm6 robot, demonstrating that a digital twin initialized from a single egocentric human play video can serve as an internal world representation to aid in deformable-object planning, highlighting egocentric RGB observations as a scalable path toward real-to-sim pipelines.