Estimating simulation-ready material parameters from visual observations is crucial for deformable-object manipulation but challenging because material properties are only indirectly revealed through physical responses. We introduce physical-field-bridged material inference, a single-pass framework that uses dense physical responses to connect multi-view RGB deformation videos with constitutive-type classification and continuous parameter regression. Unlike inverse-physics methods requiring per-scene reconstruction, online tuning, or simulation-in-the-loop optimization, our model predicts force/contact masks, projected motion-flow fields, and projected stress-response maps, then conditions material inference on these fields to reduce reliance on appearance or category shortcuts. We construct DeformFieldBench, a 3DGS MPM learning environment comprising 5,000 simulations across 100 object shapes and five mechanically informative loading modes, with dense pixel-aligned physical supervision. Our method achieves the best median errors on all four continuous parameters, improves stress-hotspot recall from 0.0762 to 0.5121, and attains 0.8676 SSIM/36.82 dB PSNR in same-simulator replay on unseen objects. Under a protocol-matched GSO evaluation, inference and future-state prediction take approximately 30 seconds, compared with 15-120 minutes for per-scene optimization. The method also yields measurable replay agreement on five real-world sequences. These results support dense physical fields as an effective bridge for fast material inference, while cross-simulator and real-world tests reveal remaining domain and identifiability gaps.
Figures & tables
Figure 1: Overview of DeformFieldBench. Each sample combines three-view RGB observations with force/contact masks, projected flow, projected stress, object masks, a constitutive family, and continuous material parameters. The left panel illustrates pixel-aligned fields over loading and recovery, the center summarizes asset and parameter diversity relative to prior datasets, and the right shows the canonical views and five loading modes.
Figure 2: Overview of the proposed model. Multi-view RGB deformation videos are encoded into static and dynamic features, decomposed through a physics bottleneck, and decoded into dense motion-flow, force/contact, and stress fields. The predicted fields are encoded and fused with the bottleneck for material inference. The implemented constitutive-type classifier runs in parallel with the illustrated continuous-parameter regressor and is omitted from the diagram for clarity.
Figure 3: Material-parameter predictions from the VLM baseline, supervised baseline, and our model. The four scatter plots show prediction versus ground truth for density, E , ν , and yield stress; the red diagonal is ideal agreement. The sorted SPE plot reports the full sample-wise error distribution, with dashed segments identifying each method’s highest-error 20%. It exposes both the improved typical-sample performance and the remaining long-tail failures.
Figure 4: Qualitative physical-field predictions for three unseen objects over representative timesteps. Rows compare ground truth (green label) and prediction (red label) for projected motion flow, force/contact response, and projected stress. The model recovers the principal moving regions, localized contacts, and high-response stress structure despite changes in geometry and action.
Figure 5: Simulation replay examples across unseen objects and loading modes, grouped from lower to higher SSIM. Each example compares ground-truth and predicted trajectories and lists the corresponding continuous parameter sets. Similar motion can arise from noticeably different parameters, illustrating why replay agreement complements but does not replace parameter-space evaluation.
Reconstructing simulation-ready deformable objects is important for vision, graphics, and robotics. Existing physics-driven methods can recover physical digital twins from videos, but they suffer from two fundamental limitations: they typically assume a homogeneous material across the whole object, and their scene-specific inverse optimization, combined with the inherent ambiguity of monocular observation, yields inconsistent parameters for the same material across different scenes or interactions. We propose MatPhys, a material-aware feed-forward framework that predicts spring-mass parameters from a single-view video, addressing these two issues with two coupled designs. To relax the homogeneous material assumption, we use DINO features to decompose the object into semantically meaningful parts and to query a part-level material prior, assigning each part its own physical behavior. To enforce cross-scene consistency, we introduce a learned material codebook of shared material embeddings as the bridge between appearance and physics, and further use the part-level prior as a reference distribution that constrains the decoder so that the same material yields consistent parameters across scenes and interactions. Together, these designs turn an under-constrained monocular problem into feed-forward inference grounded on shared, reusable material concepts. Experiments show that our method matches per-scene optimization baselines in reconstruction and future prediction, while achieving stronger generalization to unseen interactions and objects with more consistent physical parameters.
Yang Yang, Yiyan Wang, Zheming Liu +1
1The University of Osaka · 3Huawei Technologies Japan K.K. · 2The University of Tokyo
Existing feed-forward networks excel at predicting a single set of physical properties from visual appearance, but this point-estimate paradigm fundamentally fails to capture the real world's inherent physical ambiguity. We address this by reframing physics prediction as a task of learning a controllable, continuous distribution of material properties. We introduce UNIPIXIE, a framework trained to predict a continuous and parameterized path of physically plausible material properties from a single visual input. By learning a direct mapping along an object's softest-to-stiffest spectrum on our PIXIEMULTIVERSE dataset, UNIPIXIE allows for controllable generation of diverse, physically valid material fields via a single intuitive parameter. Crucially, UNIPIXIE introduces a novel unified architecture to produce simulation-ready parameters for diverse physics solvers, including continuum-based Material Point Method (MPM), reduced-order deformation based on Linear Blend Skinning (LBS), and anchor-based Spring-Mass systems, addressing a key portability issue in prior work. Experiments show our approach not only generates a rich variety of plausible dynamics but also reduces Young's Modulus prediction error by over 50% against the strongest deterministic baseline, bridging the gap between static point estimates and the continuous nature of physical reality. Project page: https://unipixie.github.io/
Qilin Huang, Quynh Anh Huynh, Long Le +5
University of Pennsylvania · Southern University of Science and Technology · Work done during an internship at UPenn +1
Learning physically plausible dynamics from visual observations is essential for interactive world models and embodied agents. However, modeling real-world deformable objects remains challenging because their dynamics often arise from complex, spatially heterogeneous material responses. To address this challenge, we propose PhysReal, a video-driven framework for learning and simulating the underlying physics of real deformable objects. PhysReal integrates a spatially varying hybrid expert-neural constitutive model with a differentiable MPM simulator and 3DGS renderer. Analytical expert models provide interpretable physical priors, while neural constitutive residuals capture material responses beyond predefined formulations. Spatially distributed patches parameterize the constitutive field, enabling a continuous representation of local material variations. To organize the identification of this model from sparse visual observations, we adopt a progressive curriculum that sequentially optimizes global material properties, spatially varying local parameters, and neural constitutive residuals, together with complementary motion and mask supervision. Extensive experiments on diverse deformable-object interactions demonstrate that PhysReal achieves superior performance in dynamic reconstruction and future-state prediction, while showing strong potential for downstream robotic applications.