Organizations: State Key Laboratory of GAI, School of IST, Peking University, China · State Key Laboratory of General Artificial Intelligence, BIGAI, China · School of Artificial Intelligence and Computer Science, Anqing Normal University
High-fidelity 4D physical simulation requires accurate physical parameter assignments that align with diverse material properties. Previous methods struggle with critical bottlenecks: they either rely on computationally intensive per-scene optimization or predict parameters from purely visual inputs. The latter suffers from inherent ambiguity, e.g., the inability to distinguish the stiffness of rubber from its appearance alone. To bridge this gap, we propose AutoPhyX, a text-conditioned framework for predicting spatially-varying physical properties. Since training robust feed-forward models requires fine-grained supervision, we first introduce a novel part-controllable data generation pipeline that decomposes complex 3D assets into semantically distinct components and pairs them with physically plausible, diverse parameters and corresponding text descriptions. Leveraging this dataset, AutoPhyX employs a cross-modal modulation mechanism where text features dynamically modulate visual features for precise physical grounding. By formulating property prediction within a voxel field, AutoPhyX ensures compatibility with diverse 3D representations, including meshes, point clouds, Gaussian Splatting, and NeRF. Experiments demonstrate that our method enables physically plausible, text-driven parameter assignment in a single forward pass. It achieves high accuracy and robust generalization to in-the-wild objects, paving the way for future physics-intensive embodied and robotic applications.
Figures & tables
Figure 1: Text-driven physical simulation with AutoPhyX. Our framework predicts dense physical parameters for 3D assets conditioned on textual descriptions. For each scenario, the leftmost images visualize the predicted voxelized physical parameters, which are normalized and mapped to RGB channels, corresponding to the input text. Under identical boundary conditions, the resulting simulations exhibit distinct behaviors, e.g . , the bonsai responding differently to the same wind force in MPM, and the box with different deformation under the same robot grasping policy in FEM.
Figure 2: Overview of our pipeline. Given a 3D object, our framework first decomposes it into semantically distinct components via multi-view feature rendering. Then, a text-conditioned neural network resolves visual ambiguity through cross-modal modulation, predicting volumetric physical properties in a single forward pass. The predicted properties are formulated as a voxel field, ensuring compatibility with diverse 3D representations, e.g . , meshes, point clouds, and Gaussian Splatting, for physics simulation.
Figure 3: Feature extraction and volumetric filling. (a) Unlike direct averaging ( e.g . , VoMP [ 13 ] ), which suffers from feature contamination by ignoring visibility, our nerf -based extraction preserves 3D structural integrity and spatial consistency. (b) Hierarchical classification for internal voxel filling. Condition 1 performs coarse enclosure detection via six-direction boundary checking. Condition 2 provides topological refinement using ray-surface intersection counts to filter candidates and achieve accurate filling in complex manifolds.
Figure 4: Distribution of object categories and physical properties in Text2PhyX. Text2PhyX contains 1,700 assets with multiple text-conditioned physical variants, covering diverse object categories as well as distributions of density, Young’s modulus, and Poisson’s ratio.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Avg. Phys. err ↓
logE err ↓
ν err ↓
logρ err ↓
DreamPhysics
1 epoch
19.412
0.881
0.212
−
2.351
−
−
25 epochs
19.515
0.882
0.185
−
1.405
−
−
50 epochs
19.223
0.881
0.178
−
1.362
−
−
OmniPhysGS
1 epoch
17.955
0.883
0.325
−
−
−
−
Table 1: Quantitative comparison on Text2PhyX. We evaluate both rendering quality (PSNR, SSIM, LPIPS) and physical property prediction accuracy (Avg. Phys. err, logE err, ν err, logρ err). “ − ” indicates that the method cannot predict the corresponding physical parameter. Bold and underline denote the best and second-best results, respectively. Our method, particularly when equipped with OpenCLIP, significantly outperforms all baselines across both rendering and physics metrics.
Figure 6
Figure 6: Qualitative verification of physical parameter prediction. To evaluate the effectiveness of our parameter estimation, we visualize simulations under identical boundary conditions but with different predicted physical properties. Each pair of rows demonstrates how various descriptions alter the simulation behavior: Rows 1–2 for rigid vs. elastic propeller blades, Rows 3–4 for stiff vs. flexible plant stems, Rows 5–6 for structural integrity vs. brittle fracture of toast, and Rows 7–8 for semi-rigid vs. large deformation of a soft body.
Figure 7: Voxel prediction across different methods. Compared to baselines that struggle with structural granularity (NeRF2Physics) or material boundaries (Pixie), our framework achieves highly precise material decomposition.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1 : Gallery of Text2PhyX. This figure illustrates the fine-grained, text-conditioned physical annotations in our dataset. From left to right: the original 3D object rendering, the part-level semantic segmentation (extracted via OpenCLIP and vlm ), and the corresponding voxel-wise physical property heatmaps for Young’s Modulus ( E ), Poisson’s Ratio ( ν ), and Density ( ρ ). Notice how distinct semantic parts within the same object ( e.g . , the tree trunk versus leaves, or the bag’s body versus handles) are assigned accurately localized and physically plausible parameters.
Figure A2 : Gallery of Text2PhyX.
Figure A3 : Failure case in segmentation. Due to the tight adherence between the trash can’s outer shell and the inner plastic bag, most voxels encapsulate both components post-voxelization. This renders perfect segmentation unattainable regardless of the part name queried in the vlm , since the features within any given voxel are effectively an average of those from both the plastic bag and the trash bin.
Figure A4 : Example of VLM-driven physical property augmentation. The figure illustrates the automated data annotation process for a 3D object. From left to right: multi-view RGB images of a soda can, its corresponding 3D point cloud representation, and the VLM-generated JSON metadata. Based on the augmented semantic description (“The soda can is now made of a very hard plastic”), the VLM successfully modifies the object’s physical parameters, such as scaling the Young’s modulus ( E ) with a specific multiplier.
Model
logE err.
ν err.
logρ err.
ρ err.
PhysDreamer
3.674
0.072
0.411
1107.99
PhysX-Anything
1.205
0.078
0.496
1399.63
Pixie
2.268
0.075
0.777
1268.79
VoMP
0.958
0.090
0.458
1330.10
Ours w/o text
0.624
0.079
0.287
603.47
Ours
0.181
0.056
0.102
296.63
Appendix
Table A1: Evaluation on the human-annotated test set. The test descriptions are written by human annotators after watching simulated object behavior, providing a cleaner evaluation of text-conditioned generalization beyond the generated Text2PhyX descriptions.
Model
Metric
Pixie dataset
VoMP dataset
Text2PhyX
Pixie
logE err.
0.022
0.330
0.059
ν err.
0.034
0.043
0.062
ρ err.
105.45
847.86
153.72
VoMP
logE err.
0.076
0.042
0.080
ν err.
0.048
0.025
0.058
ρ err.
213.6
113.31
262.78
Appendix
Table A2 : Unconditional cross-dataset evaluation. All methods are evaluated with empty text input for direct comparability with feed-forward baselines that do not support text conditioning.
Architecture
logE err ↓
ν err ↓
logρ err ↓
MLP (concat input)
0.316
0.245
0.298
Transformer ( 163 )
0.142
0.187
0.182
VisualRWKV ( 163 )
0.149
0.193
0.181
VisualRWKV ( 323 )
0.106
0.124
0.125
FiLM U-Net (Ours)
0.054
0.061
0.088
Appendix
Table A3 : Ablation Study of different model architectures. All models are trained on a subset of Text2PhyX (2,700 train / 1,000 test) for 30 epochs. The results demonstrate that the high-resolution input information is strong enough to generate accurate physics parameters. Architectures that require spatial downsampling or average pooling of the input ( e.g . , to accommodate sequence lengths) suffer a significant drop in predictive capability.
Significant advancements in 3D asset creation have enabled high-fidelity 3D generation from images. However, most existing representations remain inherently static, lacking the ability to model physically plausible behaviors under dynamic interactions. We present PhysCreator, a unified framework that directly generates simulation-ready 3D representations with spatially varying physical properties from images. Unlike prior approaches that treat physical attributes as simple post-hoc predictions or rely on expensive per-instance optimization, we reformulate physical property estimation within a unified 3D representation learning framework. We introduce a voxel-aligned latent space by lifting image observations into a spatially structured 3D representation, in which geometry, appearance, and physical properties are jointly modeled and inferred in a feed-forward manner. To enable scalable learning without densely annotated physical ground truth, we leverage vision-language distillation to transfer material-aware semantic priors from powerful vision-language models into spatially grounded voxel-level supervision. Experiments demonstrate that our method achieves superior visual quality, physical plausibility, and generalization compared to prior approaches.
Shiyan Zheng, Ming Li, Lei Han +1
Xi’an Jiaotong University · The Chinese University of Hong Kong, Shenzhen · Discover Robotics
Achieving fully automated, physically plausible 3D motion synthesis is a core objective in graphics and generative AI. However, configuring complex environmental force fields still relies entirely on manual expert intervention, creating a severe bottleneck for large-scale simulation data generation. Existing automated methods primarily focus on material optimization and exhibit severe modality gaps and technical flaws when applied to the vastly more complex force field optimization space: naive Large Language Models (LLMs) lack underlying simulation feedback, causing severe physical inaccuracies, while traditional Score Distillation Sampling (SDS) suffers from sluggish gradients, local optima entrapment, and a mathematical inability to dynamically switch discrete force fields. To address this, we propose PhysAgent, the first simulator-in-the-loop multi-agent framework that leverages multimodal inputs for automated, physically grounded 4D synthesis. By decoupling intrinsic materials from extrinsic dynamics, PhysAgent utilizes a Semantic Agent equipped with an externalized Force Field Skill module to master simulation rules and generate valid initializations. Subsequently, the Refine Agents, driven by Trajectory-Grounded Multi-Agent Feedback, leverage vision foundation models to extract dense point trajectories from rendered frames. By converting these explicit motion trajectories into structured textual descriptors, the agent harnesses LLM commonsense reasoning to execute zero-shot macroscopic leaps, effectively escaping local optima and dynamically switching discrete force fields. Extensive experiments demonstrate that PhysAgent rapidly generates stable, diverse physical scenes from arbitrary multimodal prompts, significantly outperforming existing baselines in both generation diversity and physical accuracy.
Existing feed-forward networks excel at predicting a single set of physical properties from visual appearance, but this point-estimate paradigm fundamentally fails to capture the real world's inherent physical ambiguity. We address this by reframing physics prediction as a task of learning a controllable, continuous distribution of material properties. We introduce UNIPIXIE, a framework trained to predict a continuous and parameterized path of physically plausible material properties from a single visual input. By learning a direct mapping along an object's softest-to-stiffest spectrum on our PIXIEMULTIVERSE dataset, UNIPIXIE allows for controllable generation of diverse, physically valid material fields via a single intuitive parameter. Crucially, UNIPIXIE introduces a novel unified architecture to produce simulation-ready parameters for diverse physics solvers, including continuum-based Material Point Method (MPM), reduced-order deformation based on Linear Blend Skinning (LBS), and anchor-based Spring-Mass systems, addressing a key portability issue in prior work. Experiments show our approach not only generates a rich variety of plausible dynamics but also reduces Young's Modulus prediction error by over 50% against the strongest deterministic baseline, bridging the gap between static point estimates and the continuous nature of physical reality. Project page: https://unipixie.github.io/
Qilin Huang, Quynh Anh Huynh, Long Le +5
University of Pennsylvania · Southern University of Science and Technology · Work done during an internship at UPenn +1