AutoPhyX: Automatic Text-Condition Physics Property Generation

Bei Huang; Yixin Chen; Hongbin Zha; Yuru Pei; Siyuan Huang

cs.CVOct 21, 2026

AutoPhyX: Automatic Text-Condition Physics Property Generation

Authors: Bei Huang, Yixin Chen, Hongbin Zha, Yuru Pei, Siyuan Huang

Organizations: State Key Laboratory of GAI, School of IST, Peking University, China · State Key Laboratory of General Artificial Intelligence, BIGAI, China · School of Artificial Intelligence and Computer Science, Anqing Normal University

Abstract

High-fidelity 4D physical simulation requires accurate physical parameter assignments that align with diverse material properties. Previous methods struggle with critical bottlenecks: they either rely on computationally intensive per-scene optimization or predict parameters from purely visual inputs. The latter suffers from inherent ambiguity, e.g., the inability to distinguish the stiffness of rubber from its appearance alone. To bridge this gap, we propose AutoPhyX, a text-conditioned framework for predicting spatially-varying physical properties. Since training robust feed-forward models requires fine-grained supervision, we first introduce a novel part-controllable data generation pipeline that decomposes complex 3D assets into semantically distinct components and pairs them with physically plausible, diverse parameters and corresponding text descriptions. Leveraging this dataset, AutoPhyX employs a cross-modal modulation mechanism where text features dynamically modulate visual features for precise physical grounding. By formulating property prediction within a voxel field, AutoPhyX ensures compatibility with diverse 3D representations, including meshes, point clouds, Gaussian Splatting, and NeRF. Experiments demonstrate that our method enables physically plausible, text-driven parameter assignment in a single forward pass. It achieves high accuracy and robust generalization to in-the-wild objects, paving the way for future physics-intensive embodied and robotic applications.

Figures & tables

Figure 1: Text-driven physical simulation with AutoPhyX. Our framework predicts dense physical parameters for 3D assets conditioned on textual descriptions. For each scenario, the leftmost images visualize the predicted voxelized physical parameters, which are normalized and mapped to RGB channels, corresponding to the input text. Under identical boundary conditions, the resulting simulations exhibit distinct behaviors, e.g . , the bonsai responding differently to the same wind force in MPM, and the box with different deformation under the same robot grasping policy in FEM.

Figure 2: Overview of our pipeline. Given a 3D object, our framework first decomposes it into semantically distinct components via multi-view feature rendering. Then, a text-conditioned neural network resolves visual ambiguity through cross-modal modulation, predicting volumetric physical properties in a single forward pass. The predicted properties are formulated as a voxel field, ensuring compatibility with diverse 3D representations, e.g . , meshes, point clouds, and Gaussian Splatting, for physics simulation.

Panel of Figure 3 — Figure 3: Feature extraction and volumetric filling. (a) Unlike direct averaging ( e.g . , VoMP [ 13 ] ), which suffers from feature contamination by ignoring visibility, our nerf -based extraction preserves 3D structural integrity and spatial consistency. (b) Hierarchical classification for internal voxel filling. Condition 1 performs coarse enclosure detection via six-direction boundary checking. Condition 2 provides topological refinement using ray-surface intersection counts to filter candidates and achieve accurate filling in complex manifolds.

Figure 4: Distribution of object categories and physical properties in Text2PhyX. Text2PhyX contains 1,700 assets with multiple text-conditioned physical variants, covering diverse object categories as well as distributions of density, Young’s modulus, and Poisson’s ratio.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Avg. Phys. err $\downarrow$	$\log E$ err $\downarrow$	$\nu$ err $\downarrow$	$\log\rho$ err $\downarrow$
DreamPhysics
1 epoch	19.412	0.881	0.212	$-$	2.351	$-$	$-$
25 epochs	19.515	0.882	0.185	$-$	1.405	$-$	$-$
50 epochs	19.223	0.881	0.178	$-$	1.362	$-$	$-$
OmniPhysGS
1 epoch	17.955	0.883	0.325	$-$	$-$	$-$	$-$

Table 1: Quantitative comparison on Text2PhyX. We evaluate both rendering quality (PSNR, SSIM, LPIPS) and physical property prediction accuracy (Avg. Phys. err,

\log E

err,

\nu

err,

\log\rho

err). “

-

” indicates that the method cannot predict the corresponding physical parameter. Bold and underline denote the best and second-best results, respectively. Our method, particularly when equipped with OpenCLIP, significantly outperforms all baselines across both rendering and physics metrics.

Figure 6: Qualitative verification of physical parameter prediction. To evaluate the effectiveness of our parameter estimation, we visualize simulations under identical boundary conditions but with different predicted physical properties. Each pair of rows demonstrates how various descriptions alter the simulation behavior: Rows 1–2 for rigid vs. elastic propeller blades, Rows 3–4 for stiff vs. flexible plant stems, Rows 5–6 for structural integrity vs. brittle fracture of toast, and Rows 7–8 for semi-rigid vs. large deformation of a soft body.

Figure 7: Voxel prediction across different methods. Compared to baselines that struggle with structural granularity (NeRF2Physics) or material boundaries (Pixie), our framework achieves highly precise material decomposition.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Gallery of Text2PhyX. This figure illustrates the fine-grained, text-conditioned physical annotations in our dataset. From left to right: the original 3D object rendering, the part-level semantic segmentation (extracted via OpenCLIP and vlm ), and the corresponding voxel-wise physical property heatmaps for Young’s Modulus ( $E$ ), Poisson’s Ratio ( $\nu$ ), and Density ( $\rho$ ). Notice how distinct semantic parts within the same object ( e.g . , the tree trunk versus leaves, or the bag’s body versus handles) are assigned accurately localized and physically plausible parameters.

Figure A3 : Failure case in segmentation. Due to the tight adherence between the trash can’s outer shell and the inner plastic bag, most voxels encapsulate both components post-voxelization. This renders perfect segmentation unattainable regardless of the part name queried in the vlm , since the features within any given voxel are effectively an average of those from both the plastic bag and the trash bin.

Figure A4 : Example of VLM-driven physical property augmentation. The figure illustrates the automated data annotation process for a 3D object. From left to right: multi-view RGB images of a soda can, its corresponding 3D point cloud representation, and the VLM-generated JSON metadata. Based on the augmented semantic description (“The soda can is now made of a very hard plastic”), the VLM successfully modifies the object’s physical parameters, such as scaling the Young’s modulus ( $E$ ) with a specific multiplier.

Model	$\log E$ err.	$\nu$ err.	$\log\rho$ err.	$\rho$ err.
PhysDreamer	3.674	0.072	0.411	1107.99
PhysX-Anything	1.205	0.078	0.496	1399.63
Pixie	2.268	0.075	0.777	1268.79
VoMP	0.958	0.090	0.458	1330.10
Ours w/o text	0.624	0.079	0.287	603.47
Ours	0.181	0.056	0.102	296.63

Appendix

Table A1: Evaluation on the human-annotated test set. The test descriptions are written by human annotators after watching simulated object behavior, providing a cleaner evaluation of text-conditioned generalization beyond the generated Text2PhyX descriptions.

Model	Metric	Pixie dataset	VoMP dataset	Text2PhyX
Pixie	$\log E$ err.	0.022	0.330	0.059
	$\nu$ err.	0.034	0.043	0.062
	$\rho$ err.	105.45	847.86	153.72
VoMP	$\log E$ err.	0.076	0.042	0.080
	$\nu$ err.	0.048	0.025	0.058
	$\rho$ err.	213.6	113.31	262.78

Appendix

Table A2 : Unconditional cross-dataset evaluation. All methods are evaluated with empty text input for direct comparability with feed-forward baselines that do not support text conditioning.

Architecture	$\log E$ err $\downarrow$	$\nu$ err $\downarrow$	$\log\rho$ err $\downarrow$
MLP (concat input)	0.316	0.245	0.298
Transformer ( $16^{3}$ )	0.142	0.187	0.182
VisualRWKV ( $16^{3}$ )	0.149	0.193	0.181
VisualRWKV ( $32^{3}$ )	0.106	0.124	0.125
FiLM U-Net (Ours)	0.054	0.061	0.088

Appendix

Table A3 : Ablation Study of different model architectures. All models are trained on a subset of Text2PhyX (2,700 train / 1,000 test) for 30 epochs. The results demonstrate that the high-resolution input information is strong enough to generate accurate physics parameters. Architectures that require spatial downsampling or average pooling of the input ( e.g . , to accommodate sequence lengths) suffer a significant drop in predictive capability.

Figure A7 : More mesh simulation results.

Materials Property Prediction

Materials property prediction uses data and physical models to estimate how composition, structure, and processing determine a material’s behavior.

See all 175 papers

PhysCreator: Creating Physically-Attributed 3D Assets at Will
Oct 25, 2026Shiyan Zheng, Ming Li, Lei Han +13D Asset Generation 3D Representation Learning
Xi’an Jiaotong University · The Chinese University of Hong Kong, Shenzhen · Discover Robotics
PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback
Jun 7, 2026Chunji Lv, Jiaxi Ye, Yuchen Jiang +2Multi-Agent System Optimization 3D Scene Generation
Beijing Institute of Technology
UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching
Jun 3, 2026Qilin Huang, Quynh Anh Huynh, Long Le +5Materials Property Prediction Conditional Generative Modeling
University of Pennsylvania · Southern University of Science and Technology · Work done during an internship at UPenn +1

Oct 25, 2026J/K move · Enter open · S save

PhysCreator: Creating Physically-Attributed 3D Assets at Will

Shiyan Zheng, Ming Li, Lei Han, Xiangyu Xu

Xi’an Jiaotong University · The Chinese University of Hong Kong, Shenzhen · Discover Robotics

3D Asset Generation 3D Representation Learning Physical Reasoning

Open paper AskPDF arXiv

Significant advancements in 3D asset creation have enabled high-fidelity 3D generation from images. However, most existing representations remain inherently static, lacking the ability to model physically plausible behaviors under dynamic interactions. We present PhysCreator, a unified framework that directly generates simulation-ready 3D representations with spatially varying physical properties from images. Unlike prior approaches that treat physical attributes as simple post-hoc predictions or rely on expensive per-instance optimization, we reformulate physical property estimation within a unified 3D representation learning framework. We introduce a voxel-aligned latent space by lifting image observations into a spatially structured 3D representation, in which geometry, appearance, and physical properties are jointly modeled and inferred in a feed-forward manner. To enable scalable learning without densely annotated physical ground truth, we leverage vision-language distillation to transfer material-aware semantic priors from powerful vision-language models into spatially grounded voxel-level supervision. Experiments demonstrate that our method achieves superior visual quality, physical plausibility, and generalization compared to prior approaches.

Abstract

Materials Property Prediction

Explore similar work

PhysCreator: Creating Physically-Attributed 3D Assets at Will

PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback

UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching