Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition

Mauro Comi; Jordi Serrano Berbel; Kevis-Kokitsi Maninis; Philipp Henzler; Manuel Sanchez

cs.CVAug 24, 2026

Gen2Physics: Grounding Generated 3D Meshes in Physics via Multi-View Material Decomposition

Authors: Mauro Comi, Jordi Serrano Berbel, Kevis-Kokitsi Maninis, Philipp Henzler, Manuel Sanchez

Organizations: Google DeepMind · University of Bristol · Google Research

Abstract

While state-of-the-art generative models produce high-fidelity 3D meshes, these outputs lack the physical properties required for interactive simulation, gaming, or robotics. We introduce Gen2Physics, a unified and automated framework that grounds generated meshes in physics by automatically decomposing them into their constituent material components. Unlike prior approaches, which focus on volumetric representations incompatible with standard physics engines, Gen2Physics operates directly on meshes to produce immediately simulation-ready assets. Our pipeline integrates a fine-tuned Vision Transformer for dense material segmentation, a robust 2D-to-3D consistency projection, and a Vision-Language Model (VLM) guided refinement that leverages contextual reasoning to assign physical properties and infer internal geometry (solid vs. hollow). By converting surface patches into volumes with distinct densities, our method enables physically plausible dynamic simulations. Experimental results on the ABO-500 and PartNet-Material benchmarks demonstrate that Gen2Physics more than doubles the material segmentation accuracy of prior physics-grounding pipelines (15.6 to 48.3 mIoU), while matching the mass-estimation accuracy of volumetric methods and being the only approach to output watertight per-material sub-meshes.

Figures & tables

Figure 2: Process of our 2D-to-3D consistency projection. A fine-tuned ViT predicts material segmentation masks, which are possibly noisy and not consistent across views. We project each face of our asset onto the rendered views, taking into account occlusions. For each face, we assign a label corresponding to the material with the highest frequency in the segmentation masks. The final material prediction is consistent across views. We then re-render the segmentation masks.

Figure 3: Samples of the PartNet-Material dataset. Dense segmentation masks are obtained by manually assigning material labels to each individual part of the asset.

Figure 4: We compare Gen2Physics (Ours), NeRF2Physics, and PUGS on a material identification task. The visualizations show that the 2D-to-3D consistency projection and VLM refinement proposed in Gen2Physics ensure more coherent and accurate material segmentations. In contrast, NeRF2Physics and PUGS produce fragmented masks due to their reliance on local CLIP embeddings

Method	Sim. Ready	ADE ( $\downarrow$ )	ALDE ( $\downarrow$ )	APE ( $\downarrow$ )	MnRE ( $\uparrow$ )
NeRF2Physics	$\times$	12.725	0.736	1.040	0.564
PUGS	$\times$	9.461	0.661	0.767	0.576
VoMP	$\times$	8.433	0.631	0.887	0.576
Phys4DGen	$\times$	9.961	0.664	0.825	0.566
Gen2Physics (Ours)	$\checkmark$	8.447	0.651	1.170	0.600

Table 1: Mass estimation and asset usability on the ABO-500 benchmark. Gen2Physics obtains the best MnRE, and is within 0.2% of the best ADE, while being the only method to output watertight per-material sub-meshes (Sim. Ready) compatible with rigid-body physics engines. Baselines reported from [ 35 , 11 ] .

Method	Wood	Fabric	Metal	Plastic	Glass	Ceramic	Stone	mIoU [%] $\uparrow$
NeRF2Physics	61.87	2.67	23.23	1.03	6.12	0.00	21.43	14.54
PUGS	69.50	4.94	15.66	0.61	3.90	0.00	29.84	15.56
Gen2Physics (ViT only)	66.50	10.88	38.71	33.97	5.34	0.00	25.23	22.58
Gen2Physics (Full)	86.23	35.72	84.38	71.93	40.11	43.49	24.32	48.27

Table 2: Per-class Intersection-over-Union (IoU [%]

\uparrow

) and mean IoU (mIoU [%]

\uparrow

) on the PartNet-Material benchmark. The 8 reported categories are the foreground material classes populated across the test split. Organic and Leather never occur and are excluded, while Paper has 0.00 IoU for all methods and is omitted for space. Overall mIoU is calculated as the unweighted mean across these active classes. Colors denote the best and second-best results respectively.

Figure 5: Qualitative result showing part-reconstruction, material and physics assignment for a TRELLIS generated multi-composite asset. While both methods estimate plausible total mass, PUGS treats the mesh as a monolithic asset. In contrast, our framework decomposes and reconstructs watertight sub-meshes for a physically plausible dynamic behavior, as shown by the MuJoCo simulation [ 39 ] .

Method	ADE ( $\downarrow$ )	ALDE ( $\downarrow$ )	APE ( $\downarrow$ )	MnRE ( $\uparrow$ )
w/o Solid/Hollow	7.636	0.699	2.189	0.595
Full	8.447	0.651	1.170	0.600

Table 3: Ablation study on the ABO-500 mass estimation benchmark. Results show that internal structure reasoning is important for achieving robust mass calculation.

3D Asset Generation

3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.

See all 170 papers

PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
May 6, 2026Yunhan Yang, Chunshi Wang, Junliang Ye +73D Asset Generation Diffusion Models
HKU · Tencent Hunyuan · ZJU +3
PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets
Oct 29, 2026Bi'an Du, Fanpeng Meng, Jiemin Fang +53D Asset Generation Articulated 3D Asset Generation
Peking University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Huawei +1
PhysCreator: Creating Physically-Attributed 3D Assets at Will
Oct 25, 2026Shiyan Zheng, Ming Li, Lei Han +13D Asset Generation 3D Representation Learning
Xi’an Jiaotong University · The Chinese University of Hong Kong, Shenzhen · Discover Robotics

May 6, 2026J/K move · Enter open · S save

PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World

Yunhan Yang, Chunshi Wang, Junliang Ye, Yang Li+6

HKU · Tencent Hunyuan · ZJU+3

3D Asset Generation Diffusion Models

Open paper AskPDF arXiv

Synthesizing physics-grounded 3D assets is a critical bottleneck for interactive virtual worlds and embodied AI. Existing methods predominantly focus on static geometry, overlooking the functional properties essential for interaction. We propose that interactive asset generation must be rooted in functional logic and hierarchical physics. To bridge this gap, we introduce PhysForge, a decoupled two-stage framework supported by PhysDB, a large-scale dataset of 150,000 assets with four-tier physical annotations. First, a VLM acts as a "physical architect" to plan a "Hierarchical Physical Blueprint" defining material, functional, and kinematic constraints. Second, a physics-grounded diffusion model realizes this blueprint by synthesizing high-fidelity geometry alongside precise kinematic parameters via a novel KineVoxel Injection (KVI) mechanism. Experiments demonstrate that PhysForge produces functionally plausible, simulation-ready assets, providing a robust data engine for interactive 3D content and embodied agents.

Abstract

3D Asset Generation

Explore similar work

PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World

PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets

PhysCreator: Creating Physically-Attributed 3D Assets at Will