PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets

Bi'an Du; Fanpeng Meng; Jiemin Fang; Chen Yang; Pufan Li; Daizong Liu; Qi Tian; Wei Hu

cs.CVOct 29, 2026

PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets

Authors: Bi'an Du, Fanpeng Meng, Jiemin Fang, Chen Yang, Pufan Li, Daizong Liu, Qi Tian, Wei Hu

Organizations: Peking University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Huawei · Wuhan University

Abstract

Generating interactive 3D assets from a single image requires realistic appearance, fidelity to the depicted object, and plausible physical behavior. Yet detailed geometry alone does not determine how an object should move or deform. We introduce PhysAg3D, which couples 3D generative models with agentic reasoning and tool use. A vision-language agent first infers semantic parts, candidate interactions, and physical priors from the image; a 3D generator then produces a textured whole-object mesh from the same image. A hierarchical interaction representation links these revisable priors to mechanisms and the generated surfaces they control, organizing interaction construction and refinement. Guided by this representation and observations of the mesh, the agent uses geometric and physical tools to construct parts, joints, deformable regions, and surface bindings while retaining useful generated geometry and materials. Rendered interactions and execution measurements guide targeted revisions within a fixed budget. The framework accommodates both rigid articulation and compliant deformation. Quantitative and qualitative evaluations on real photographs and external benchmarks demonstrate detailed, image-aligned assets with plausible interactions. Ablations support bounded refinement and source-surface preservation. Project Page: supra-3d.github.io/PhysAg3D.

Figures & tables

Figure 1: Detailed appearance with region-specific interaction. Each example shows an input image, its generated asset, and either appearance details or a selected interaction state. (a) A camera shows detailed surface appearance; insets compare corresponding lens-barrel regions visible in the input image and generated asset. (b) A laptop stand adjusts its tray tilt and support-arm elevation. (c) Headphones combine a rigid support with locally compliant ear pads. These development examples show recorded simulated states in (b) and (c); ear-pad deformation uses an approximate compliance model.

Figure 2: Overview of PhysAg3D with an excavator example. (1) Image-derived priors are combined with a generated textured mesh. (2) Interaction intent $S$ specifies lifting, reaching, and curling; physical structure $K$ specifies the corresponding joints; concrete realization $X$ implements them with asset entities and bindings, illustrated here by its visible surfaces. (3) Asset construction binds surfaces to physical entities, and evidence attribution links observations back to the representation for local revision. Mechanism reasoning, asset construction, and evidence attribution operate within one agentic loop. Close-ups show body–boom, boom–stick, and stick–bucket connections; reference and posed states illustrate articulation.

Method	Object Type Support		Input Consistency			Action Quality		Range	Stability
Method	Art.	Def.	FG-IoU $\uparrow$	PSNR $\uparrow$	CLIP $\uparrow$	ACR $\uparrow$	MMR $\downarrow$	RAR $\uparrow$	FVSP $\uparrow$
Particulate	$\checkmark$	$\times$	0.607	14.24	0.853	46.7%	57.5%	34.7%	24.0%
PAct	$\checkmark$	$\times$	0.531	12.53	0.806	35.0%	87.2%	18.0%	16.0%
ArtiCraft	$\checkmark$	$\times$	0.514	12.25	0.768	56.8%	51.8%	34.3%	28.0%
PhysX-Omni	$\checkmark$	$\checkmark$	0.504	13.06	0.817	41.2%	68.5%	26.0%	16.0%
aDSL	$\checkmark$	$\times$	0.520	12.37	0.779	66.8 %	40.2 %	48.8 %	42.0 %

Table 1: Single-image interactive generation on 50 in-the-wild objects with 110 predefined actions. Image metrics use camera-grid-aligned views. ACR, MMR, and RAR are object-level macro averages; FVSP requires all actions to complete without video-visible instability. Art./Def.: articulated/deformable support;

\checkmark

\times

: reported or evaluated/no reported support. Best and second-best values are bold and italic, respectively.

Figure 3: Qualitative comparison on real photographs. Printer and chair panels compare appearance and displayed motion across aDSL, PhysX-Omni, and PhysAg3D. The right column shows initial, pressed, and released states of a deformable asset.

Dataset: PhysXVerse
Method	PSNR $\uparrow$	CD $\downarrow$	F-score $\uparrow$	Scale $\downarrow$	Mat. $\uparrow$	Aff. $\uparrow$	Kin. $\uparrow$	Desc. $\uparrow$
Articulate-Anything ( Le et al., 2025 )	14.03	48.77	46.44	-	-	-	0.30	-
MonoArt ( Li et al., 2026a )	19.68	7.03	85.27	-	-	-	0.38	-
PhysXGen ( Cao et al., 2025a )	19.41	15.19	83.56	309.31	16.51	9.40	0.35	11.84
PhysX-Anything ( Cao et al., 2025b )	15.97	37.06	40.46	298.19	15.65	10.50	0.42	21.38
PhysX-Omni ( Cao et al., 2026 )	21.52	2.95	91.28	2.79	27.23	21.47	0.92	31.05

Table 2: Reference-based comparison on PhysXVerse and PhysX-Mobility. CD is in

10^{-3}

and F-score in

10^{-2}

at threshold

0.05

. Mat., Aff., Kin., and Desc. denote material, affordance, kinematics, and description. Bold and underlined values indicate the best and second-best scores. A dash indicates an entry with no reported score.

Method	CLIP	RQS	MCS	DQS	APS	KPS	MPS	DCS
Articulate-Anything ( Le et al., 2025 )	0.55	88.46	55.27	-	-	71.25	-	-
MonoArt ( Li et al., 2026a )	0.84	96.20	82.56	-	-	68.32	-	-
PhysXGen ( Cao et al., 2025a )	0.80	85.93	73.50	24.21	66.07	69.17	NR	22.24
PhysX-Anything ( Cao et al., 2025b )	0.55	70.81	52.71	50.20	59.96	65.99	44.70	26.89
PhysX-Omni ( Cao et al., 2026 )	0.77	90.00	64.48	64.26	70.57	80.72	59.89	39.02
PAct ( Liu et al., 2026 )	0.74	88.50	62.00	61.50	68.00	82.00	58.50	36.50

Table 3: PhysX-Bench comparison. RQS: visual quality; MCS: 3D consistency; DQS: absolute scale; APS, KPS, MPS, and DCS denote affordance, kinematic, material, and description scores. All columns are higher-is-better. Bold and underlined values indicate the best and second-best scores, including ties. A dash or NR denotes an unreported score.

Figure 4: Qualitative comparison on PhysXVerse. Each row shows the input and two states per method, with fixed viewpoint and scale within each pair. Red arrows and arcs indicate translation or rotation in our results. The states show appearance and articulation playback, not calibrated dynamics or contact stability.

Method	Mean $\uparrow$	#1 $\uparrow$	Mean $\uparrow$	#1 $\uparrow$	Collapse $\downarrow$
	Input-image consistency		Motion plausibility
Particulate	2.98	7	2.84	14	21
PAct	1.99	1	1.73	0	47
ArtiCraft	2.01	1	2.06	1	28
PhysX-Omni	2.03	5	2.24	7	15
aDSL	2.38	4	2.75	11	11

Table 4: Human perceptual evaluation over 102 valid evaluations. “#1” denotes first-place selections. “Collapse” counts evaluations in which visible motion collapse was flagged; multiple methods may be selected. Best and second-best results are bolded and underlined, respectively.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Configuration
3D generator	TRELLIS.2-4B; 1024_cascade route
Sampling and texture	24 sampling steps; 2048-pixel texture size
Generator seed	42
Reasoning model	gpt-6-astra ; high reasoning effort
Geometry / simulation	Blender / MuJoCo
Builder attempts	3 total: initial construction and at most 2 repairs

Appendix

Table 6: Documented implementation configuration and construction budgets. Host validation attempts and simulations called inside construction tools are counted separately.

Evaluation	Evidence and interpretation
In-the-wild interaction	ACR, MMR, and RAR: macro averages over 50 objects, computed from 110 predefined actions. FVSP: object-level pass rate requiring all actions to complete without video-visible instability.
PhysXVerse / PhysX-Mobility	Reference assets support geometry, appearance, scale, and attribute comparisons. CD uses $10^{-3}$ units; F-score uses $10^{-2}$ units at threshold $0.05$ , as in Table 2 .
PhysX-Bench	CLIP, RQS, and MCS address alignment, visual quality, and multi-view consistency. DQS addresses scale; APS, KPS, MPS, and DCS address affordance, kinematics, materials, and description.
Component ablation	The same PhysX-Bench setting compares Full pipeline, One-shot, and w/o surface preservation using CLIP, RQS, KPS, and MPS.
Extended visualization	Additional real-world cases and methods support inspection of photorealistic appearance and correspondence to the input image.

Appendix

Table 7: Evidence and interpretation for the evaluations reported in the main paper. The rows describe evaluation settings, not additional experiments.

Figure 5: Extended qualitative comparison on real-world images. Complementing Figure 3 , the expanded comparison focuses on photorealistic appearance and alignment with the input photograph across additional cases and methods.

3D Asset Generation

3D asset generation creates and edits objects, characters, materials, and scenes from text, images, video, or geometric constraints.

See all 170 papers

PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World
May 6, 2026Yunhan Yang, Chunshi Wang, Junliang Ye +73D Asset Generation Diffusion Models
HKU · Tencent Hunyuan · ZJU +3
PhysCreator: Creating Physically-Attributed 3D Assets at Will
Oct 25, 2026Shiyan Zheng, Ming Li, Lei Han +13D Asset Generation 3D Representation Learning
Xi’an Jiaotong University · The Chinese University of Hong Kong, Shenzhen · Discover Robotics
Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction
Sep 21, 2026Lintao Wang, Mingyang Sun, Yang Liu +23D Reconstruction 3D Scene Generation
Physical Superintelligence Lab, Fysics AI · College of Intelligent Robotics and Advanced Manufacturing, Fudan University · Multimedia Laboratory (MMLab), The Chinese University of Hong Kong +1

May 6, 2026J/K move · Enter open · S save

PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World

Yunhan Yang, Chunshi Wang, Junliang Ye, Yang Li+6

HKU · Tencent Hunyuan · ZJU+3

3D Asset Generation Diffusion Models

Open paper AskPDF arXiv

Synthesizing physics-grounded 3D assets is a critical bottleneck for interactive virtual worlds and embodied AI. Existing methods predominantly focus on static geometry, overlooking the functional properties essential for interaction. We propose that interactive asset generation must be rooted in functional logic and hierarchical physics. To bridge this gap, we introduce PhysForge, a decoupled two-stage framework supported by PhysDB, a large-scale dataset of 150,000 assets with four-tier physical annotations. First, a VLM acts as a "physical architect" to plan a "Hierarchical Physical Blueprint" defining material, functional, and kinematic constraints. Second, a physics-grounded diffusion model realizes this blueprint by synthesizing high-fidelity geometry alongside precise kinematic parameters via a novel KineVoxel Injection (KVI) mechanism. Experiments demonstrate that PhysForge produces functionally plausible, simulation-ready assets, providing a robust data engine for interactive 3D content and embodied agents.

Abstract

3D Asset Generation

Explore similar work

PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World

PhysCreator: Creating Physically-Attributed 3D Assets at Will

Fysiverse-3D-SimReady Technical Report: Agentic Physical Simulation for Pragmatic 3D World Reconstruction