A Unified Framework for Image-to-3D Part Generation via Variable Granularity

Qiwen Gu; Xin Zhou; Xiwu Chen; Zhongyang Zhu; Zhuo Zhang; Junqiao Zhao

cs.CVOct 22, 2026

A Unified Framework for Image-to-3D Part Generation via Variable Granularity

Authors: Qiwen Gu, Xin Zhou, Xiwu Chen, Zhongyang Zhu, Zhuo Zhang, Junqiao Zhao

Organizations: Tongji University · Chongqing Afari Intelligent Drive Co., Ltd · Huazhong University of Science and Technology

Abstract

Controllable 3D part generation is pivotal for digital asset creation, yet existing methods lack the flexibility to accommodate diverse user workflows requiring varying levels of control. In this paper, we introduce FlexPart, a unified image-to-3D part generation framework that supports variable-granularity geometric conditions within a single model. By seamlessly integrating heterogeneous inputs, ranging from sparse points to dense bounding boxes and masks, via gated adaptive modulation, FlexPart unlocks a powerful cross-modal synergy. We also propose an asymmetric geometric guidance mechanism that leverages strong spatial priors to align representations across granularities, improving geometric consistency and facilitating inter-task synergy. Under mask conditioning on PartVerseXL, FlexPart reduces Part CD by 16.6% relative to TripoSG + SAM3D and by 10.5% relative to OmniPart, while maintaining competitive whole-mesh quality. The code can be found at https://github.com/Gucci233/FlexPart

Figures & tables

Figure 1: (a) Our unified framework integrates variable-granularity geometric prompts from sparse points to dense masks into one model for flexible part control. (b) The proposed method produces high-fidelity, semantically coherent 3D parts across diverse object categories.

Figure 2: Overview of the FlexPart framework. FlexPart supports point, box, and mask prompts, with points as the minimal input. The noisy latent, part ID, and timestep are jointly conditioned on encoded prompts, where learnable null embeddings replace missing inputs. The fused condition modulates each DiT layer for part generation via gated adaptive layer normalization. During training, a multi-granularity geometric guidance branch aligns weak and strong priors via optimal transport.

Method	Reference	Cond.	PartVerseXL			PartObjaverse-Tiny
Method	Reference	Cond.	Part CD $\downarrow$	Part IoU V $\uparrow$	Part IoU B $\uparrow$	Part CD $\downarrow$	Part IoU V $\uparrow$	Part IoU B $\uparrow$
TRELLIS + SAM3D	CVPR 25	MV Mask	0.3471	0.1885	0.2173	0.3916	0.1226	0.2031
TripoSG + SAM3D	TPAMI 25	MV Mask	0.3523	0.1727	0.1896	0.4235	0.1250	0.2078
PartCrafter	NeurIPS 25	Number	0.5235	0.0907	0.1257	0.5193	0.0802	0.1455
PartPacker	NeurIPS 25	-	0.3933	0.1717	0.2210	0.3932	0.1588	0.2327
OmniPart	SIGGRAPH Asia 25	Mask	0.3283	0.1961	0.2831	0.3681	0.1568	0.2649

Table 1: Part-level structure evaluation. Cond. denotes input conditions; PartPacker requires none. For whole-mesh baselines, we use SAM3D to lift multi-view 2D masks to 3D segmentations. Part IoU V and Part IoU B denote voxel IoU and 3D bounding box IoU. Bold and underlined indicate the best and second-best results, respectively.

Method	Reference	PartVerseXL			PartObjaverse-Tiny
Method	Reference	Mean CD $\downarrow$	F-Score $\uparrow$	ULIP $\uparrow$	Mean CD $\downarrow$	F-Score $\uparrow$	ULIP $\uparrow$
TRELLIS	CVPR 25	0.2125	0.7514	0.2370	0.2038	0.7478	0.2434
TripoSG	TPAMI 25	0.1227	0.8176	0.2331	0.1504	0.7811	0.2410
PartCrafter	NeurIPS 25	0.1666	0.7346	0.1950	0.1731	0.7280	0.2136
PartPacker	NeurIPS 25	0.1489	0.7899	0.2266	0.1612	0.7682	0.2397
OmniPart	SIGGRAPH Asia 25	0.1547	0.7636	0.2179	0.1585	0.7501	0.2335

Table 2: Whole-mesh quality evaluation. TRELLIS and TripoSG are full-mesh generation methods, while FullPart overlaps with the PartVerseXL training set; these entries are grayed out and reported for reference only, and are excluded from direct comparison. Bold and underlined indicate the best and second-best results.

Figure 3: Qualitative results of different methods. The upper row shows exploded views for inspecting individual parts, and the lower row shows the assembled meshes. Our method uses masks as input, while other methods are evaluated with their respective input conditions.

Eval Cond.	Train Setting	Whole Mesh			Part-level
Eval Cond.	Train Setting	CD $\downarrow$	F-Score $\uparrow$	ULIP $\uparrow$	CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
Point	Point only	0.1623	0.7368	0.2014	0.3550	0.1763	0.2320
	Point + Mask	0.1587	0.7505	0.2112	0.3421	0.1816	0.2396
	Point + Box + Mask	0.1525	0.7689	0.2171	0.3334	0.1902	0.2506
Box	Box only	0.1489	0.7756	0.2171	0.3155	0.2011	0.2756
	Point + Box	0.1458	0.7812	0.2201	0.3102	0.2023	0.2763

Table 3: Ablation on multi-granularity prompt synergy. Eval Cond. denotes the evaluation condition, while Train Setting lists the prompt modalities used to train each model. Point only , Box only , and Mask only denote single-modality training baselines. We compare whole-mesh and part-level fidelity within each evaluation condition.

Figure 4: Qualitative generation results of our method under varying geometric prompts.

Cond.	Guid.	Part-level
Cond.	Guid.	CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
Points	-	0.3406	0.1811	0.2323
Points	✓	0.3334	0.1902	0.2506
Masks	-	0.3011	0.2078	0.2813
Masks	✓	0.2938	0.2049	0.2840

Table 4: Ablation on Asymmetric Geometric Guidance under point and mask prompts.

Method	Cond.	Whole Mesh			Part-level
Method	Cond.	CD $\downarrow$	F-Score $\uparrow$	ULIP $\uparrow$	CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
FullPart	3D Box	0.1625	0.7577	0.1775	0.2488	0.2064	0.4370
Ours	3D Box	0.1385	0.8158	0.2209	0.2356	0.2653	0.4027

Table 5: Comparison with FullPart under 3D box conditioning on the PartObjaverse-Tiny benchmark.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Conditioning route	Part CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
Gated AdaLN (ours)	0.2938	0.2049	0.2840
Prompt tokens in cross-attention	0.6147	0.0818	0.1089
PartCrafter (reference)	0.5235	0.0907	0.1257

Appendix

Table 6: Conditioning routes under mask input on PartVerseXL. PartCrafter is a reference using part count and no spatial prompts, not a matched-condition ablation.

Condition	Setting	CD $\downarrow$	F-Score $\uparrow$	ULIP $\uparrow$
Point	w/ Grad	0.1518	0.7702	0.2166
Point	wo/ Grad	0.1525	0.7689	0.2171
Box	w/ Grad	0.1531	0.7762	0.2198
Box	wo/ Grad	0.1447	0.7828	0.2219
Mask	w/ Grad	0.1512	0.7761	0.2212
Mask	wo/ Grad	0.1418	0.7877	0.2206

Appendix

Table 7: Ablation on the Stop-Gradient Design in Geometric Guidance. "w/ Grad" allows the alignment loss to update the teacher prompt embeddings, while "wo/ Grad" (our default) stops gradients at the box/mask embeddings before the Visual Bridge.

Condition	Alignment loss	Part CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
Mask	Partial OT (ours)	0.2938	0.2049	0.2840
	InfoNCE	0.3053	0.1924	0.2673
Box	Partial OT (ours)	0.3042	0.2078	0.2810
	InfoNCE	0.3074	0.1874	0.2653
Point	Partial OT (ours)	0.3334	0.1902	0.2506
	InfoNCE	0.3499	0.1807	0.2346

Appendix

Table 8: End-to-end comparison of alignment losses on PartVerseXL. Only the alignment loss is changed; initialization and training budget are held fixed. IoU V and IoU B denote part-level voxel and bounding-box IoU, respectively.

Assignment operator	Top-1 matching $\uparrow$	Many-to-one collapse $\downarrow$
Row-wise similarity matching	0.803	0.133
Sinkhorn global assignment	0.895	0.024

Appendix

Table 9: Assignment-operator diagnostic on the same frozen features. These are feature-matching results, rather than generation metrics or separate L2/InfoNCE training experiments.

Transport formulation	Top-1 matching $\uparrow$
Strict OT	0.677
Partial OT with dustbin	0.747

Appendix

Table 10: Frozen-feature diagnostic with four cross-object box-prior distractors. Both variants receive the same augmented set of priors; the distractors have no valid correspondence in the current object.

Method	Part-level
Method	CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
2D Box w/ Perturbation	0.3171	0.2032	0.2759
Half Box + Point	0.3287	0.1998	0.2609
Half Mask + Half Box	0.3190	0.2053	0.2706
Part Count Only	0.4786	0.1255	0.1608
Ours (2D Box)	0.3042	0.2078	0.2810

Appendix

Table 11: Analysis of the adaptability to user input variations.

Prompt source	Part CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
Human points + 2D boxes	$0.384\pm 0.022$	$0.131\pm 0.010$	$0.231\pm 0.013$

Appendix

Table 12: Human-provided prompts on 50 PartObjaverse-Tiny objects. All inputs contain both points and 2D boxes. Values are mean

\pm

standard deviation across three annotators. IoU V and IoU B denote part-level voxel and bounding-box IoU, respectively.

Figure 5: Efficiency comparison between FlexPart and other methods.

Figure 6: Qualitative analysis of point placement under four sampling strategies: Center, Offset, Edge, and Cluster.

Input Type	Point Strategy	CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
Point Only	Center	0.3302	0.1905	0.2513
	Edge	0.4190	0.1342	0.1697
	Offset	0.3476	0.1898	0.2455
	Cluster	0.3803	0.1632	0.1905
Point + Box	Center	0.3118	0.2074	0.2728
	Edge	0.3178	0.1996	0.2703

Appendix

Table 13: Robustness analysis of point prompt placement across different input modalities. Metrics are evaluated on the PartVerseXL test set.

Setting	Condition	CD $\downarrow$	IoU V $\uparrow$	IoU B $\uparrow$
SAM (Random)	Mask	0.3498	0.1796	0.2375
SAM (Center)	Mask	0.3387	0.1731	0.2244
Direct point (ours)	Point	0.3334	0.1902	0.2506
Ground Truth (Random)	Mask	0.2938	0.2049	0.2840

Appendix

Table 14: Comparison of direct point conditioning and mask-conditioned generation using SAM-derived or ground-truth masks. Direct point conditioning outperforms both SAM cascades on the reported part-level metrics.

Figure 7: Qualitative evaluation of the SAM-based interface. We compare generation results under two input conditions across diverse examples. SAM : masks generated by SAM from a single point prompt sampled near the visual center. GT : ground-truth segmentation masks.

Figure 8: Additional qualitative results of FlexPart across objects and geometric prompts.

Figure 9: Generalization to complex topologies. Our method directly processes user-provided point or box prompts to generate 3D part assets.

Part-Level 3D Generation

Part-level 3D generation creates and edits 3D objects as structured components with controllable geometry, relationships, materials, and motion.

See all 32 papers

Seg3DParts: Segmentation-Grounded Controllable Part-Level 3D Generation
Sep 29, 2026Jiantao Lin, Meixi Chen, Yingjie Xu +53D Mesh Generation Part-Level 3D Generation
The Hong Kong University of Science and Technology (Guangzhou) · Knowin AI · The Hong Kong University of Science and Technology
SAM3D-Part: Interactive Part Selection and Generation from 3D Objects
Sep 14, 2026Jiahao Chang, Dong Du, Wanhu Sun +63D Mesh Generation Part-Level 3D Generation
SSE, CUHKSZ and FNii-Shenzhen and Meshy AI, China · Meshy AI, China · Nanjing University of Science and Technology, China +5
PartRAG: Retrieval-Augmented Part-Level 3D Generation and Editing
Feb 19, 2026Peize Li, Zeyu Zhang, Hao Tang3D Shape Retrieval Part-Level 3D Generation
King’s College London · Peking University

Sep 29, 2026J/K move · Enter open · S save

Seg3DParts: Segmentation-Grounded Controllable Part-Level 3D Generation

Jiantao Lin, Meixi Chen, Yingjie Xu, Chenbo Fu+4

The Hong Kong University of Science and Technology (Guangzhou) · Knowin AI · The Hong Kong University of Science and Technology

3D Mesh Generation Part-Level 3D Generation 3D Asset Generation 3D Shape Generation

Open paper AskPDF arXiv

Part-level 3D assets are essential for editing, reassembly, and interaction, yet recovering such structure from a single image remains challenging due to occlusion, ambiguous boundaries, and the need for coherent multi-part reasoning. Existing approaches struggle to achieve both controllable part-level generation and coherent multi-part structure, as part identity and spatial allocation are typically inferred implicitly. We present Seg3DParts, a segmentation-grounded framework for controllable part-level 3D generation from a single image. By treating segmentation as an explicit grounding signal, our method defines part identity during generation, enabling each component to be anchored to a corresponding image region. To ensure coherent assemblies, we introduce structured cross-part interaction that allows components to exchange global context throughout the generative process. As a result, Seg3DParts directly generates well-aligned part meshes in a shared canonical space without post-hoc alignment, supporting flexible and controllable decomposition. We further introduce PartObjectNet, a large-scale dataset with over 200K objects and 1M annotated parts. Experiments demonstrate that Seg3DParts achieves superior geometry quality, cross-part coherence, and part-level controllability over existing methods.

Abstract

Part-Level 3D Generation

Explore similar work

Seg3DParts: Segmentation-Grounded Controllable Part-Level 3D Generation

SAM3D-Part: Interactive Part Selection and Generation from 3D Objects

PartRAG: Retrieval-Augmented Part-Level 3D Generation and Editing