Single-image 3D generation lies at the core of vision-to-graphics models in the real world. However, it remains a fundamental challenge to achieve reliable generalization across diverse semantic categories and highly variable structural complexity under sparse supervision. Existing approaches typically model objects in a monolithic manner or rely on a fixed number of parts, including recent part-aware models such as PartCrafter, which still require a labor-intensive user-specified part count. Such designs easily lead to overfitting, fragmented or missing structural components, and limited compositional generalization when encountering novel object layouts. To this end, this paper rethinks single-image 3D generation as learning an adaptive part-whole hierarchy in the flexible 3D latent space. We present a novel part-to-whole 3D generative world model that autonomously discovers latent structural slots by inferring soft and compositional masks directly from image tokens. Specifically, an adaptive slot-gating mechanism dynamically determines the slot-wise activation probabilities and smoothly consolidates redundant slots within different objects, ensuring that the emergent structure remains compact yet expressive across categories. Each distilled slot is then aligned to a learnable, class-agnostic prototype bank, enabling powerful cross-category shape sharing and denoising through universal geometric prototypes in the real world. Furthermore, a lightweight 3D denoiser is introduced to reconstruct geometry and appearance via unified diffusion objectives. Experiments show consistent gains in cross-category transfer and part-count extrapolation, and ablations confirm complementary benefits of the prototype bank for shape-prior sharing as well as slot-gating for structural adaptation.
Figures & tables
Fig. 1 : We rethink single-image 3D generation as learning a structured 3D world state with an adaptive part–whole hierarchy. Specifically, latent structural slots are dynamically gated per instance, and a class-agnostic prototype bank provides shared part-level priors for improved structural coherence and generalization.
Fig. 2 : Framework of our part-to-whole 3D generative world model. Given a single-view RGB image, our model builds a structured 3D world state with flexible-capacity part slots. An adaptive slot-gating head selects active slots, a prototype bank injects shared part priors, and a pretrained 3D DiT rectified-flow backbone denoises slot latents with canonical-masked supervision. Decoded part meshes are finally composed into the 3D object or scene for generation.
Fig. 3 : Generalization across diverse 3D object cases. Given an input RGB image (left), our approach reconstructs more complete and structurally coherent shapes with cleaner part boundaries and less part interpenetration.
Method
Segment Mask
Fixed Part Number
Objaverse [ 24 ]
ShapeNet [ 25 ]
ABO [ 26 ]
IoU ↓
F-Score ↑
CD ↓
IoU ↓
F-Score ↑
CD ↓
IoU ↓
F-Score ↑
CD ↓
Dataset
/
/
0.080
/
/
0.183
/
/
0.014
/
/
TripoSG [ 15 ]
/
/
/
0.594
0.311
/
0.505
0.375
/
0.710
0.202
HoloPart [ 28 ]
✓
✓
0.044
0.692
0.192
0.111
0.550
0.351
0.045
0.809
0.134
PartCrafter [ 14 ]
×
✓
0.036
0.747
0.173
0.029
0.567
0.321
0.024
0.862
0.105
Ours
×
×
0.032
0.781
0.159
0.026
0.573
0.317
0.022
0.891
0.089
TABLE I : Evaluation on Part-based 3D Object Generation on the Objaverse [ 24 ] , ShapeNet [ 25 ] and ABO [ 26 ] datasets.
Method
3D-Front [ 27 ]
3D-Front (Occluded) [ 27 ]
IoU ↓
F-Score ↑
CD ↓
IoU ↓
F-Score ↑
CD ↓
MIDI [ 21 ]
0.001
0.793
0.160
0.002
0.662
0.259
PartCrafter [ 14 ]
0.003
0.815
0.149
0.004
0.780
0.151
Ours
0.002
0.852
0.128
0.003
0.785
0.124
TABLE II : Evaluation on Compositional 3D Scene Generation.
Fig. 4 : Generalization across complicated 3D scene cases.
Setting
Metrics
ASG
PB
Warm-up
IoU ↓
F-Score ↑
CD ↓
✓
✓
0.035
0.751
0.168
✓
✓
0.039
0.710
0.184
✓
✓
0.048
0.602
0.265
✓
✓
✓
0.032
0.781
0.159
TABLE III : Ablation study on the key components of our method over the Objaverse [ 24 ] dataset.
Generating 3D assets with explicit parts requires two properties at once: geometric detail inside each part and structural coherence across the whole object. Recent image-conditioned 3D generation models adopt a structured 3D representation that anchors tokens to explicit spatial positions, capturing both properties, but their output remains a single fused mesh. We ask how a part-decomposed generation model can inherit both properties of these holistic priors, and answer with STRUCTPARTS, a framework whose core technical contribution is a dual-frame coordinate interleaving mechanism: each shape latent is interpreted under both a part-local and an object-global reference frame, and processed through a global-local inter-leaved transformer that simultaneously reuses the prior's geometric detail at the part scale and its structural coherence at the whole-object scale. As its structural input, STRUCTPARTS uses a segmented mesh scaffold, a coarse mesh paired with a face-level segmentation mask kept entirely within the prior's native 3D representation. With only lightweight finetuning, both intra-part detail and inter-part coherence emerge from the same pretrained weights. On PartObjaverse-Tiny, STRUCTPARTS matches or surpasses prior part-decomposed models at a fraction of their training cost, while naturally supporting part-mesh refinement and part-level editing.
Yiqun Zhao, Binbin Huang, Haobin Duan +2
The University of Hong Kong · Transengram · Tencent +1
Part-level 3D assets are essential for editing, reassembly, and interaction, yet recovering such structure from a single image remains challenging due to occlusion, ambiguous boundaries, and the need for coherent multi-part reasoning. Existing approaches struggle to achieve both controllable part-level generation and coherent multi-part structure, as part identity and spatial allocation are typically inferred implicitly. We present Seg3DParts, a segmentation-grounded framework for controllable part-level 3D generation from a single image. By treating segmentation as an explicit grounding signal, our method defines part identity during generation, enabling each component to be anchored to a corresponding image region. To ensure coherent assemblies, we introduce structured cross-part interaction that allows components to exchange global context throughout the generative process. As a result, Seg3DParts directly generates well-aligned part meshes in a shared canonical space without post-hoc alignment, supporting flexible and controllable decomposition. We further introduce PartObjectNet, a large-scale dataset with over 200K objects and 1M annotated parts. Experiments demonstrate that Seg3DParts achieves superior geometry quality, cross-part coherence, and part-level controllability over existing methods.
Jiantao Lin, Meixi Chen, Yingjie Xu +5
The Hong Kong University of Science and Technology (Guangzhou) · Knowin AI · The Hong Kong University of Science and Technology
Part-aware 3D generation aims to synthesize structured objects with semantically meaningful components, yet often suffers from structural ambiguity due to identity-layout entanglement. Existing methods either infer part identity and spatial layout implicitly, which can lead to unstable part allocation (e.g., slot swapping or part merging), or rely on strong layout conditions that are difficult to obtain in practice. We attribute this ambiguity to identity-slot permutation freedom: without explicit identity-slot alignment, the correspondence between semantic parts and generation slots is not identifiable during training, allowing multiple slot assignments to fit the same supervision and leading to inconsistent decomposition. Based on this insight, we argue that stable part-aware generation requires identity-aligned one-to-one slot modelling. We therefore propose an identity-slot aligned framework, ISAP-3D, which anchors each part with semantic identity tokens and performs identity-conditioned one-to-one layout prediction, followed by layout-conditioned geometry synthesis. Structured local-global conditioning maintains identity alignment across semantic, spatial, and geometric stages. We also construct a part-level dataset with a unified semantic protocol to enable learnable and consistent identity-slot alignment. Extensive experiments demonstrate improved structural stability, controllability, and robustness over state-of-the-art part-aware generation baselines.