CRAG: Can 3D Generative Models Help 3D Assembly?

Zeyu Jiang; Sihang Li; Siqi Tan; Chenyang Xu; Juexiao Zhang; Julia Galway-Witham; Xue Wang; Scott A. Williams

cs.CVFeb 26, 2026

CRAG: Can 3D Generative Models Help 3D Assembly?

Authors: Zeyu Jiang, Sihang Li, Siqi Tan, Chenyang Xu, Juexiao Zhang, Julia Galway-Witham, Xue Wang, Scott A. Williams,

Organizations: New York University

Abstract

Most existing 3D assembly methods treat the problem as pure pose estimation, rearranging observed parts via rigid transformations. In contrast, human assembly naturally couples structural reasoning with holistic shape inference. Inspired by this intuition, we reformulate 3D assembly as a joint problem of assembly and generation. We show that these two processes are mutually reinforcing: assembly provides part-level structural priors for generation, while generation injects holistic shape context that resolves ambiguities in assembly. Unlike prior methods that cannot synthesize missing geometry, we propose CRAG, which simultaneously generates plausible complete shapes and predicts poses for input parts. Extensive experiments demonstrate state-of-the-art performance across in-the-wild objects with diverse geometries, varying part counts, and missing pieces. Project Page: https://ai4ce.github.io/CRAG/

Figures & tables

Figure 2: Overview of our approach Crag . We propose a unified framework for 3D assembly and whole-shape generation. Our model consists of two interacting branches: an Assembly Branch that predicts the pose for each part via SE(3) flow matching, and a Generation Branch that synthesizes the complete shape via flow matching. A Joint Adapter bridges these branches, enabling bidirectional information flow. We employ a two-stage training strategy: learning assembly first, and then jointly finetuning both tasks.

	PartNeXt ( Wang et al., 2025a )								Breaking Bad ( Sellán et al., 2022 )
Part Status	Complete				Missing				Complete				Missing
Method	RE ↓	TE ↓	PA ↑	CD ↓	RE ↓	TE ↓	PA ↑	CD ↓	RE ↓	TE ↓	PA ↑	CD ↓	RE ↓	TE ↓	PA ↑	CD ↓
GARF ( Li et al., 2025a )	52.52	10.68	58.19	3.10	49.40	9.73	60.71	5.78	8.75	1.72	93.56	0.42	14.59	3.01	85.55	3.43
RPF ( Sun et al., 2025 )	54.99	29.54	46.17	10.46	42.49	22.95	57.20	8.21	30.59	13.67	80.23	1.01	31.04	14.96	77.05	1.20
Crag w/o img (Ours)	45.45	9.82	61.67	3.31	42.46	8.70	66.74	5.17	8.00	1.37	94.64	0.22	11.43	1.79	92.03	0.52
Assembler ( Zhao et al., 2025a )	85.82	17.14	44.18	27.93	83.80	15.00	49.86	24.71	74.92	13.22	48.38	7.46	74.45	13.43	45.93	7.78

Table 1: Quantitative comparison on PartNeXt ( Wang et al., 2025a ) and Breaking Bad ( Sellán et al., 2022 ) under two part-status settings: Complete (all parts observed) and Missing (with missing parts). CRAG consistently achieves the best overall performance across datasets and remains robust in the challenging missing-part setting. The best and second best results are highlighted.

Figure 3: Qualitative results across PartNeXt ( Wang et al., 2025a ) , Breaking Bad ( Sellán et al., 2022 ) , and MorphoSource ( Boyer et al., 2016 ) . We first compare methods without reference images by contrasting GARF ( Li et al., 2025a ) , RPF ( Sun et al., 2025 ) , and Crag w/o image, where Crag produces more coherent assemblies and more complete shapes from the same observed parts. We then compare image-conditioned methods by showing Assembler and full Crag given the reference image, where Crag better aligns parts and yields shapes that more closely match the ground truth.

Figure 4: Qualitative results on PartNeXt ( Wang et al., 2025a ) , Breaking Bad ( Sellán et al., 2022 ) , and MorphoSource ( Boyer et al., 2016 ) with missing parts. We compare Assembler, Crag without reference images, and Crag given a reference image. Crag simultaneously assembles the observed parts and synthesizes a plausible, complete shape, and reference images further improve fidelity when available.

Method	CD-L1 ( $\times 10^{-2}$ ) $\downarrow$	CD-L2 ( $\times 10^{-3}$ ) $\downarrow$	F-Score@1% (%) $\uparrow$	F-Score@5% (%) $\uparrow$	EMD ( $\times 10^{-3}$ ) $\downarrow$
SDFusion ( Cheng et al., 2023 )	22.51	23.93	1.90	28.58	28.60
MGPC ( Liu et al., 2026 )	12.59	11.54	12.75	70.54	16.99
Crag w/o img	12.31	10.32	13.84	74.11	16.12
Crag	9.39	5.76	18.23	82.82	13.09

Table 2: Comparison with shape-completion baselines on PartNeXt ( Wang et al., 2025a ) under the missing-part setting (2–20 parts). SDFusion ( Cheng et al., 2023 ) and MGPC ( Liu et al., 2026 ) operate as downstream completion modules: they receive fragments already pre-aligned at ground-truth poses (equivalent to an oracle upstream assembly) as a single point cloud with missing regions, together with a reference image. Crag starts from unaligned fragments and must solve both assembly and shape synthesis jointly, yet still substantially outperforms both.

Figure 5: Qualitative results under ambiguous reference images on PartNeXt ( Wang et al., 2025a ) . We compare image-only generation with TripoSG ( Li et al., 2025b ) against Crag by visualizing Crag ’s assembled parts and generated shapes alongside the ground truth. When the reference view is incomplete and does not reveal the full object, part-level evidence helps, to some extent, resolve ambiguity and yields a better shape.

Method	CD-L1 ( $\times 10^{-2}$ ) $\downarrow$	CD-L2 ( $\times 10^{-3}$ ) $\downarrow$	F-Score@1% (%) $\uparrow$	F-Score@5% (%) $\uparrow$	EMD ( $\times 10^{-3}$ ) $\downarrow$	Failure Rate $\downarrow$
TripoSG ( Li et al., 2025b )	9.12	11.23	14.63	73.18	16.67	25.0% (1157/4626)
Crag w/o img	9.26	11.46	15.02	72.61	16.96	0%
Crag	6.53	5.83	20.49	83.87	12.92	0%

Table 3: Generation quality on the PartNeXt ( Wang et al., 2025a ) evaluation set (4,626 shapes). Part-level evidence from the assembly branch substantially improves image-conditioned generation across all metrics and prevents the VAE-decoding failures observed in image-only TripoSG ( Li et al., 2025b ) .

Panel of Table 4 — Table 4: Ablation study (PartNeXt ( Wang et al., 2025a ) , Complete) with key design choices. “A” and “G” refer to the assembly branch and the generation branch, respectively.

# Parts	2	3	4	5	6	7	8	9	10
Assembler	83.6	76.0	64.7	54.0	44.5	38.8	33.6	34.0	31.1
Crag	95.0	91.2	84.7	70.9	65.6	62.0	57.0	58.9	53.9
$\Delta$	$+11.4$	$+15.2$	$+20.0$	$+16.9$	$+21.1$	$+23.2$	$+23.4$	$+24.9$	$+22.8$

Table 5: PA on PartNeXt ( Wang et al., 2025a ) (Complete) broken down by the number of parts per shape. Crag ’s lead over Assembler ( Zhao et al., 2025a ) (

\Delta=\text{{Crag}{}}-\text{Assembler}

, last row) widens substantially as the number of parts grows, roughly doubling from

+11.4

at 2 parts to

+22.8

at 10 parts.

Figure 6: Qualitative results of Crag on FRACTURA ( Li et al., 2025a ) , demonstrating robustness on real-world fractures. All parts are real scanned fragments; colors are rendered for visualization.

Figure 7: Our representative failure cases.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Main Anatomical Composition	Count
Cranium	Cranium elements (cranium, maxilla, frontal, zygomatic)	1,130
Mandible	Mandible elements (mandible, hemimandible)	1,015
Forearm	Ulna, radius	489
Shoulder Girdle	Scapula, clavicle	198
Leg & Knee	Tibia, fibula, patella, femur fragments	173
Vertebral Column	Vertebrae (cervical, thoracic, lumbar), sacrum	129

Appendix

Table 6: Distribution of osteological samples by anatomical category in the MorphoSource ( Boyer et al., 2016 ) collection.

Hyperparameter	Assembly Branch	Generation Branch	Joint Adapter
Model Type	Custom DiT	TripoSG DiT ( Li et al., 2025b )	Bi-dir Cross-Attn
Number of Layers	21	21	21
Hidden Dimension	768	2048	512
Attention Heads	12	16	8
Dim per Head	64	128	64

Appendix

Table 7: Network Architecture

Figure 8: Qualitative results on PartNeXt ( Wang et al., 2025a ) and Breaking Bad ( Sellán et al., 2022 ) We compare Assembler ( Zhao et al., 2025a ) , Crag without reference images, and Crag given a reference image. Crag simultaneously assembles the observed parts and synthesizes a plausible complete shape, and reference images further improve fidelity when available.

3D Shape Generation

3D shape generation creates and edits 3D objects from learned geometric priors and text, images, sketches, or structural constraints.

See all 106 papers

PART: Learning 3D Part Assembly and Retrieval with Transformers
Sep 17, 2026Ruchao Bao, Wenzheng Wu, Chucheng Xiang +53D Reconstruction
University of Science and Technology of China, China · Tencent, China · Hong Kong University of Science and Technology, China +1
AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents
Sep 30, 2026Jiahao Zhang, Yeying Fan, Moitreya Chatterjee +5Robot Manipulation Benchmarks Robotic Assembly
The Australian National University · Tsinghua University · Mitsubishi Electric Research Laboratories (MERL) +1
AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
May 13, 2026Danrui Li, Jiahao Zhang, Bernhard Egger +4Object Pose Estimation Robot Manipulation Benchmarks
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1

Sep 17, 2026J/K move · Enter open · S save

PART: Learning 3D Part Assembly and Retrieval with Transformers

Ruchao Bao, Wenzheng Wu, Chucheng Xiang, Zhongyuan Liu+4

University of Science and Technology of China, China · Tencent, China · Hong Kong University of Science and Technology, China+1

3D Reconstruction

Open paper AskPDF arXiv

3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified transformer-based framework for 3D part retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate retrieval and assembly as a set prediction problem and design a novel transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target segmentation through joint training and a novel segmentation-enhanced optimization module. Finally, We curate a large-scale dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Project Page: https://iambrc.github.io/PART-project-page/.

Abstract

3D Shape Generation

Explore similar work

PART: Learning 3D Part Assembly and Retrieval with Transformers

AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects