Most existing 3D assembly methods treat the problem as pure pose estimation, rearranging observed parts via rigid transformations. In contrast, human assembly naturally couples structural reasoning with holistic shape inference. Inspired by this intuition, we reformulate 3D assembly as a joint problem of assembly and generation. We show that these two processes are mutually reinforcing: assembly provides part-level structural priors for generation, while generation injects holistic shape context that resolves ambiguities in assembly. Unlike prior methods that cannot synthesize missing geometry, we propose CRAG, which simultaneously generates plausible complete shapes and predicts poses for input parts. Extensive experiments demonstrate state-of-the-art performance across in-the-wild objects with diverse geometries, varying part counts, and missing pieces. Project Page: https://ai4ce.github.io/CRAG/
Figures & tables
Figure 2: Overview of our approach Crag . We propose a unified framework for 3D assembly and whole-shape generation. Our model consists of two interacting branches: an Assembly Branch that predicts the pose for each part via SE(3) flow matching, and a Generation Branch that synthesizes the complete shape via flow matching. A Joint Adapter bridges these branches, enabling bidirectional information flow. We employ a two-stage training strategy: learning assembly first, and then jointly finetuning both tasks.
PartNeXt ( Wang et al., 2025a )
Breaking Bad ( Sellán et al., 2022 )
Part Status
Complete
Missing
Complete
Missing
Method
RE ↓
TE ↓
PA ↑
CD ↓
RE ↓
TE ↓
PA ↑
CD ↓
RE ↓
TE ↓
PA ↑
CD ↓
RE ↓
TE ↓
PA ↑
CD ↓
GARF ( Li et al., 2025a )
52.52
10.68
58.19
3.10
49.40
9.73
60.71
5.78
8.75
1.72
93.56
0.42
14.59
3.01
85.55
3.43
RPF ( Sun et al., 2025 )
54.99
29.54
46.17
10.46
42.49
22.95
57.20
8.21
30.59
13.67
80.23
1.01
31.04
14.96
77.05
1.20
Crag w/o img (Ours)
45.45
9.82
61.67
3.31
42.46
8.70
66.74
5.17
8.00
1.37
94.64
0.22
11.43
1.79
92.03
0.52
Assembler ( Zhao et al., 2025a )
85.82
17.14
44.18
27.93
83.80
15.00
49.86
24.71
74.92
13.22
48.38
7.46
74.45
13.43
45.93
7.78
Table 1: Quantitative comparison on PartNeXt ( Wang et al., 2025a ) and Breaking Bad ( Sellán et al., 2022 ) under two part-status settings: Complete (all parts observed) and Missing (with missing parts). CRAG consistently achieves the best overall performance across datasets and remains robust in the challenging missing-part setting. The best and second best results are highlighted.
Figure 3: Qualitative results across PartNeXt ( Wang et al., 2025a ) , Breaking Bad ( Sellán et al., 2022 ) , and MorphoSource ( Boyer et al., 2016 ) . We first compare methods without reference images by contrasting GARF ( Li et al., 2025a ) , RPF ( Sun et al., 2025 ) , and Crag w/o image, where Crag produces more coherent assemblies and more complete shapes from the same observed parts. We then compare image-conditioned methods by showing Assembler and full Crag given the reference image, where Crag better aligns parts and yields shapes that more closely match the ground truth.
Figure 4: Qualitative results on PartNeXt ( Wang et al., 2025a ) , Breaking Bad ( Sellán et al., 2022 ) , and MorphoSource ( Boyer et al., 2016 ) with missing parts. We compare Assembler, Crag without reference images, and Crag given a reference image. Crag simultaneously assembles the observed parts and synthesizes a plausible, complete shape, and reference images further improve fidelity when available.
Method
CD-L1 ( ×10−2 ) ↓
CD-L2 ( ×10−3 ) ↓
F-Score@1% (%) ↑
F-Score@5% (%) ↑
EMD ( ×10−3 ) ↓
SDFusion ( Cheng et al., 2023 )
22.51
23.93
1.90
28.58
28.60
MGPC ( Liu et al., 2026 )
12.59
11.54
12.75
70.54
16.99
Crag w/o img
12.31
10.32
13.84
74.11
16.12
Crag
9.39
5.76
18.23
82.82
13.09
Table 2: Comparison with shape-completion baselines on PartNeXt ( Wang et al., 2025a ) under the missing-part setting (2–20 parts). SDFusion ( Cheng et al., 2023 ) and MGPC ( Liu et al., 2026 ) operate as downstream completion modules: they receive fragments already pre-aligned at ground-truth poses (equivalent to an oracle upstream assembly) as a single point cloud with missing regions, together with a reference image. Crag starts from unaligned fragments and must solve both assembly and shape synthesis jointly, yet still substantially outperforms both.
Figure 5: Qualitative results under ambiguous reference images on PartNeXt ( Wang et al., 2025a ) . We compare image-only generation with TripoSG ( Li et al., 2025b ) against Crag by visualizing Crag ’s assembled parts and generated shapes alongside the ground truth. When the reference view is incomplete and does not reveal the full object, part-level evidence helps, to some extent, resolve ambiguity and yields a better shape.
Method
CD-L1 ( ×10−2 ) ↓
CD-L2 ( ×10−3 ) ↓
F-Score@1% (%) ↑
F-Score@5% (%) ↑
EMD ( ×10−3 ) ↓
Failure Rate ↓
TripoSG ( Li et al., 2025b )
9.12
11.23
14.63
73.18
16.67
25.0% (1157/4626)
Crag w/o img
9.26
11.46
15.02
72.61
16.96
0%
Crag
6.53
5.83
20.49
83.87
12.92
0%
Table 3: Generation quality on the PartNeXt ( Wang et al., 2025a ) evaluation set (4,626 shapes). Part-level evidence from the assembly branch substantially improves image-conditioned generation across all metrics and prevents the VAE-decoding failures observed in image-only TripoSG ( Li et al., 2025b ) .
Table 4: Ablation study (PartNeXt ( Wang et al., 2025a ) , Complete) with key design choices. “A” and “G” refer to the assembly branch and the generation branch, respectively.
# Parts
2
3
4
5
6
7
8
9
10
Assembler
83.6
76.0
64.7
54.0
44.5
38.8
33.6
34.0
31.1
Crag
95.0
91.2
84.7
70.9
65.6
62.0
57.0
58.9
53.9
Δ
+11.4
+15.2
+20.0
+16.9
+21.1
+23.2
+23.4
+24.9
+22.8
Table 5: PA on PartNeXt ( Wang et al., 2025a ) (Complete) broken down by the number of parts per shape. Crag ’s lead over Assembler ( Zhao et al., 2025a ) ( Δ=Crag−Assembler , last row) widens substantially as the number of parts grows, roughly doubling from +11.4 at 2 parts to +22.8 at 10 parts.
Figure 6: Qualitative results of Crag on FRACTURA ( Li et al., 2025a ) , demonstrating robustness on real-world fractures. All parts are real scanned fragments; colors are rendered for visualization.
Figure 7: Our representative failure cases.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Main Anatomical Composition
Count
Cranium
Cranium elements (cranium, maxilla, frontal, zygomatic)
1,130
Mandible
Mandible elements (mandible, hemimandible)
1,015
Forearm
Ulna, radius
489
Shoulder Girdle
Scapula, clavicle
198
Leg & Knee
Tibia, fibula, patella, femur fragments
173
Vertebral Column
Vertebrae (cervical, thoracic, lumbar), sacrum
129
Appendix
Table 6: Distribution of osteological samples by anatomical category in the MorphoSource ( Boyer et al., 2016 ) collection.
Hyperparameter
Assembly Branch
Generation Branch
Joint Adapter
Model Type
Custom DiT
TripoSG DiT ( Li et al., 2025b )
Bi-dir Cross-Attn
Number of Layers
21
21
21
Hidden Dimension
768
2048
512
Attention Heads
12
16
8
Dim per Head
64
128
64
Appendix
Table 7: Network Architecture
Figure 8: Qualitative results on PartNeXt ( Wang et al., 2025a ) and Breaking Bad ( Sellán et al., 2022 ) We compare Assembler ( Zhao et al., 2025a ) , Crag without reference images, and Crag given a reference image. Crag simultaneously assembles the observed parts and synthesizes a plausible complete shape, and reference images further improve fidelity when available.
3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified transformer-based framework for 3D part retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate retrieval and assembly as a set prediction problem and design a novel transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target segmentation through joint training and a novel segmentation-enhanced optimization module. Finally, We curate a large-scale dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Project Page: https://iambrc.github.io/PART-project-page/.
Ruchao Bao, Wenzheng Wu, Chucheng Xiang +5
University of Science and Technology of China, China · Tencent, China · Hong Kong University of Science and Technology, China +1
The task of 3D assembly requires translating an understanding of parts and their relationships into precise spatial arrangements. Can pretrained general-purpose agents assemble objects through visual interaction without additional assembly-specific fine-tuning? To investigate this question, we introduce AssemblyWorld, an interactive 3D environment in which agents inspect rendered views and manipulate supplied rigid parts, guided by images or assembly manuals when available. Agents perceive part geometry through 2D views rather than direct access to mesh vertices or faces, while their resulting assemblies are evaluated geometrically. Building on this environment, we construct AssemblyWorldBench, comprising 100 assembly tasks across 80 objects spanning furniture, industrial assembly, and fracture reassembly. Evaluating eight agent systems reveals substantial differences in their capabilities. The strongest system achieves 80.9% part accuracy but 59.4% complete-assembly success. The evaluated open-source systems lag substantially behind their stronger closed-source peers in both execution reliability and assembly accuracy. Analyses of visual references, interaction trajectories, and failures show how agents revise assemblies while leaving residual positioning errors. AssemblyWorld provides a common setting for both assessing the capabilities of interactive assembly agents and characterizing the gap between approximate structure recovery and precise reconstruction.
Jiahao Zhang, Yeying Fan, Moitreya Chatterjee +5
The Australian National University · Tsinghua University · Mitsubishi Electric Research Laboratories (MERL) +1
Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies. We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and part assembly trajectories. We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories. AssemblyDyno outperforms prior works in both assembly pose estimation and trajectory feasibility, where the latter is evaluated by our physics-based simulations.
Danrui Li, Jiahao Zhang, Bernhard Egger +4
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1