Image-to-3D foundation models have improved 3D reconstruction, but typically produce monolithic representations without explicit part-level structure. Part-aware generation and learning-based assembly methods often require part-level supervision, category-specific priors, or paired part-layout training data. We instead consider a practical but underexplored setting in which independently prepared or generated 3D parts have no predefined inter-part relationships, shared coordinate frames, or consistent scales, and must be assembled into a reference-consistent object from a single image. We present VA-Assembly, a training-free framework for 3D part assembly through part-level transformation estimation using view-aligned diffusion guidance. Given a reference image and fixed 3D part geometries, VA-Assembly optimizes each part's translation, rotation, and scale. We condition a frozen view-conditioned diffusion model on same-view renderings of a reference-derived monolithic 3D representation, producing score gradients geometrically aligned with the part configuration. This guidance provides multi-view structural cues under large initial misalignment. By repurposing pretrained visual-structural priors as transformation-level supervision, VA-Assembly requires no annotated part layouts, predefined part taxonomies, category-specific assembly training, or bounding-box-based layout initialization. Experiments compare VA-Assembly with geometric registration and compositional layout baselines using each baseline's supported transformation components, showing improved part-wise transformation estimation. Ablation studies show that view-aligned conditioning, rather than diffusion guidance alone, is critical for reliable transformation gradients. Updating only part transformations preserves explicit component separability and supports component-level editing, replacement, and reconfiguration.
Figures & tables
Figure 1: Overview of the VA-Assembly pipeline. 3D assets are generated from a reference image by using a vision-language model and an image-to-3D foundation model. The resulting 3D parts are arranged into a coherent structure via transformation optimization guided by the proposed view-aligned score guidance.
Figure 2: Effect of view-aligned score guidance on target-view generation. Compared with original Zero-1-to-3, the proposed guidance better preserves target-view structure. Darker difference maps indicate smaller discrepancies.
Figure 3: Clean-latent estimates decoded from noisy rendered latents at different diffusion timesteps. High-noise guidance (t=850) yields broad, structure-oriented corrections over the object region, while lower-noise guidance (t=450,250) retains more local details and boundary information.
Figure 4: View-aligned score distillation for part transformation optimization. For each sampled view, the reference-view render provides the Zero-1-to-3 conditioning, while the current render is encoded and noised to form xv,t . The frozen diffusion U-Net predicts the noise under this same-view condition, from which the view-aligned SDS loss is computed. Its gradients are backpropagated to the current render to guide part transformation optimization.
Figure 5: View-Dependent 3D SDS Gradients from Single-View Rendering. Single-view SDS produces projection-biased gradients on 3D points, motivating multi-view aggregation for robust 3D arrangement.
Method
Task Type
Category Dependency
Translation
Rotation
Scale
CAP-Net ( Huang et al., 2025 )
Object Pose/shape Estimation
Category-specific
✓
✓
✓
CRISP ( Shi et al., 2025 )
Object Pose/shape Estimation
Category-agnostic
✓
✓
×
TEASER++ ( Yang et al., 2020 )
3D Registration
N/A
✓
✓
✓
Go-ICP ( Yang et al., 2015 )
3D Registration
N/A
✓
✓
×
CFPA ( Zhang et al., 2026 )
3D Arrangement
Category-specific
✓
✓
×
Assembler ( Zhao et al., 2025 )
3D Arrangement
Category-agnostic
✓
✓
×
Table 1: Transformation estimation capabilities of related methods
Figure 6: Qualitative comparison on the Breaking Bad and GSO datasets. The top three rows correspond to Breaking Bad, and the bottom three rows correspond to GSO. All methods start from the same challenging initialization, in which the initial 3D part transformations differ from the target in translation, rotation, and scale. VA-Assembly optimizes the full transformation and produces layouts that more closely match the target.
Optimization parameters
Method
Chamf. [NCS] ↓
Trans. [NCS] ↓
Rot. [deg] ↓
Scale ↓
Mean
SD
Mean
SD
Mean
SD
Mean
SD
Translation, Rotation
Go-ICP
0.181
0.179
0.269
0.280
44.61
28.324
–
–
Assembler
0.037
0.004
0.049
0.001
61.038
25.156
–
–
Ours
0.019
0.007
0.051
0.013
14.061
37.415
–
–
Translation Scale
REPARO
0.257
0.077
0.406
0.093
–
–
0.150
0.034
Ours
0.017
0.009
0.039
0.021
–
–
0.005
0.001
Table 2: Quantitative comparison within baseline-supported transformation spaces
Figure 7: Qualitative comparison with part-aware 3D generation methods. Using the same input image, we compare Part123, OmniPart, and VA-Assembly. Assembled and exploded views are provided to visualize the part-level structure. VA-Assembly preserves explicit part organization and direct part separability.
Method
CLIP ↑
PSNR ↑
SSIM ↑
OmniPart
0.9021
23.2988
0.9172
Part123
0.7460
18.5328
0.9113
Ours
0.9310
23.0371
0.9198
Table 3: Quantitative comparison with part-aware 3D generation methods
Init.
Config.
Trans. [NCS] ↓
Rot. [deg] ↓
Scale ↓
Local
LRGB
0.039
4.557
0.072
LVA−SDS
0.035
3.471
0.023
Global
LRGB
0.441
44.543
0.124
LVA−SDS
0.053
13.763
0.061
Table 4: Ablation study of RGB supervision and view-aligned score guidance under local and global initialization
SDS Config
Chamf. ↓ [NCS]
Trans. ↓ [NCS]
Rot. ↓ [deg]
Scale ↓
Single view Zero-1-to-3
0.102
0.183
45.426
0.112
Multi view Zero-1-to-3
0.111
0.234
57.881
0.078
Ours: View-aligned SDS
0.038
0.057
12.795
0.018
Table 5: Ablation study of SDS guidance strategies for part transformation optimization
Figure 8: Translation-gradient alignment under global initialization. Cosine similarity is computed between each guidance-induced translation update vector and the ground-truth vector from the current part position to the target position. Higher values indicate better alignment with the correct update direction.
Figure 9: Hierarchical 3D Part Assembly for Product Design.
3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified transformer-based framework for 3D part retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate retrieval and assembly as a set prediction problem and design a novel transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target segmentation through joint training and a novel segmentation-enhanced optimization module. Finally, We curate a large-scale dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Project Page: https://iambrc.github.io/PART-project-page/.
Ruchao Bao, Wenzheng Wu, Chucheng Xiang +5
University of Science and Technology of China, China · Tencent, China · Hong Kong University of Science and Technology, China +1
Most existing 3D assembly methods treat the problem as pure pose estimation, rearranging observed parts via rigid transformations. In contrast, human assembly naturally couples structural reasoning with holistic shape inference. Inspired by this intuition, we reformulate 3D assembly as a joint problem of assembly and generation. We show that these two processes are mutually reinforcing: assembly provides part-level structural priors for generation, while generation injects holistic shape context that resolves ambiguities in assembly. Unlike prior methods that cannot synthesize missing geometry, we propose CRAG, which simultaneously generates plausible complete shapes and predicts poses for input parts. Extensive experiments demonstrate state-of-the-art performance across in-the-wild objects with diverse geometries, varying part counts, and missing pieces. Project Page: https://ai4ce.github.io/CRAG/
3D part assembly aims to reconstruct a complete shape from a set of unassembled parts by predicting a rigid SE(3) transformation for each part. Because the placement of a part is rarely determined by its own geometry, the task demands reasoning over the entire part set, and existing methods, which rely on pairwise relations or iterative generation and verification, scale poorly to objects with many parts. We present LLM2R, which casts part assembly as sequence modeling with a pretrained decoder-only large language model (LLM): a Point-BERT encoder and a Q-Former compress each part into a block of geometric tokens, the blocks of all parts are concatenated, and a causal LLM adapted with Low-Rank Adaptation (LoRA) processes the sequence in one forward pass. Causal attention, however, lets the hidden states of a part observe only the parts that precede it. LLM2R therefore appends a block of learnable summary tokens after the last part: these tokens follow every part, so their states aggregate the whole assembly at a constant token cost, and a part-aware pose head fuses each part's states with them before predicting its rotation and translation. One architecture thus handles semantic and geometric assembly and predicts all poses in a single non-autoregressive pass. LLM2R achieves the highest part accuracy and success rate on all three PartNet categories, the best rotation error, translation error, and part accuracy on PartNeXt, and the lowest translation error and fastest inference on Breaking Bad, where only a fracture-specialized method is more accurate. Its advantage grows with the number of parts: at 100 parts it exceeds GARF by 33.5 points in part accuracy on Breaking Bad and Assembler by 29.1 points on PartNet, and it transfers zero-shot to unseen categories with more than 16 points higher part accuracy than the strongest of three baselines.
Zhouhan Zhong, Qiuhong Shen, Xingyi Yang +2
National University of Singapore · The Hong Kong Polytechnic University