Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity. Project webpage: https://xishuxishu.github.io/uniforge3d-project/
Figures & tables
Figure 1: UniForge3D generates faithful 3D assets from sparse, unposed multi-view images.
Figure 2: Different views provide different evidence for the same 3D region. The 3D box marks three blueberries, which are clearly visible, seen at an oblique angle, or fully occluded across the three views, motivating region-specific view selection.
Figure 3: Overview of UniForge3D. Left: The reconstruction and generation branches interact bidirectionally to predict multi-view point maps and a complete sparse structure in a shared anchor-guided space. Right: Structure-conditioned point maps establish voxel-to-pixel correspondences, enabling the model to select informative views and use their image features to guide detailed shape generation.
Figure 4: Qualitative comparisons on Toys4K and GSO. Each row shows four input views, ground-truth renderings, and results from different methods. To enable direct comparison, each generated mesh is geometrically aligned with the ground-truth asset and rendered using the corresponding input-view cameras. Our method better preserves the observed appearance and local structures while producing complete and consistent geometry.
Figure 5: Qualitative comparisons on real-world sparse, unposed images. All results are aligned and rendered from common viewpoints for visual comparison.
Dataset
Toys4K ( Stojanov et al., 2021 )
GSO ( Downs et al., 2022 )
Metric
PSNR ↑
SSIM ↑
LPIPS ↓
CD ( ×10−2 ) ↓
F-score ↑
PSNR ↑
SSIM ↑
LPIPS ↓
CD ( ×10−2 ) ↓
F-score ↑
Stream3D + TRELLIS.2 ( Zhou et al., 2026 )
20.955
0.9351
0.0789
6.3608
0.5003
19.576
0.9119
0.0990
6.2441
0.4700
Hunyuan3D-2MV ( Team, 2025 )
22.726
0.9470
0.0576
2.9175
0.6723
20.526
0.9206
0.0780
3.2713
0.6139
MV-SAM3D ( Li et al., 2026a )
23.071
0.9469
0.0590
2.7147
0.6830
20.653
0.9222
0.0791
2.7447
0.6517
ReconViaGen v0.2 ( Chang et al., 2026 )
22.713
0.9451
0.0548
2.6155
0.6786
20.498
0.9196
0.0738
2.5905
0.6697
ReconViaGen v0.5 ( Chang et al., 2026 )
25.020
0.9553
0.0519
2.7007
0.6840
22.774
0.9313
0.0731
2.7002
0.6489
Table 1: Quantitative comparison on the Toys4K ( Stojanov et al., 2021 ) and GSO ( Downs et al., 2022 ) datasets. Best results are shown in bold and second-best results are underlined. PSNR, SSIM, and LPIPS are computed on RGB renderings.
Method
VGGT Space
DiT Space
IoU ↑
CD ( ×10−2 ) ↓
DINO Conditioning
N/A
Anchor-Guided Space
0.4261
2.0019
DINO–VGGT Conditioning
Anchor-Guided Space
Anchor-Guided Space
0.5533
1.1710
Canonical-Space Joint Training
Canonical Space
Canonical Space
0.4601
1.5825
Cross-Space Joint Training
First-Camera Space
Canonical Space
0.4563
1.6291
Anchor-Guided Joint Training (Ours)
Anchor-Guided Space
Anchor-Guided Space
0.5827
1.0563
Table 2: Ablation of conditioning schemes and coordinate spaces for sparse-structure generation on GSO ( Downs et al., 2022 ) .
Method
Acc. ↓
Comp. ↓
Overall ↓
Image-Only VGGT
5.025
4.667
4.846
Jointly Trained VGGT
4.053
4.250
4.152
Structure-Conditioned VGGT
3.539
3.086
3.313
Table 3: Point-map reconstruction results on the GSO dataset ( Downs et al., 2022 ) . Distances are reported in normalized object coordinates ( ×10−3 ).
Method
PSNR ↑
SSIM ↑
LPIPS ↓
CD ( ×10−2 ) ↓
F-score ↑
Full Multi-View Conditioning
26.892
0.9649
0.0401
1.4739
0.8758
Voxel-Wise View Selection and Conditioning (Ours)
27.207
0.9657
0.0388
1.4047
0.8897
Table 4: Ablation of view-conditioning strategies for detailed shape generation on GSO ( Downs et al., 2022 ) . Both variants take the same predicted 643 sparse structure as input. PSNR, SSIM, and LPIPS are computed on rendered normal maps.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Qualitative comparison of point-map reconstruction. Each row shows one object. From left to right: four input views, point maps predicted by image-only VGGT, jointly trained VGGT, and structure-conditioned VGGT (ours), followed by ground-truth visible point maps.
Figure 7: Qualitative ablation of voxel-wise view selection and conditioning. Each row shows four input views followed by rendered normal maps of shapes generated with full multi-view conditioning and our method, respectively. Both variants use the same predicted sparse structure. Red boxes highlight reference details in the input images, and black boxes mark the corresponding regions in the generated shapes.
Figure 8: Additional qualitative results on rendered 3D assets. Each group shows four input views on a white background and the corresponding renderings of our generated mesh on a light-blue background. For direct visual comparison, we geometrically align the generated mesh with the ground-truth asset and render it using the same cameras as the input views.
Figure 9: Visualization of voxel-wise view selection from our inference results. Each row shows a generated sparse structure with a query voxel highlighted in magenta, followed by four input views. Colored pixels and enlarged insets show the predicted pixel-to-voxel correspondences, with numbers indicating highlighted pixels within each inset. Green checkmarks identify the views selected by our model.
Figure 10: Shared-space consistency of Structure-Conditioned VGGT . Each row shows the reconstructed colored point cloud, the generated mesh, and their direct overlay from left to right. No object-specific alignment or registration is applied.
Figure 11: Shared-space consistency of Jointly Trained VGGT . Each row shows the reconstructed colored point cloud, the generated mesh, and their direct overlay from left to right. No object-specific alignment or registration is applied.
Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input alignment, it often lacks the global priors needed for structural completeness. Conversely, diffusion-based generation provides rich geometric details but struggles with multi-view consistency. We present RecGen3D, a framework that combines these two paradigms into a cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ decoupled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical geometric anchors, while the diffusion generator leverages latent-augmented conditioning to refine and complete the geometric structure. Experimental results demonstrate that RecGen3D achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations.
Zhisheng Huang, Jiahao Chen, Cheng Lin +10
Texas A&M University, College Station, USA · Macau University of Science and Technology, Macau, Macau · HKUST, Hong Kong, Hong Kong +2
Generating pose-aligned 3D objects is challenging due to the spatial mismatches and transformation ambiguities inherent in decoupled canonical-then-rotate paradigms. To this end, we introduce Pose-Aware Diffusion (PAD), a novel end-to-end diffusion framework that synthesizes 3D geometry directly within the observation space. By unprojecting monocular depth into a partial point cloud and explicitly injecting it as a 3D geometric anchor, PAD abandons canonical assumptions to enforce rigorous spatial supervision. This native generation intrinsically resolves pose ambiguity, producing high-fidelity pose-aligned assets. Extensive experiments demonstrate that PAD achieves superior geometric alignment and image-to-3D correspondence compared to state-of-the-art methods. Additionally, PAD naturally extends to compositional 3D scene reconstruction via a simple union of independently generated objects, highlighting its robust ability to preserve precise spatial layouts.
Zihan Zhou, Luxi Chen, Jingzhi Zhou +4
Gaoling School of AI, Renmin University of China · VCIP, School of Computer Science, Nankai University · THU-Bosch MLCenter, Tsinghua University +1
Recent trends in sparse-view 3D reconstruction have taken two different paths: feed-forward reconstruction that predicts pixel-aligned point maps without a complete geometry, and generative 3D reconstruction that generates complete geometry but often with poor input-alignment. We present Mix3R, a novel generative 3D reconstruction method which mixes feed-forward reconstruction and 3D generation into a single framework in an aligned manner. Mix3R generates a 3D shape in two stages: a sparse voxel generation stage and a textured geometry generation stage. Unlike pure generative methods, our first-stage generation jointly produces a coarse 3D structure (sparse voxels), per-view point maps and camera parameters aligned to that 3D structure. This is made possible by introducing a Mixture-of-Transformers architecture that inserts global self-attentions to a feed-forward reconstruction model and a 3D generative model, both pretrained on large-scale data. This design effectively retains the pretrained priors but enables better 2D-3D alignment. Based on the initial aligned generations of sparse 3D voxels and point maps, we compute an overlap-based attention bias that is directly added to another pretrained textured geometry generation model, enabling it to correctly place input textures onto generated shapes in a training-free manner. Our design brings mutual benefits to both feed-forward reconstruction and 3D generation: The feed-forward branch learns to ground its predictions to a generative 3D prior, and conversely, the 3D generation branch is conditioned on geometrically informative features from the feed-forward branch. As a result, our method produces 3D shapes with better input alignment compared with pure 3D generative methods, together with camera pose estimations more accurate than previous feed-forward reconstruction methods. Our project page is at https://jsnln.github.io/mix3r/
Siyou Lin, Zhou Xue, Hongwen Zhang +4
Tsinghua University · Beijing Normal University · ByteDance