RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space

Zhisheng Huang; Jiahao Chen; Cheng Lin; Chenyu Hu; Hanzhuo Huang; Zhengming Yu; Mengfei Li; Yuheng Liu

cs.CVApr 1, 2026

RecGen3D: Reconstruction-Guided 3D Generation in a Shared Canonical Space

Authors: Zhisheng Huang, Jiahao Chen, Cheng Lin, Chenyu Hu, Hanzhuo Huang, Zhengming Yu, Mengfei Li, Yuheng Liu, +5 more

Organizations: Texas A&M University, College Station, USA · Macau University of Science and Technology, Macau, Macau · HKUST, Hong Kong, Hong Kong · ShanghaiTech University, Shanghai, China · University of California Irvine, Irvine, USA

Abstract

Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input alignment, it often lacks the global priors needed for structural completeness. Conversely, diffusion-based generation provides rich geometric details but struggles with multi-view consistency. We present RecGen3D, a framework that combines these two paradigms into a cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ decoupled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical geometric anchors, while the diffusion generator leverages latent-augmented conditioning to refine and complete the geometric structure. Experimental results demonstrate that RecGen3D achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations.

Figures & tables

Figure 1. RecGen3D enables high-fidelity 3D object reconstruction from sparse, unposed images. It first establishes a deterministic point-cloud “geometric anchor” within a canonical coordinate system. Guided by this anchor, a generative model synthesizes a detailed mesh that preserves instance-specific structural features while producing plausible geometric completions for unobserved regions, ensuring high alignment with the ground-truth shape.

Figure 2. Overview of our method. Given $N$ unposed input views, we first canonicalize feed-forward multi-view geometry predictions via branch repurposing and similarity alignment to obtain a canonical point cloud (top). We then train a controllable 3D generator conditioned on this point cloud together with multi-view image features, geometry latents, and camera embeddings to synthesize a high-fidelity mesh (bottom).

Dataset	Method	Chamfer- $L_{2}\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	F-Score $\uparrow$	Normal $\uparrow$	IoU $\uparrow$
Toys4K	LucidFusion ( He et al., 2024b )	0.1333	0.2143	0.1378	0.1471	0.6240	0.1035
	Hunyuan3D-MV ( Zhao et al., 2025a )	0.0513	0.4930	0.4305	0.4540	0.8124	0.6311
	TRELLIS-M ( Xiang et al., 2025 )	0.0343	0.5952	0.5600	0.5674	0.8309	0.6318
	TRELLIS-S ( Xiang et al., 2025 )	0.0309	0.5762	0.5430	0.5525	0.8324	0.6583
	SAM 3D ( Chen et al., 2025a )	0.0354	0.5416	0.6575	0.5711	0.8218	0.6061
	ReconViaGen ( Chang et al., 2026 )	0.0281	0.6224	0.6086	0.6105	0.8543	0.7229

Table 1. Quantitative Results for 3D Object Reconstruction. We report geometric metrics across the Toys4K and GSO datasets. Our framework consistently outperforms existing generative and reconstruction-based methods across all metrics.

Method	GSO			Toys4k
Method	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$
VGGT ( Wang et al., 2025a )	0.0799	0.1856	8.18	0.1209	0.2931	19.14
ReconViaGen ( Chang et al., 2026 )	0.0190	0.0443	2.12	0.0425	0.1099	8.53
Ours	0.0151	0.0338	2.16	0.0255	0.0637	3.67

Table 2. Quantitative Comparison of Camera Pose Estimation. We evaluate Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) for translation (

t

) and rotation (

r

) on the GSO and Toys4k benchmarks.

Method	GSO		Toys4k
Method	Abs Rel $\downarrow$	RMSE $\downarrow$	Abs Rel $\downarrow$	RMSE $\downarrow$
VGGT ( Wang et al., 2025a )	0.0512	0.0686	0.0522	0.0735
ReconViaGen ( Chang et al., 2026 )	0.0051	0.0081	0.0065	0.0109
Ours	0.0033	0.0057	0.0039	0.0075

Table 3. Quantitative Comparison of Depth Estimation. Our method demonstrates superior accuracy in depth prediction compared to other baselines on both GSO and Toys4k datasets.

Figure 3. Qualitative Comparison of Canonical Alignment. Our strategy achieves better geometric quality while aligning the canonical object space.

Strategy	Camera Pose			Depth
Strategy	ATE $\downarrow$	RPE t $\downarrow$	RPE r $\downarrow$	Abs Rel $\downarrow$	RMSE $\downarrow$
Direct Supervision Transferring	0.2517	0.1414	4.2376	0.0062	0.0210
Explicit Rigid Transformation	0.5934	0.1474	4.4596	0.0049	0.0191
Branch Repurposing (Ours)	0.0353	0.0453	1.2181	0.0054	0.0200

Table 4. Quantitative Comparison of Canonical Alignment. Compared with other strategies, our strategy is more accurate in camera pose prediction while remaining comparable in depth estimation.

Figure 4. Qualitative Comparison of Multi-view Condition. Our strategy (right) preserves dense image context more effectively than point-guided sampling (left), leading to improved input alignment.

Strategy	Chamfer- $L_{2}\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	F-Score $\uparrow$	Normal $\uparrow$	IoU $\uparrow$
Point-Guided Sampling	0.0260	0.7677	0.7087	0.7208	0.8419	0.6304
Latent-Augmented (Ours)	0.0259	0.7890	0.7325	0.7428	0.8350	0.6707

Table 5. Quantitative Comparison of Multi-view Condition. Our strategy achieves the best performance across most geometric metrics.

Figure 5. Failure cases of our full method. Top to bottom: mutual occlusion loses fine facial detail; thin, perforated structures are partially recovered; reflective objects are less robust under real-world illumination.

Figure 6. Qualitative Comparison on Toys4K and GSO. Compared to state-of-the-art reconstruction and generative baselines, our method produces 3D meshes with higher structural fidelity and superior multi-view consistency from sparse inputs.

Figure 7. Generalization to Real-world Environments. Our framework demonstrates robustness and superior performance compared to SOTA method.

Figure 8. Additional qualitative results. We show six multi-view shape modeling examples. For each of four input views, from left to right we show: the input image, our mesh rendering overlaid on the input image, our mesh rendering, and the ground-truth mesh rendering.

Method	Branch	VGGT	CD $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	F-Score $\uparrow$
	Repurposing	Features
w/o Branch Repurposing	✗	✓	0.0339	0.7229	0.6407	0.6535
w/o VGGT Features	✓	✗	0.0301	0.7189	0.6711	0.6761
Full Method (Ours)	✓	✓	0.0259	0.7890	0.7325	0.7428

Table S1. Component-wise ablation study. We isolate the contribution of Branch Repurposing (canonical alignment) and VGGT Feature Injection (multi-view geometric conditioning). ✓ indicates the component is enabled. Removing either component leads to consistent degradation across all geometric metrics, confirming that both designs are necessary and complementary.

Figure S1. Qualitative comparison of VGGT feature injection. From left to right: input images, results without VGGT features, our full method, and ground truth. Without VGGT geometric tokens, the generator relies solely on DINO appearance features and fails to disambiguate geometrically distinct viewpoints. This manifests as front-back inversion in articulated objects (rows 1 and 3) and geometric aliasing across structurally asymmetric surfaces (row 2, teapot). Our full method injects VGGT latent tokens as spatial anchors, resolving viewpoint ambiguity and preserving instance-specific structural fidelity.

$2$ input views
Method	CD $\downarrow$	Prec. $\uparrow$	Rec. $\uparrow$	F-Score $\uparrow$	Normal $\uparrow$	IoU $\uparrow$
ReconViaGen	0.0297	0.5311	0.5247	0.5222	0.8733	0.6992
Ours	0.0221	0.6491	0.6850	0.6622	0.8824	0.7805
$8$ input views
ReconViaGen	0.0234	0.6654	0.6567	0.6544	0.8881	0.7462
Ours	0.0177	0.7569	0.7964	0.7711	0.9064	0.8263

Table S2. Generalization to varying numbers of input views on GSO. Both methods are trained with

4

input views. Our framework maintains a consistent advantage over ReconViaGen across both the sparse (

2

-view) and denser (

8

-view) regimes.

Figure S2. Qualitative comparison with ReconViaGen under $2$ input views on GSO. Each group shows ReconViaGen, ours, and ground truth rendered from two viewpoints. Under this challenging sparse-view setting, our method exhibits stronger generative completion: global shape structure is better preserved (row 1, gear puzzle; row 2, fingerless gloves) and fine geometric details remain coherent even when large portions of the object are unobserved (row 3, character with umbrella; row 4, airplane set). ReconViaGen tends to produce over-smoothed or topologically incorrect surfaces under such limited input.

Figure S3. Qualitative comparison with ReconViaGen under $8$ input views on GSO. Each group shows ReconViaGen, ours, and ground truth rendered from two viewpoints. With denser input coverage, our method consistently produces more complete and geometrically faithful meshes, recovering fine-grained structures such as thin appendages (row 1, insect figurines), articulated components (row 2, bunny car), and complex part assemblies (row 4, robot toy) that ReconViaGen fails to reconstruct accurately.

Figure S4. Additional in-the-wild qualitative results. Each group shows input images (left) and our reconstructed mesh (right). Input view counts range from $2$ to $16$ unposed images captured under uncontrolled real-world conditions. Despite being trained on fixed $4$ -view synthetic data, RecGen3D generalizes robustly to varying view counts, uncontrolled camera poses, and diverse object categories, including architectural models, vehicles, organic objects, and articulated figurines.

Method	CD $\downarrow$	F-Score $\uparrow$
ReconViaGen	0.037	0.80
RecGen3D (Ours)	0.029	0.84

Table S3. Quantitative real-world evaluation on CO3D. Real casual captures with cluttered backgrounds (

20

Explore similar work

UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space
Oct 11, 2026Wenjue Chen, Wenyue Chen, Heliang Zheng +33D Asset Generation 3D Shape Generation
Peking University · Math Magic · The Hong Kong University of Science and Technology
Sparse auto-regressive modeling for scene generation from multi-view images
Sep 3, 2026Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4Sparse-View 3D Reconstruction 3D Scene Generation
1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA
AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model
Apr 21, 2026Yutian Chen, Shi Guo, Renbiao Jin +7Video Diffusion Models 3D Reconstruction
Shanghai AI Lab · Shanghai Jiao Tong University · The University of Hong Kong +2

Oct 11, 2026J/K move · Enter open · S save

UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space

Wenjue Chen, Wenyue Chen, Heliang Zheng, Rongfei Jia+2

Peking University · Math Magic · The Hong Kong University of Science and Technology

3D Asset Generation 3D Shape Generation Sparse-View 3D Reconstruction

Open paper AskPDF arXiv

Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity. Project webpage: https://xishuxishu.github.io/uniforge3d-project/

Configuration	F-Score $\uparrow$	CD $\downarrow$
(a) Base generator + off-the-shelf VGGT	0.155	0.153
(b) Base generator + canonicalized VGGT (ours)	0.567	0.038
(c) Full method (ours)	0.738	0.019

Conditioning scheme	F-Score $\uparrow$	CD $\downarrow$	IoU $\uparrow$	Normal $\uparrow$
Implicit feature injection (ReconViaGen-style)	0.399	0.0506	0.548	0.723
Explicit canonical geometry (ours)	0.771	0.0193	0.820	0.875

Training mode	F-Score $\uparrow$	CD $\downarrow$	IoU $\uparrow$
(a) Decoupled modular (ours)	0.70	0.0202	0.773
(b) Joint, same pretrained init	0.59	0.0229	0.737
(c) Joint, from scratch	0.54	0.0257	0.706

Strategy	CD $\downarrow$	F-Score $\uparrow$	IoU $\uparrow$
Point-Guided Sampling	0.0207 $\pm$ 0.0017	0.692 $\pm$ 0.015	0.742 $\pm$ 0.015
Latent-Augmented (ours)	0.0190 $\pm$ 0.0015	0.720 $\pm$ 0.010	0.760 $\pm$ 0.016

Backbone	F-Score $\uparrow$	CD $\downarrow$	IoU $\uparrow$	Normal $\uparrow$
$\pi^{3}$	0.646	0.023	0.741	0.847
VGGT	0.762	0.020	0.816	0.873

Abstract

Explore similar work

UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space

Sparse auto-regressive modeling for scene generation from multi-view images

AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model