Organizations: Texas A&M University, College Station, USA · Macau University of Science and Technology, Macau, Macau · HKUST, Hong Kong, Hong Kong · ShanghaiTech University, Shanghai, China · University of California Irvine, Irvine, USA
Sparse-view 3D modeling represents a fundamental tension between reconstruction fidelity and generative plausibility. While feed-forward reconstruction excels in efficiency and input alignment, it often lacks the global priors needed for structural completeness. Conversely, diffusion-based generation provides rich geometric details but struggles with multi-view consistency. We present RecGen3D, a framework that combines these two paradigms into a cooperative system. To overcome inherent conflicts in coordinate spaces, 3D representations, and training objectives, we align both models within a shared canonical space. We employ decoupled cooperative learning, which maintains stable training while enabling seamless collaboration during inference. Specifically, the reconstruction module is adapted to provide canonical geometric anchors, while the diffusion generator leverages latent-augmented conditioning to refine and complete the geometric structure. Experimental results demonstrate that RecGen3D achieves superior fidelity and robustness, outperforming existing methods in creating complete and consistent 3D models from sparse observations.
Figures & tables
Figure 1. RecGen3D enables high-fidelity 3D object reconstruction from sparse, unposed images. It first establishes a deterministic point-cloud “geometric anchor” within a canonical coordinate system. Guided by this anchor, a generative model synthesizes a detailed mesh that preserves instance-specific structural features while producing plausible geometric completions for unobserved regions, ensuring high alignment with the ground-truth shape.
Figure 2. Overview of our method. Given N unposed input views, we first canonicalize feed-forward multi-view geometry predictions via branch repurposing and similarity alignment to obtain a canonical point cloud (top). We then train a controllable 3D generator conditioned on this point cloud together with multi-view image features, geometry latents, and camera embeddings to synthesize a high-fidelity mesh (bottom).
Dataset
Method
Chamfer- L2↓
Prec. ↑
Rec. ↑
F-Score ↑
Normal ↑
IoU ↑
Toys4K
LucidFusion ( He et al., 2024b )
0.1333
0.2143
0.1378
0.1471
0.6240
0.1035
Hunyuan3D-MV ( Zhao et al., 2025a )
0.0513
0.4930
0.4305
0.4540
0.8124
0.6311
TRELLIS-M ( Xiang et al., 2025 )
0.0343
0.5952
0.5600
0.5674
0.8309
0.6318
TRELLIS-S ( Xiang et al., 2025 )
0.0309
0.5762
0.5430
0.5525
0.8324
0.6583
SAM 3D ( Chen et al., 2025a )
0.0354
0.5416
0.6575
0.5711
0.8218
0.6061
ReconViaGen ( Chang et al., 2026 )
0.0281
0.6224
0.6086
0.6105
0.8543
0.7229
Table 1. Quantitative Results for 3D Object Reconstruction. We report geometric metrics across the Toys4K and GSO datasets. Our framework consistently outperforms existing generative and reconstruction-based methods across all metrics.
Method
GSO
Toys4k
ATE ↓
RPE t ↓
RPE r ↓
ATE ↓
RPE t ↓
RPE r ↓
VGGT ( Wang et al., 2025a )
0.0799
0.1856
8.18
0.1209
0.2931
19.14
ReconViaGen ( Chang et al., 2026 )
0.0190
0.0443
2.12
0.0425
0.1099
8.53
Ours
0.0151
0.0338
2.16
0.0255
0.0637
3.67
Table 2. Quantitative Comparison of Camera Pose Estimation. We evaluate Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) for translation ( t ) and rotation ( r ) on the GSO and Toys4k benchmarks.
Method
GSO
Toys4k
Abs Rel ↓
RMSE ↓
Abs Rel ↓
RMSE ↓
VGGT ( Wang et al., 2025a )
0.0512
0.0686
0.0522
0.0735
ReconViaGen ( Chang et al., 2026 )
0.0051
0.0081
0.0065
0.0109
Ours
0.0033
0.0057
0.0039
0.0075
Table 3. Quantitative Comparison of Depth Estimation. Our method demonstrates superior accuracy in depth prediction compared to other baselines on both GSO and Toys4k datasets.
Figure 3. Qualitative Comparison of Canonical Alignment. Our strategy achieves better geometric quality while aligning the canonical object space.
Strategy
Camera Pose
Depth
ATE ↓
RPE t ↓
RPE r ↓
Abs Rel ↓
RMSE ↓
Direct Supervision Transferring
0.2517
0.1414
4.2376
0.0062
0.0210
Explicit Rigid Transformation
0.5934
0.1474
4.4596
0.0049
0.0191
Branch Repurposing (Ours)
0.0353
0.0453
1.2181
0.0054
0.0200
Table 4. Quantitative Comparison of Canonical Alignment. Compared with other strategies, our strategy is more accurate in camera pose prediction while remaining comparable in depth estimation.
Figure 4. Qualitative Comparison of Multi-view Condition. Our strategy (right) preserves dense image context more effectively than point-guided sampling (left), leading to improved input alignment.
Strategy
Chamfer- L2↓
Prec. ↑
Rec. ↑
F-Score ↑
Normal ↑
IoU ↑
Point-Guided Sampling
0.0260
0.7677
0.7087
0.7208
0.8419
0.6304
Latent-Augmented (Ours)
0.0259
0.7890
0.7325
0.7428
0.8350
0.6707
Table 5. Quantitative Comparison of Multi-view Condition. Our strategy achieves the best performance across most geometric metrics.
Figure 5. Failure cases of our full method. Top to bottom: mutual occlusion loses fine facial detail; thin, perforated structures are partially recovered; reflective objects are less robust under real-world illumination.
Figure 6. Qualitative Comparison on Toys4K and GSO. Compared to state-of-the-art reconstruction and generative baselines, our method produces 3D meshes with higher structural fidelity and superior multi-view consistency from sparse inputs.
Figure 7. Generalization to Real-world Environments. Our framework demonstrates robustness and superior performance compared to SOTA method.
Figure 8. Additional qualitative results. We show six multi-view shape modeling examples. For each of four input views, from left to right we show: the input image, our mesh rendering overlaid on the input image, our mesh rendering, and the ground-truth mesh rendering.
Method
Branch
VGGT
CD ↓
Prec. ↑
Rec. ↑
F-Score ↑
Repurposing
Features
w/o Branch Repurposing
✗
✓
0.0339
0.7229
0.6407
0.6535
w/o VGGT Features
✓
✗
0.0301
0.7189
0.6711
0.6761
Full Method (Ours)
✓
✓
0.0259
0.7890
0.7325
0.7428
Table S1. Component-wise ablation study. We isolate the contribution of Branch Repurposing (canonical alignment) and VGGT Feature Injection (multi-view geometric conditioning). ✓ indicates the component is enabled. Removing either component leads to consistent degradation across all geometric metrics, confirming that both designs are necessary and complementary.
Figure S1. Qualitative comparison of VGGT feature injection. From left to right: input images, results without VGGT features, our full method, and ground truth. Without VGGT geometric tokens, the generator relies solely on DINO appearance features and fails to disambiguate geometrically distinct viewpoints. This manifests as front-back inversion in articulated objects (rows 1 and 3) and geometric aliasing across structurally asymmetric surfaces (row 2, teapot). Our full method injects VGGT latent tokens as spatial anchors, resolving viewpoint ambiguity and preserving instance-specific structural fidelity.
Method
CD ↓
Prec. ↑
Rec. ↑
F-Score ↑
Normal ↑
IoU ↑
2 input views
ReconViaGen
0.0297
0.5311
0.5247
0.5222
0.8733
0.6992
Ours
0.0221
0.6491
0.6850
0.6622
0.8824
0.7805
8 input views
ReconViaGen
0.0234
0.6654
0.6567
0.6544
0.8881
0.7462
Ours
0.0177
0.7569
0.7964
0.7711
0.9064
0.8263
Table S2. Generalization to varying numbers of input views on GSO. Both methods are trained with 4 input views. Our framework maintains a consistent advantage over ReconViaGen across both the sparse ( 2 -view) and denser ( 8 -view) regimes.
Figure S2. Qualitative comparison with ReconViaGen under 2 input views on GSO. Each group shows ReconViaGen, ours, and ground truth rendered from two viewpoints. Under this challenging sparse-view setting, our method exhibits stronger generative completion: global shape structure is better preserved (row 1, gear puzzle; row 2, fingerless gloves) and fine geometric details remain coherent even when large portions of the object are unobserved (row 3, character with umbrella; row 4, airplane set). ReconViaGen tends to produce over-smoothed or topologically incorrect surfaces under such limited input.
Figure S3. Qualitative comparison with ReconViaGen under 8 input views on GSO. Each group shows ReconViaGen, ours, and ground truth rendered from two viewpoints. With denser input coverage, our method consistently produces more complete and geometrically faithful meshes, recovering fine-grained structures such as thin appendages (row 1, insect figurines), articulated components (row 2, bunny car), and complex part assemblies (row 4, robot toy) that ReconViaGen fails to reconstruct accurately.
Figure S4. Additional in-the-wild qualitative results. Each group shows input images (left) and our reconstructed mesh (right). Input view counts range from 2 to 16 unposed images captured under uncontrolled real-world conditions. Despite being trained on fixed 4 -view synthetic data, RecGen3D generalizes robustly to varying view counts, uncontrolled camera poses, and diverse object categories, including architectural models, vehicles, organic objects, and articulated figurines.
Method
CD ↓
F-Score ↑
ReconViaGen
0.037
0.80
RecGen3D (Ours)
0.029
0.84
Table S3. Quantitative real-world evaluation on CO3D. Real casual captures with cluttered backgrounds ( 20 categories ×2 sequences, 4 unposed views each). Predictions of both methods are aligned to the SfM ground-truth point clouds with an identical Sim(3) + ICP procedure.
Configuration
F-Score ↑
CD ↓
(a) Base generator + off-the-shelf VGGT
0.155
0.153
(b) Base generator + canonicalized VGGT (ours)
0.567
0.038
(c) Full method (ours)
0.738
0.019
Table S4. Source-of-gain analysis on GSO. The pretrained Hunyuan3D-Omni (official weights) is conditioned through its native point-cloud control; only the reconstruction source varies between (a) and (b). Configuration (c) is our full method with generator fine-tuning. (a) → (b) isolates the reconstruction-side design; (b) → (c) isolates the generator-side design.
Conditioning scheme
F-Score ↑
CD ↓
IoU ↑
Normal ↑
Implicit feature injection (ReconViaGen-style)
0.399
0.0506
0.548
0.723
Explicit canonical geometry (ours)
0.771
0.0193
0.820
0.875
Table S5. Backbone-matched comparison. ReconViaGen’s implicit feature-injection is re-implemented on the same Hunyuan3D-Omni backbone, with identical encoder, image tokens, data, training budget, and evaluation. Both variants are trained for the identical full schedule.
Training mode
F-Score ↑
CD ↓
IoU ↑
(a) Decoupled modular (ours)
0.70
0.0202
0.773
(b) Joint, same pretrained init
0.59
0.0229
0.737
(c) Joint, from scratch
0.54
0.0257
0.706
Table S6. Joint training vs. decoupled modular training. All runs share the same backbone, data, and training schedule, differing only in training mode. Coupling the two objectives degrades every metric, even from identical pretrained initialization.
Backbone
F-Score ↑
CD ↓
IoU ↑
Normal ↑
π3
0.646
0.023
0.741
0.847
VGGT
0.762
0.020
0.816
0.873
Table S7. Reconstruction backbone swap. We replace VGGT with π3 , keeping everything else identical; both variants are trained under the same training schedule.
Strategy
CD ↓
F-Score ↑
IoU ↑
Point-Guided Sampling
0.0207 ± 0.0017
0.692 ± 0.015
0.742 ± 0.015
Latent-Augmented (ours)
0.0190 ± 0.0015
0.720 ± 0.010
0.760 ± 0.016
Table S8. Multi-run conditioning ablation. Mean ± standard deviation over three inference runs with different random seeds, evaluated on an extended 100 -object set.
Figure S5. Qualitative comparison on CO3D. From four unposed real captures, RecGen3D recovers cleaner and more complete geometry than ReconViaGen.
Figure S6. Qualitative results with the π3 backbone. Replacing VGGT with π3 yields accurate and complete meshes.
Generating complete 3D assets from sparse, unposed images requires preserving observed geometry while completing unseen regions. Joint reconstruction and generation offers a way to address both goals: reconstruction recovers the geometry observed in the input images, while generation contributes learned shape priors. However, combining camera-relative reconstruction with canonical-space generation introduces a coordinate mismatch that makes direct geometric interaction more difficult. Moreover, conditioning each 3D location on all views without accounting for their relevance may mix useful geometric evidence with unrelated image content, hindering local detail recovery. We present UniForge3D, which jointly models reconstruction and generation in a shared anchor-guided space, resolving their coordinate mismatch. This space follows the anchor view's azimuth while keeping the object upright. The two branches exchange features bidirectionally to predict per-view point maps and a complete sparse voxel structure. We further condition point-map prediction on the generated structure to establish pixel-to-voxel correspondences and select the most informative input view for each voxel. Features from the selected views then guide detailed shape generation, helping preserve observed geometric details. Our method achieves state-of-the-art appearance and geometry results on Toys4K and GSO, generating complete 3D assets with improved fidelity. Project webpage: https://xishuxishu.github.io/uniforge3d-project/
Wenjue Chen, Wenyue Chen, Heliang Zheng +3
Peking University · Math Magic · The Hong Kong University of Science and Technology
Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.
Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel +4
1NAVER LABS Europe, France · 2NAVER LABS, South Korea · 3Carnegie Mellon University, USA
Sparse-view 3D reconstruction is essential for modeling scenes from casual captures, but remain challenging for non-generative reconstruction. Existing diffusion-based approaches mitigates this issues by synthesizing novel views, but they often condition on only one or two capture frames, which restricts geometric consistency and limits scalability to large or diverse scenes. We propose AnyRecon, a scalable framework for reconstruction from arbitrary and unordered sparse inputs that preserves explicit geometric control while supporting flexible conditioning cardinality. To support long-range conditioning, our method constructs a persistent global scene memory via a prepended capture view cache, and removes temporal compression to maintain frame-level correspondence under large viewpoint changes. Beyond better generative model, we also find that the interplay between generation and reconstruction is crucial for large-scale 3D scenes. Thus, we introduce a geometry-aware conditioning strategy that couples generation and reconstruction through an explicit 3D geometric memory and geometry-driven capture-view retrieval. To ensure efficiency, we combine 4-step diffusion distillation with context-window sparse attention to reduce quadratic complexity. Extensive experiments demonstrate robust and scalable reconstruction across irregular inputs, large viewpoint gaps, and long trajectories.
Yutian Chen, Shi Guo, Renbiao Jin +7
Shanghai AI Lab · Shanghai Jiao Tong University · The University of Hong Kong +2