Generative Structure from Motion with Native 3D Diffusion

Haobin Duan; Binbin Huang; Mochu Xiang; Yiqun Zhao; Zibo Zhao; Yi Ma; Shenghua Gao

cs.CVOct 26, 2026

Generative Structure from Motion with Native 3D Diffusion

Authors: Haobin Duan, Binbin Huang, Mochu Xiang, Yiqun Zhao, Zibo Zhao, Yi Ma, Shenghua Gao

Organizations: The University of Hong Kong · Transengram · Tencent

Abstract

Recovering an object's 3D structure and camera motion from a few uncalibrated views is a fundamental problem in computer vision. Classical structure from motion methods recover 3D geometry consistent with the input images but cannot recover unseen regions; native generation methods produce plausible 3D objects which are related but not necessarily faithful to the input images. Methods simply concatenating reconstruction and generation allow error to cascade across the two stages, instead of producing a 3D object consistent with both the distribution and the input. We propose GenSfM, which integrates image-based 3D reconstruction with a native 3D diffusion model. In particular, we use a multi-modal diffusion transformer to jointly denoise the shape latent and per-view pose latents under shared self-attention, grounding generated geometry in the observed views while using object-level priors to infer unobserved regions. A compact 3D-native UV-volume tokenization keeps the joint sequence small as views are added, and a pose-aware stage refines geometry and texture by sampling per-voxel image features at the recovered camera poses. On Toys4k and GSO, for $4$ uncalibrated views, GenSfM improves novel-view PSNR by $3.2$ - $4.5$ dB and reduces median Chamfer distance by $2.8$ - $3.8\times$ over a strong native-3D generation baseline, while recovering input-view poses over $96\%$ Acc@ $5^\circ$ without any external pose-estimation backbone, post-hoc optimization, or known intrinsics. Project page: https://dotb1ue.github.io/GenSfm/.

Figures & tables

Figure 1: Generative object shape and camera motion recovery. Given multiview input images, GenSfM recovers the object shape and camera poses in a unified 3D space. A diffusion transformer performs multi-view reasoning through iterative structure-and-pose denoising (the trajectory of the camera), emulating an end-to-end SfM process. The denoised cameras ( blue ) align accurately with the ground-truth poses ( red ) and the recovered 3D structure gives a holistic object representation.

Method family	Where:	What: object structure		Unified joint	Native
	camera motion	observed	hidden	inference	3D prior
Classical SfM/MVS Schönberger and Frahm (2016) ; Schönberger et al. (2016) ; Furukawa and Ponce (2009)	✓	✓	–	$\triangle$	–
Pose-centric methods Zhang et al. (2022) ; Wang et al. (2023) ; Sinha et al. (2023) ; Zhang et al. (2024a)	✓	–	–	–	–
Feed-forward reconstruction Wang et al. (2024) ; Leroy et al. (2024) ; Wang et al. (2025)	✓	✓	$\triangle$	$\triangle$	–
Native 3D generation Tochilkin et al. (2024) ; Xiang et al. (2025) ; Zhao et al. (2025b)	–	$\triangle$	✓	–	✓
View-based generative SfM Zhao et al. (2025a) ; Zhang et al. (2024b) ; Xu et al. (2024a)	✓	$\triangle$	$\triangle$	$\triangle$	–

Table 1: Comparison of paradigms for sparse-view object structure-and-motion recovery. “Where” denotes camera-motion recovery and “what” denotes object-structure recovery, with observed and hidden geometry referring to visible and unobserved object regions, respectively. “Unified joint inference” means jointly recovering complete object structure and camera motion in one model, while “native 3D prior” denotes a generative prior over complete 3D shapes. ✓: explicit support;

\triangle

: partial or indirect support; –: not a primary objective.

Figure 2: Overview of GenSfM . We cast pose-free multi-view 3D recovery as a single coarse-to-fine generative problem. The coarse stage (Sec. 4.1 ) jointly denoises the sparse-structure latent and per-view 3D-native pose tokens with a multi-modal shape-pose DiT, recovering shape and multi-view camera poses together. The fine stage (Sec. 4.2 ) refines geometry and appearance with per-voxel pose-sampled image conditioning, producing the final structured-latent set.

Method	Toys4k						GSO
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD-med $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD-med $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
FreeSplatter Xu et al. (2024b)	22.00	0.9183	0.0913	2.7160	0.3163	0.7112	20.07	0.8948	0.1135	3.2859	0.2713	0.6736
LucidFusion He et al. (2025)	22.24	0.9251	0.0827	4.1596	0.1527	0.6670	20.94	0.9097	0.0943	3.8702	0.1755	0.7164
ReconViaGen Chang et al. (2025)	21.88	0.9216	0.0721	1.1237	0.3369	0.8434	20.88	0.9021	0.0856	1.0495	0.3524	0.8789
GenSfM	28.61	0.9609	0.0308	0.2184	0.6701	0.9823	26.36	0.9411	0.0472	0.2801	0.6095	0.9842

Table 2: Comparison with feed-forward reconstruction methods. We align predictions to the ground truth via a similarity transform fit by least squares between predicted and ground-truth input-view pointmaps, then evaluate photometric quality on novel views and 3D geometric accuracy in the ground-truth coordinate frame. CD is bidirectional symmetric (squared L2,

\times 10^{-3}

), reported as the median across objects.

Panel of Figure 3 — Figure 3: Qualitative comparison with feed-forward reconstruction methods on Toys4k (top two rows) and GSO (bottom two). All non-input columns render the same held-out novel view in the GT camera frame after the input-view pointmap LSQ alignment of Table 2 .

Method	Toys4k						GSO
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD-med $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	CD-med $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
SAM 3D Team et al. (2025)	20.49	0.9155	0.0926	1.7935	0.3292	0.8250	19.49	0.8995	0.1112	1.3716	0.3387	0.8505
TRELLIS -S Xiang et al. (2025)	21.78	0.9205	0.0874	1.2032	0.4064	0.8881	20.00	0.8943	0.1127	1.2418	0.3663	0.8695
TRELLIS -M Xiang et al. (2025)	24.01	0.9355	0.0658	0.4329	0.5570	0.9604	22.28	0.9138	0.0867	0.5200	0.5079	0.9655
ReconViaGen Chang et al. (2025)	23.26	0.9316	0.0587	0.6955	0.4374	0.9555	21.98	0.9108	0.0778	0.6500	0.4428	0.9434
GenSfM	28.54	0.9599	0.0315	0.1133	0.8279	0.9970	25.50	0.9349	0.0533	0.1882	0.7283	0.9914

Table 3: Comparison with 3D generation methods. For methods that produce a canonical 3D output without input camera poses, we align predictions to the ground truth via similarity ICP on

100

K points sampled from the rendered surface of the output mesh of each side, then evaluate photometric quality on novel views and 3D geometric accuracy in the ground-truth coordinate frame.

Panel of Figure 4 — Figure 4: Qualitative comparison with 3D generation methods on Toys4k (top row) and GSO (bottom three). All non-input columns render the same held-out novel view in the GT camera frame after the similarity ICP alignment of Tab. 3 .

Method	Toys4k					GSO
Method	RRE $\downarrow$	Acc.@5° $\uparrow$	Acc.@15° $\uparrow$	RTE $\downarrow$	RFOV $\downarrow$	RRE $\downarrow$	Acc.@5° $\uparrow$	Acc.@15° $\uparrow$	RTE $\downarrow$	RFOV $\downarrow$
FreeSplatter Xu et al. (2024b)	26.45	0.2438	0.4996	32.32	12.51	29.10	0.2327	0.4646	32.81	13.13
ReconViaGen Chang et al. (2025)	16.54	0.2044	0.6881	19.56	9.54	14.48	0.2260	0.7343	16.16	8.95
VGGT Wang et al. (2025)	10.29	0.5969	0.8406	18.61	37.66	9.33	0.6310	0.8702	17.69	36.08
GenSfM (predicted intrinsics)	1.60	0.9723	0.9913	5.51	2.90	1.27	0.9821	0.9985	5.08	2.86
GenSfM (known intrinsics)	1.44	0.9798	0.9910	2.38	–	1.13	0.9892	0.9981	2.11	–

Table 4: Camera-pose prediction accuracy. We report relative rotation error (RRE) with Acc.@

\{5^{\circ},15^{\circ}\}

thresholds, relative translation error (RTE; the angular error between predicted and GT relative-translation directions), and mean absolute FOV error (RFOV; intrinsic-only); see supplementary for the full evaluation protocol. GenSfM reports two rows: known intrinsics (GT FOV fed in; RFOV not applicable, marked –) and predicted intrinsics. Angular pose metrics use raw, gauge-invariant pairwise relative poses and the same ambiguity-tolerant score for all methods.

Stage	Variant	PSNR $\uparrow$	LPIPS $\downarrow$	CD-med $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$	RRE $\downarrow$	Acc.@5° $\uparrow$
Stage 1	Base (per-view cross-attention)	27.00	0.0383	0.2811	0.6078	0.9677	5.62	0.8653
	+ full cross-attention	27.49	0.0359	0.2568	0.6324	0.9748	5.20	0.9069
	+ visual conditioning	27.65	0.0353	0.2488	0.6346	0.9783	4.59	0.9228
	+ register tokens	28.23	0.0327	0.2204	0.6670	0.9824	3.55	0.9478
Stage 2	+ native multi-view training ( Full model )	28.61	0.0308	0.2184	0.6701	0.9823	3.65	0.9416

Table 5: Ablation study on Toys4k. Rows add components cumulatively, ending with the full model. Stage 1 uses a single-view-trained fine decoder extended to multi-view at inference; Stage 2 replaces it with a natively multi-view-trained decoder. All rows use the input-view pointmap alignment from Table 2 ; RRE and Acc.@

5^{\circ}

use a similarity fit from the final pointmap, so the fine decoder can shift them without changing raw coarse cameras. CD is the median across objects.

Panel of Figure 5 — Figure 5: Qualitative ablation on Toys4k. For each instance we show the four held-out test views as a $2{\times}2$ grid (top), with two zoomed-in regions of largest cross-variant disagreement below (red / green borders). Columns left-to-right correspond to the cumulative ablation rows of Table 5 : Base, $+$ cross, $+$ visual, $+$ reg, the full model, and ground truth. Each added component sharpens local geometry and texture, with the full model best matching GT.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Latent	UV Dist. $\downarrow$ ( $\times 10^{-4}$ )	RRE $\downarrow$	RTE $\downarrow$	RFOV $\downarrow$
$4^{3}$	4.61 / 5.48 / 10.12	0.217 / 0.388 / 0.987	0.209 / 0.380 / 0.986	0.077 / 0.136 / 0.264
$8^{3}$	7.06 / 7.85 / 12.41	0.202 / 0.373 / 0.919	0.200 / 0.366 / 0.924	0.063 / 0.121 / 0.234
$16^{3}$	4.58 / 4.88 / 6.78	0.241 / 0.402 / 1.031	0.235 / 0.393 / 1.028	0.090 / 0.147 / 0.282

Appendix

Table 6: Pose VAE autoencoding accuracy on Toys4k across latent spatial resolutions. We sweep the latent grid in

\{4^{3},8^{3},16^{3}\}

at a fixed

8

channels per token (yielding

\{64,\,512,\,4096\}

latent tokens respectively), encode each held-out

64^{3}

ground-truth UV volume, decode, and recover camera pose via the DLT solve. Each cell reports median / mean /

95

% quantile over

\sim\!77

K (image, pose) pairs. UV distance is in normalized image coordinates (

\times 10^{-4}

); pose errors are in degrees.

Structure from Motion

Structure from Motion estimates camera poses and 3D scene geometry from overlapping images by matching visual features across views.

See all 46 papers

GenIA: Generative Reconstruction with Test-Time Input Alignment
Oct 8, 2026Stefano Esposito, Naama Pearl, Polina Karpikova +53D Reconstruction Object-Centric 3D Reconstruction
Tübingen AI Center, University of Tübingen · Meta Reality Labs
UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space
Oct 11, 2026Wenjue Chen, Wenyue Chen, Heliang Zheng +33D Asset Generation 3D Shape Generation
Peking University · Math Magic · The Hong Kong University of Science and Technology
MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generation
Mar 12, 2026Baicheng Li, Dong Wu, Jun Li +43D Shape Generation 3D Scene Generation
Peking University · JD Explore Academy

Oct 8, 2026J/K move · Enter open · S save

GenIA: Generative Reconstruction with Test-Time Input Alignment

Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò+4

Tübingen AI Center, University of Tübingen · Meta Reality Labs

3D Reconstruction Object-Centric 3D Reconstruction Sparse-View 3D Reconstruction Single-View 3D Reconstruction 3D Diffusion Models Test-Time Adaptation+1

Open paper AskPDF arXiv

Reconstructing complete 3D object assets from monocular or sparse multi-view observations remains challenging. Generative 3D foundation models can complete object geometry beyond the observed views, but their predictions may not faithfully reproduce the observed geometry, appearance, or pose. We introduce GenIA, a framework for test-time input-aligned generation that grounds SAM3D's generative prior in geometric and photometric observations without retraining the foundation model. We improve object pose by deriving translation and scale from geometry while retaining the learned rotation prior, and align appearance through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance during denoising. An optional post-denoising refinement further adapts the appearance latent, lightweight decoder adapters, and object placement to the observations. Our framework also supports externally supplied geometry; when given temporal shapes of dynamic objects, it recovers a shared, input-aligned canonical appearance and stable world-space placement. Across synthetic and real benchmarks, GenIA improves pose prediction and object reconstruction from monocular, multi-view, and dynamic inputs, outperforming recent optimization-based, per-frame image-to-3D, and video-to-4D methods. Our project page is available at https://facebookresearch.github.io/GenIA.

Abstract

Structure from Motion

Explore similar work

GenIA: Generative Reconstruction with Test-Time Input Alignment

UniForge3D: Joint 3D Reconstruction and Generation in a Shared Anchor-Guided Space

MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generation