Fus3D: Decoding Consolidated 3D Geometry from Feed-forward Geometry Transformer Latents

Laura Fink; Linus Franke; George Kopanas; Marc Stamminger; Peter Hedman

cs.CVMar 26, 2026

Fus3D: Decoding Consolidated 3D Geometry from Feed-forward Geometry Transformer Latents

Authors: Laura Fink, Linus Franke, George Kopanas, Marc Stamminger, Peter Hedman

Organizations: Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany · Inria, Université Côte d’Azur, France · Google DeepMind, UK

Abstract

We propose a feed-forward method for dense Signed Distance Field (SDF) regression from unstructured image collections in less than three seconds, without camera calibration or post-hoc fusion. Our key insight is that the intermediate feature space of pretrained multi-view feed-forward geometry transformers already encodes a powerful joint world representation; yet, existing pipelines discard it, routing features through per-view prediction heads before assembling 3D geometry post-hoc, which discards valuable completeness information and accumulates inaccuracies. We instead perform 3D extraction directly from geometry transformer features via learned volumetric extraction: voxelized canonical embeddings that progressively absorb multi-view geometry information through interleaved cross- and self-attention into a structured volumetric latent grid. A simple convolutional decoder then maps this grid to a dense SDF. We additionally propose a scalable, validity-aware supervision scheme directly using SDFs derived from depth maps or 3D assets, tackling practical issues like non-watertight meshes. Our approach yields complete and well-defined distance values across sparse- and dense-view settings and demonstrates geometrically plausible completions. Code and further material can be found at https://lorafib.github.io/fus3d.

Figures & tables

Figure 1: Fus3D directly regresses a 3D representation from the latent space of a multi-view geometry transformer, bypassing per-view prediction and post-hoc fusion. This, compared to VGGT [ 46 ] using TSDF fusion, yields improved surface completeness under sparse views (left, 2 views) and avoids error accumulation reducing details at scale (right, 24 views), also reflected in the F-score curves across view counts (center).

Figure 2: Juxtaposition of geometric extraction metholodogies. Common pipelines like VGGT [ 46 ] (top) route transformer features through per-view 2D decoder heads, discarding the joint multi-view representation before 3D assembly. We instead extract dense 3D features directly from the transformer’s intermediate feature space, preserving the full multi-view information.

Figure 3: Architecture of Fus3D: The geometry transformer

\mathcal{G}

(beige) processes tokenized input images, yielding a list of 2D intermediate features

\{z^{b}_{\textrm{2D}}\}^{B}_{b=1}

extracted from different stages. The extraction transformer

\mathcal{E}

(orange) leverages 2D-to-3D cross attention (green) to absorb the 3D information into features of the learned canonical embedding

z_{\textrm{3D}}

, and distributes this information via 3D self-attention (blue) throughout the volumetric latent. The head

\mathcal{H}_{3D}

decodes the resulting structured latent

\hat{z}_{\textrm{3D}}

into a dense SDF grid.

	favorable (23, 24, 33)			unfavorable (1, 16, 36)
	$\downarrow$ CD	D ${}_{\text{GT2P}}$	D ${}_{\text{P2GT}}$	CD	D ${}_{\text{GT2P}}$	D ${}_{\text{P2GT}}$
VolRecon $\dagger$	2.905	2.331	3.478	5.781	3.811	7.751
UFORecon $\dagger$	2.771	2.101	3.440	3.275	2.252	4.298
VolRecon	3.678	3.369	3.987	8.878	5.869	11.887
UFORecon	3.415	2.989	3.841	4.942	3.737	6.146
Fus3D	2.432	1.804	3.059	3.525	2.814	4.236

Table 1: Quantitative results with direct SDF prediction baselines in a feed-forward scenario. For reference,

\dagger

indicated non-feed forward (COLMAP) poses for competing methods.

Figure 4: Qualitative results on DTU . Beige boxes indicate input views. Numbers in brackets correspond to image indices of “favorable” (23, 24, 33) and “unfavorable” (1, 16, 36) view combinations. (Following the baselines’ evaluation protocols, only visible geometry is evaluated.)

Figure 5: Comparison vs ground-truth TSDF isosurfaces (green) on 170 Objaverse test scenes, w.r.t number of input views. Left: F-scores with threshold

0.5\epsilon

. Right: Chamfer distances.

	F ϵ	F 0.5ϵ	CD	EMD	SDF
					MAE
Fus3D	0.83	0.51	0.021	0.019	0.004
VGGT ft + TSDF	0.80	0.43	0.022	0.014	0.023
VGGT ft + Poiss.	0.70	0.38	0.044	0.052	–
TSDF VGGT	0.35	0.12	0.080	0.082	0.136
Poiss. VGGT	0.29	0.10	0.121	0.788	–

Table 2: Quantitative comparison on the 170 test scenes of the Objaverse dataset for 8 input views. Lower segment shows non-finetuned baselines.

Figure 6: Visual examples from two input views depicted in the beige boxes.

Figure 7: Visual examples with varying number of input views. Two exemplary views shown on the left.

Panel of Figure 8 — Figure 8: Left: Visualization of the eikonal term for the center slice of the x-axis. Right: Plot of the F 0.5ϵ -score for surfaces extracted at isovalue +0.5 $\epsilon$ .

Figure 9: Visualization of the PCA of

\hat{z}_{\textrm{3D}}

of 3 different avatar-like assets. Images show the central slices, with the first three components used as RGB channels. The contours indicate the zero-crossing of the SDF min-projection along the visualized axis.

Figure 10: PCA Visualization of occupied voxels of $\hat{z}_{\textrm{3D}}$ on four car-like assets. First three components are used as RGB channels and visualized as surface colors.

Figure 11: Inference times in seconds w.r.t the number of input views.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 12 — Figure 12: Visual example of Dtu ground-truth mesh.

Figure 13: Reliability of SDF from Meshes vs TSDFs from Depth. Non-watertight meshes may lead to sign flips, while TSDFs contain unobserved regions (green boxes). Difference images visualize differences also in values between the two.

Method	24	37	40	55	63	65	69	83	97	105	106	110	114	118	122	Mean
Favorable
GT
VolRecon	1.652	2.612	1.704	1.190	1.538	2.027	1.340	1.532	1.412	1.046	1.383	1.739	0.883	1.324	1.322	1.514
UFORecon	0.774	2.105	1.348	0.858	1.154	1.160	0.709	1.243	1.172	0.808	0.890	0.568	0.521	0.869	0.980	1.011
Offline
VolRecon	2.284	3.244	2.484	2.537	4.200	4.108	2.645	3.580	2.817	2.446	2.449	3.088	2.238	2.546	2.903	2.905

Appendix

Table 3: Quantitative results in offline vs online scenarios, and favorable vs unfavorable view combinations, separated by pose-estimation type. Values are Chamfer distances (

\downarrow

CD). GT indicates Dtu dataset poses, Offline refers to COLMAP estimation, and Online to feed-forward methods using VGGT or ours.

	Cam.	SDF	Total
Offline
UFORecon	66.4	31.5	97.9
VolRecon	66.4	21.6	88.0
Online
UFORecon	0.31	31.5	31.8
VolRecon	0.31	21.6	21.9

Appendix

Table 4: Measured execution time in seconds. "Cam." indicates time for camera parameter estimation required for baselines, SDF for direct SDF estimation.

Figure 14: Qualitative results of CUT3R on our Objaverse test set.

	Variable $\Omega$		Default $\Omega$
	F 0.5ϵ	CD	F 0.5ϵ	CD
Fus3D+	0.281	0.018	0.467	0.021
Fus3D+ wo. Q	0.274	0.021	0.466	0.022

Appendix

Table 19

Panel of Figure 15 — Figure 15: Varying the query domain $\Omega$ , potentially leading to zoomed or cropped results. Left: extracted isosurfaces. Right: slices of the SDF volumes.

Figure 16: Effect of the loss configuration, evaluated with F 0.5ϵ -Score. Fus3D denotes our final model.

*

-variants omit

{\mathcal{L}}_{\text{SDF}}(\textrm{X}_{S})

; (M) is trained on mesh-derived SDFs, (T) is trained on TSDFs fused from 24 ground-truth depth maps and camera poses.

Panel of Figure 17 — Figure 17: Left: isosurface extracted from our prediction. Center: slices of our predicted SDF and the corresponding error map. Right: slices of the TSDF reconstructed from VGGT ${}_{\text{ft}}$ predictions and the corresponding error map.

Figure 18: Further qualitative results on Objaverse , supplementing Main Fig. 1 .

Panel of Figure 19 — Figure 19: Additional qualitative results on Objaverse , supplementing Main Fig. 6 .

Panel of Figure 20 — Figure 20: Additional variants of the scene shown in Main Fig. 7 . Top: surface extracted at isovalue $0$ . Bottom: surface extracted at isovalue $+0.5\epsilon$ (Poisson reconstruction is omitted here, as it only reconstructs the zero level set).

Figure 21: Additional qualitative results on Objaverse .

Figure 22: Further qualitative results. Spurious sign flips in mesh-derived SDFs can lead to broken isosurfaces (see, e.g., the legs).

Explore similar work

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views
Apr 10, 2026Chaoyi Zhou, Run Wang, Feng Luo +43D Reconstruction Feed-Forward 3D Reconstruction
Microsoft · Clemson University · Texas A&M University
GGPT: Geometry Grounded Point Transformer
Mar 11, 2026Yutong Chen, Yiming Wang, Xucong Zhang +2Sparse-View 3D Reconstruction Feed-Forward 3D Reconstruction
ETH Zurich · Delft University of Technology
Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction
May 12, 2026Haoyu Zhang, Zeyu Zhang, Zedong Zhou +2Quantization-Aware Training 3D Reconstruction
1Peking University · 2La Trobe University

Apr 10, 2026J/K move · Enter open · S save

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

Chaoyi Zhou, Run Wang, Feng Luo, Mert D. Pesé+3

Microsoft · Clemson University · Texas A&M University

3D Reconstruction Feed-Forward 3D Reconstruction Multi-View 3D Reconstruction Open-Vocabulary 3D Segmentation

Open paper AskPDF arXiv

Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.

Abstract

Explore similar work

FF3R: Feedforward Feature 3D Reconstruction from Unconstrained views

GGPT: Geometry Grounded Point Transformer

Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction