We propose a feed-forward method for dense Signed Distance Field (SDF) regression from unstructured image collections in less than three seconds, without camera calibration or post-hoc fusion. Our key insight is that the intermediate feature space of pretrained multi-view feed-forward geometry transformers already encodes a powerful joint world representation; yet, existing pipelines discard it, routing features through per-view prediction heads before assembling 3D geometry post-hoc, which discards valuable completeness information and accumulates inaccuracies. We instead perform 3D extraction directly from geometry transformer features via learned volumetric extraction: voxelized canonical embeddings that progressively absorb multi-view geometry information through interleaved cross- and self-attention into a structured volumetric latent grid. A simple convolutional decoder then maps this grid to a dense SDF. We additionally propose a scalable, validity-aware supervision scheme directly using SDFs derived from depth maps or 3D assets, tackling practical issues like non-watertight meshes. Our approach yields complete and well-defined distance values across sparse- and dense-view settings and demonstrates geometrically plausible completions. Code and further material can be found at https://lorafib.github.io/fus3d.
Figures & tables
Figure 1: Fus3D directly regresses a 3D representation from the latent space of a multi-view geometry transformer, bypassing per-view prediction and post-hoc fusion. This, compared to VGGT [ 46 ] using TSDF fusion, yields improved surface completeness under sparse views (left, 2 views) and avoids error accumulation reducing details at scale (right, 24 views), also reflected in the F-score curves across view counts (center).
Figure 2: Juxtaposition of geometric extraction metholodogies. Common pipelines like VGGT [ 46 ] (top) route transformer features through per-view 2D decoder heads, discarding the joint multi-view representation before 3D assembly. We instead extract dense 3D features directly from the transformer’s intermediate feature space, preserving the full multi-view information.
Figure 3: Architecture of Fus3D: The geometry transformer G (beige) processes tokenized input images, yielding a list of 2D intermediate features {z2Db}b=1B extracted from different stages. The extraction transformer E (orange) leverages 2D-to-3D cross attention (green) to absorb the 3D information into features of the learned canonical embedding z3D , and distributes this information via 3D self-attention (blue) throughout the volumetric latent. The head H3D decodes the resulting structured latent z^3D into a dense SDF grid.
favorable (23, 24, 33)
unfavorable (1, 16, 36)
↓ CD
D GT2P
D P2GT
CD
D GT2P
D P2GT
VolRecon †
2.905
2.331
3.478
5.781
3.811
7.751
UFORecon †
2.771
2.101
3.440
3.275
2.252
4.298
VolRecon
3.678
3.369
3.987
8.878
5.869
11.887
UFORecon
3.415
2.989
3.841
4.942
3.737
6.146
Fus3D
2.432
1.804
3.059
3.525
2.814
4.236
Table 1: Quantitative results with direct SDF prediction baselines in a feed-forward scenario. For reference, † indicated non-feed forward (COLMAP) poses for competing methods.
Figure 4: Qualitative results on DTU . Beige boxes indicate input views. Numbers in brackets correspond to image indices of “favorable” (23, 24, 33) and “unfavorable” (1, 16, 36) view combinations. (Following the baselines’ evaluation protocols, only visible geometry is evaluated.)
Figure 5: Comparison vs ground-truth TSDF isosurfaces (green) on 170 Objaverse test scenes, w.r.t number of input views. Left: F-scores with threshold 0.5ϵ . Right: Chamfer distances.
F ϵ
F 0.5ϵ
CD
EMD
SDF
MAE
Fus3D
0.83
0.51
0.021
0.019
0.004
VGGT ft + TSDF
0.80
0.43
0.022
0.014
0.023
VGGT ft + Poiss.
0.70
0.38
0.044
0.052
–
TSDF VGGT
0.35
0.12
0.080
0.082
0.136
Poiss. VGGT
0.29
0.10
0.121
0.788
–
Table 2: Quantitative comparison on the 170 test scenes of the Objaverse dataset for 8 input views. Lower segment shows non-finetuned baselines.
Figure 6: Visual examples from two input views depicted in the beige boxes.
Figure 7: Visual examples with varying number of input views. Two exemplary views shown on the left.
Figure 8: Left: Visualization of the eikonal term for the center slice of the x-axis. Right: Plot of the F 0.5ϵ -score for surfaces extracted at isovalue +0.5 ϵ .
Figure 9: Visualization of the PCA of z^3D of 3 different avatar-like assets. Images show the central slices, with the first three components used as RGB channels. The contours indicate the zero-crossing of the SDF min-projection along the visualized axis.
Figure 10: PCA Visualization of occupied voxels of z^3D on four car-like assets. First three components are used as RGB channels and visualized as surface colors.
Figure 11: Inference times in seconds w.r.t the number of input views.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Visual example of Dtu ground-truth mesh.
Figure 13: Reliability of SDF from Meshes vs TSDFs from Depth. Non-watertight meshes may lead to sign flips, while TSDFs contain unobserved regions (green boxes). Difference images visualize differences also in values between the two.
Method
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
Mean
Favorable
GT
VolRecon
1.652
2.612
1.704
1.190
1.538
2.027
1.340
1.532
1.412
1.046
1.383
1.739
0.883
1.324
1.322
1.514
UFORecon
0.774
2.105
1.348
0.858
1.154
1.160
0.709
1.243
1.172
0.808
0.890
0.568
0.521
0.869
0.980
1.011
Offline
VolRecon
2.284
3.244
2.484
2.537
4.200
4.108
2.645
3.580
2.817
2.446
2.449
3.088
2.238
2.546
2.903
2.905
Appendix
Table 3: Quantitative results in offline vs online scenarios, and favorable vs unfavorable view combinations, separated by pose-estimation type. Values are Chamfer distances ( ↓ CD). GT indicates Dtu dataset poses, Offline refers to COLMAP estimation, and Online to feed-forward methods using VGGT or ours.
Cam.
SDF
Total
Offline
UFORecon
66.4
31.5
97.9
VolRecon
66.4
21.6
88.0
Online
UFORecon
0.31
31.5
31.8
VolRecon
0.31
21.6
21.9
Appendix
Table 4: Measured execution time in seconds. "Cam." indicates time for camera parameter estimation required for baselines, SDF for direct SDF estimation.
Figure 14: Qualitative results of CUT3R on our Objaverse test set.
Variable Ω
Default Ω
F 0.5ϵ
CD
F 0.5ϵ
CD
Fus3D+
0.281
0.018
0.467
0.021
Fus3D+ wo. Q
0.274
0.021
0.466
0.022
Appendix
Table 19
Figure 15: Varying the query domain Ω , potentially leading to zoomed or cropped results. Left: extracted isosurfaces. Right: slices of the SDF volumes.
Figure 16: Effect of the loss configuration, evaluated with F 0.5ϵ -Score. Fus3D denotes our final model. ∗ -variants omit LSDF(XS) ; (M) is trained on mesh-derived SDFs, (T) is trained on TSDFs fused from 24 ground-truth depth maps and camera poses.
Figure 17: Left: isosurface extracted from our prediction. Center: slices of our predicted SDF and the corresponding error map. Right: slices of the TSDF reconstructed from VGGT ft predictions and the corresponding error map.
Figure 18: Further qualitative results on Objaverse , supplementing Main Fig. 1 .
Figure 19: Additional qualitative results on Objaverse , supplementing Main Fig. 6 .
Figure 20: Additional variants of the scene shown in Main Fig. 7 . Top: surface extracted at isovalue 0 . Bottom: surface extracted at isovalue +0.5ϵ (Poisson reconstruction is omitted here, as it only reconstructs the zero level set).
Figure 21: Additional qualitative results on Objaverse .
Figure 22: Further qualitative results. Spurious sign flips in mesh-derived SDFs can lead to broken isosurfaces (see, e.g., the legs).
Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.
Chaoyi Zhou, Run Wang, Feng Luo +4
Microsoft · Clemson University · Texas A&M University
Recent feed-forward networks have achieved remarkable progress in sparse-view 3D reconstruction by predicting dense point maps directly from RGB images. However, they often suffer from geometric inconsistencies and limited fine-grained accuracy due to the absence of explicit multi-view constraints. We introduce the Geometry-Grounded Point Transformer (GGPT), a framework that augments feed-forward reconstruction with reliable sparse geometric guidance. We first propose an improved Structure-from-Motion pipeline based on dense feature matching and lightweight geometric optimisation to efficiently estimate accurate camera poses and partial 3D point clouds from sparse input views. Building on this foundation, we propose a geometry-guided 3D point transformer that refines dense point maps under explicit partial-geometry supervision using an optimised guidance encoding. Extensive experiments demonstrate that our method provides a principled mechanism for integrating geometric priors with dense feed-forward predictions, producing reconstructions that are both geometrically consistent and spatially complete, recovering fine structures and filling gaps in textureless areas. Trained solely on ScanNet++ with VGGT predictions, GGPT generalises across architectures and datasets, substantially outperforming state-of-the-art feed-forward 3D reconstruction models in both in-domain and out-of-domain settings.
Transformer-based 3D reconstruction has emerged as a powerful paradigm for recovering geometry and appearance from multi-view observations, offering strong performance across challenging visual conditions. As these models scale to larger backbones and higher-resolution inputs, improving their efficiency becomes increasingly important for practical deployment. However, modern 3D transformer pipelines face two coupled challenges: dense multi-view attention creates substantial token-mixing overhead, and low-precision execution can destabilize geometry-sensitive representations and degrade depth, pose, and 3D consistency. To address the first challenge, we propose Lite3R, a model-agnostic teacher-student framework that replaces dense attention with Sparse Linear Attention to preserve important geometric interactions while reducing attention cost. To address the second challenge, we introduce a parameter-efficient FP8-aware quantization-aware training (FP8-aware QAT) strategy with partial attention distillation, which freezes the vast majority of pretrained backbone parameters and trains only lightweight linear-branch projection layers, enabling stable low-precision deployment while retaining pretrained geometric priors. We further evaluate Lite3R on two representative backbones, VGGT and DA3-Large, over BlendedMVS and DTU64, showing that it substantially reduces latency (1.7-2.0x) and memory usage (1.9-2.4x) while preserving competitive reconstruction quality overall. These results demonstrate that Lite3R provides an effective algorithm-system co-design approach for practical transformer-based 3D reconstruction. Code: https://github.com/AIGeeksGroup/Lite3R. Website: https://aigeeksgroup.github.io/Lite3R.