Recent feed-forward networks have achieved remarkable progress in sparse-view 3D reconstruction by predicting dense point maps directly from RGB images. However, they often suffer from geometric inconsistencies and limited fine-grained accuracy due to the absence of explicit multi-view constraints. We introduce the Geometry-Grounded Point Transformer (GGPT), a framework that augments feed-forward reconstruction with reliable sparse geometric guidance. We first propose an improved Structure-from-Motion pipeline based on dense feature matching and lightweight geometric optimisation to efficiently estimate accurate camera poses and partial 3D point clouds from sparse input views. Building on this foundation, we propose a geometry-guided 3D point transformer that refines dense point maps under explicit partial-geometry supervision using an optimised guidance encoding. Extensive experiments demonstrate that our method provides a principled mechanism for integrating geometric priors with dense feed-forward predictions, producing reconstructions that are both geometrically consistent and spatially complete, recovering fine structures and filling gaps in textureless areas. Trained solely on ScanNet++ with VGGT predictions, GGPT generalises across architectures and datasets, substantially outperforming state-of-the-art feed-forward 3D reconstruction models in both in-domain and out-of-domain settings.
Figures & tables
Figure 2: Overview of our method. We utilise off-the-shelf feed-forward multi-view transformers [ 54 , 22 ] and dense matchers [ 10 , 69 ] to predict dense point maps {Pi∈RH×W×3}i=1N , camera parameters {gi}i=1N , and multi-view correspondences T∈RN×N×H×W×2 . These correspondences are used for sparse bundle adjustment and multi-view DLT triangulation, producing a geometrically consistent yet incomplete sparse cloud Xs via a lightweight alternative to standard SfM pipelines (Section 3.1 ). The dense yet multi-view inconsistent point cloud Xd , obtained by combining the predicted per-view point maps, exhibits local misalignments (red boxes) that are then refined under the geometric guidance of Xs by our Geometry-Grounded Point Transformer (GGPT) (Section 3.2 ), a modified point transformer [ 61 ] equipped with specialised geometric embeddings PE(xd→s) and Δd→s , which encode spatial relations between dense and sparse points to provide stable geometric guidance. The final output is a refined, globally aligned, and geometry-consistent dense point cloud X^d .
ScanNet++ [ 63 ]
ETH3D [ 44 ]
T&T [ 25 ]
4 imgs
8 imgs
16 imgs
4 imgs
8 imgs
16 imgs
4 imgs
8 imgs
16 imgs
VGGT [ 54 ]
23/37
19/32
16/29
27/41
23/36
19/32
26/40
25/39
24/38
[ 54 ] + Ours
38/53
45/60
50/66
41/55
47/61
49/63
34/47
42/57
43/57
Pi3 [ 59 ]
54/69
56 /71
58/ 74
31/47
25/41
23/38
25/39
26/42
25/40
[ 59 ] + Ours
54 /68
56/72
59/74
36/53
36/53
37/54
27/43
32/50
33/50
MapAnything [ 22 ]
40/57
38/57
38/58
10/20
7/15
5/12
10/21
9/20
8/17
Table 1: Multi-view 3D Reconstruction on Standard Test Sets. We report AUC@5/10 cm (% ↑ ). Our method (our SfM and GGPT) can improve dense predictions of various models. Despite GGPT being trained solely on ScanNet++ and VGGT’s dense prediction, as we highlight the top-left cells as within-domain , it generalises well to cross-domain datasets and other methods’ predictions (rows 3–10).
Figure 3: 3D reconstruction results on T&T [ 25 ] and ETH3D [ 44 ] . Points with confidence above the 90% quantile are visualised. We compare the error maps of the reconstruction before and after our refinement. In the zoomed-in regions, the ground truth (colored by input RGBs) is overlaid with the predictions (colored by error) to highlight how our method corrects the misalignment of input points.
Figure 4: Comparison between SfMs on ETH3D [ 44 ] . Across 4/8/16-view setups, our SfM pipeline achieves consistently better camera pose accuracy, points accuracy, and good points completeness, while retaining the shortest running time.
Figure 5: Examples on out-of-domain 4D-DRESS [ 58 ] with visualisation (top row), and MV-dVRK [ 47 ] with both visualisation and reconstruction error (bottom row), predicted by feed-forward networks and enhanced by our method. Our GGPT can significantly improve the multi-view consistency in the reconstruction result.
4D-DRESS [ 58 ]
MV-dVRK [ 47 ]
VGGT [ 54 ]
10/45
8/33
[ 54 ] + Ours
66/77
45/61
Pi3 [ 59 ]
8/50
18/51
[ 59 ] + Ours
63/80
40/67
MapAnything [ 22 ]
2/12
3/13
[ 22 ] + Ours
42/52
35/47
Table 2: Multi-view 3D reconstruction on out-of-domain datasets. AUC@1/5 cm (% ↑ ) for human body reconstruction (4D-DRESS [ 58 ] ) and AUC@1/5 mm (% ↑ ) for surgical scene reconstruction (MV-dVRK [ 47 ] ).
Method
Xs from [ 9 ]
Xs from our SfM
Murre [ 14 ]
9/23
26/40 (+17/+17)
OMNI-DC [ 73 ]
25/44
31/44 (+6/+0)
POW3R [ 20 ]
13/29
32/45 (+19/+16)
MapAnything [ 22 ]
13/32
27/40 (+14/+8)
VGGT [ 54 ] + GGPT
36/50
47/61 (+11/+11)
Table 3: Comparison of geometry conditions and geometry-grounded models. Points AUC@5/10 cm (% ↑ ) on the ETH3D [ 44 ] 8-view test set. Two conclusions can be drawn here. (1) Column-wise, our SfM Xs provides a stronger geometry signal than prior SfM [ 9 ] (We highlight the gain from using our SfM). (2) Row-wise, given the same SfM condition, our 3D GGPT outperforms prior 2D-based methods.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
4D-DRESS
MV-dVRK
VGGT [ 54 ]
10/45
8/33
[ 54 ] + Ours (RoMa + UFM)
66/77
45/61
[ 54 ] + Ours (RoMa v2)
75 / 87
62 / 77
Pi3 [ 59 ]
8/50
18/51
[ 59 ] + Ours (RoMa + UFM)
63/80
40/67
[ 59 ] + Ours (RoMa v2)
76 / 88
67 / 82
Appendix
Table 4: Multi-view 3D reconstruction on out-of-domain datasets. AUC@1/5 cm (% ↑ ) for human body reconstruction (4D-DRESS [ 58 ] ) and AUC@1/5 mm (% ↑ ) for surgical scene reconstruction (MV-dVRK [ 47 ] ). Here we report additional results with latest dense matcher RoMa v2 [ 11 ] , which further boosts the performance.
ScanNet++ [ 63 ]
ETH3D [ 44 ]
T&T [ 25 ]
4 imgs
8 imgs
16 imgs
4 imgs
8 imgs
16 imgs
4 imgs
8 imgs
16 imgs
VGGT [ 54 ]
23/37
19/32
16/29
27/41
23/36
19/32
26/40
25/39
24/38
[ 54 ] + Ours (RoMa + UFM)
38/53
45/60
50/66
41/55
47/61
49/63
34/47
42/57
43/57
[ 54 ] + Ours (RoMa v2)
47 / 62
56 / 70
62 / 76
54 / 66
57 / 69
62 / 75
39 / 52
50 / 63
49 / 62
Pi3 [ 59 ]
54/69
56 /71
58/ 74
31/47
25/41
23/38
25/39
26/42
25/40
[ 59 ] + Ours (RoMa + UFM)
54 /68
56/72
59/74
36/53
36/53
37/54
27/43
32/50
33/50
Appendix
Table 5: Multi-view 3D Reconstruction on Standard Test Sets. We report AUC@5/10 cm (% ↑ ). Our method (our SfM and GGPT) can improve dense predictions of various models. Despite GGPT is trained solely on ScanNet++ and VGGT’s dense prediction, as we highlight the top-left cells as within-domain , it generalises well to cross-domain datasets and other methods’ predictions (rows 3–10). For the results reported in the main paper, we ensemble the outputs of RoMa [ 10 ] and UFM [ 69 ] for dense matchings. Here we report additional results with latest dense matcher RoMa v2 [ 11 ] , which further boosts the performance.
4 Views
8 Views
16 Views
Time (s)
Cam. (%)
Pts. (%)
Time (s)
Cam. (%)
Pts. (%)
Time (s)
Cam. (%)
Pts. (%)
(a) Dense Matcher
MASt3R [ 28 ] (Sparse)
8.4
75/86
19/ 2
21
75/85
21/ 4
57
76/86
26/ 8
RoMa [ 10 ] (Dense)
2.3
90/93
47 /20
8.8
87 /91
46 /25
36
89/93
46 /29
UFM [ 69 ] (Dense)
1.0
64/84
26/19
1.7
81/90
30/25
8.5
77/87
33/30
RoMa [ 10 ] + UFM [ 69 ] (Ours, Dense)
3.3
86/ 93
35/ 22
11
87/92
37/ 30
44
85/91
40/ 35
Appendix
Table 6: Ablation of SfM Components on ETH3D. We evaluate design variants across three stages of our SfM—matching, bundle adjustment, and triangulation—and highlight the configuration with the best efficiency or accuracy. For each variant, we report: (1) Time denotes the running time (s ↓ ) of each stage. (2) Cam. denotes the AUC@1/5 ° (% ↑ ) for camera-pose estimation. (3) Pts. (% ↑ ) denotes the Accuracy/Completeness@1 cm (% ↑ ) of reconstructed points.
SNt++
ETH3D
4DDS.
VGGT [ 54 ]
19/32
23/36
10/45
(a) Refinement Architecture
2D Transformers
Xs→ VGGT [ 54 , 20 ]
55/70
25/40
12/52
Xs→ MapAnything [ 22 ]
39/57
11/21
5/34
3D Networks
Appendix
Table 7: Ablation Studies on GGPT. We report AUC@5/10 cm on within-domain ScanNet++ and cross-domain ETH3D 8-view splits and AUC@1/5 cm on out-of-domain 4D-DRESS .
Dense Matcher
ϵDLT
ϵreproj
Point Cloud
SNt++
ETH3D
4DDS
Coverage (%)
AUC (%)
Coverage (%)
AUC (%)
Coverage (%)
AUC (%)
–
–
–
VGGT [ 54 ] ( Xd )
100
19/32
100
23/36
100
10/45
RoMa
0.6
2
Our SfM ( Xs )
8.5
6 / 7
29
20/23
60
55/56
RoMa
0.6
2
+ GGPT ( X^d )
100
26/41
100
39/53
100
59/67
RoMa
0.1
4
Our SfM ( Xs )
21
12/15
54
35/41
73
62/64
RoMa
0.1
4
+ GGPT ( X^d )
100
31/45
100
46/59
100
66/77
Appendix
Table 8: Performance of GGPT across Varying SfM Input Densities. We vary the SfM configuration, including the matcher type, DLT threshold ϵDLT , and reprojection threshold ϵreproj , to obtain SfM point clouds of different coverage (%). We report AUC@5/10 cm on within-domain ScanNet++ and cross-domain ETH3D 8-view splits and AUC@1/5 cm (% ↑ ) on out-of-domain 4D-DRESS . Our GGPT is robust to SfM points of various densities, and can improve the complete reconstruction even when the SfM point is extremely sparse, e.g . 8.5% on ScanNet++.
Figure 6: GGPT’s Robustness to Sparse Guidance. Test examples from ETH3D [ 44 ] (1st row), ScanNet++ [ 63 ] (2nd and 3rd rows), and 4D-DRESS [ 58 ] (4th row). Points with predicted confidence below the 10% quantile are filtered. SfM points are produced with matchings from RoMa [ 10 ] . We visualise the SfM points and compare error maps before and after GGPT refinement. In the zoomed-in regions, the ground truth (coloured by input RGBs) is overlaid with the predictions (coloured by error) to highlight how our method corrects misaligned input points. SfM guidance provides little support in textureless regions, such as white walls or black cloth, but GGPT effectively propagates sparse guidance to unguided areas, improving the overall accuracy of the point map. Quantitative evaluation can be found in Tab. 8 .
Peak GPU Memory (GB) ↓
Inference Time (s) ↓
AUC@5/10 cm (%) ↑
4 Views
8 Views
16 Views
4 Views
8 Views
16 Views
4 Views
8 Views
16 Views
(a) Pipeline Breakdown
VGGT Initialisation
6.9
7.0
8.0
0.1
0.2
0.3
–
–
–
Our SfM
Dense Matcher - RoMa [ 10 ]
1.4
1.9
3.1
1.9
9.0
36
–
–
–
Dense Matcher - RoMa v2 [ 11 ]
4.9
5.3
6.1
1.5
3.1
10
–
–
–
Appendix
Table 9: Comparison of GPU Memory Consumption and Inference Speed. (a) We show the per-stage GPU memory and running time of our pipeline. (b) We compare our method with feed-forward method [ 54 ] and hybrid methods [ 9 , 13 ] across varying view counts.
Figure 7: Performance vs. number of views on ETH3D. Using the latest dense matcher RoMa v2 [ 11 ] , our method consistently surpasses the baselines with a favourable runtime profile across a wide range of input views, scaling effectively up to 50.
Method
ETH3D ( ∼36 views)
IMC ( ∼9 views)
AUC@1/3 °
Runtime (s)
AUC@3/5 °
Runtime (s)
Dense-SfM
61/78
> 300
48/61
unavailable
Our SfM
81/85
38
56/66
1.5
Appendix
Table 10: Comparison with Dense-SfM.
ETH3D
T&T
4DDress
MV-dVRK
Rel ↓
τ↑
Rel ↓
τ↑
Rel ↓
τ↑
Rel ↓
τ↑
VGGT
4.3
27/57
2.9
40/79
75.4
0/0
31.8
1/3
+ Ours
2.3
63/85
2.3
58/85
7.9
72/81
11.4
40/60
Pi3
3.7
27/60
2.9
38/75
8.6
7/20
9.4
10/29
+ Ours
2.2
59/86
2.1
58/87
4.8
27/66
5.9
34/63
Appendix
Table 11: Depth metrics. We report relative absolute error (Rel %) and inlier ratio at 1.01/1.03 ( τ %) on cross-domain and out-of-domain test sets.
Figure 8: Visual Results on ETH3D [ 44 ] . We present error maps for all baselines alongside our refined predictions and report the AUC@10 cm (%) for each scene. Points with confidence below the 10 % quantile are discarded. Because the laser ground truth is incomplete, some pixels in the error maps have no valid values.
Figure 9: Visual Results on ScanNet++ [ 63 ] . We compare the error maps between baselines and our refinement and report the AUC@10 cm (%) for each scene. We filter points with predicted confidence below the 10% quantile. On the within-domain ScanNet++, Pi3 [ 59 ] achieves saturated performance as it is trained on a large scale of similar indoor scenes, leaving limited room for further improvement. Nevertheless, our method still enhances its performance on other datasets, as demonstrated in the accompanying figures.
Figure 10: More Visual Results on T&T [ 44 ] . We compare the error maps of the baselines with those of our refinement and report the AUC@10 cm (%) for each scene. Points with predicted confidence below the 10% quantile are removed. As the laser ground truth does not cover all pixels, some regions in the error maps remain unlabelled.
Figure 11: More Visual Results on MV-dVRK [ 47 ] . We compare the error maps of the baselines with those of our refinement and report the AUC@5 mm (%) for each scene. Points with predicted confidence below the 10% quantile are discarded.
Figure 12: More Visual Results on 4D - DRESS [ 58 ] . We compare the error maps of the baselines with those of our refinement and report the AUC@5 cm (%) for each scene. Points with predicted confidence below the 10% quantile are discarded.
Feed-forward 3D reconstruction models are efficient but rigid: once trained, they perform inference in a zero-shot manner and cannot adapt to the test scene. As a result, visually plausible reconstructions often contain errors, particularly under occlusions, specularities, and ambiguous cues. To address this, we introduce Free Geometry, a framework that enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth. Our key insight is that, when the model receives more views, it produces more reliable and view-consistent reconstructions. Leveraging this property, given a testing sequence, we mask a subset of frames to construct a self-supervised task. Free Geometry enforces cross-view feature consistency between representations from full and partial observations, while maintaining the pairwise relations implied by the held-out frames. This self-supervision allows for fast recalibration via lightweight LoRA updates, taking less than 2 minutes per dataset on a single GPU. Our approach consistently improves state-of-the-art foundation models, including Depth Anything 3 and VGGT, across 4 benchmark datasets, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Code is available at https://github.com/hiteacherIamhumble/Free-Geometry .
Yuhang Dai, Xingyi Yang
The Hong Kong Polytechnic University, Hong Kong SAR
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.
Haofei Xu, Rundi Wu, Philipp Henzler +7
Google · ETH Zurich · University of Tübingen, Tübingen AI Center +3
Recent advances in vision foundation models have revolutionized geometry reconstruction and semantic understanding. Yet, most of the existing approaches treat these capabilities in isolation, leading to redundant pipelines and compounded errors. This paper introduces FF3R, a fully annotation-free feed-forward framework that unifies geometric and semantic reasoning from unconstrained multi-view image sequences. Unlike previous methods, FF3R does not require camera poses, depth maps, or semantic labels, relying solely on rendering supervision for RGB and feature maps, establishing a scalable paradigm for unified 3D reasoning. In addition, we address two critical challenges in feedforward feature reconstruction pipelines, namely global semantic inconsistency and local structural inconsistency, through two key innovations: (i) a Token-wise Fusion Module that enriches geometry tokens with semantic context via cross-attention, and (ii) a Semantic-Geometry Mutual Boosting mechanism combining geometry-guided feature warping for global consistency with semantic-aware voxelization for local coherence. Extensive experiments on ScanNet and DL3DV-10K demonstrate FF3R's superior performance in novel-view synthesis, open-vocabulary semantic segmentation, and depth estimation, with strong generalization to in-the-wild scenarios, paving the way for embodied intelligence systems that demand both spatial and semantic understanding.
Chaoyi Zhou, Run Wang, Feng Luo +4
Microsoft · Clemson University · Texas A&M University