Feed-forward models have been very successful in 3D reconstruction with ad hoc attention designs that relate all view patches densely. However, the cost of this dense global attention grows quadratically with the number of views, preventing the model from applying to larger-scale reconstruction settings. In this paper, we argue that most dense attention computations are unnecessary from a classical structure-from-motion perspective: 3D structures stem from geometrically matching regions in 2D. We bring this principle to learning-based feed-forward reconstruction with our Sparse Geometric Attention Reconstruction (SGAR) method. In SGAR, each patch token attends to only a small set of patch tokens from other views, chosen by a lightweight learned key selection module trained to predict cross-view 3D proximity of tokens. Thus, dense global attention is replaced by much more efficient geometric-similarity-guided sparse attention. SGAR is a general method that can be applied to different feed-forward models by fine-tuning. In our experiments, we show that SGAR is the fastest among both state-of-the-art dense and sparse methods and the most accurate sparse one. It runs up to 12-times faster than dense attention at 1,000 views with stable pose accuracy where existing sparse methods fail, and shows the best performance in reconstructing more complete geometries on the same time budget.
Figures & tables
Figure 2: Dense attention is redundant. By selecting oracle top-k keys based on ground-truth geometry, most of the pose accuracy can be retained even when 99% of attention weights are zeroed out. Evaluating DejaView [ 8 ] on an NRGBD [ 3 ] scene.
Figure 3: Overview of SGAR. We replace the global dense attention of DejaView [ 8 ] with our sparse geometric attention as an example. The images are encoded by DINOv2 and augmented with camera and register tokens. A single shared block, unrolled over its L steps, runs frame attention, key selection and sparse attention. In sparse attention, each patch attends to the special tokens of all views, the patches of the reference view, a summary of each cluster, and all the patches of the nearest clusters, so that its cost grows near-linearly with the number of views. Key Selection projects the patch tokens by Wadj into a geometric embedding and clusters the embeddings by spherical k -means into N clusters, re-clustered at every step. Each cluster probes its m=4 nearest clusters, itself first. With the patches sorted by cluster, the selected keys form a block-sparse mask over tiles of 128 tokens. A contrastive loss pulls together the embeddings of patches that observe the same surface in the ground-truth geometry and pushes the others apart.
Short ( <100 )
Mid-range (125–311)
Long (1,000)
AUC@30
Mean AUC
Mip360 AUC
ScanNet AUC
Method
Params
7-Scenes
ETH3D
LaMAR
NRGBD
@5
@30
Time
@5
@30
Time
@5
@30
Time
Mem
VGGT [ 60 ]
1158
81.0
70.6
27.3
97.2
40.2
69.0
6.98
74.5
94.9
54.1
oom
–
–
75.3
π3 X [ 65 ]
1360
82.0
83.7
59.1
98.6
51.8
80.8
3.82
oom
–
–
oom
–
–
–
DA3 [ 29 ]
1356
82.8
83.6
47.7
99.8
53.5
78.5
3.81
90.9
98.3
24.3
oom
–
–
39.4
VGGT- Ω [ 61 ]
1144
88.2
90.1
59.7
98.6
57.9
84.2
2.27
86.3
97.4
14.2
27.9
85.3
303.8
22.9
Table 1: Camera pose estimation. SGAR (VGGT- Ω ) and SGAR (DejaView) are the fastest models at every sequence length and the most accurate efficient ones, close to their dense counterparts. AUC in %, time in s per scene, Params in millions, Mem in peak GiB on Mip-NeRF 360; oom : out of memory; fail : kernel compile error; –: not available. Green, yellow-green, yellow: first, second, third; ties share a place.
Figure 4: Qualitative comparison. Point clouds predicted by FastVGGT, ZipMap and SGAR (DejaView), with the ground truth, on one scene of each of 7-Scenes (chess/seq-03), ETH3D (courtyard, laser scan), LaMAR (HGE) and Mip-NeRF 360 (bonsai, SfM points). Left: four of the input views. Below each reconstruction: the AUC@30 of its cameras. SGAR (DejaView) keeps the structure of the scene where the other efficient models fragment it, most visibly on the long LaMAR trajectory.
Time budget (s)
Method
1
2
3
5
7.5
10
15
20
VGGT [ 60 ]
59.5
69.9
73.2
78.6
81.2
84.4
87.0
88.3
π3 X [ 65 ]
55.9
67.5
73.5
80.1
80.6
81.9
84.9
85.5
DA3 [ 29 ]
59.8
74.4
75.9
83.6
84.7
87.3
88.8
89.9
VGGT- Ω [ 61 ]
71.0
79.6
82.5
86.4
88.7
90.7
90.9
92.4
DejaView [ 8 ]
73.8
78.6
84.2
86.2
87.0
88.2
89.2
91.3
Table 2: Reconstruction at a fixed time budget. Coverage @5 cm (%): share of the ground-truth surface of ScanNet scene0770_00 within 5 cm of a predicted point after ICP alignment, with each model given as many views as fit in the budget. Of each column, over all models, green marks the best value, yellow-green the second and yellow the third. Our model can recover the most complete geometries at all time budgets.
δ1.25↑
Method
7-Scenes
LaMAR
NRGBD
Mean
VGGT [ 60 ]
95.4
81.1
99.5
92.0
π3 X [ 65 ]
98.5
94.0
99.6
97.3
DA3 [ 29 ]
97.5
91.6
99.5
96.2
VGGT- Ω [ 61 ]
97.4
91.8
100.0
96.4
DejaView [ 8 ]
97.6
87.3
99.3
94.8
Table 3: Depth estimation. Depth on the short benchmark ( K=100 views per scene) for the three datasets with ground-truth depth. The predicted depth is aligned to the ground truth with one median scale per sequence, so it must be consistent across views. δ1.25 : share of pixels with δ<1.25 , in per cent (higher is better). Mean is over the three datasets, each weighted equally. Of each column, over all models, green marks the best value, yellow-green the second and yellow the third; tied values share a place.
Figure 5: Scaling with the number of views. Time per scene (a, log scale) and AUC@30 (b) on the ScanNet test capture scene0770_00, with 100 to 1,000 views evenly spaced over its 3,383 frames, so that more views sample the same trajectory more densely. A line ends where the model runs out of memory on the A100 80 GB or fails; the cross marks its last size.
K=100
Mip360
ScanNet
Variant
@30
Time
@30
Time
@30
Time
DejaView, dense
82.3
2.10
97.5
14.1
86.5
319.6
top- 256
68.8
1.85
88.4
5.6
75.1
33.6
k -means on raw tokens
65.3
1.39
86.7
4.4
71.9
30.3
w/o reference view R
74.1
1.30
93.1
4.0
84.7
25.6
w/o summaries M(i)
76.3
1.29
95.0
3.9
83.5
24.3
Table 4: Ablations. An ablation study on the SGAR (DejaView) model. AUC@30 (%) and time (s per scene). K=100 : mean over the four short datasets; ScanNet: 1,000 views.
Recent feed-forward networks have achieved remarkable progress in sparse-view 3D reconstruction by predicting dense point maps directly from RGB images. However, they often suffer from geometric inconsistencies and limited fine-grained accuracy due to the absence of explicit multi-view constraints. We introduce the Geometry-Grounded Point Transformer (GGPT), a framework that augments feed-forward reconstruction with reliable sparse geometric guidance. We first propose an improved Structure-from-Motion pipeline based on dense feature matching and lightweight geometric optimisation to efficiently estimate accurate camera poses and partial 3D point clouds from sparse input views. Building on this foundation, we propose a geometry-guided 3D point transformer that refines dense point maps under explicit partial-geometry supervision using an optimised guidance encoding. Extensive experiments demonstrate that our method provides a principled mechanism for integrating geometric priors with dense feed-forward predictions, producing reconstructions that are both geometrically consistent and spatially complete, recovering fine structures and filling gaps in textureless areas. Trained solely on ScanNet++ with VGGT predictions, GGPT generalises across architectures and datasets, substantially outperforming state-of-the-art feed-forward 3D reconstruction models in both in-domain and out-of-domain settings.
While recent feed-forward 3D reconstruction models accelerate 3D reconstruction by jointly inferring dense geometry and camera poses in a single pass, their reliance on dense attention imposes a quadratic complexity, creating a prohibitive computational bottleneck that severely limits inference speed. To resolve this, we introduce Speed3R, an end-to-end trainable model inspired by the core principle of Structure-from-Motion: that a sparse set of keypoints is sufficient for robust pose estimation. Speed3R features a dual-branch attention mechanism where a compression branch creates a coarse contextual prior to guide a selection branch, which performs fine-grained attention only on the most informative image tokens. This strategy mimics the efficiency of traditional keypoint matching, achieving a remarkable 12.4x inference speedup on 1000-view sequences, while introducing a minimal, controlled trade-off in geometric accuracy. Validated on standard benchmarks with both VGGT and π3 backbones, our method delivers high-quality reconstructions at a fraction of computational cost, paving the way for efficient large-scale scene modeling.
VGGT reconstructs cameras and geometry from unposed images in one pass, but its global attention grows quadratically with the number of views. Training-free accelerators thin this attention and work well on redundant video, but lose pose accuracy when views share little geometry. This paper traces these losses to two kinds of cross-view information that a uniform sparse approximation can fail to preserve together: diffuse layers aggregate distributed context, whereas matching layers depend on query-specific interactions that carry little attention mass. A per-layer statistic measures the log-mass missed by a mean token. Its row-free proxy is the logit variance, and it selects the estimator for each layer. Diffuse layers use shared, importance-sampled hub keys, with log-weights that make the softmax numerator and denominator unbiased. Matching layers stay exact: every aggressive thinning tested cost accuracy. At equal cost, the selected layers outperform the four shifted windows tested. On 13 unseen low-redundancy scenes with 32 views, LMSA stays within 1.4 AUC@5 of dense attention, compared with 10.3 for AVGGT and 4.5 for SAF3R; its difference from contemporaneous GPA-VGGT is not significant, and only its gains over AVGGT pass both scene-level tests. With 100 views, it stays within 1.9 AUC@5 while its aggregator runs 2.0× faster. Used post hoc to choose the layers GPA-VGGT protects, the statistic improves it by 1.2 AUC@5 for 5% more FLOPs. Applied untuned to π3, it beats AVGGT but remains further from dense accuracy.
Sammam Mahdi, Fariha Binta Salim, K. M. Jubair Sami +1