Feed-forward models have been very successful in 3D reconstruction with ad hoc attention designs that relate all view patches densely. However, the cost of this dense global attention grows quadratically with the number of views, preventing the model from applying to larger-scale reconstruction settings. In this paper, we argue that most dense attention computations are unnecessary from a classical structure-from-motion perspective: 3D structures stem from geometrically matching regions in 2D. We bring this principle to learning-based feed-forward reconstruction with our Sparse Geometric Attention Reconstruction (SGAR) method. In SGAR, each patch token attends to only a small set of patch tokens from other views, chosen by a lightweight learned key selection module trained to predict cross-view 3D proximity of tokens. Thus, dense global attention is replaced by much more efficient geometric-similarity-guided sparse attention. SGAR is a general method that can be applied to different feed-forward models by fine-tuning. In our experiments, we show that SGAR is the fastest among both state-of-the-art dense and sparse methods and the most accurate sparse one. It runs up to 12-times faster than dense attention at 1,000 views with stable pose accuracy where existing sparse methods fail, and shows the best performance in reconstructing more complete geometries on the same time budget.
Figures & tables
Figure 2: Dense attention is redundant. By selecting oracle top-k keys based on ground-truth geometry, most of the pose accuracy can be retained even when 99% of attention weights are zeroed out. Evaluating DejaView [ 8 ] on an NRGBD [ 3 ] scene.
Figure 3: Overview of SGAR. We replace the global dense attention of DejaView [ 8 ] with our sparse geometric attention as an example. The images are encoded by DINOv2 and augmented with camera and register tokens. A single shared block, unrolled over its L steps, runs frame attention, key selection and sparse attention. In sparse attention, each patch attends to the special tokens of all views, the patches of the reference view, a summary of each cluster, and all the patches of the nearest clusters, so that its cost grows near-linearly with the number of views. Key Selection projects the patch tokens by Wadj into a geometric embedding and clusters the embeddings by spherical k -means into N clusters, re-clustered at every step. Each cluster probes its m=4 nearest clusters, itself first. With the patches sorted by cluster, the selected keys form a block-sparse mask over tiles of 128 tokens. A contrastive loss pulls together the embeddings of patches that observe the same surface in the ground-truth geometry and pushes the others apart.
Short ( <100 )
Mid-range (125–311)
Long (1,000)
AUC@30
Mean AUC
Mip360 AUC
ScanNet AUC
Method
Params
7-Scenes
ETH3D
LaMAR
NRGBD
@5
@30
Time
@5
@30
Time
@5
@30
Time
Mem
VGGT [ 60 ]
1158
81.0
70.6
27.3
97.2
40.2
69.0
6.98
74.5
94.9
54.1
oom
–
–
75.3
π3 X [ 65 ]
1360
82.0
83.7
59.1
98.6
51.8
80.8
3.82
oom
–
–
oom
–
–
–
DA3 [ 29 ]
1356
82.8
83.6
47.7
99.8
53.5
78.5
3.81
90.9
98.3
24.3
oom
–
–
39.4
VGGT- Ω [ 61 ]
1144
88.2
90.1
59.7
98.6
57.9
84.2
2.27
86.3
97.4
14.2
27.9
85.3
303.8
22.9
Table 1: Camera pose estimation. SGAR (VGGT- Ω ) and SGAR (DejaView) are the fastest models at every sequence length and the most accurate efficient ones, close to their dense counterparts. AUC in %, time in s per scene, Params in millions, Mem in peak GiB on Mip-NeRF 360; oom : out of memory; fail : kernel compile error; –: not available. Green, yellow-green, yellow: first, second, third; ties share a place.
Figure 4: Qualitative comparison. Point clouds predicted by FastVGGT, ZipMap and SGAR (DejaView), with the ground truth, on one scene of each of 7-Scenes (chess/seq-03), ETH3D (courtyard, laser scan), LaMAR (HGE) and Mip-NeRF 360 (bonsai, SfM points). Left: four of the input views. Below each reconstruction: the AUC@30 of its cameras. SGAR (DejaView) keeps the structure of the scene where the other efficient models fragment it, most visibly on the long LaMAR trajectory.
Time budget (s)
Method
1
2
3
5
7.5
10
15
20
VGGT [ 60 ]
59.5
69.9
73.2
78.6
81.2
84.4
87.0
88.3
π3 X [ 65 ]
55.9
67.5
73.5
80.1
80.6
81.9
84.9
85.5
DA3 [ 29 ]
59.8
74.4
75.9
83.6
84.7
87.3
88.8
89.9
VGGT- Ω [ 61 ]
71.0
79.6
82.5
86.4
88.7
90.7
90.9
92.4
DejaView [ 8 ]
73.8
78.6
84.2
86.2
87.0
88.2
89.2
91.3
Table 2: Reconstruction at a fixed time budget. Coverage @5 cm (%): share of the ground-truth surface of ScanNet scene0770_00 within 5 cm of a predicted point after ICP alignment, with each model given as many views as fit in the budget. Of each column, over all models, green marks the best value, yellow-green the second and yellow the third. Our model can recover the most complete geometries at all time budgets.
δ1.25↑
Method
7-Scenes
LaMAR
NRGBD
Mean
VGGT [ 60 ]
95.4
81.1
99.5
92.0
π3 X [ 65 ]
98.5
94.0
99.6
97.3
DA3 [ 29 ]
97.5
91.6
99.5
96.2
VGGT- Ω [ 61 ]
97.4
91.8
100.0
96.4
DejaView [ 8 ]
97.6
87.3
99.3
94.8
Table 3: Depth estimation. Depth on the short benchmark ( K=100 views per scene) for the three datasets with ground-truth depth. The predicted depth is aligned to the ground truth with one median scale per sequence, so it must be consistent across views. δ1.25 : share of pixels with δ<1.25 , in per cent (higher is better). Mean is over the three datasets, each weighted equally. Of each column, over all models, green marks the best value, yellow-green the second and yellow the third; tied values share a place.
Figure 5: Scaling with the number of views. Time per scene (a, log scale) and AUC@30 (b) on the ScanNet test capture scene0770_00, with 100 to 1,000 views evenly spaced over its 3,383 frames, so that more views sample the same trajectory more densely. A line ends where the model runs out of memory on the A100 80 GB or fails; the cross marks its last size.
K=100
Mip360
ScanNet
Variant
@30
Time
@30
Time
@30
Time
DejaView, dense
82.3
2.10
97.5
14.1
86.5
319.6
top- 256
68.8
1.85
88.4
5.6
75.1
33.6
k -means on raw tokens
65.3
1.39
86.7
4.4
71.9
30.3
w/o reference view R
74.1
1.30
93.1
4.0
84.7
25.6
w/o summaries M(i)
76.3
1.29
95.0
3.9
83.5
24.3
Table 4: Ablations. An ablation study on the SGAR (DejaView) model. AUC@30 (%) and time (s per scene). K=100 : mean over the four short datasets; ScanNet: 1,000 views.