VGGT reconstructs cameras and geometry from unposed images in one pass, but its global attention grows quadratically with the number of views. Training-free accelerators thin this attention and work well on redundant video, but lose pose accuracy when views share little geometry. This paper traces these losses to two kinds of cross-view information that a uniform sparse approximation can fail to preserve together: diffuse layers aggregate distributed context, whereas matching layers depend on query-specific interactions that carry little attention mass. A per-layer statistic measures the log-mass missed by a mean token. Its row-free proxy is the logit variance, and it selects the estimator for each layer. Diffuse layers use shared, importance-sampled hub keys, with log-weights that make the softmax numerator and denominator unbiased. Matching layers stay exact: every aggressive thinning tested cost accuracy. At equal cost, the selected layers outperform the four shifted windows tested. On 13 unseen low-redundancy scenes with 32 views, LMSA stays within 1.4 AUC@5 of dense attention, compared with 10.3 for AVGGT and 4.5 for SAF3R; its difference from contemporaneous GPA-VGGT is not significant, and only its gains over AVGGT pass both scene-level tests. With 100 views, it stays within 1.9 AUC@5 while its aggregator runs 2.0× faster. Used post hoc to choose the layers GPA-VGGT protects, the statistic improves it by 1.2 AUC@5 for 5% more FLOPs. Applied untuned to π3, it beats AVGGT but remains further from dense accuracy.
Figures & tables
Figure 1: LMSA on a frozen VGGT-1B. (a) N unposed views (ETH3D courtyard [ 35 ] ), each P patch and 5 special tokens. (b) Frame attention (grey) is unchanged. The offline statistic Γˉℓ (black bars, once per model, Section 3.3 ) sets each global layer’s estimator: above 1.5 (dotted), matching layers (blue) keep exact attention; diffuse layers (orange) use 256 or 1024 shared hubs. Block width: keys per query at N=32 , log scale. (c) Cameras, point cloud and depth predicted by LMSA. (d) A diffuse-layer query (black dot) attends to its own view, all special tokens and m hubs drawn per input ( Section 3.4 ); hub counts to scale, positions illustrative. (e) In a matching layer every token is a key (magenta: true correspondences). (f) Exact cost ( Section 3.6 ); pale: the same layers at dense cost.
Figure 2: One statistic assigns the layer roles; attention mass is not importance. (a) Log-mass deficit Γ of a count-biased mean token against the row-free logit variance s2 ( Eqs. 3 and 5 ), for 1.15M (query, layer, head) triples from layers 9–23 of VGGT-1B on 8 scenes (median, IQR); it follows s2/2 up to s2≈10 and falls below it at larger variance (grey), as the Gaussian REM, or light-tailed or correlated logits, would. (b) Offline layer statistic Γˉℓ (median head, mean of the 5 development scenes), coloured by the estimator LMSA assigns; dotted: the empirical cut-off (1.5 nats). (c) AUC@5 lost when one key set is removed from layers 10–16 only (three low-redundancy development scenes, 5 seeds; Tab. S15 ); in brackets, the attention mass removed.
Scene ( ρ )
E (0–8)
B (10–16)
L (9, 17–23)
all
TUM fr1_room (4.3)
5.6
11.7
−0.4
13.3
ETH3D courtyard (7.7)
33.1
28.5
5.4
50.2
ETH3D office (5.5)
1.5
6.0
3.5
12.3
TUM fr3_long_office (15.4)
0.5
4.4
14.8
22.1
7-Scenes office (13.0)
−1.1
0.8
1.6
1.8
Table 1: Three groups fail for different reasons. AUC@5 lost by frozen VGGT-1B when one group of global layers is thinned AVGGT-style and the rest stay dense: E, layers 0–8 made frame-local; B, layers 10–16 on a σ=9 lattice with a mean token; L, layers 9 and 17–23 likewise. Mean of 5 frame subsets, N=32 ( N=22 for ETH3D office); ρ as in Section 4.1 ; bold: largest single-group loss.
Unseen low- ρ (13 scenes)
Unseen high- ρ (8 scenes)
Dev. (5)
Method
AUC@5 ↑
AUC@30 ↑
AbsRel ↓
AUC@5 ↑
AUC@30 ↑
AbsRel ↓
AUC@5 ↑
GFLOP/tok. ↓
Dense VGGT-1B
57.4
85.4
4.34
72.9
92.6
3.95
49.2
5.29
AVGGT ( σ=4 ) [ 40 ]
47.1
81.1
4.08
67.4
91.4
3.87
35.2
2.64
AVGGT ( σ=9 ) [ 40 ]
38.5
72.8
4.32
62.4
90.5
3.79
29.2
2.38
GPA-VGGT [ 41 ]
55.0
83.8
4.65
72.8
92.6
3.93
47.8
2.96
SAF3R [ 12 ]
52.8
84.3
5.48
71.1
92.2
3.80
41.5
2.60
Table 2: Main results on the 21 pre-named test scenes, the frozen recipe’s single evaluation ( N=32 , scene means of 5 seeds), and on the development scenes. AbsRel ×100 , without Tanks and Temples (no depth). GFLOP/token at N=32 : exact cost model (reference T=1041 ) for VGGT-based recipes, measured kept fractions otherwise. Bold: best sparse value per column (not a significant difference).
GFLOP / token ↓
Speed-up vs. dense ↑
T&T loss ↓
Method
N=32
100
32
64
100
N=100
Dense
5.29
12.25
1.00
1.00
1.00
0
AVGGT ( σ=4 )
2.64
3.77
1.54
1.99
2.41
7.10
AVGGT ( σ=9 )
2.38
2.94
1.63
2.22
2.78
12.61
GPA-VGGT
2.96
4.79
1.40
1.73
2.00
1.61
SAF3R
2.60
3.67
1.34
1.80
2.19
5.65
Table 3: Cost, speed and accuracy at scale. GFLOP per token from the exact cost model ( Eq. 8 ) at the reference T=1041 ; speed-up of the aggregator over dense attention timed in the same session; AUC@5 lost against dense attention on 5 Tanks and Temples scenes at N=100 . Absolute times: Tab. S10 . Bold: best sparse value per column.
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
Layers
Key set per query (query in frame i )
Units
0–8
Own frame (all tokens) + special tokens of every frame + 256 hubs importance-sampled from a 512-query pilot (head-mean full attention, column mass, floor 10−3/∣C∣ ), log-weight ln(cj/(mpj)) ; hubs in the own frame get no weight
1.6
9, 10
Own frame + special tokens + 1024 hubs (same estimator)
2.4
11–16
Exact dense attention (the layers with Γˉℓ≥1.5 )
N
17–23
As layers 9–10
2.4
LMSA-fast, 11–16
Eq. S2 : frame 0 + special tokens + σ=4 lattice + diagonal + mean token + switched 6×10 blocks
9–10
LMSA-fast, router
Once, on layer 9’s attention input; top-4 cosine per (query, frame) at full width; 4×8 tiles; 3×3 -smoothed displacement mode; switch: share of the tile’s top-1 votes in the smoothed peak ≥0.3
≈0.5N
Appendix
Table S1: LMSA on VGGT-1B (frozen definition) and LMSA-fast. T=1041 tokens per frame at 518 px, 16 heads, dh=64 . Units at N=32 from the exact cost model (one unit = one frame’s keys for every query). Bold: the layers kept exact.
Layer
Γˉ
s2
<3
≥7
Tube
Role
0
0.12
–
–
–
–
hubs, 256
1
0.05
–
–
–
–
hubs, 256
2
0.30
–
–
–
–
hubs, 256
3
0.71
–
–
–
–
hubs, 256
4
0.78
–
–
–
–
hubs, 256
5
0.86
–
–
–
–
hubs, 256
Appendix
Table S2: Per-layer statistics behind Fig. 2 . Γˉ : pilot statistic (median head, mean of 5 development scenes). s2 : median-head row-free logit variance. “ <3 ”: heads (of 16) with s2<3 . “ ≥7 ”: share of (query, head) pairs with s2≥7 . Tube: attention mass on the 3×3 correspondence window (low- ρ ). Layers 0–8 were not in the attention audit. Bold: the layers kept exact (those above the cut-off) and their Γˉ .
Method
Original order
Random order
Change
LMSA
1.57
1.65
+0.08 [ −1.05 , 1.21]
GPA-VGGT †
3.01
2.83
−0.19 [ −2.08 , 1.35]
AVGGT(4)
10.44
12.31
+1.87 [ −1.02 , 4.95]
Appendix
Table S3: Random input order on the 8 unseen low- ρ ETH3D scenes (5 seeds): AUC@5 lost against dense attention on the same order, mean and 95% CI; change = random minus original. Bold: lowest loss per column and the change for LMSA.
Scene
Method
AUC@5
AUC@30
AbsRel
TUM fr1_room
dense
29.15
77.90
0.0418
SAF3R
23.02
73.50
0.0414
GPA-VGGT
26.94
76.13
0.0429
ETH3D courtyard
dense
54.40
90.22
0.0198
SAF3R
30.97
80.34
0.0245
GPA-VGGT
41.37
84.69
0.0222
Appendix
Table S4: Baseline bring-up check (single seed, N=32 , one environment). Not used for any claim; the main comparison is Table 2 . Bold: the better baseline per scene and metric.
Test
Plan hash (prefix)
Fixed
π3 predictions, newer release
20947731e395
Nov 13, 15:02
π3 predictions, original checkpoint
d338191c0597
Nov 13, 15:44
π3 transfer evaluation
5ed86d7cbc44
Nov 13, 18:19
Selector applied to GPA-VGGT
1d5d99725695
Nov 13, 21:37
7-Scenes long sequences
d360a911d5f0
Nov 13, 21:52
Confirmatory test (first version)
2ee8727dd5a1
Nov 14, 04:31
Appendix
Table S5: Registered plans. SHA-256 prefix of each plan file and the time it was fixed, before any run of that test (2026, UTC + 6). Bold: the confirmatory test.
Method
AUC@5 loss, low- ρ
AUC@30 loss, low- ρ
AUC@5 loss, high- ρ
Δ AbsRel ×100 , low- ρ
AVGGT ( σ=4 )
10.27 [7.98, 13.22]
4.30 [2.25, 6.85]
5.50 [1.40, 9.53]
-0.26
AVGGT ( σ=9 )
18.91 [14.64, 24.51]
12.63 [7.73, 18.59]
10.50 [4.01, 16.65]
-0.02
GPA-VGGT
2.42 [1.28, 3.71]
1.67 [0.50, 3.09]
0.08 [ −0.20 , 0.42]
+0.31
SAF3R
4.53 [1.73, 7.73]
1.09 [ −0.52 , 2.73]
1.73 [0.60, 3.20]
+1.14
LMSA
1.38 [0.71, 2.01]
0.70 [0.17, 1.26]
0.73 [0.25, 1.27]
+0.34
LMSA-fast
4.65 [2.74, 6.72]
1.54 [0.71, 2.49]
1.81 [0.24, 3.67]
-0.37
Appendix
Table S6: Paired differences to dense on the unseen scenes (pooled; hierarchical bootstrap 95% CI), N=32 , 5 seeds. Loss = dense − method; Δ AbsRel = method − dense. Bold: lowest value per column.
Split
Scene
Dense
AVGGT(4)
AVGGT(9)
GPA-VGGT
SAF3R
LMSA
LMSA-fast
Unseen low- ρ
ETH3D delivery_area
25.1
14.8
8.9
24.1
24.4
24.6
19.0
ETH3D electro
61.2
48.1
35.5
58.0
44.5
58.3
54.6
ETH3D kicker
90.2
84.8
79.0
89.1
88.6
88.4
86.7
ETH3D playground
25.8
17.9
9.0
20.9
21.5
24.0
25.8
ETH3D relief
23.5
13.1
4.5
18.6
21.2
22.7
19.0
ETH3D relief_2
10.1
4.9
1.7
7.0
7.3
9.0
7.2
Appendix
Table S7: Per-scene AUC@5 (5-seed means, N=32 unless the scene has fewer views). Rows are generated from the main-table results file. Bold: highest sparse value per scene (ties all bold).
Dataset
Scenes
Dense AUC@5
AVGGT(4)
AVGGT(9)
GPA-VGGT
SAF3R
LMSA
LMSA-fast
ETH3D
8 (8)
43.4
10.44
18.93
3.01
3.14
1.57
4.84
Tanks and Temples
5 (4)
84.1
9.40
18.78
1.25
5.50
1.10
3.78
TUM RGB-D
3 (1)
82.1
11.81
19.20
0.73
4.46
1.04
5.04
7-Scenes
3 (0)
38.4
−1.08
−0.46
−0.21
2.04
0.80
0.64
DTU
2 (0)
99.9
7.36
14.13
0.06
0.30
0.05
0.06
RealEstate10K outdoor ∗
9 (0)
33.2
2.11
5.14
−0.76
2.49
0.22
–
Appendix
Table S8: Per-dataset comparison. AUC@5 lost against dense attention, mean over each dataset’s unseen test scenes of the 5-seed scene means ( N=32 ; in brackets, how many of the scenes are low- ρ ). Negative: above dense attention. Bold: lowest loss in the row. Descriptive: datasets with 2–3 scenes carry no interval. ∗ A separate post-hoc test ( Section C.6 ): 9 outdoor test clips, 5 frame subsets each, all of high redundancy; LMSA-fast was not run.
A (cameras)
B (points)
LMSA − dense
+0.007 [ −0.014 , 0.029]
+0.001 [ −0.014 , 0.015]
LMSA-fast − dense
−0.057 [ −0.116 , −0.010 ]
+0.034 [ −0.010 , 0.095]
GPA-VGGT − dense
+0.066 [0.018, 0.130]
+0.011 [ −0.006 , 0.035]
SAF3R − dense
+0.064 [0.031, 0.098]
+0.020 [0.005, 0.038]
AVGGT(4) − dense
+0.120 [0.033, 0.219]
–
AVGGT(9) − dense
+0.283 [0.108, 0.541]
–
Appendix
Table S9: Point maps on the 9 unseen low- ρ scenes with ground truth (post hoc). Change in Chamfer distance (m) against dense attention, and paired contrasts, with hierarchical 95% CIs; negative is better. A: alignment on camera centres (includes pose error); B: alignment on points. Bold: LMSA against dense attention, and its two contrasts that are significant (alignment A).
GFLOP / token
Agg. time (s)
Full time (s)
Method
N=32
100
32
64
100
32
64
Dense
5.29
12.25
5.27
15.02
31.32
5.71
16.05
AVGGT ( σ=4 )
2.64
3.77
3.43
7.54
12.98
3.90
8.63
AVGGT ( σ=9 )
2.38
2.94
3.22
6.76
11.26
3.72
7.88
GPA-VGGT
2.96
4.79
3.76
8.68
15.69
4.03
9.41
SAF3R
2.60
3.67
3.94
8.34
14.28
4.41
9.34
Appendix
Table S10: Cost and time . GFLOP per token from the exact cost model ( Eq. 8 ). Times: one session per N in which every method, dense included, was timed interleaved on the same input (1 warm-up and 3 timed rounds, median, CUDA events) on one 16 GB GPU. “Agg.”: aggregator only, including the hub pilot; “full”: aggregator (which includes the DINOv2 patch embedding) plus camera and depth heads, i.e. the whole network; image loading and resizing are excluded (at N=100 , in a separate session: dense 33.26 s, LMSA 17.19 s, GPA-VGGT 16.84 s). † Own sessions, in which dense took 5.14, 14.88 and 31.30 s (aggregator). Bold: lowest sparse value per column.
Method
chess
office
fire
kitchen
∣Δ5∣
Δ30
Speed-up
AVGGT(4)
−0.85
+0.96
+7.85
+4.28
3.48
−0.11
5.1 ×
AVGGT(9)
−1.92
+0.32
+5.93
+5.04
3.30
−0.90
8.1 ×
LMSA
−0.33
+0.94
−1.57
+1.35
1.05
+0.59
3.3 ×
LMSA-auto
−0.13
+0.94
−0.97
+1.35
0.85
+0.70
3.4 ×
Appendix
Table S11: 800 frames on four 7-Scenes sequences, one frame subset each. Per-sequence AUC@5 minus that of dense attention (positive = above dense), mean ∣Δ∣ of AUC@5, mean Δ of AUC@30, and mean end-to-end speed-up over dense attention. Dense AUC@5: 43.2, 15.4, 53.1 and 6.7. Bold: the two lowest mean absolute differences (both LMSA variants), the largest departure from dense attention and the largest speed-up.
Accurate − dense
Efficient − LMSA
Draw SD of AUC@5
Dataset
AUC@5
AUC@30
AUC@5
AUC@30
Acc.
Eff.
LMSA
NRGBD
−0.41 [ −0.78 , −0.10 ]
−0.08
+0.32 [ −0.25 , 0.94]
+0.07
0.68
1.44
1.40
Sintel
−0.30 [ −0.65 , 0.02]
−0.08
+0.27 [ −0.38 , 0.93]
+0.04
1.30
2.01
2.04
7-Scenes
+0.00 [ −0.55 , 0.48]
−0.06
+0.55 [ −0.77 , 2.44]
−0.00
0.79
0.85
1.11
Appendix
Table S12: Confirmatory test (registered before the runs; mean and nested 95% CI). Left: LMSA-Accurate minus dense attention. Middle: LMSA-Efficient minus LMSA. Right: draw-to-draw SD of AUC@5 within a subset. Bold: the pose-accuracy difference of LMSA-Accurate to dense attention (first column), and the lowest draw SD per dataset.
Baseline
Mean d
Ahead
Wilcoxon (Holm)
Sign-flip (Holm)
AVGGT ( σ=4 )
+8.89
13/13
< 0.001 (0.001)
< 0.001 (0.001)
AVGGT ( σ=9 )
+17.54
13/13
< 0.001 (0.001)
< 0.001 (0.001)
GPA-VGGT
+1.04
8/13
0.340 (0.340)
0.165 (0.165)
SAF3R
+3.15
11/13
0.017 (0.034)
0.064 (0.128)
Appendix
Table S13: Scene-level tests on the 13 unseen low- ρ scenes: d = AUC@5 of LMSA minus that of the baseline, per scene (5-seed means). Two-sided Wilcoxon signed-rank and exact sign-flip tests on d ; Holm correction over the four baselines within each test. Bold: significant after Holm correction.
Figure S1: Reconstructions on two ETH3D scenes [ 35 ] . Each panel shows the predicted depth, unprojected with the predicted cameras (86% most confident pixels), and the predicted cameras (frusta in the column colour), both mapped to the ground-truth frame by the least-squares similarity transform between predicted and ground-truth camera centres; black frusta are the ground-truth cameras, and all panels of a row share one viewpoint. Top: courtyard, a development scene, subset 0. Bottom: terrace, an unseen low-redundancy test scene, subset 2. In each scene the subset is the one on which GPA-VGGT loses the most AUC@5 against dense attention (13.0 and 14.6 points), chosen by GPA-VGGT’s loss, not by the result of LMSA. Titles give that subset’s pose AUC@5 and, in brackets, its difference from dense VGGT-1B. On courtyard the cameras of AVGGT(4) are visibly displaced and its facades blurred; smaller pose errors are hard to see at this scale. Courtyard was used to design LMSA, so its +9.4 there is not a test result.
Layers and key set
fr1
court
ETH-o
fr3
7S
Early, 0–8 (own frame exact)
frame-local (AVGGT)
5.6
33.1
1.5
0.5
−1.1
+ specials
5.9
24.2
–
0.1
−1.0
+ specials, σ=16 lattice, mean
−0.6
1.6
–
1.0
−0.4
+ specials, 256 hubs (LMSA)
0.2
0.9
–
0.1
0.0
Late, 9 and 17–23
Appendix
Table S14: Diffuse layers in isolation . AUC@5 lost against dense when only the named layers are sparse; development scenes, 5 seeds, † : 2 seeds. “mean”: AVGGT’s count-biased mean token. Bold: the key sets of LMSA.
Removed set
fr1
court
ETH-o
Mass
Correspondence
22.1
49.2
17.7
0.03–0.04
Heaviest
6.5
23.0
6.0
0.07–0.11
Highest lift
29.7
58.2
42.5
0.01–0.14
Random
0.1
−0.3
0.0
–
Appendix
Table S15: Which keys matter in layers 10–16. AUC@5 lost against dense when a key set is removed from layers 10–16 only (5 seeds, low- ρ development scenes). “Correspondence”: the model’s own 3×3 correspondence windows. “Heaviest”: as many of the highest-attention non-correspondence keys. “Highest lift”: as many keys ranked by A(t,j)/πj . “Random”: as many random keys. Mass: head-averaged attention mass removed (for highest lift: range over layers). Bold: the two largest losses per scene.
Arm
fr1
court
ETH-o
fr3
7S
low- ρ
LMSA-fast with one group dense
layers 0–8 dense
0.36
2.10
1.30
5.91
1.61
1.25
layers 9–10 dense
0.91
5.30
2.51
7.66
1.55
2.91
layers 11–16 dense
0.10
−0.93
1.17
0.00
0.77
0.11
layers 17–23 dense
−0.28
6.11
2.29
4.68
1.67
2.71
only the named layers sparse
Appendix
Table S16: Composition study: swap-backs and single groups . AUC@5 lost against dense, development scenes, seeds 0–1. Top: LMSA-fast with one group returned to dense attention. Bottom: only the named group sparse. Bold: in the TUM long-office and pooled low- ρ columns, the matching-layer rows have the lowest loss of the top block and the highest of the bottom block.
Layers 11–16
fr1
court
ETH-o
fr3
7S
low- ρ
exact (LMSA)
1.41
0.24
0.81
0.14
0.55
0.82
σ=4 + blocks †
4.06
2.88
7.01
0.95
1.93
4.65
σ=2 + blocks
2.76
1.21
2.68
0.37
0.58
2.22
13–15 exact
7.10
0.80
3.62
0.41
1.27
3.84
only 11–16 sparse †
5.65
1.25
6.83
0.91
1.41
4.57
Appendix
Table S17: AUC@30 lost against dense in the composition study (development scenes; † : seeds 0–1). Bold: lowest loss per column.
Scene
s0
s1
s2
s3
s4
TUM fr1_room
0.52
−0.32
0.81
7.26
−1.81
ETH3D courtyard
−3.51
1.65
8.19
2.46
−3.31
ETH3D office
−0.09
2.42
−1.13
−0.17
3.55
TUM fr3_long_office
−2.50
2.50
−0.52
1.90
2.18
7-Scenes office
1.01
0.52
0.36
0.20
−0.32
Appendix
Table S18: LMSA per seed (AUC@5 lost against dense, seeds 0–4, masked implementation). Bold: the two largest seed-level losses.
Layers 11–16
fr1
court
ETH-o
fr3
7S
low- ρ
G 100
exact (LMSA)
1.29
1.10
0.92
0.71
0.35
1.10
4.76
σ=4 + blocks (fast) †
−1.21
11.55
1.73
5.56
1.79
4.02
3.14
σ=9 + blocks †
6.41
16.77
3.38
7.44
2.96
8.85
2.82
σ=2 + blocks
1.99
4.92
1.32
2.06
0.56
2.74
3.77
13–15 exact, rest σ=4
4.10
3.69
1.09
2.15
0.86
2.96
4.05
14 exact, rest σ=4 †
0.36
10.28
4.33
4.50
1.53
4.99
3.44
Appendix
Table S19: Design study (development scenes): AUC@5 lost against dense when layers 11–16 are exact or sparse and all other layers use hubs. fr1: TUM fr1_room; court: ETH3D courtyard; ETH-o: ETH3D office; fr3: TUM fr3_long_office; 7S: 7-Scenes office; low- ρ : pooled over the first three; G 100 : GFLOP per token at N=100 . 5 seeds, † : seeds 0–1. “Blocks”: routed correspondence blocks. Red: above the pre-registered acceptance threshold (4 on the low- ρ scenes, 3 on the others). Further arms: Table S16 . Bold: the frozen recipe and its pooled low- ρ loss, the lowest of all arms.
Figure S2: Break-even precision p⋆=1/(1+r) of a routed slot ( Eq. S3 ) against the benefit-to-harm ratio r=og/H . Black dots show the development scenes. Horizontal lines show measured slot precisions; above the curve, switching a slot on pays. On courtyard the per-slot rule predicts a net loss for always-on slots ( −1.05 AUC@5, Table S20 ), so the switch helps by removing wrong slots. On 7-Scenes office it should be off.
Signal
Slots kept
Visible kept
Right-slot precision, fr1 / court / ETH-o
Wrong / query
Predicted net
Always on
1.00
1.00
0.06 / 0.09 / 0.12
24.3
−1.05
Mutual nearest neighbour at layer 9
0.21
0.38
0.20 / 0.20 / 0.36
3.9
+2.63
Layer-12 attention top-1 agrees with hit
0.25
0.36
0.20 / 0.20 / 0.33
5.5
+1.98
same, inside the epipolar band
0.29
0.43
0.23 / 0.20 / 0.36
6.0
+2.53
Mutual NN and layer-12 agreement
0.07
0.17
0.41 / 0.35 / 0.59
1.2
+1.93
Hit lands on a non-hub (in-degree ≤2 )
0.35
0.53
visibility 0.32
7.4
+2.2
Appendix
Table S20: Switch signals on saved layer-9 router outputs (courtyard unless stated). “Right-slot precision” counts a slot as right if its frame sees the point and its window holds the target. The last column is the net AUC@5 change on courtyard predicted by the break-even model, not a measurement. The vote-peak gate is a block-level rule; for it, and for the in-degree rule, only visibility precision is available. Bold: the rule used by LMSA-fast, the predicted net loss of always-on slots, and the highest predicted net among the signals with a right-slot precision.
Layers 11–16
Keys
FLOPs
Loss
Δ low- ρ
Δ high- ρ
Exact (LMSA)
100%
2.58 ×
1.38
–
–
Merge 0.2
80%
2.86 ×
1.86
−0.49 [ −1.24 , 0.12]
+0.06 [ −0.35 , 0.49]
Merge 0.5
50%
3.49 ×
3.00
−1.62 [ −2.53 , −0.80 ]
+0.22 [ −0.20 , 0.67]
Merge 0.8
20%
4.25 ×
7.11
−5.74 [ −7.57 , −3.93 ]
−0.43 [ −1.14 , 0.13]
Top- k frames
21–27%
4.34 ×
5.60
−4.22 [ −5.70 , −2.91 ]
−0.75 [ −1.77 , 0.33]
Shared hubs (1024)
own frame + 1024
5.40 ×
29.17
−27.79 [ −34.62 , −21.22 ]
−14.88 [ −23.04 , −7.49 ]
Appendix
Table S21: Thinner matching layers (post hoc), N=32 , 5 seeds. Keys: share of keys kept in layers 11–16. FLOPs: reduction against dense attention at N=100 . Loss: AUC@5 lost against dense attention on the 13 unseen low- ρ scenes. Δ : paired AUC@5 difference to LMSA (negative = less accurate) with hierarchical 95% CIs. Red: CI below zero. Bold: the lowest and highest loss, and the only thinner arm that is not significantly worse on the low- ρ scenes.
Layer
Peaked share
neff (peaked)
k50
k90
Cross-frame
Band ±9
3×3 window
neff (diffuse)
10 (hubs)
0.22
3480
491
5806
0.93
0.27
0.016
10231
11
0.77
1121
158
2061
0.88
0.32
0.045
10537
12
0.79
1510
223
2540
0.90
0.38
0.044
8972
13
0.73
1138
172
1734
0.87
0.47
0.079
12590
14
0.91
284
48
349
0.85
0.56
0.120
10437
15
0.87
1007
171
1302
0.89
0.43
0.050
11332
Appendix
Table S22: Breadth of attention in the matching layers (post hoc; 5 development scenes × 2 seeds, 2048 random queries, all heads). Peaked: (query, head) pairs with Γrow≥1.5 . neff : exponential of the attention entropy, median. k50 , k90 : median number of keys holding 50% and 90% of the mass. Band and window: share of the cross-frame mass inside the ground-truth epipolar band ( ±9 patches, 16% of cross-frame keys) and the 3×3 correspondence window (0.17%). Bold: the most concentrated matching layer, which still needs hundreds of keys to hold most of its mass.
Development (5)
Unseen low- ρ (13)
Exact layers
Loss
Δ
Loss
Δ
9–14
7.60
+7.07 [2.70, 13.10]
8.51
+7.14 [5.36, 9.18]
10–15
2.89
+2.36 [0.80, 4.89]
4.19
+2.81 [1.68, 4.11]
11–16 (LMSA)
0.53
–
1.38
–
12–17
3.03
+2.50 [0.31, 4.73]
2.72
+1.34 [0.49, 2.31]
13–18
6.09
+5.56 [0.72, 12.31]
4.61
+3.24 [1.91, 4.85]
Appendix
Table S23: Equal-cost windows of six exact layers (AUC@5 lost against dense attention, mean and 95% hierarchical-bootstrap CI; Δ : arm minus layers 11–16). Development: 5 scenes × 5 seeds; test: the 13 unseen low- ρ scenes × 5 seeds. All arms: 2.99 GFLOP per token at N=32 . Bold: the selected window, which has the lowest loss.
Exact layers
Dev. (5)
Test (13)
11–16 ( Γˉ , s2 , HX , neff )
0.53
1.38
0, 1, 4, 8, 22, 23 (mass)
25.21
28.91
Δ
+24.68 [13.76, 39.68]
+27.54 [21.03, 34.42]
Appendix
Table S24: Mass-selected against Γˉ -selected exact layers (AUC@5 lost against dense attention, mean and 95% hierarchical-bootstrap CI; Δ : arm minus layers 11–16). Development: 5 scenes × 5 seeds; test: the 13 unseen low- ρ scenes × 5 seeds. GFLOP per token at N=32 : 2.993 (11–16) and 3.005 (mass). Bold: the layers selected by Γˉ , which have the lower loss.
Arm
Loss
Δ vs GPA-VGGT
Δ vs LMSA
AbsRel
GFLOP/tok
GPA-VGGT
2.42
–
−1.04
.0465
2.96
LMSA
1.38
+1.04 [ −0.34 , 2.54]
–
.0468
2.99
GPA+sel
1.21
+1.20 [0.41, 2.12]
+0.16 [ −0.73 , 1.00]
.0451
3.10
GPA+hubs
2.26
+0.15 [ −0.30 , 0.60]
−0.89 [ −2.33 , 0.31]
.0494
2.96
GPA+sel+hubs
1.13
+1.28 [0.03, 2.55]
+0.24 [ −0.41 , 0.90]
.0480
3.10
Appendix
Table S25: The selector applied to GPA-VGGT (post hoc), 13 unseen low- ρ scenes, N=32 , 5 seeds. Loss: AUC@5 lost against dense attention; Δ : paired AUC@5 difference (positive = the arm is more accurate), with hierarchical 95% CIs. Bold: the lowest loss and AbsRel, and the gain of GPA+sel over GPA-VGGT, the main result of this test.
Component
Variant
Dev loss
Low- ρ test loss
GFLOP/token
LMSA (frozen recipe)
0.54 [ −0.61 , 1.81]
1.38 [0.71, 2.01]
2.99
Early layers 0–8
own frame only
8.41 [0.10, 22.26]
–
2.97
own frame + special tokens
5.69 [0.24, 14.18]
–
2.97
+ count-biased mean token
0.95 [ −0.69 , 3.03]
–
2.97
Hub estimator
uniform proposal (weights kept)
1.34 [0.00, 3.64]
3.34 [2.26, 4.74]
2.97
no log-weight correction
24.22 [9.73, 38.90]
36.17 [25.88, 46.31]
2.99
Appendix
Table S26: Component ablation , N=32 , 5 seeds. AUC@5 lost against dense attention, with hierarchical 95% CIs. Dev: 5 development scenes. Test: the 13 unseen low- ρ scenes (post hoc). Each row changes one component of the frozen recipe (first row). Bold: the frozen recipe and the lowest test loss.
Figure S3: Post-hoc robustness analysis (not pre-registered). (a) AUC@5 of LMSA minus that of GPA-VGGT on the 21 test scenes (filled) and the 5 development scenes (open) of Table 2 against the scene’s redundancy ρ (log axis); shaded: ρ<10.5 ; dashed lines and box: mean on each side of the split and Spearman correlation with a bootstrap 95% CI, over the test scenes only. (b) The same against SAF3R. (c) Loss against dense attention on the 13 unseen low- ρ scenes: worst scene (bar) and 90th-percentile scene (tick).
Unseen low- ρ (13 scenes)
All 26 scenes
Method
mean
90th pct. scene
worst scene
95th pct. run
worst 5% of runs
worst scene
AVGGT ( σ=4 )
10.3
13.1
24.2 (terrace)
20.9
25.2
37.5 (courtyard)
AVGGT ( σ=9 )
18.9
25.3
44.8 (terrace)
38.0
47.0
50.4 (courtyard)
GPA-VGGT
2.4
4.9
6.8 (terrace)
6.3
10.2
6.8 (terrace)
SAF3R
4.5
10.9
16.7 (electro)
18.3
22.6
22.0 (courtyard)
LMSA
1.4
2.5
2.9 (electro)
5.2
5.9
2.9 (electro)
Appendix
Table S27: Tail losses (post hoc): AUC@5 lost against dense attention. Scene losses use 5-seed means; run losses use single scene-seed runs (65 on the unseen low- ρ scenes; “worst 5%” is the mean of the 4 worst). Dense loses 0 by definition. Bold: lowest per column.
Component
Levels
Var.
SD
Share
Scene
5
0.00 †
0.00
0% [0, 23]
Frame subset
10
1.00
1.00
16% [0, 66]
Hub draw
30
5.21
2.28
84% [34, 100]
Total per run
30
6.21
2.49
100%
Hub draw, more hubs ‡
30
1.62
1.27
37% [19, 100]
Appendix
Table S28: Variance components of the per-run AUC@5 loss of LMSA: 5 development scenes × 2 frame subsets × 3 hub draws, nested method of moments; share with a bootstrap 95% interval over scenes. † Negative estimate ( −0.70 ) truncated at 0. ‡ With more hubs; share of that arm’s total (4.41). Bold: the dominant component, the hub draw, and its SD with more hubs.
Group ( π3 layers)
Γˉ
Predicted
Loss share
Orig. π3 ( Γˉ )
early (0–6)
0.76
diffuse
0.6%
diffuse (0.82)
middle (7)
1.42
diffuse
0%
matching (1.54)
middle (8–12)
1.64
diffuse ∗
14.6%
diffuse ∗ (1.70)
late (13–17)
2.96
matching
84.8%
matching (3.02)
Appendix
Table S29: Selector on π3 . Γˉ (group mean) and predicted role for the Pi3X release, and for the original π3 checkpoint (last column); loss share: share of the summed positive AUC@5 loss when only that group uses hubs (original checkpoint, 5 development scenes × 3 subsets). Groups are mapped from VGGT’s by relative depth (VGGT 0–8 / 9–10 / 11–16 / 17–23). Predictions were written and hashed before any sparse run. Bold: groups labelled matching, and the observed loss share of the late group.
Feed-forward 3D vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass. However, their quadratic global attention makes long image sequences expensive, while existing sparse methods may favor highly attended yet value-redundant regions. To address these limitations, we introduce VASC, a training-free sparse attention method combining value-aware block selection and execution-aware cross-layer memory. Our value-aware block selection integrates pooled query--key relevance with neighboring value contrast, reducing redundancy while preserving query-relevant and distinctive content. Cross-layer memory tracks unserved demand across layers and updates this state according to actual execution, enabling previously underserved blocks to compete under a fixed computation budget. Experiments on 7Scenes and NeuralRGB-D with VGGT and π3 demonstrate improved pose estimation and reconstruction quality compared with FasterVGGT, together with up to 2.29× faster inference than dense VGGT. Code is available at https://github.com/kosakayamahoo-design/VASC.
Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators exploit token, frame, or attention redundancy; we investigate how layer-wise functional roles can guide compression. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Without any training, RegimeVGGT achieves a 6.7x speedup over the memory-optimized VGGT* at matched reconstruction quality. Project page: https://tracyou07.github.io/regimevggt/
Jinhao You, Shuo Lyu, Zhuohang Lyu +5
University of Pennsylvania · University of California, Irvine · Nanyang Technological University
We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T3 (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a 1k image collection in just 54 seconds, achieving a 11.6× speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.