Layer-Matched Sparse Attention for Low-Redundancy 3D Reconstruction
Organizations: BRAC University
Abstract
VGGT reconstructs cameras and geometry from unposed images in one pass, but its global attention grows quadratically with the number of views. Training-free accelerators thin this attention and work well on redundant video, but lose pose accuracy when views share little geometry. This paper traces these losses to two kinds of cross-view information that a uniform sparse approximation can fail to preserve together: diffuse layers aggregate distributed context, whereas matching layers depend on query-specific interactions that carry little attention mass. A per-layer statistic measures the log-mass missed by a mean token. Its row-free proxy is the logit variance, and it selects the estimator for each layer. Diffuse layers use shared, importance-sampled hub keys, with log-weights that make the softmax numerator and denominator unbiased. Matching layers stay exact: every aggressive thinning tested cost accuracy. At equal cost, the selected layers outperform the four shifted windows tested. On 13 unseen low-redundancy scenes with 32 views, LMSA stays within 1.4 AUC@5 of dense attention, compared with 10.3 for AVGGT and 4.5 for SAF3R; its difference from contemporaneous GPA-VGGT is not significant, and only its gains over AVGGT pass both scene-level tests. With 100 views, it stays within 1.9 AUC@5 while its aggregator runs faster. Used post hoc to choose the layers GPA-VGGT protects, the statistic improves it by 1.2 AUC@5 for 5% more FLOPs. Applied untuned to , it beats AVGGT but remains further from dense accuracy.
Figures & tables
| Scene ( ) | E (0–8) | B (10–16) | L (9, 17–23) | all |
|---|---|---|---|---|
| TUM fr1_room (4.3) | 5.6 | 11.7 | 13.3 | |
| ETH3D courtyard (7.7) | 33.1 | 28.5 | 5.4 | 50.2 |
| ETH3D office (5.5) | 1.5 | 6.0 | 3.5 | 12.3 |
| TUM fr3_long_office (15.4) | 0.5 | 4.4 | 14.8 | 22.1 |
| 7-Scenes office (13.0) | 0.8 | 1.6 | 1.8 |
| Unseen low- (13 scenes) | Unseen high- (8 scenes) | Dev. (5) | ||||||
| Method | AUC@5 | AUC@30 | AbsRel | AUC@5 | AUC@30 | AbsRel | AUC@5 | GFLOP/tok. |
| Dense VGGT-1B | 57.4 | 85.4 | 4.34 | 72.9 | 92.6 | 3.95 | 49.2 | 5.29 |
| AVGGT ( ) [ 40 ] | 47.1 | 81.1 | 4.08 | 67.4 | 91.4 | 3.87 | 35.2 | 2.64 |
| AVGGT ( ) [ 40 ] | 38.5 | 72.8 | 4.32 | 62.4 | 90.5 | 3.79 | 29.2 | 2.38 |
| GPA-VGGT [ 41 ] | 55.0 | 83.8 | 4.65 | 72.8 | 92.6 | 3.93 | 47.8 | 2.96 |
| SAF3R [ 12 ] | 52.8 | 84.3 | 5.48 | 71.1 | 92.2 | 3.80 | 41.5 | 2.60 |
| GFLOP / token | Speed-up vs. dense | T&T loss | ||||
| Method | 100 | 32 | 64 | 100 | ||
| Dense | 5.29 | 12.25 | 1.00 | 1.00 | 1.00 | 0 |
| AVGGT ( ) | 2.64 | 3.77 | 1.54 | 1.99 | 2.41 | 7.10 |
| AVGGT ( ) | 2.38 | 2.94 | 1.63 | 2.22 | 2.78 | 12.61 |
| GPA-VGGT | 2.96 | 4.79 | 1.40 | 1.73 | 2.00 | 1.61 |
| SAF3R | 2.60 | 3.67 | 1.34 | 1.80 | 2.19 | 5.65 |
Appendix figures & tables32 assets
Supplementary material from the paper’s appendix.
Appendix
| Layers | Key set per query (query in frame ) | Units |
| 0–8 | Own frame (all tokens) + special tokens of every frame + 256 hubs importance-sampled from a 512-query pilot (head-mean full attention, column mass, floor ), log-weight ; hubs in the own frame get no weight | 1.6 |
| 9, 10 | Own frame + special tokens + 1024 hubs (same estimator) | 2.4 |
| 11–16 | Exact dense attention (the layers with ) | |
| 17–23 | As layers 9–10 | 2.4 |
| LMSA-fast, 11–16 | Eq. S2 : frame 0 + special tokens + lattice + diagonal + mean token + switched blocks | 9–10 |
| LMSA-fast, router | Once, on layer 9’s attention input; top-4 cosine per (query, frame) at full width; tiles; -smoothed displacement mode; switch: share of the tile’s top-1 votes in the smoothed peak |
| Layer | Tube | Role | ||||
|---|---|---|---|---|---|---|
| 0 | 0.12 | – | – | – | – | hubs, 256 |
| 1 | 0.05 | – | – | – | – | hubs, 256 |
| 2 | 0.30 | – | – | – | – | hubs, 256 |
| 3 | 0.71 | – | – | – | – | hubs, 256 |
| 4 | 0.78 | – | – | – | – | hubs, 256 |
| 5 | 0.86 | – | – | – | – | hubs, 256 |
| Method | Original order | Random order | Change |
|---|---|---|---|
| LMSA | 1.57 | 1.65 | [ , 1.21] |
| GPA-VGGT † | 3.01 | 2.83 | [ , 1.35] |
| AVGGT(4) | 10.44 | 12.31 | [ , 4.95] |
| Scene | Method | AUC@5 | AUC@30 | AbsRel |
|---|---|---|---|---|
| TUM fr1_room | dense | 29.15 | 77.90 | 0.0418 |
| SAF3R | 23.02 | 73.50 | 0.0414 | |
| GPA-VGGT | 26.94 | 76.13 | 0.0429 | |
| ETH3D courtyard | dense | 54.40 | 90.22 | 0.0198 |
| SAF3R | 30.97 | 80.34 | 0.0245 | |
| GPA-VGGT | 41.37 | 84.69 | 0.0222 |
| Test | Plan hash (prefix) | Fixed |
|---|---|---|
| predictions, newer release | 20947731e395 | Nov 13, 15:02 |
| predictions, original checkpoint | d338191c0597 | Nov 13, 15:44 |
| transfer evaluation | 5ed86d7cbc44 | Nov 13, 18:19 |
| Selector applied to GPA-VGGT | 1d5d99725695 | Nov 13, 21:37 |
| 7-Scenes long sequences | d360a911d5f0 | Nov 13, 21:52 |
| Confirmatory test (first version) | 2ee8727dd5a1 | Nov 14, 04:31 |
| Method | AUC@5 loss, low- | AUC@30 loss, low- | AUC@5 loss, high- | AbsRel , low- |
|---|---|---|---|---|
| AVGGT ( ) | 10.27 [7.98, 13.22] | 4.30 [2.25, 6.85] | 5.50 [1.40, 9.53] | -0.26 |
| AVGGT ( ) | 18.91 [14.64, 24.51] | 12.63 [7.73, 18.59] | 10.50 [4.01, 16.65] | -0.02 |
| GPA-VGGT | 2.42 [1.28, 3.71] | 1.67 [0.50, 3.09] | 0.08 [ , 0.42] | +0.31 |
| SAF3R | 4.53 [1.73, 7.73] | 1.09 [ , 2.73] | 1.73 [0.60, 3.20] | +1.14 |
| LMSA | 1.38 [0.71, 2.01] | 0.70 [0.17, 1.26] | 0.73 [0.25, 1.27] | +0.34 |
| LMSA-fast | 4.65 [2.74, 6.72] | 1.54 [0.71, 2.49] | 1.81 [0.24, 3.67] | -0.37 |
| Split | Scene | Dense | AVGGT(4) | AVGGT(9) | GPA-VGGT | SAF3R | LMSA | LMSA-fast |
|---|---|---|---|---|---|---|---|---|
| Unseen low- | ETH3D delivery_area | 25.1 | 14.8 | 8.9 | 24.1 | 24.4 | 24.6 | 19.0 |
| ETH3D electro | 61.2 | 48.1 | 35.5 | 58.0 | 44.5 | 58.3 | 54.6 | |
| ETH3D kicker | 90.2 | 84.8 | 79.0 | 89.1 | 88.6 | 88.4 | 86.7 | |
| ETH3D playground | 25.8 | 17.9 | 9.0 | 20.9 | 21.5 | 24.0 | 25.8 | |
| ETH3D relief | 23.5 | 13.1 | 4.5 | 18.6 | 21.2 | 22.7 | 19.0 | |
| ETH3D relief_2 | 10.1 | 4.9 | 1.7 | 7.0 | 7.3 | 9.0 | 7.2 |
| Dataset | Scenes | Dense AUC@5 | AVGGT(4) | AVGGT(9) | GPA-VGGT | SAF3R | LMSA | LMSA-fast |
|---|---|---|---|---|---|---|---|---|
| ETH3D | 8 (8) | 43.4 | ||||||
| Tanks and Temples | 5 (4) | 84.1 | ||||||
| TUM RGB-D | 3 (1) | 82.1 | ||||||
| 7-Scenes | 3 (0) | 38.4 | ||||||
| DTU | 2 (0) | 99.9 | ||||||
| RealEstate10K outdoor ∗ | 9 (0) | 33.2 | – |
| A (cameras) | B (points) | |
| LMSA dense | [ , 0.029] | [ , 0.015] |
| LMSA-fast dense | [ , ] | [ , 0.095] |
| GPA-VGGT dense | [0.018, 0.130] | [ , 0.035] |
| SAF3R dense | [0.031, 0.098] | [0.005, 0.038] |
| AVGGT(4) dense | [0.033, 0.219] | – |
| AVGGT(9) dense | [0.108, 0.541] | – |
| GFLOP / token | Agg. time (s) | Full time (s) | |||||
| Method | 100 | 32 | 64 | 100 | 32 | 64 | |
| Dense | 5.29 | 12.25 | 5.27 | 15.02 | 31.32 | 5.71 | 16.05 |
| AVGGT ( ) | 2.64 | 3.77 | 3.43 | 7.54 | 12.98 | 3.90 | 8.63 |
| AVGGT ( ) | 2.38 | 2.94 | 3.22 | 6.76 | 11.26 | 3.72 | 7.88 |
| GPA-VGGT | 2.96 | 4.79 | 3.76 | 8.68 | 15.69 | 4.03 | 9.41 |
| SAF3R | 2.60 | 3.67 | 3.94 | 8.34 | 14.28 | 4.41 | 9.34 |
| Method | chess | office | fire | kitchen | Speed-up | ||
|---|---|---|---|---|---|---|---|
| AVGGT(4) | 3.48 | 5.1 | |||||
| AVGGT(9) | 3.30 | 8.1 | |||||
| LMSA | 1.05 | 3.3 | |||||
| LMSA-auto | 0.85 | 3.4 |
| Accurate dense | Efficient LMSA | Draw SD of AUC@5 | |||||
|---|---|---|---|---|---|---|---|
| Dataset | AUC@5 | AUC@30 | AUC@5 | AUC@30 | Acc. | Eff. | LMSA |
| NRGBD | [ , ] | [ , 0.94] | 0.68 | 1.44 | 1.40 | ||
| Sintel | [ , 0.02] | [ , 0.93] | 1.30 | 2.01 | 2.04 | ||
| 7-Scenes | [ , 0.48] | [ , 2.44] | 0.79 | 0.85 | 1.11 | ||
| Baseline | Mean | Ahead | Wilcoxon (Holm) | Sign-flip (Holm) |
|---|---|---|---|---|
| AVGGT ( ) | +8.89 | 13/13 | 0.001 (0.001) | 0.001 (0.001) |
| AVGGT ( ) | +17.54 | 13/13 | 0.001 (0.001) | 0.001 (0.001) |
| GPA-VGGT | +1.04 | 8/13 | 0.340 (0.340) | 0.165 (0.165) |
| SAF3R | +3.15 | 11/13 | 0.017 (0.034) | 0.064 (0.128) |
| Layers and key set | fr1 | court | ETH-o | fr3 | 7S |
|---|---|---|---|---|---|
| Early, 0–8 (own frame exact) | |||||
| frame-local (AVGGT) | 5.6 | 33.1 | 1.5 | 0.5 | |
| + specials | 5.9 | 24.2 | – | 0.1 | |
| + specials, lattice, mean | 1.6 | – | 1.0 | ||
| + specials, 256 hubs (LMSA) | 0.2 | 0.9 | – | 0.1 | 0.0 |
| Late, 9 and 17–23 | |||||
| Removed set | fr1 | court | ETH-o | Mass |
|---|---|---|---|---|
| Correspondence | 22.1 | 49.2 | 17.7 | 0.03–0.04 |
| Heaviest | 6.5 | 23.0 | 6.0 | 0.07–0.11 |
| Highest lift | 29.7 | 58.2 | 42.5 | 0.01–0.14 |
| Random | 0.1 | 0.0 | – |
| Arm | fr1 | court | ETH-o | fr3 | 7S | low- |
|---|---|---|---|---|---|---|
| LMSA-fast with one group dense | ||||||
| layers 0–8 dense | 0.36 | 2.10 | 1.30 | 5.91 | 1.61 | 1.25 |
| layers 9–10 dense | 0.91 | 5.30 | 2.51 | 7.66 | 1.55 | 2.91 |
| layers 11–16 dense | 0.10 | 1.17 | 0.00 | 0.77 | 0.11 | |
| layers 17–23 dense | 6.11 | 2.29 | 4.68 | 1.67 | 2.71 | |
| only the named layers sparse | ||||||
| Layers 11–16 | fr1 | court | ETH-o | fr3 | 7S | low- |
|---|---|---|---|---|---|---|
| exact (LMSA) | 1.41 | 0.24 | 0.81 | 0.14 | 0.55 | 0.82 |
| + blocks † | 4.06 | 2.88 | 7.01 | 0.95 | 1.93 | 4.65 |
| + blocks | 2.76 | 1.21 | 2.68 | 0.37 | 0.58 | 2.22 |
| 13–15 exact | 7.10 | 0.80 | 3.62 | 0.41 | 1.27 | 3.84 |
| only 11–16 sparse † | 5.65 | 1.25 | 6.83 | 0.91 | 1.41 | 4.57 |
| Scene | s0 | s1 | s2 | s3 | s4 |
|---|---|---|---|---|---|
| TUM fr1_room | 0.52 | 0.81 | 7.26 | ||
| ETH3D courtyard | 1.65 | 8.19 | 2.46 | ||
| ETH3D office | 2.42 | 3.55 | |||
| TUM fr3_long_office | 2.50 | 1.90 | 2.18 | ||
| 7-Scenes office | 1.01 | 0.52 | 0.36 | 0.20 |
| Layers 11–16 | fr1 | court | ETH-o | fr3 | 7S | low- | G 100 |
|---|---|---|---|---|---|---|---|
| exact (LMSA) | 1.29 | 1.10 | 0.92 | 0.71 | 0.35 | 1.10 | 4.76 |
| + blocks (fast) † | 11.55 | 1.73 | 5.56 | 1.79 | 4.02 | 3.14 | |
| + blocks † | 6.41 | 16.77 | 3.38 | 7.44 | 2.96 | 8.85 | 2.82 |
| + blocks | 1.99 | 4.92 | 1.32 | 2.06 | 0.56 | 2.74 | 3.77 |
| 13–15 exact, rest | 4.10 | 3.69 | 1.09 | 2.15 | 0.86 | 2.96 | 4.05 |
| 14 exact, rest † | 0.36 | 10.28 | 4.33 | 4.50 | 1.53 | 4.99 | 3.44 |
| Signal | Slots kept | Visible kept | Right-slot precision, fr1 / court / ETH-o | Wrong / query | Predicted net |
|---|---|---|---|---|---|
| Always on | 1.00 | 1.00 | 0.06 / 0.09 / 0.12 | 24.3 | |
| Mutual nearest neighbour at layer 9 | 0.21 | 0.38 | 0.20 / 0.20 / 0.36 | 3.9 | |
| Layer-12 attention top-1 agrees with hit | 0.25 | 0.36 | 0.20 / 0.20 / 0.33 | 5.5 | |
| same, inside the epipolar band | 0.29 | 0.43 | 0.23 / 0.20 / 0.36 | 6.0 | |
| Mutual NN and layer-12 agreement | 0.07 | 0.17 | 0.41 / 0.35 / 0.59 | 1.2 | |
| Hit lands on a non-hub (in-degree ) | 0.35 | 0.53 | visibility 0.32 | 7.4 |
| Layers 11–16 | Keys | FLOPs | Loss | low- | high- |
|---|---|---|---|---|---|
| Exact (LMSA) | 100% | 2.58 | 1.38 | – | – |
| Merge 0.2 | 80% | 2.86 | 1.86 | [ , 0.12] | [ , 0.49] |
| Merge 0.5 | 50% | 3.49 | 3.00 | [ , ] | [ , 0.67] |
| Merge 0.8 | 20% | 4.25 | 7.11 | [ , ] | [ , 0.13] |
| Top- frames | 21–27% | 4.34 | 5.60 | [ , ] | [ , 0.33] |
| Shared hubs (1024) | own frame + 1024 | 5.40 | 29.17 | [ , ] | [ , ] |
| Layer | Peaked share | (peaked) | Cross-frame | Band | window | (diffuse) | ||
|---|---|---|---|---|---|---|---|---|
| 10 (hubs) | 0.22 | 3480 | 491 | 5806 | 0.93 | 0.27 | 0.016 | 10231 |
| 11 | 0.77 | 1121 | 158 | 2061 | 0.88 | 0.32 | 0.045 | 10537 |
| 12 | 0.79 | 1510 | 223 | 2540 | 0.90 | 0.38 | 0.044 | 8972 |
| 13 | 0.73 | 1138 | 172 | 1734 | 0.87 | 0.47 | 0.079 | 12590 |
| 14 | 0.91 | 284 | 48 | 349 | 0.85 | 0.56 | 0.120 | 10437 |
| 15 | 0.87 | 1007 | 171 | 1302 | 0.89 | 0.43 | 0.050 | 11332 |
| Development (5) | Unseen low- (13) | |||
| Exact layers | Loss | Loss | ||
| 9–14 | 7.60 | [2.70, 13.10] | 8.51 | [5.36, 9.18] |
| 10–15 | 2.89 | [0.80, 4.89] | 4.19 | [1.68, 4.11] |
| 11–16 (LMSA) | 0.53 | – | 1.38 | – |
| 12–17 | 3.03 | [0.31, 4.73] | 2.72 | [0.49, 2.31] |
| 13–18 | 6.09 | [0.72, 12.31] | 4.61 | [1.91, 4.85] |
| Exact layers | Dev. (5) | Test (13) |
|---|---|---|
| 11–16 ( , , , ) | 0.53 | 1.38 |
| 0, 1, 4, 8, 22, 23 (mass) | 25.21 | 28.91 |
| [13.76, 39.68] | [21.03, 34.42] |
| Arm | Loss | vs GPA-VGGT | vs LMSA | AbsRel | GFLOP/tok |
|---|---|---|---|---|---|
| GPA-VGGT | 2.42 | – | .0465 | 2.96 | |
| LMSA | 1.38 | [ , 2.54] | – | .0468 | 2.99 |
| GPA+sel | 1.21 | +1.20 [0.41, 2.12] | [ , 1.00] | .0451 | 3.10 |
| GPA+hubs | 2.26 | [ , 0.60] | [ , 0.31] | .0494 | 2.96 |
| GPA+sel+hubs | 1.13 | [0.03, 2.55] | [ , 0.90] | .0480 | 3.10 |
| Component | Variant | Dev loss | Low- test loss | GFLOP/token |
|---|---|---|---|---|
| LMSA (frozen recipe) | 0.54 [ , 1.81] | 1.38 [0.71, 2.01] | 2.99 | |
| Early layers 0–8 | own frame only | 8.41 [0.10, 22.26] | – | 2.97 |
| own frame + special tokens | 5.69 [0.24, 14.18] | – | 2.97 | |
| + count-biased mean token | 0.95 [ , 3.03] | – | 2.97 | |
| Hub estimator | uniform proposal (weights kept) | 1.34 [0.00, 3.64] | 3.34 [2.26, 4.74] | 2.97 |
| no log-weight correction | 24.22 [9.73, 38.90] | 36.17 [25.88, 46.31] | 2.99 | |
| Unseen low- (13 scenes) | All 26 scenes | |||||
|---|---|---|---|---|---|---|
| Method | mean | 90th pct. scene | worst scene | 95th pct. run | worst 5% of runs | worst scene |
| AVGGT ( ) | 10.3 | 13.1 | 24.2 (terrace) | 20.9 | 25.2 | 37.5 (courtyard) |
| AVGGT ( ) | 18.9 | 25.3 | 44.8 (terrace) | 38.0 | 47.0 | 50.4 (courtyard) |
| GPA-VGGT | 2.4 | 4.9 | 6.8 (terrace) | 6.3 | 10.2 | 6.8 (terrace) |
| SAF3R | 4.5 | 10.9 | 16.7 (electro) | 18.3 | 22.6 | 22.0 (courtyard) |
| LMSA | 1.4 | 2.5 | 2.9 (electro) | 5.2 | 5.9 | 2.9 (electro) |
| Component | Levels | Var. | SD | Share |
|---|---|---|---|---|
| Scene | 5 | 0.00 † | 0.00 | 0% [0, 23] |
| Frame subset | 10 | 1.00 | 1.00 | 16% [0, 66] |
| Hub draw | 30 | 5.21 | 2.28 | 84% [34, 100] |
| Total per run | 30 | 6.21 | 2.49 | 100% |
| Hub draw, more hubs ‡ | 30 | 1.62 | 1.27 | 37% [19, 100] |
| Group ( layers) | Predicted | Loss share | Orig. ( ) | |
|---|---|---|---|---|
| early (0–6) | 0.76 | diffuse | 0.6% | diffuse (0.82) |
| middle (7) | 1.42 | diffuse | 0% | matching (1.54) |
| middle (8–12) | 1.64 | diffuse ∗ | 14.6% | diffuse ∗ (1.70) |
| late (13–17) | 2.96 | matching | 84.8% | matching (3.02) |
3D Reconstruction
3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.
See all 1,025 papers