Visual Geometry Grounded Transformer (VGGT) has advanced 3D vision, yet its global attention layers suffer from quadratic computational costs that hinder scalability. Several sparsification-based acceleration techniques have been proposed to alleviate this issue, but they often suffer from substantial accuracy degradation. We hypothesize that the accuracy degradation stems from the heterogeneity in head-wise sparsification sensitivity, as the existing methods apply a uniform sparsity pattern across all heads. Motivated by this hypothesis, we present a two-stage sparsification pipeline that effectively quantifies and exploits headwise sparsification sensitivity. In the first stage, we measure head-wise sparsification sensitivity using a novel metric, the Head Sensitivity Score (HeSS), which approximates the Hessian with respect to two distinct error terms on a small calibration set. In the inference stage, we perform HeSS-Guided Sparsification, leveraging the pre-computed HeSS to reallocate the total attention budget-assigning denser attention to sensitive heads and sparser attention to more robust ones. We demonstrate that HeSS effectively captures head-wise sparsification sensitivity and empirically confirm that attention heads in the global attention layers exhibit heterogeneous sensitivity characteristics. Extensive experiments further show that our method effectively mitigates performance degradation under high sparsity, demonstrating strong robustness across varying sparsification levels. Code is available at https://github.com/libary753/HeSS.
Figures & tables
Figure 1: Comparison with SparseVGGT [ 33 ] . SparseVGGT suffers from severe performance degradation because it applies uniform sparsity across all heads, overly sparsifying sensitive ones. Our HeSS-Guided Sparsification preserves these highly sensitive heads, retaining performance even under high sparsity.
Figure 2: Pipeline Overview. Our pipeline consists of two stages: (a) Calibration stage, which computes Head Sensitivity Score (HeSS) of all attention heads in VGGT’s Global Attention layers. The Hessian with respect to two errors, the camera pose error ecam and the point cloud error epc , is used to compute HeSS. HeSS is obtained from a calibration set, and these scores are fixed during the inference stage. (b) Inference stage, based on HeSS, different masking ratios are assigned throughout each attention head.
Figure 3: HeSS-Guided Budget Reallocation. The total budget ( Ctotal ) is obtained by summing baseline per-head budgets ( chn ) (top). This budget is then reallocated based on HeSS scores ( red ). An iterative Budget Capping process redistributes surplus ( orange ) from heads exceeding their capacity to the remaining uncapped heads, yielding the final budget ( chnfinal , right).
Figure 4: HeSS Distribution in VGGT. The horizontal axis represents the Global Attention (GA) layers from GA 1 to GA 21. Each layer contains two columns, corresponding to HeSScam (Cam) and HeSSpc (PC). The vertical axis lists the attention heads (H1–H16). Darker colors indicate higher sensitivity.
Figure 5: Error visualization on DTU [ 12 ] For each scene, we visualize predicted point clouds from VGGT [ 34 ] , SparseVGGT [ 33 ] , and our method at various sparsity levels. Points whose 3D error exceeds 5 mm from the DTU ground truth are highlighted in green. Our method produces fewer highlighted points than SparseVGGT, reflecting a more stable reconstruction as sparsity increases.
Figure 6: Comparison with SparseVGGT. We evaluate performance across various sparsity levels. On CO3Dv2 [ 23 ] , we measure camera pose estimation (AUC@30 ↑ , ATE ↓ ). On DTU [ 12 ] , we measure MVS performance (Chamfer Distance ↓ ).
Figure 7: Comparison with ViT [ 6 ] Sparsification. Our method significantly outperforms baselines [ 20 , 15 ] across all sparsity levels, maintaining robust performance even at extreme sparsity.
Figure 8: Generalization to π3 [ 37 ] . Performance comparison across sparsity levels on the π3 . Our method shows improved robustness at higher sparsity levels compared to SparseVGGT [ 33 ]
Figure 10: Effect of budget reallocation. Performance comparison with and without reallocation under the same sparsity budget. Without reallocation, surplus budgets remain unused, leading to degraded performance.
Chamfer. ↓
Acc. ↓
Comp. ↓
WKh
1.9167
3.4498
0.3842
WVh
1.9661
3.5400
0.3921
Linear
1.8398
3.2717
0.4078
Softmax
1.9685
3.5536
0.3834
Log
2.1413
3.8816
0.4010
Ours
1.6030
2.8392
0.3667
Table 1: Ablation Study. We compare different design choices for computing and combining HeSScam and HeSSpc . The top block evaluates the use of alternative projection matrices ( WKh and WVh ), while the bottom block compares combination strategies for HeSScam and HeSSpc . Our final configuration (Ours) achieves the best performance.
Method
ρ
τ
Sparsity
Runtime
VGGT [ 34 ]
-
-
-
10.35
SparseVGGT [ 33 ]
0.5
0.9
0.43
8.42
Ours
8.37
SparseVGGT [ 33 ]
0.8
0.4
0.73
6.59
Ours
6.58
Table 2: Runtime Analysis. Comparison of inference runtime on an A40 GPU using the CO3Dv2 [ 23 ] dataset. At identical sparsity levels (0.43 and 0.73), HeSS-Guided Sparsification incurs negligible computational overhead.
Figure 11: Ablation of HeSS weight λ . Performance comparison across sparsity levels for λ=1 (HeSS cam only, blue ), λ=0 (HeSS pc only, green ), and λ=0.5 (Ours, red ). Ours achieves the most stable performance across the entire sparsity range.
Figure S1 : Dataset sensitivity of HeSS calibration. The performance remains largely unchanged regardless of the dataset used for calibration, indicating low sensitivity to the choice of calibration data.
Figure S2 : HeSS Distribution in π3 [ 37 ] . The horizontal axis represents the Global Attention (GA) layers from GA 1 to GA 18. Each layer contains two columns, corresponding to HeSScam (Cam) and HeSSpc (PC). The vertical axis lists the attention heads (H1–H16). Dark colors indicate higher sensitivity.
Figure S3 : Analysis of HeSS weight λ on π3 [ 37 ] . Performance comparison across sparsity levels for λ=1 (HeSS cam only, blue ), λ=0 (HeSS pc only, green ), and λ=0.5 (Both, red ). In contrast to VGGT [ 34 ] , π3 achieves its highest accuracy with HeSS pc only.
Figure S4 : Error visualization on DTU [ 12 ] For each scene, we visualize predicted point clouds from VGGT [ 34 ] , SparseVGGT [ 33 ] , and our method at various sparsity levels. Points whose 3D error exceeds 5 mm from the DTU ground truth are highlighted in green. Our method produces fewer highlighted points than SparseVGGT, reflecting a more stable reconstruction as sparsity increases.
Figure S5 : Error visualization on DTU [ 12 ] For each scene, we visualize predicted point clouds from VGGT [ 34 ] , SparseVGGT [ 33 ] , and our method at various sparsity levels. Points whose 3D error exceeds 5 mm from the DTU ground truth are highlighted in green. Our method produces fewer highlighted points than SparseVGGT, reflecting a more stable reconstruction as sparsity increases.
DTU (Chamfer Dist.)
CO3Dv2 (AUC@30)
CO3Dv2 (ATE)
Sparse Ratio
CDF Thr.
Sparsity
SparseVGGT
Ours
SparseVGGT
Ours
SparseVGGT
Ours
0.1
0.97
0.07
1.2109
1.2156
0.9545
0.9560
0.1498
0.1493
0.3
0.97
0.10
1.1908
1.1996
0.9542
0.9558
0.1493
0.1476
0.5
0.97
0.11
1.2618
1.2050
0.9539
0.9560
0.1520
0.1453
0.4
0.95
0.14
1.2281
1.2219
0.9517
0.9553
0.1493
0.1444
0.6
0.93
0.19
1.2164
1.2419
0.9482
0.9516
0.1608
0.1550
Table S1: Comparison of SparseVGGT [ 33 ] and our method with the VGGT [ 34 ] backbone. Performance is evaluated on DTU [ 12 ] (Chamfer Distance) and CO3Dv2 [ 23 ] (AUC@30, ATE) across varying sparsity levels.
DTU (Chamfer Dist.)
CO3Dv2 (AUC@30)
CO3Dv2 (ATE)
Sparse Ratio
CDF Thr.
Sparsity
SparseVGGT
Ours
SparseVGGT
Ours
SparseVGGT
Ours
0.1
0.97
0.08
1.3320
1.3151
0.9722
0.9728
5.581
5.4822
0.3
0.97
0.11
1.3515
1.3217
0.9713
0.9726
5.7365
5.6254
0.5
0.97
0.13
1.3536
1.3286
0.9709
0.9716
5.8080
5.6833
0.4
0.95
0.16
1.3680
1.3260
0.9704
0.9722
5.9064
5.7517
0.6
0.93
0.22
1.3962
1.3559
0.9688
0.9708
6.1670
5.9403
Table S2: Comparison of SparseVGGT [ 33 ] and our method with the π3 [ 37 ] backbone. Performance is evaluated on DTU [ 12 ] (Chamfer Distance) and CO3Dv2 [ 23 ] (AUC@30, ATE) across varying sparsity levels.
3D vision transformers such as VGGT predict camera poses and scene geometry from multi-view images in a single forward pass, but their global attention over all concatenated view tokens dominates computation as the number of views grows. To reduce this cost, SparseVGGT and HeSS sparsify attention at the block level, and both retain blocks with high attention probability. However, we observe that attention probability poorly predicts how much the model's behavior actually changes when a block is removed, and we show that this mismatch is why performance collapses as sparsity increases. In this paper, we propose ReSS (ReSidual-ReStoring Sparse Attention), which recasts block selection from a problem of maximizing the retained attention mass to one of minimizing the drift that sparsification leaves in the residual stream. We introduce a drift score that quantifies how much each block shifts the residual, and, since the drift of a drop set depends on the directions of the contribution vectors rather than on their magnitudes alone, an iterative residual restoration procedure that refines the drop set as a whole. Across three backbones and five datasets, ReSS preserves dense performance better than prior methods at matched sparsity. Two further results support drift as the quantity that governs the cost of sparsification: maximizing drift degrades performance faster than random selection, and plotted against realized drift instead of sparsity, all methods fall approximately onto a single curve. Code is available at https://github.com/libary753/ReSS.
Yongsung Kim, Jaehoon Lee, Minjun Park +3
IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University
Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a 6.7x speedup over VGGT* at matched reconstruction quality.
Jinhao You, Shuo Lyu, Zhuohang Lyu +5
University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1
Multi-view geometry transformers are feed-forward 3D foundation models that jointly predict depth maps, point maps, and camera poses for N images in a single forward pass. Most of them build on an alternating-attention (AA) stack whose cross-view attention runs over the patch tokens of all N views in every block, so its cost grows quadratically in N: on 300-frame ScanNetv2 clips, VGGT peaks at 72.3 GiB, and MapAnything does not fit on a 93 GiB card. Existing methods reduce tokens or attention inside the AA stack, so the full patch grid still enters it, and the pre-AA interface, where the encoder hands its tokens to the first AA block, has been overlooked. We propose PrePARE, which prunes patch tokens once at the pre-AA interface and restores the dense grid once after the last AA block. Every block of the frozen AA stack then runs on the reduced tokens, while the prediction heads receive the full grid. A Token Scorer, supervised by how the unpruned AA stack uses each token, and a Feature-guided Restoration module are the only trained parts. The Token Routing module is rule-based. On ScanNetv2 at N = 300, PrePARE reduces the peak memory of VGGT by 80%, from 72.3 to 14.4 GiB, below every in-AA method we ran, and runs 5.4 times faster, while its per-scene Chamfer distance is lower than the unpruned model's on 34 of 50 scenes (sign test p = 0.015). On pi^3 and Depth Anything 3 it saves 15% and 20% of peak memory, where the in-AA methods save at most 9%, and at 1000 frames it is the only method we ran that fits Depth Anything 3 and MapAnything on one GPU.
Haotang Li, Zhenyu Qi, Shaohan Henry Wang +5
University of Arizona · East Carolina University · Augusta University +1