HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT

Yongsung Kim; Wooseok Song; Jaihyun Lew; Hun Hwangbo; Jaehoon Lee; Sungroh Yoon

cs.CVMar 26, 2026

HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT

Authors: Yongsung Kim, Wooseok Song, Jaihyun Lew, Hun Hwangbo, Jaehoon Lee, Sungroh Yoon

Organizations: IPAI, Seoul National University · ECE, Seoul National University · AIIS, ASRI, INMC, ISRC, Seoul National University

Abstract

Visual Geometry Grounded Transformer (VGGT) has advanced 3D vision, yet its global attention layers suffer from quadratic computational costs that hinder scalability. Several sparsification-based acceleration techniques have been proposed to alleviate this issue, but they often suffer from substantial accuracy degradation. We hypothesize that the accuracy degradation stems from the heterogeneity in head-wise sparsification sensitivity, as the existing methods apply a uniform sparsity pattern across all heads. Motivated by this hypothesis, we present a two-stage sparsification pipeline that effectively quantifies and exploits headwise sparsification sensitivity. In the first stage, we measure head-wise sparsification sensitivity using a novel metric, the Head Sensitivity Score (HeSS), which approximates the Hessian with respect to two distinct error terms on a small calibration set. In the inference stage, we perform HeSS-Guided Sparsification, leveraging the pre-computed HeSS to reallocate the total attention budget-assigning denser attention to sensitive heads and sparser attention to more robust ones. We demonstrate that HeSS effectively captures head-wise sparsification sensitivity and empirically confirm that attention heads in the global attention layers exhibit heterogeneous sensitivity characteristics. Extensive experiments further show that our method effectively mitigates performance degradation under high sparsity, demonstrating strong robustness across varying sparsification levels. Code is available at https://github.com/libary753/HeSS.

Figures & tables

Figure 1: Comparison with SparseVGGT [ 33 ] . SparseVGGT suffers from severe performance degradation because it applies uniform sparsity across all heads, overly sparsifying sensitive ones. Our HeSS-Guided Sparsification preserves these highly sensitive heads, retaining performance even under high sparsity.

Figure 2: Pipeline Overview. Our pipeline consists of two stages: (a) Calibration stage, which computes Head Sensitivity Score (HeSS) of all attention heads in VGGT’s Global Attention layers. The Hessian with respect to two errors, the camera pose error $e_{\text{cam}}$ and the point cloud error $e_{\text{pc}}$ , is used to compute HeSS. HeSS is obtained from a calibration set, and these scores are fixed during the inference stage. (b) Inference stage, based on HeSS, different masking ratios are assigned throughout each attention head.

Figure 3: HeSS-Guided Budget Reallocation. The total budget ( $C_{\text{total}}$ ) is obtained by summing baseline per-head budgets ( $c_{h_{n}}$ ) (top). This budget is then reallocated based on HeSS scores ( red ). An iterative Budget Capping process redistributes surplus ( orange ) from heads exceeding their capacity to the remaining uncapped heads, yielding the final budget ( $c_{h_{n}}^{\text{final}}$ , right).

Figure 4: HeSS Distribution in VGGT. The horizontal axis represents the Global Attention (GA) layers from GA 1 to GA 21. Each layer contains two columns, corresponding to $\mathrm{HeSS}_{\text{cam}}$ (Cam) and $\mathrm{HeSS}_{\text{pc}}$ (PC). The vertical axis lists the attention heads (H1–H16). Darker colors indicate higher sensitivity.

Figure 5: Error visualization on DTU [ 12 ] For each scene, we visualize predicted point clouds from VGGT [ 34 ] , SparseVGGT [ 33 ] , and our method at various sparsity levels. Points whose 3D error exceeds 5 mm from the DTU ground truth are highlighted in green. Our method produces fewer highlighted points than SparseVGGT, reflecting a more stable reconstruction as sparsity increases.

Figure 6: Comparison with SparseVGGT. We evaluate performance across various sparsity levels. On CO3Dv2 [ 23 ] , we measure camera pose estimation (AUC@30

\uparrow

, ATE

\downarrow

). On DTU [ 12 ] , we measure MVS performance (Chamfer Distance

\downarrow

Figure 7: Comparison with ViT [ 6 ] Sparsification. Our method significantly outperforms baselines [ 20 , 15 ] across all sparsity levels, maintaining robust performance even at extreme sparsity.

Figure 8: Generalization to

\mathrm{\pi}^{3}

[ 37 ] . Performance comparison across sparsity levels on the

\mathrm{\pi}^{3}

. Our method shows improved robustness at higher sparsity levels compared to SparseVGGT [ 33 ]

Figure 9: Reversed HeSS Ranking. Pruning sensitive heads (Reverse) triggers catastrophic failure, whereas HeSS maintains robust geometry by preserving high-impact heads.

Figure 10: Effect of budget reallocation. Performance comparison with and without reallocation under the same sparsity budget. Without reallocation, surplus budgets remain unused, leading to degraded performance.

	Chamfer. $\downarrow$	Acc. $\downarrow$	Comp. $\downarrow$
$W^{h}_{K}$	1.9167	3.4498	0.3842
$W^{h}_{V}$	1.9661	3.5400	0.3921
Linear	1.8398	3.2717	0.4078
Softmax	1.9685	3.5536	0.3834
Log	2.1413	3.8816	0.4010
Ours	1.6030	2.8392	0.3667

Table 1: Ablation Study. We compare different design choices for computing and combining

\mathrm{HeSS}_{\text{cam}}

and

\mathrm{HeSS}_{\text{pc}}

. The top block evaluates the use of alternative projection matrices (

W^{h}_{K}

and

W^{h}_{V}

), while the bottom block compares combination strategies for

\mathrm{HeSS}_{\text{cam}}

and

\mathrm{HeSS}_{\text{pc}}

. Our final configuration (Ours) achieves the best performance.

Method	$\rho$	$\tau$	Sparsity	Runtime
VGGT [ 34 ]	-	-	-	10.35
SparseVGGT [ 33 ]	0.5	0.9	0.43	8.42
Ours	0.5	0.9	0.43	8.37
SparseVGGT [ 33 ]	0.8	0.4	0.73	6.59
Ours	0.8	0.4	0.73	6.58

Table 2: Runtime Analysis. Comparison of inference runtime on an A40 GPU using the CO3Dv2 [ 23 ] dataset. At identical sparsity levels (0.43 and 0.73), HeSS-Guided Sparsification incurs negligible computational overhead.

Figure 11: Ablation of HeSS weight

\lambda

. Performance comparison across sparsity levels for

\lambda\!=\!1

(HeSS

{}_{\text{cam}}

only, blue ),

\lambda\!=\!0

(HeSS

{}_{\text{pc}}

only, green ), and

\lambda\!=\!0.5

(Ours, red ). Ours achieves the most stable performance across the entire sparsity range.

Figure S1 : Dataset sensitivity of HeSS calibration. The performance remains largely unchanged regardless of the dataset used for calibration, indicating low sensitivity to the choice of calibration data.

Figure S2 : HeSS Distribution in $\mathbf{\pi}^{3}$ [ 37 ] . The horizontal axis represents the Global Attention (GA) layers from GA 1 to GA 18. Each layer contains two columns, corresponding to $\mathrm{HeSS}_{\text{cam}}$ (Cam) and $\mathrm{HeSS}_{\text{pc}}$ (PC). The vertical axis lists the attention heads (H1–H16). Dark colors indicate higher sensitivity.

Figure S3 : Analysis of HeSS weight

\lambda

\pi^{3}

[ 37 ] . Performance comparison across sparsity levels for

\lambda\!=\!1

(HeSS

{}_{\text{cam}}

only, blue ),

\lambda\!=\!0

(HeSS

{}_{\text{pc}}

only, green ), and

\lambda\!=\!0.5

(Both, red ). In contrast to VGGT [ 34 ] ,

\pi^{3}

achieves its highest accuracy with HeSS

{}_{\text{pc}}

only.

Figure S4 : Error visualization on DTU [ 12 ] For each scene, we visualize predicted point clouds from VGGT [ 34 ] , SparseVGGT [ 33 ] , and our method at various sparsity levels. Points whose 3D error exceeds 5 mm from the DTU ground truth are highlighted in green. Our method produces fewer highlighted points than SparseVGGT, reflecting a more stable reconstruction as sparsity increases.

Figure S5 : Error visualization on DTU [ 12 ] For each scene, we visualize predicted point clouds from VGGT [ 34 ] , SparseVGGT [ 33 ] , and our method at various sparsity levels. Points whose 3D error exceeds 5 mm from the DTU ground truth are highlighted in green. Our method produces fewer highlighted points than SparseVGGT, reflecting a more stable reconstruction as sparsity increases.

			DTU (Chamfer Dist.)		CO3Dv2 (AUC@30)		CO3Dv2 (ATE)
Sparse Ratio	CDF Thr.	Sparsity	SparseVGGT	Ours	SparseVGGT	Ours	SparseVGGT	Ours
0.1	0.97	0.07	1.2109	1.2156	0.9545	0.9560	0.1498	0.1493
0.3	0.97	0.10	1.1908	1.1996	0.9542	0.9558	0.1493	0.1476
0.5	0.97	0.11	1.2618	1.2050	0.9539	0.9560	0.1520	0.1453
0.4	0.95	0.14	1.2281	1.2219	0.9517	0.9553	0.1493	0.1444
0.6	0.93	0.19	1.2164	1.2419	0.9482	0.9516	0.1608	0.1550

Table S1: Comparison of SparseVGGT [ 33 ] and our method with the VGGT [ 34 ] backbone. Performance is evaluated on DTU [ 12 ] (Chamfer Distance) and CO3Dv2 [ 23 ] (AUC@30, ATE) across varying sparsity levels.

			DTU (Chamfer Dist.)		CO3Dv2 (AUC@30)		CO3Dv2 (ATE)
Sparse Ratio	CDF Thr.	Sparsity	SparseVGGT	Ours	SparseVGGT	Ours	SparseVGGT	Ours
0.1	0.97	0.08	1.3320	1.3151	0.9722	0.9728	5.581	5.4822
0.3	0.97	0.11	1.3515	1.3217	0.9713	0.9726	5.7365	5.6254
0.5	0.97	0.13	1.3536	1.3286	0.9709	0.9716	5.8080	5.6833
0.4	0.95	0.16	1.3680	1.3260	0.9704	0.9722	5.9064	5.7517
0.6	0.93	0.22	1.3962	1.3559	0.9688	0.9708	6.1670	5.9403

Table S2: Comparison of SparseVGGT [ 33 ] and our method with the

\pi^{3}

[ 37 ] backbone. Performance is evaluated on DTU [ 12 ] (Chamfer Distance) and CO3Dv2 [ 23 ] (AUC@30, ATE) across varying sparsity levels.

Sparse Attention

Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.

See all 243 papers

ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers
Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +33D ViTs Efficient ViTs
IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University
RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer
Jun 16, 2026Jinhao You, Shuo Lyu, Zhuohang Lyu +53D ViTs Multi-View 3D Reconstruction
University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1
PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers
May 8, 2026Haotang Li, Zhenyu Qi, Shaohan Henry Wang +5Multi-View Geometry Efficient ViTs
University of Arizona · East Carolina University · Augusta University +1

Sep 28, 2026J/K move · Enter open · S save

ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

Yongsung Kim, Jaehoon Lee, Minjun Park, Wooseok Song+2

IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University

3D ViTs Efficient ViTs Sparse Attention Block-Sparse Attention

Open paper AskPDF arXiv

3D vision transformers such as VGGT predict camera poses and scene geometry from multi-view images in a single forward pass, but their global attention over all concatenated view tokens dominates computation as the number of views grows. To reduce this cost, SparseVGGT and HeSS sparsify attention at the block level, and both retain blocks with high attention probability. However, we observe that attention probability poorly predicts how much the model's behavior actually changes when a block is removed, and we show that this mismatch is why performance collapses as sparsity increases. In this paper, we propose ReSS (ReSidual-ReStoring Sparse Attention), which recasts block selection from a problem of maximizing the retained attention mass to one of minimizing the drift that sparsification leaves in the residual stream. We introduce a drift score that quantifies how much each block shifts the residual, and, since the drift of a drop set depends on the directions of the contribution vectors rather than on their magnitudes alone, an iterative residual restoration procedure that refines the drop set as a whole. Across three backbones and five datasets, ReSS preserves dense performance better than prior methods at matched sparsity. Two further results support drift as the quantity that governs the cost of sparsification: maximizing drift degrades performance faster than random selection, and plotted against realized drift instead of sparsity, all methods fall approximately onto a single curve. Code is available at https://github.com/libary753/ReSS.

Abstract

Sparse Attention

Explore similar work

ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers