VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction

Junyi Wu; Fanqing Kong; Leyang Chen; Shaoqiu Zhang; Yulun Zhang

cs.CVOct 1, 2026

VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction

Authors: Junyi Wu, Fanqing Kong, Leyang Chen, Shaoqiu Zhang, Yulun Zhang

Organizations: Shanghai Jiao Tong University

Abstract

Feed-forward 3D vision models such as VGGT have achieved remarkable progress, unifying camera estimation and dense scene reconstruction in a single pass. However, their quadratic global attention makes long image sequences expensive, while existing sparse methods may favor highly attended yet value-redundant regions. To address these limitations, we introduce VASC, a training-free sparse attention method combining value-aware block selection and execution-aware cross-layer memory. Our value-aware block selection integrates pooled query--key relevance with neighboring value contrast, reducing redundancy while preserving query-relevant and distinctive content. Cross-layer memory tracks unserved demand across layers and updates this state according to actual execution, enabling previously underserved blocks to compete under a fixed computation budget. Experiments on 7Scenes and NeuralRGB-D with VGGT and $π^3$ demonstrate improved pose estimation and reconstruction quality compared with FasterVGGT, together with up to $2.29\times$ faster inference than dense VGGT. Code is available at https://github.com/kosakayamahoo-design/VASC.

Figures & tables

Figure 1: Reconstruction quality with sparse attention. Left: RGB references and aligned stride-10 reconstructions; red boxes mark shared regions. For each scene, FasterVGGT and VASC use the same sparsity. RGB and rendered views are not pixel-aligned. Right: dataset-level metrics normalized to dense VGGT. Higher values indicate lower errors or higher normal consistency (NC).

Figure 2: Quality–latency comparison. VGGT at stride 10. Higher NC and lower latency are better.

Figure 3: Attention priority and local value variation. (a) Dense and attention-only reconstruction with 25% of connections retained. (b) Attention and neighboring-value similarity reveal different spatial patterns; lower similarity indicates greater directional variation.

Figure 4: VASC overview. Value contrast and cross-layer memory jointly guide block selection within a fixed budget. Actual execution updates memory for the next layer.

Method	Sparsity (%)	Stride	7Scenes						NeuralRGB-D
			ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
VGGT
Dense	0	10	0.069	0.031	0.859	0.017	0.028	0.652	0.030	0.016	0.179	0.014	0.015	0.908
FasterVGGT	65	10	0.081	0.035	0.960	0.017	0.029	0.647	0.059	0.026	0.427	0.034	0.024	0.832
VASC	65	10	0.073	0.033	0.911	0.018	0.028	0.650	0.044	0.019	0.239	0.018	0.017	0.880
FasterVGGT	75	10	0.084	0.038	1.080	0.018	0.028	0.646	0.073	0.030	0.503	0.041	0.027	0.798

Table 1: Comparison at strides 10 and 5. Sequence means on 7Scenes (18 sequences) and NRGBD (9). Bold marks the better unrounded value within each FasterVGGT–VASC pair at the same backbone, stride, and sparsity. FasterVGGT ( Wang et al., 2026a ) is adapted to

{\pi}^{3}

in its rows. Dense is the unpruned reference; OOM: out of memory.

Figure 5: Qualitative reconstruction at sparsity 75. Kitchen and Green Room from NRGBD, and Heads from 7Scenes, using full stride-10 sequences. Each method shows an overview and a detail crop; yellow boxes mark shared regions. Cameras and crop bounds are shared after rigid alignment to GT.

GPU	Sparsity (%)	Dense (s)	FasterVGGT (s)	VASC (s)	Speedup	Overhead
A6000	75	23.849	11.981	12.239	$1.950\times$	2.150%
	85	23.849	10.506	10.739	$2.220\times$	2.220%
RTX 4090	75	15.631	7.256	7.446	$2.100\times$	2.620%
	85	15.631	6.650	6.822	$2.290\times$	2.600%

Table 2: Full-model efficiency on NeuralRGB-D. Mean seconds per complete stride-10 sequence. Speedup is relative to dense VGGT; overhead is relative to FasterVGGT ( Wang et al., 2026a ) .

Backbone	Dataset	Setting	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
VGGT	7Scenes	FasterVGGT	0.084	0.038	1.080	0.018	0.028	0.646
		Value-aware only	0.078	0.034	0.957	0.018	0.028	0.649
		Faster+memory	0.078	0.035	0.975	0.018	0.028	0.648
		Full VASC	0.074	0.034	0.963	0.018	0.028	0.649
VGGT	NRGBD	FasterVGGT	0.073	0.030	0.503	0.041	0.027	0.798
		Value-aware only	0.045	0.020	0.278	0.020	0.018	0.871

Table 3: S75 component ablation. Frame stride 10. FasterVGGT results are from Table 1 . Bold marks the best unrounded value within each backbone–dataset group.

Figure 6: Component trends across sparsity. Relative improvements over FasterVGGT at stride 10; higher is better. Shading marks gains over the stronger single component. Each panel uses its own y-axis scale.

Host	Dataset	Configuration	Acc $\downarrow$	Comp $\downarrow$	NC1 $\uparrow$	NC2 $\uparrow$
FastVGGT	7Scenes	Official	0.018	0.027	0.616	0.648
		Memory adapter	0.018	0.027	0.617	0.650
AVGGT-R	7Scenes	w/o memory	0.018	0.028	0.647	0.650
		w/ memory	0.017	0.027	0.646	0.651
AVGGT-R	NRGBD	w/o memory	0.020	0.018	0.844	0.841
		w/ memory	0.019	0.018	0.849	0.842

Table 4: Memory adaptation across hosts. All 18 7Scenes and nine NRGBD sequences at stride 10; settings in Section 5.3 . FastVGGT follows Shen et al. (2025) . Bold marks the better value in each pair.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Region	Keys	Retained	Mean attention percentile	Mean V cosine
A	45	9	54.500	0.610
B	32	26	83.400	0.810
C	42	8	43.900	0.660

Appendix

Table 5: Regional attention, value similarity, and actual selection. Statistics for the spatial boxes in Figure 3 (b). Retained counts refer to query–key connections in the illustrated head, not tokens omitted from all network computation.

Dataset	Method	S	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Gain
7Scenes	Dense VGGT	0	0.069	0.031	0.859	0.017	0.028	0.652	–
	FasterVGGT	S55	0.079	0.033	0.922	0.017	0.028	0.650	–
	VASC	S55	0.071	0.032	0.895	0.018	0.028	0.650	$+2.500\%$ (5/6)
	FasterVGGT	S65	0.081	0.035	0.960	0.017	0.029	0.647	–
	VASC	S65	0.073	0.033	0.911	0.018	0.028	0.650	$+3.600\%$ (5/6)
	FasterVGGT	S75	0.084	0.038	1.080	0.018	0.028	0.646	–

Appendix

Table 6: Complete stride-10 comparison on VGGT and

{\pi}^{3}

. Dense is the unpruned reference for each backbone. FasterVGGT ( Wang et al., 2026a ) denotes the original baseline on VGGT and our adaptation on

{\pi}^{3}

. Sparse baseline/VASC pairs have matched realized capacity; bold marks the better raw value. Gain is VASC’s arithmetic mean over six direction-normalized relative changes, with metric wins in parentheses. Every row uses complete native sequences (18/18 for 7Scenes, 9/9 for NRGBD).

Dataset	Method	S	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Gain
7Scenes	FasterVGGT	S55	0.080	0.024	0.650	0.016	0.027	0.611	–
	VASC	S55	0.072	0.022	0.609	0.016	0.027	0.612	$+3.000\%$ (4/6)
	FasterVGGT	S65	0.081	0.024	0.682	0.016	0.027	0.610	–
	VASC	S65	0.073	0.023	0.623	0.016	0.027	0.612	$+4.100\%$ (5/6)
	FasterVGGT	S75	0.084	0.027	0.757	0.017	0.027	0.608	–
	VASC	S75	0.075	0.024	0.670	0.017	0.027	0.611	$+6.200\%$ (6/6)

Appendix

Table 7: Complete stride-5 comparison across sparsity. FasterVGGT ( Wang et al., 2026a ) and VASC are paired at identical realized block capacity. Bold marks the better raw value within each pair. Gain uses the same six-metric signed mean as Table 6 , with wins in parentheses. All sequences are native and uncapped (18/18 for 7Scenes, 9/9 for NRGBD).

Method	ATE	RPE-t	RPE-r	ARE	Chamfer
Dense VGGT	0.092	0.055	1.206	3.950	0.441
FasterVGGT, S75	0.119	0.077	1.590	5.012	0.447
VASC, S75	0.102	0.066	1.303	4.253	0.447
VASC wins vs. FasterVGGT	46/50	48/50	50/50	42/50	23/50

Appendix

Table 8: ScanNet-50 quality. Macro means across all 50 scenes; 100 frames per scene. All metrics are lower-is-better.

Method	Stride	7Scenes			NeuralRGB-D
		Acc Med $\downarrow$	Comp Med $\downarrow$	NC Med $\uparrow$	Acc Med $\downarrow$	Comp Med $\downarrow$	NC Med $\uparrow$
FastVGGT		0.008	0.010	0.716	0.011	0.010	0.793
VASC (S65)	10	0.007	0.010	0.733	0.011	0.007	0.974
VASC (S75)		0.007	0.009	0.732	0.012	0.008	0.969
FastVGGT		0.008	0.010	0.709	0.012	0.010	0.790
VASC (S65)	5	0.006	0.008	0.674	0.018	0.008	0.945

Appendix

Table 9: Reconstruction medians versus FastVGGT. Sequence-averaged point-level medians on all 18/9 sequences. FastVGGT ( Shen et al., 2025 ) uses merge ratio 0.9; VASC uses VGGT. Bold: best unrounded value per stride and dataset.

GPU	Dataset	S	Dense (s)	FasterVGGT (s)	VASC (s)	Speedup	Overhead (%)
A6000	7Scenes	55	14.865	9.951	10.124	1.470 $\times$	1.740
A6000	7Scenes	65	14.865	9.175	9.347	1.590 $\times$	1.870
A6000	7Scenes	75	14.865	8.373	8.528	1.740 $\times$	1.850
A6000	NRGBD	55	23.849	14.830	15.076	1.580 $\times$	1.660
A6000	NRGBD	65	23.849	13.402	13.666	1.750 $\times$	1.970
A6000	NRGBD	75	23.849	11.981	12.239	1.950 $\times$	2.150

Appendix

Table 10: Full-model forward latency. Seconds per complete stride-10 sequence, averaged over the dataset. Speedup is Dense/VASC; overhead is VASC/FasterVGGT

-1

( Wang et al., 2026a ) . Each sequence uses one warm-up and the median of three measured forwards. Only complete, uncontaminated matched comparisons are included.

S	Variant	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
55	Value-aware only	0.075	0.032	0.886	0.017	0.028	0.650
	Faster+memory	0.070	0.031	0.873	0.017	0.028	0.650
	Full VASC	0.071	0.032	0.895	0.018	0.028	0.650
65	Value-aware only	0.076	0.033	0.910	0.017	0.028	0.649
	Faster+memory	0.074	0.033	0.927	0.018	0.028	0.649
	Full VASC	0.073	0.033	0.911	0.018	0.028	0.650

Appendix

Table 11: VGGT components on 7Scenes. Frame stride 10.

S	Variant	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
55	Value-aware only	0.036	0.018	0.228	0.017	0.016	0.888
	Faster+memory	0.038	0.018	0.212	0.016	0.016	0.884
	Full VASC	0.036	0.018	0.203	0.016	0.016	0.888
65	Value-aware only	0.041	0.019	0.254	0.018	0.017	0.880
	Faster+memory	0.043	0.020	0.253	0.020	0.019	0.871
	Full VASC	0.044	0.019	0.239	0.018	0.017	0.880

Appendix

Table 12: VGGT components on NeuralRGB-D. Frame stride 10.

S	Variant	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
55	Value-aware only	0.060	0.025	0.801	0.024	0.025	0.755
	Faster+memory	0.059	0.025	0.774	0.021	0.024	0.795
	Full VASC	0.058	0.024	0.774	0.021	0.024	0.800
65	Value-aware only	0.061	0.025	0.811	0.025	0.025	0.746
	Faster+memory	0.063	0.026	0.803	0.023	0.025	0.768
	Full VASC	0.060	0.025	0.789	0.022	0.025	0.786

Appendix

Table 13:

{\pi}^{3}

components on 7Scenes. Frame stride 10.

S	Variant	ATE $\downarrow$	RPE-t $\downarrow$	RPE-r $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
55	Value-aware only	0.042	0.016	0.194	0.036	0.026	0.852
	Faster+memory	0.042	0.015	0.160	0.026	0.022	0.918
	Full VASC	0.037	0.014	0.154	0.025	0.021	0.927
65	Value-aware only	0.054	0.017	0.217	0.040	0.028	0.836
	Faster+memory	0.056	0.017	0.196	0.034	0.025	0.869
	Full VASC	0.046	0.015	0.173	0.029	0.024	0.900

Appendix

Table 14:

{\pi}^{3}

components on NeuralRGB-D. Frame stride 10.

Variant	Acc $\downarrow$	Comp $\downarrow$	NC1 $\uparrow$	NC2 $\uparrow$
Official, mean	0.018	0.027	0.616	0.648
Adapted, mean	0.018	0.027	0.617	0.650
Official, median	0.008	0.010	0.682	0.750
Adapted, median	0.008	0.010	0.684	0.754

Appendix

Table 15: FastVGGT adaptation on 7Scenes. Frame stride 10. Means are macro averages of per-sequence statistics; “median” rows first take the within-sequence median, then average across sequences.

Explore similar work

ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers
Sep 28, 2026Yongsung Kim, Jaehoon Lee, Minjun Park +3Visual Geometry Grounded Transformer Dynamic Sparse Attention
IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University
Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
May 10, 2026Zichen Zou, Xiaosong Jia, Zuxuan Wu +1Visual Geometry Grounded Transformer 3D Reconstruction
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI
RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer
Jun 16, 2026Jinhao You, Shuo Lyu, Zhuohang Lyu +5Visual Geometry Grounded Transformer Self-Supervised Vision Transformers
University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1

Sep 28, 2026, cs.CVJ/K move · Enter open · S save

ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

Yongsung Kim, Jaehoon Lee, Minjun Park, Wooseok Song+2

IPAI, 2ECE, 3AIIS, ASRI, INMC, ISRC Seoul National University · ECE Seoul National University · AIIS Seoul National University

Visual Geometry Grounded Transformer Dynamic Sparse Attention Self-Supervised Vision Transformers Sparse-View Vision Transformer Residual Stream

Open paper AskPDF arXiv

3D vision transformers such as VGGT predict camera poses and scene geometry from multi-view images in a single forward pass, but their global attention over all concatenated view tokens dominates computation as the number of views grows. To reduce this cost, SparseVGGT and HeSS sparsify attention at the block level, and both retain blocks with high attention probability. However, we observe that attention probability poorly predicts how much the model's behavior actually changes when a block is removed, and we show that this mismatch is why performance collapses as sparsity increases. In this paper, we propose ReSS (ReSidual-ReStoring Sparse Attention), which recasts block selection from a problem of maximizing the retained attention mass to one of minimizing the drift that sparsification leaves in the residual stream. We introduce a drift score that quantifies how much each block shifts the residual, and, since the drift of a drop set depends on the directions of the contribution vectors rather than on their magnitudes alone, an iterative residual restoration procedure that refines the drop set as a whole. Across three backbones and five datasets, ReSS preserves dense performance better than prior methods at matched sparsity. Two further results support drift as the quantity that governs the cost of sparsification: maximizing drift degrades performance faster than random selection, and plotted against realized drift instead of sparsity, all methods fall approximately onto a single curve. Code is available at https://github.com/libary753/ReSS.

Abstract

Explore similar work

ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers

Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer