GPA-VGGT: Geometry-Preserving Acceleration of VGGT

Muyeol Sung; Jangwon Kim; Jongwoo Lim

cs.CVOct 27, 2026

GPA-VGGT: Geometry-Preserving Acceleration of VGGT

Authors: Muyeol Sung, Jangwon Kim, Jongwoo Lim

Organizations: Seoul National University

Abstract

Feed-forward geometry models such as VGGT recover camera parameters, depth maps, and point maps from one to hundreds of views in a single forward pass, making them powerful backbones for multi-view 3D perception. However, VGGT's global attention is quadratic in the number of frames and dominates end-to-end runtime on long sequences, which has motivated a range of training-free efficiency methods such as token merging. These methods, however, come with a drop in accuracy that makes them hard to use in place of VGGT. We propose GPA-VGGT, a training-free acceleration method that replaces global attention in early blocks with frame attention augmented by a shared cross-frame K/V set. In the remaining blocks, we merge only keys and values while preserving all queries, restore the attention mass of the tokens each merged key represents, and randomly project the matching metric to 128 dimensions to reduce matching cost. Across diverse multi-view geometry tasks, GPA-VGGT provides training-free acceleration at speedups comparable to prior work while preserving the information required for precise geometric estimation, retaining over 99% of VGGT's pose accuracy under AUC@3.

Figures & tables

Figure 1: Accuracy and runtime scaling with the number of input frames. Although existing acceleration methods substantially reduce end-to-end runtime, they incur pronounced camera-pose degradation under the strict AUC@3 metric. GPA-VGGT achieves comparable runtime while maintaining camera-pose accuracy close to that of vanilla VGGT.

Figure 2: Two sources of accuracy degradation in existing token-merging methods. Top: Query merging collapses the block-12 outputs of $83$ tokens into $12$ shared outputs, whereas retaining all queries while merging only K/V preserves the output diversity of the unmerged model. Bottom: Omitting the $+\log s$ correction disproportionately allocates attention to the first frame. Restoring the correction recovers an attention distribution close to that of the unmerged model.

Figure 3: Effect of cross-frame context in the early blocks. Top: Replacing global attention with frame attention substantially displaces the block-8 output field from the full-global-attention reference, whereas the shared cross-frame K/V set largely preserves it. Bottom: Pairwise alignment of per-frame drift directions. Cross-frame special tokens partially align the drift, and image-token summaries further reduce its variation, preserving a coherent geometric reference across views.

Figure 4: Overview of GPA-VGGT. Early blocks $0$ – $8$ replace global attention with frame attention augmented by a shared cross-frame K/V set containing special tokens and content-adaptive pooled image-token summaries. Later blocks $9$ – $23$ preserve all queries and merge only K/V tokens, using a weaker merge ratio in the core blocks $13$ – $16$ . (a) Pooled summaries are constructed by assigning patch tokens to stride-8 seeds according to cosine similarity and averaging the K/V tokens within each content-adaptive cluster; the summary size determines the $+\log s$ correction. (b) K/V matching is computed after a $128$ -dimensional Gaussian projection, while all queries are retained for global attention.

	CO3D				RE10K
	AUC@3	AUC@5	AUC@10	AUC@30	AUC@3	AUC@5	AUC@10	AUC@30
vanilla VGGT	57.49	68.88	80.66	91.71	44.78	56.67	70.79	85.76
FastVGGT	41.56	55.97	72.61	88.38	37.56	50.52	66.38	83.69
FasterVGGT	16.05	33.80	60.00	83.91	22.00	34.09	51.47	73.85
AVGGT(2)	51.87	64.55	78.02	90.65	44.68	56.49	70.66	85.70
AVGGT(4)	47.08	60.75	75.72	89.72	42.53	54.78	69.50	85.21

Table 1: Camera-pose accuracy on CO3D and RE10K , with ten frames per sequence and relative pose errors computed over all frame pairs. Bold and underline mark the best and second-best value in each column, excluding vanilla VGGT.

Figure 5: Qualitative reconstruction under estimated camera poses. Each column shows the point cloud produced by the corresponding method, and red boxes indicate the enlarged regions below. GPA-VGGT most closely matches the geometry reconstructed by vanilla VGGT.

	AUC@3 $\uparrow$			AUC@30 $\uparrow$
	64	100	200	64	100	200
vanilla VGGT	62.57	62.45	61.47	92.58	92.59	92.42
FastVGGT	55.18	55.97	56.28	91.33	91.54	91.68
FasterVGGT	20.66	20.40	19.59	85.62	85.57	85.42
AVGGT(2)	58.49	58.59	57.71	91.94	92.03	91.85
AVGGT(4)	54.27	54.76	54.40	91.35	91.50	91.45

Table 2: Camera-pose accuracy on CO3D at

64

100

, and

200

input frames. Bold and underline mark the best and second-best value in each column, excluding vanilla VGGT.

	NRGBD sparse			NRGBD dense			7-Scenes sparse			7-Scenes dense
	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$
vanilla VGGT	0.0510	0.0665	0.8843	0.0144	0.0152	0.8743	0.0432	0.0557	0.7337	0.0215	0.0259	0.6673
FastVGGT	0.0560	0.0819	0.8601	0.0308	0.0328	0.8302	0.0510	0.0696	0.7215	0.0195	0.0235	0.6646
FasterVGGT	0.1551	0.2173	0.7749	0.0759	0.0642	0.7988	0.0994	0.1527	0.6637	0.0246	0.0254	0.6695
AVGGT(2)	0.0355	0.0396	0.8981	0.0154	0.0157	0.8562	0.0613	0.0765	0.7115	0.0210	0.0257	0.6653
AVGGT(4)	0.0465	0.0554	0.8818	0.0207	0.0182	0.8466	0.0656	0.0866	0.7063	0.0218	0.0248	0.6665

Table 3: Point-map reconstruction on NRGBD and 7-Scenes , under the sparse and dense frame samplings of

\pi^{3}

. Bold and underline mark the best and second-best value in each column, excluding vanilla VGGT.

	number of input frames
time (s) $\downarrow$	10	32	64	100	200	500
vanilla VGGT	0.84	3.84	12.46	24.01	86.78	513.78
FastVGGT	0.85	2.81	6.46	11.71	33.91	169.43
FasterVGGT	0.78	2.66	6.18	11.29	32.27	OOM
AVGGT(2)	0.78	2.76	6.70	12.66	37.02	184.03
AVGGT(4)	0.76	2.51	5.58	9.86	25.82	111.74

Table 4: End-to-end runtime for the whole forward pass, camera and depth heads included, on a single A6000.

	time $\downarrow$	AUC@3 $\uparrow$
	$200$ f	CO3D	RE10K
vanilla VGGT	86.8 s	57.49	44.78
$+$ frame attention in early blocks	59.1 s	54.43	44.45
$+$ pooled cross-frame broadcast	59.8 s	57.25	45.20
$+$ query-preserving K/V merging	40.1 s	57.13	44.45
on blocks 9–12 and 17–23

Table 5: Ablation of GPA-VGGT components. Each row adds one component to the row above it.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

frames	vanilla	FastVGGT	FasterVGGT	AVGGT	GPA-VGGT
10	4.58	4.58	4.58	4.58	4.58
32	5.12	5.12	5.12	5.12	5.12
64	6.08	6.85	7.48	6.08	6.08
100	8.27	9.48	10.50	8.27	8.28
200	14.37	16.78	19.46	14.37	14.38
300	20.47	24.08	29.43	20.47	20.48

Appendix

Table S1 : Peak GPU memory (GiB) during the full forward pass on a single A6000, measured using torch.cuda.max_memory_allocated() after warm-up. Vanilla denotes the VRAM-efficient VGGT implementation introduced by FastVGGT, and the AVGGT column covers both

\sigma=2

and

\sigma=4

, which produced identical measurements at every length.

	vanilla	mean	stride	cluster	cluster-h
CO3D AUC@3	$57.49$	$56.44$	$56.61$	$56.90$	$56.94$
$\Delta$ vs. vanilla	–	$-1.04$	$-0.88$	$-0.59$	$-0.55$

Appendix

Table S2 : Pooling strategy for the shared cross-frame summaries on CO3D, using the same stride-

8

seed grid and summary-token budget. Bold marks the adopted setting.

	preserve	$+\log s$	output	CO3D
	Q	bias	error	AUC@3
no merging	–	–	–	57.25
(Pooled Cross-Frame Broadcast only)
neither component	–	–	0.394	49.93
query preservation only	✓	–	0.374	51.09
$+\log s$ correction only	–	✓	0.112	55.36

Appendix

Table S3 : Merge-operator components on CO3D, under a fixed merge window and merge ratio. Output error is the relative deviation of the merged attention output from unmerged attention on the same input. Bold marks the adopted operator.

Figure S1 : Core-block selection on VGGT. Each point shows how much the final global-block output changes from the unmerged model when the indicated contiguous block set is protected from merging; lower values mean smaller changes. The criterion is computed without ground-truth annotations across four datasets. Blocks

13

–

16

consistently give the smallest change and are selected as the core-block set.

dimension	32	64	128	256	512	1024
matching (ms)	37	38	64	106	194	469
same destination (%)	54.4	67.1	76.6	82.8	87.9	100
similarity loss (%)	3.30	1.28	0.56	0.28	0.13	0
CO3D AUC@3	56.47	56.48	56.94	56.87	56.89	57.09

Appendix

Table S4 : Projected feature dimension for K/V matching , with matching cost measured at

200

input frames and AUC@3 on CO3D. Same destination and similarity loss are both measured against the full

1{,}024

-dimensional metric. Bold marks the adopted dimension.

	CO3D AUC@ $\tau$				RE10K AUC@ $\tau$				$200$ frames
	$3$	$5$	$10$	$30$	$3$	$5$	$10$	$30$	time (s)	speedup
vanilla	$58.01$	$68.36$	$79.16$	$89.71$	$50.90$	$60.67$	$72.43$	$85.74$	$65.5$	$1.00\times$
$+$ frame attention in early blocks	$56.25$	$67.01$	$78.35$	$89.32$	$50.64$	$60.39$	$72.18$	$85.63$	$35.4$	$1.85\times$
$+$ pooled cross-frame broadcast	$58.06$	$68.42$	$79.18$	$89.66$	$51.34$	$60.85$	$72.42$	$85.67$	$36.5$	$1.80\times$
$+$ query-preserving K/V merging, $r{=}0.50$	$\mathbf{57.29}$	$\mathbf{67.92}$	$\mathbf{78.97}$	$\mathbf{89.56}$	$\mathbf{51.04}$	$\mathbf{60.66}$	$\mathbf{72.35}$	$\mathbf{85.73}$	$\mathbf{27.7}$	$\mathbf{2.36\times}$
$+$ query-preserving K/V merging, $r{=}0.90$	$56.40$	$67.34$	$78.58$	$89.35$	$49.85$	$59.71$	$71.72$	$85.45$	$23.8$	$2.75\times$

Appendix

Table S5 : Ablation of the proposed design on

\pi^{3}

. Bold marks the adopted configuration.

	CO3D AUC@ $\tau$				RE10K AUC@ $\tau$				$200$ frames
	$3$	$5$	$10$	$30$	$3$	$5$	$10$	$30$	time (s)	speedup
vanilla	$54.66$	$66.09$	$77.95$	$89.14$	$63.22$	$71.26$	$80.59$	$90.58$	$41.5$	$1.00\times$
$+$ frame attention in early blocks	$53.98$	$65.59$	$77.62$	$88.97$	$63.01$	$71.09$	$80.47$	$90.57$	$29.4$	$1.41\times$
$+$ pooled cross-frame broadcast	$54.56$	$66.01$	$77.90$	$89.11$	$63.21$	$71.23$	$80.59$	$90.61$	$29.9$	$1.39\times$
$+$ query-preserving K/V merging, $r{=}0.50$	$\mathbf{53.84}$	$\mathbf{65.44}$	$\mathbf{77.52}$	$\mathbf{89.00}$	$\mathbf{62.95}$	$\mathbf{71.05}$	$\mathbf{80.50}$	$\mathbf{90.56}$	$\mathbf{22.5}$	$\mathbf{1.84\times}$
$+$ query-preserving K/V merging, $r{=}0.90$	$52.97$	$64.81$	$77.08$	$88.81$	$62.64$	$70.82$	$80.36$	$90.52$	$19.2$	$2.17\times$

Appendix

Table S6 : Ablation of the proposed design on VGGT-

\Omega

. Bold marks the adopted configuration.

Figure S2 : Core-block selection on

\pi^{3}

and VGGT-

\Omega

. Core blocks are selected using the same measurement as in Figure S1 . The criterion selects blocks

13

–

15

\pi^{3}

and raw blocks

12

13

15

, and

16

on VGGT-

\Omega

, where raw block

14

is a register block and is therefore excluded from merging.

Figure S3 : A small-room scene from 7-Scenes ( heads-seq-02#0 ).

Figure S4 : A green-room scene from NRGBD ( green_room#11 ).

Figure S5 : An outdoor scene from RealEstate10K ( 2c805f56d92a2e22 ).

Efficient Transformer Inference

Efficient Transformer Inference studies methods and systems that reduce the latency, memory, and energy needed to run Transformer models.

See all 121 papers

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer
Jun 16, 2026Jinhao You, Shuo Lyu, Zhuohang Lyu +53D ViTs Multi-View 3D Reconstruction
University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1
VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers
Sep 20, 2026Abteen Arab, Guile Wu, Chengjie Huang +13D ViTs Multi-View 3D Reconstruction
Huawei Noah’s Ark Lab · University of British Columbia
BlazeVGGT: 13x Faster VGGT at 2,000 Frames via Camera Token without Training
Oct 21, 2026Shujun Yao, Junhui Lin, Congying Han +3Multi-View 3D Reconstruction Feed-Forward 3D Reconstruction
University of Chinese Academy of Sciences, Beijing, China · the University of Maryland, College Park, MD, USA · Beijing University of Posts and Telecommunications, Beijing, China

Jun 16, 2026J/K move · Enter open · S save

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

Jinhao You, Shuo Lyu, Zhuohang Lyu, Tanxuan Li+4

University of Pennsylvania · University of California, Irvine · Nanyang Technological University+1

3D ViTs Multi-View 3D Reconstruction Efficient ViTs Visual Token Compression

Open paper AskPDF arXiv

Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a 6.7x speedup over VGGT* at matched reconstruction quality.

Abstract

Efficient Transformer Inference

Explore similar work

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers

BlazeVGGT: 13x Faster VGGT at 2,000 Frames via Camera Token without Training