PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers

Haotang Li; Zhenyu Qi; Shaohan Henry Wang; Kebin Peng; Zi Wang; Qing Guo; Sen He; Huanrui Yang

cs.CVMay 8, 2026

PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers

Authors: Haotang Li, Zhenyu Qi, Shaohan Henry Wang, Kebin Peng, Zi Wang, Qing Guo, Sen He, Huanrui Yang

Organizations: University of Arizona · East Carolina University · Augusta University · Nankai University

Abstract

Visual Geometry Transformer (VGGT) is a strong feed-forward model for multiple 3D tasks, but its Alternating-Attention (AA) stack scales quadratically in the total token count, making long clips expensive. Existing token-reduction accelerators operate inside AA, leaving the patch grid that enters AA uncompressed. We introduce PaceVGGT, a pre-AA token pruning framework that prunes DINO patch tokens before the first AA block of a frozen VGGT. PaceVGGT trains a lightweight Token Scorer that estimates per-token importance from DINO features. The scorer is first distilled against an AA-internal attention target from the unpruned backbone, then refined under downstream camera, depth, and point-map losses. A per-frame keep budget fixes the backbone-visible sequence length, while an importance-adaptive merge/prune assignment preserves residual content from high-saliency frames under a fixed total merge budget. A Feature-guided Restoration module reconstructs the dense spatial grid required by the prediction heads. On ScanNet-50 and 7-Scenes, PaceVGGT remains on the reconstruction quality--latency frontier while reducing inference latency. On ScanNet-50, it reduces latency by $5.1\times$ over unmodified VGGT at $N=300$ and $1.47\times$ over LiteVGGT at $N=1000$ . These results identify pre-AA pruning as a viable acceleration route for frozen VGGT-style geometry transformers.

Explore similar work

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers
Sep 20, 2026Abteen Arab, Guile Wu, Chengjie Huang +1Feed-Forward Visual Geometry Estimation Self-Supervised Vision Transformers
RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer
Jun 16, 2026Jinhao You, Shuo Lyu, Zhuohang Lyu +5Self-Supervised Vision Transformers Cross-Attention
QVGGT: Post-Training Quantized Visual Geometry Grounded Transformer
May 29, 2026Zhizhen Pan, Hesong Wang, Huan Wang3D Perception Self-Supervised Vision Transformers

Sep 20, 2026 · cs.CVJ/K move · Enter open · S save

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers

Abteen Arab, Guile Wu, Chengjie Huang, Dongfeng Bai

Feed-forward visual geometry models such as the Visual Geometry Grounded Transformer (VGGT) have recently enabled direct 3D reconstruction from multi-view images. Despite their promising performance, these models scale quadratically with the number of input views due to their global attention mechanism, resulting in substantial latency for long sequence inputs. There have been some recent efforts to accelerate VGGT, but they primarily focus on reducing \emph{token redundancy} through token merging or key/value sparsification. Our work resolves this bottleneck from a different perspective by investigating \emph{architectural redundancy} in visual geometry transformers. We show that the multi-head attention modules in VGGT's global-attention layers contain substantial architectural redundancy, with only a subset of heads carrying critical geometric information. In light of this observation, we propose VGGT-Prime, a compute-adaptive mixture-of-heads model that resolves this redundancy to accelerate visual geometry transformers while maintaining competitive reconstruction quality. The key idea of VGGT-Prime is to estimate the appropriate computation level for each global-attention head using a lightweight router and then dynamically assign each head to different computation modes. Extensive experiments on multiple datasets demonstrate that VGGT-Prime can achieve an {

8\times

} inference speedup over VGGT while maintaining competitive performance on camera pose, depth, and point-cloud predictions. We further show that VGGT-Prime is complementary to existing acceleration methods, such as token merging, further improving inference speed by up to

14{\times}

over VGGT. An overview of our work is available on our \href{https://vggt-prime.github.io}{project page}.

Feed-Forward Visual Geometry Estimation Self-Supervised Vision Transformers Feed-Forward 3D Reconstruction

Open paper Ask about it PDF

Abstract

Explore similar work

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

QVGGT: Post-Training Quantized Visual Geometry Grounded Transformer