RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

Jinhao You; Shuo Lyu; Zhuohang Lyu; Tanxuan Li; Zibo Zhao; Jiaxiang Hu; Kai Tang; Yichen Guo

cs.CVJun 16, 2026

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

Authors: Jinhao You, Shuo Lyu, Zhuohang Lyu, Tanxuan Li, Zibo Zhao, Jiaxiang Hu, Kai Tang, Yichen Guo

Organizations: University of Pennsylvania · University of California, Irvine · Nanyang Technological University · 3Nanyang Technological University

Abstract

Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a 6.7x speedup over VGGT* at matched reconstruction quality.

Explore similar work

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers
Sep 20, 2026Abteen Arab, Guile Wu, Chengjie Huang +1Feed-Forward Visual Geometry Estimation Self-Supervised Vision Transformers
PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers
May 8, 2026Haotang Li, Zhenyu Qi, Shaohan Henry Wang +5Visual Token Pruning Self-Supervised Vision Transformers
Diversity-aware View Partitioning for Scalable VGGT
Jul 2, 2026Jinsoo Park, Donggyu Choi, Ahyun Seo +2Self-Supervised Vision Transformers Partitioning

Sep 20, 2026 · cs.CVJ/K move · Enter open · S save

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers

Abteen Arab, Guile Wu, Chengjie Huang, Dongfeng Bai

Feed-forward visual geometry models such as the Visual Geometry Grounded Transformer (VGGT) have recently enabled direct 3D reconstruction from multi-view images. Despite their promising performance, these models scale quadratically with the number of input views due to their global attention mechanism, resulting in substantial latency for long sequence inputs. There have been some recent efforts to accelerate VGGT, but they primarily focus on reducing \emph{token redundancy} through token merging or key/value sparsification. Our work resolves this bottleneck from a different perspective by investigating \emph{architectural redundancy} in visual geometry transformers. We show that the multi-head attention modules in VGGT's global-attention layers contain substantial architectural redundancy, with only a subset of heads carrying critical geometric information. In light of this observation, we propose VGGT-Prime, a compute-adaptive mixture-of-heads model that resolves this redundancy to accelerate visual geometry transformers while maintaining competitive reconstruction quality. The key idea of VGGT-Prime is to estimate the appropriate computation level for each global-attention head using a lightweight router and then dynamically assign each head to different computation modes. Extensive experiments on multiple datasets demonstrate that VGGT-Prime can achieve an {

8\times

} inference speedup over VGGT while maintaining competitive performance on camera pose, depth, and point-cloud predictions. We further show that VGGT-Prime is complementary to existing acceleration methods, such as token merging, further improving inference speed by up to

14{\times}

over VGGT. An overview of our work is available on our \href{https://vggt-prime.github.io}{project page}.

Feed-Forward Visual Geometry Estimation Self-Supervised Vision Transformers Feed-Forward 3D Reconstruction

Open paper Ask about it PDF

Abstract

Explore similar work

VGGT-Prime: Compute-Adaptive Mixture-of-Heads for Efficient Visual Geometry Transformers

PaceVGGT: Pre-Alternating-Attention Token Pruning for Visual Geometry Transformers

Diversity-aware View Partitioning for Scalable VGGT