cs.CVSep 29, 2026

Does the VGGT Family Need All Its Layers?

Authors: Fengyi Zhang, Holger Caesar, Xiangyu Sun, Zheng Zhang, Zi Huang, Yadan Luo

Organizations: The University of Queensland · Delft University of Technology · Harbin Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Which layers of a feed-forward geometry model are needed to preserve both camera poses and dense 3D structure? We study layer redundancy in VGGT, π3π^3, and VGGT-ΩΩ: 3,018 pruned configurations, scored on seven camera-pose and dense-geometry metrics across four indoor and outdoor datasets. Four findings follow: (i) Removable layers cluster in two redundancy regions: a dominant early region and a narrower late one, while deletions spanning the intervening layers are consistently more disruptive. This recurring pattern holds across models, datasets, and metrics, and contrasts with the middle-to-late redundancy commonly reported in the literature. (ii) Within these regions, we observe that the joint degradation from deleting two intervals is approximately the sum of their individual degradations, reducing the number of model evaluations for pruning search from O(L4)O(L^4) to O(L2)O(L^2), where LL is the aggregator depth. (iii) We find that CKA provides a cheaper representation-based proxy for interval degradation, offering a practical trade-off between pruning quality and calibration cost. (iv) Closed-form linear calibration recovers accuracy after pruning without end-to-end retraining. A least-squares analysis shows that using a shared map for special and patch tokens generally incurs excess reconstruction loss, motivating token-aware recovery. Recovery maps fitted on just 100 calibration scenes generalize to held-out scenes and unseen datasets. The resulting models reduce aggregator parameters by up to 44% while maintaining accuracy comparable to their intact counterparts. Code and experimental results will be available at our project page: https://xian-bei.github.io/vggt-family-layer-redundancy/

Figures & tables

Figure 1: Overview of our analysis and pruning framework. (a) Layer-redundancy patterns in the VGGT family. (b) Structured pruning based on approximate additivity and CKA. (c) Calibration-based recovery after pruning. (d) Accuracy and efficiency of the resulting compressed models.
Figure 2: Two separated low-degradation regions consistently appear across models, domains and metrics, with the dominant region occurring early in the aggregator. Each cell reports normalized degradation after deleting the contiguous interval [i,j][i,j] of aggregator layers. Scores are mirrored across the diagonal, with π3\pi^{3} aligned to the first 36 layers and the remaining 12 positions hatched. Trajectory and geometry panels average over their four and three metrics, respectively. Indoor and outdoor panels average equally over ScanNet/7Scenes and nuScenes/Waymo, respectively.
Figure 3: Approximate additivity of interval-wise degradation. For each backbone, the intact-model landscape is compared with the marginal landscape after removing an interval AA within the early redundancy region. The marginal normalized degradation DM(B∣A)D_{\mathcal{M}}(B\mid A) closely matches the original DM(B)D_{\mathcal{M}}(B) in both values and rankings.
Figure 4: Pruning quality and generalization. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to: (a) the held-out evaluation sets, and (b) the unseen 7Scenes and Waymo datasets. “Single” restricts removal to one contiguous interval, whereas “Multi” allows multiple intervals according to each method’s search space; our method allows up to two separated intervals.
Figure 5: Representation-based proxies versus measured normalized degradation on VGGT- Ω\Omega . CKA achieves the lowest MAE, while cosine similarity and CCA yield higher correlations.
Figure 6: Pruning quality and generalization with representation-based proxies. The protocol and row definitions follow Fig. 4 .
Figure 7: Post-pruning recovery and generalization. Recovery is fitted on 100 ScanNet+nuScenes calibration scenes and evaluated on (a) held-out and (b) unseen 7Scenes+Waymo data.
  Trajectory ErrorsGeometric ErrorsSystem Efficiency
MethodLayersAPE ↓\downarrowARE ↓\downarrowRPE t ↓\downarrowRPE r ↓\downarrowAcc. ↓\downarrowComp. ↓\downarrowCD ↓\downarrowParams ↓\downarrowLatency ↓\downarrowMemory ↓\downarrow
  (m)( ∘ )(m)( ∘ )(m)(m)(m)(M)(s)(GiB)
VGGT- Ω\Omega480.319.990.180.931.151.081.116051.988.2
++ FastVGGT480.5711.440.331.041.151.101.136051.70 ↓\downarrow 14.0%8.2
++ ReplaceMe300.9623.460.534.591.381.301.34378 ↓\downarrow 37.5%1.57 ↓\downarrow 20.9%6.9 ↓\downarrow 15.9%
++ Ours300.339.070.191.001.131.091.11378 ↓\downarrow 37.5%1.58 ↓\downarrow 20.4%6.9 ↓\downarrow 15.9%
Table 1: Comparison averaged over the held-out splits of ScanNet, 7Scenes, nuScenes, and Waymo. Bold error values indicate the best results among compressed variants of each backbone.
Figure 8: Qualitative reconstruction comparison. After removing 18 of 48 aggregator layers, ReplaceMe exhibits substantial geometric misalignment and trajectory errors, while our recovery preserves accurate geometry and trajectories, matching or even improving upon the intact VGGT- Ω\Omega .
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Scope of approximate additivity. Original degradation DM(B)D_{\mathcal{M}}(B) is compared with marginal degradation DM(B∣A)D_{\mathcal{M}}(B\mid A) using the same normalization constants. Upper: shorter deletions within the early redundancy region largely preserve the original landscape. Lower: longer deletions reaching its boundary weaken the later redundancy region and reduce agreement.
Figure 10: Selected pruning intervals across budgets on VGGT- Ω\Omega . Left: single-interval pruning progressively extends the early deletion as the budget grows. Right: guided by approximate additivity, our two-interval selection keeps the early interval relatively stable and allocates additional removals to the later redundancy region, avoiding excessive pruning near the early-region boundary.
Figure 11: Representation similarity versus measured normalized degradation.
Figure 12: Effect of calibration-set size on post-pruning recovery.
Figure 13: Effect of token-region handling on similarity-based pruning. Using special or patch token type alone generally performs poorly. Neither the all-token nor the special–patch mean variant consistently dominates, motivating selection based on measured calibration degradation.
   Trajectory ErrorsGeometric ErrorsSystem Efficiency
MethodFramesLayersAPE ↓\downarrowARE ↓\downarrowRPE t ↓\downarrowRPE r ↓\downarrowAcc. ↓\downarrowComp. ↓\downarrowCD ↓\downarrowParams ↓\downarrowLatency ↓\downarrowMemory ↓\downarrow
   (m)( ∘ )(m)( ∘ )(m)(m)(m)(M)(s)(GiB)
VGGT- Ω\Omega30480.319.990.180.931.151.081.116051.988.2
++ Ours30300.339.070.191.001.131.091.11378 ↓\downarrow 37.5%1.58 ↓\downarrow 20.4%6.9 ↓\downarrow 15.9%
VGGT- Ω\Omega100480.319.970.140.521.131.071.106056.5811.3
++ Ours100300.328.510.140.541.121.081.10378 ↓\downarrow 37.5%4.87 ↓\downarrow 26.1%10.0 ↓\downarrow 11.3%
Appendix
Table 2: Generalization across scenes and sequence lengths. Pruning configurations and recovery transformations are calibrated using only 30-frame sequences and applied unchanged to held-out test scenes with 30, 100, and 200 frames. The results demonstrate generalization to both unseen scenes and longer sequences. Parameter counts include only the aggregator, whereas latency and peak allocated GPU memory are measured over the complete forward pass.
Figure 14: A shared redundancy structure across the VGGT family. Two separated low-degradation regions consistently appear across dataset domains and evaluation metrics, with the dominant region occurring early in the aggregator. Scores are mirrored across the diagonal. This is the unnormalized, per-metric counterpart to Fig. 2 .
Figure 15: Pruning quality and same-dataset generalization. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the held-out evaluation sets for same-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Multi” allows multiple intervals according to each method’s search space; our method allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 4 .
Figure 16: Pruning quality and cross-dataset generalization. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the unseen datasets for cross-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Multi” allows multiple intervals according to each method’s search space; our method allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 4 .
Figure 17: Pruning quality and same-dataset generalization with representation-based proxies. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the held-out evaluation sets for same-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Double” allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 6 .
Figure 18: Pruning quality and cross-dataset generalization with representation-based proxies. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the unseen datasets for cross-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Double” allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 6 .
Figure 19: Calibration-based post-pruning recovery benchmark. Recovery transformations are fitted on the combined 100-scene calibration set from ScanNet and nuScenes and evaluated on their corresponding held-out evaluation sets. This is the unnormalized, per-metric counterpart to Fig. 7 .
Figure 20: Calibration-based post-pruning recovery benchmark. Recovery transformations are fitted on the combined 100-scene calibration set from ScanNet and nuScenes and evaluated on the unseen 7Scenes and Waymo datasets. This is the unnormalized, per-metric counterpart to Fig. 7 .

Explore similar work

CardsList
Jun 16, 2026cs.CV

RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

Visual Geometry Grounded Transformer (VGGT) recovers dense 3D scene structure from multi-view images in one forward pass, but quadratic cross-frame attention limits its scalability. Existing training-free accelerators reduce computation uniformly along one axis, missing layer heterogeneity. Our spectral, probing, and causal analyses reveal three regimes: shallow layers lack cross-view structure, middle layers drive cross-view alignment, and deep layers are redundant for dense geometry yet their cross-frame attention remains essential for pose. RegimeVGGT applies layer-wise U-shaped compression along two axes: Saliency-Guided Banded Merging protects geometry- and edge-salient tokens, while Selectively Protected K/V Downsampling preserves cross-frame spatial coverage and the pose-critical path through a phase-shifted spatial grid, a reference-frame anchor, and uncompressed camera/register tokens. Training-free, RegimeVGGT achieves a 6.7x speedup over VGGT* at matched reconstruction quality.

Jinhao You, Shuo Lyu, Zhuohang Lyu +5

University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1

Visual Geometry Grounded TransformerSelf-Supervised Vision TransformersLayer-WiseTask-Aware CompressionRedundancy
AskPDF
May 8, 2026cs.CV

PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers

Multi-view geometry transformers are feed-forward 3D foundation models that jointly predict depth maps, point maps, and camera poses for N images in a single forward pass. Most of them build on an alternating-attention (AA) stack whose cross-view attention runs over the patch tokens of all N views in every block, so its cost grows quadratically in N: on 300-frame ScanNetv2 clips, VGGT peaks at 72.3 GiB, and MapAnything does not fit on a 93 GiB card. Existing methods reduce tokens or attention inside the AA stack, so the full patch grid still enters it, and the pre-AA interface, where the encoder hands its tokens to the first AA block, has been overlooked. We propose PrePARE, which prunes patch tokens once at the pre-AA interface and restores the dense grid once after the last AA block. Every block of the frozen AA stack then runs on the reduced tokens, while the prediction heads receive the full grid. A Token Scorer, supervised by how the unpruned AA stack uses each token, and a Feature-guided Restoration module are the only trained parts. The Token Routing module is rule-based. On ScanNetv2 at N = 300, PrePARE reduces the peak memory of VGGT by 80%, from 72.3 to 14.4 GiB, below every in-AA method we ran, and runs 5.4 times faster, while its per-scene Chamfer distance is lower than the unpruned model's on 34 of 50 scenes (sign test p = 0.015). On pi^3 and Depth Anything 3 it saves 15% and 20% of peak memory, where the in-AA methods save at most 9%, and at 1000 frames it is the only method we ran that fits Depth Anything 3 and MapAnything on one GPU.

Haotang Li, Zhenyu Qi, Shaohan Henry Wang +5

University of Arizona · East Carolina University · Augusta University +1

Visual Geometry Grounded TransformerVisual Token PruningSelf-Supervised Vision TransformersAttention LayersSaliency
AskPDF
May 14, 2026cs.CV

VGGT-ΩΩ

Recent feed-forward reconstruction models, such as VGGT, have proven competitive with traditional optimization-based reconstructors while also providing geometry-aware features useful for other tasks. Here, we show that the quality of these models scales predictably with model and data size. We do so by introducing VGGT-ΩΩ, which substantially improves reconstruction accuracy, efficiency, and capabilities for both static and dynamic scenes. To enable training this model at an unprecedented scale, we introduce architectural changes that improve training efficiency, a high-quality data annotation pipeline that supports dynamic scenes, and a self-supervised learning protocol. We simplify VGGT's architecture by using a single dense prediction head with multi-task supervision and removing the expensive high-resolution convolutional layers. We also use registers to aggregate scene information into a compact representation and introduce register attention, which restricts inter-frame information exchange to these registers, in part replacing global attention. In this way, during training, VGGT-ΩΩ uses only about 30% of the GPU memory of its predecessor, allowing us to train with 15x more supervised data than prior work and to leverage vast amounts of unlabeled video data. VGGT-ΩΩ achieves strong results for reconstruction of static and dynamic scenes across multiple benchmarks, for example, improving over the previous best camera estimation accuracy on Sintel by 77%. We also show that the learned registers can improve vision-language-action models and support alignment with language, suggesting that reconstruction can be a powerful and scalable proxy task for spatial understanding. Project Page: http://vggt-omega.github.io/

Jianyuan Wang, Minghao Chen, Shangzhan Zhang +7

1Visual Geometry Group, University of Oxford · 2Meta AI

Visual Geometry Grounded TransformerScene ReconstructionGeometry-AwareImagenet
AskPDF