cs.CVSep 29, 2026

Does the VGGT Family Need All Its Layers?

Authors: Fengyi Zhang, Holger Caesar, Xiangyu Sun, Zheng Zhang, Zi Huang, Yadan Luo

Organizations: The University of Queensland · Delft University of Technology · Harbin Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Which layers of a feed-forward geometry model are needed to preserve both camera poses and dense 3D structure? We study layer redundancy in VGGT, π3π^3, and VGGT-ΩΩ: 3,018 pruned configurations, scored on seven camera-pose and dense-geometry metrics across four indoor and outdoor datasets. Four findings follow: (i) Removable layers cluster in two redundancy regions: a dominant early region and a narrower late one, while deletions spanning the intervening layers are consistently more disruptive. This recurring pattern holds across models, datasets, and metrics, and contrasts with the middle-to-late redundancy commonly reported in the literature. (ii) Within these regions, we observe that the joint degradation from deleting two intervals is approximately the sum of their individual degradations, reducing the number of model evaluations for pruning search from O(L4)O(L^4) to O(L2)O(L^2), where LL is the aggregator depth. (iii) We find that CKA provides a cheaper representation-based proxy for interval degradation, offering a practical trade-off between pruning quality and calibration cost. (iv) Closed-form linear calibration recovers accuracy after pruning without end-to-end retraining. A least-squares analysis shows that using a shared map for special and patch tokens generally incurs excess reconstruction loss, motivating token-aware recovery. Recovery maps fitted on just 100 calibration scenes generalize to held-out scenes and unseen datasets. The resulting models reduce aggregator parameters by up to 44% while maintaining accuracy comparable to their intact counterparts. Code and experimental results will be available at our project page: https://xian-bei.github.io/vggt-family-layer-redundancy/

Figures & tables

Figure 1: Overview of our analysis and pruning framework. (a) Layer-redundancy patterns in the VGGT family. (b) Structured pruning based on approximate additivity and CKA. (c) Calibration-based recovery after pruning. (d) Accuracy and efficiency of the resulting compressed models.
Figure 2: Two separated low-degradation regions consistently appear across models, domains and metrics, with the dominant region occurring early in the aggregator. Each cell reports normalized degradation after deleting the contiguous interval [i,j][i,j] of aggregator layers. Scores are mirrored across the diagonal, with π3\pi^{3} aligned to the first 36 layers and the remaining 12 positions hatched. Trajectory and geometry panels average over their four and three metrics, respectively. Indoor and outdoor panels average equally over ScanNet/7Scenes and nuScenes/Waymo, respectively.
Figure 3: Approximate additivity of interval-wise degradation. For each backbone, the intact-model landscape is compared with the marginal landscape after removing an interval AA within the early redundancy region. The marginal normalized degradation DM(B∣A)D_{\mathcal{M}}(B\mid A) closely matches the original DM(B)D_{\mathcal{M}}(B) in both values and rankings.
Figure 4: Pruning quality and generalization. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to: (a) the held-out evaluation sets, and (b) the unseen 7Scenes and Waymo datasets. “Single” restricts removal to one contiguous interval, whereas “Multi” allows multiple intervals according to each method’s search space; our method allows up to two separated intervals.
Figure 5: Representation-based proxies versus measured normalized degradation on VGGT- Ω\Omega . CKA achieves the lowest MAE, while cosine similarity and CCA yield higher correlations.
Figure 6: Pruning quality and generalization with representation-based proxies. The protocol and row definitions follow Fig. 4 .
Figure 7: Post-pruning recovery and generalization. Recovery is fitted on 100 ScanNet+nuScenes calibration scenes and evaluated on (a) held-out and (b) unseen 7Scenes+Waymo data.
  Trajectory ErrorsGeometric ErrorsSystem Efficiency
MethodLayersAPE ↓\downarrowARE ↓\downarrowRPE t ↓\downarrowRPE r ↓\downarrowAcc. ↓\downarrowComp. ↓\downarrowCD ↓\downarrowParams ↓\downarrowLatency ↓\downarrowMemory ↓\downarrow
  (m)( ∘ )(m)( ∘ )(m)(m)(m)(M)(s)(GiB)
VGGT- Ω\Omega480.319.990.180.931.151.081.116051.988.2
++ FastVGGT480.5711.440.331.041.151.101.136051.70 ↓\downarrow 14.0%8.2
++ ReplaceMe300.9623.460.534.591.381.301.34378 ↓\downarrow 37.5%1.57 ↓\downarrow 20.9%6.9 ↓\downarrow 15.9%
++ Ours300.339.070.191.001.131.091.11378 ↓\downarrow 37.5%1.58 ↓\downarrow 20.4%6.9 ↓\downarrow 15.9%
Table 1: Comparison averaged over the held-out splits of ScanNet, 7Scenes, nuScenes, and Waymo. Bold error values indicate the best results among compressed variants of each backbone.
Figure 8: Qualitative reconstruction comparison. After removing 18 of 48 aggregator layers, ReplaceMe exhibits substantial geometric misalignment and trajectory errors, while our recovery preserves accurate geometry and trajectories, matching or even improving upon the intact VGGT- Ω\Omega .
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Scope of approximate additivity. Original degradation DM(B)D_{\mathcal{M}}(B) is compared with marginal degradation DM(B∣A)D_{\mathcal{M}}(B\mid A) using the same normalization constants. Upper: shorter deletions within the early redundancy region largely preserve the original landscape. Lower: longer deletions reaching its boundary weaken the later redundancy region and reduce agreement.
Figure 10: Selected pruning intervals across budgets on VGGT- Ω\Omega . Left: single-interval pruning progressively extends the early deletion as the budget grows. Right: guided by approximate additivity, our two-interval selection keeps the early interval relatively stable and allocates additional removals to the later redundancy region, avoiding excessive pruning near the early-region boundary.
Figure 11: Representation similarity versus measured normalized degradation.
Figure 12: Effect of calibration-set size on post-pruning recovery.
Figure 13: Effect of token-region handling on similarity-based pruning. Using special or patch token type alone generally performs poorly. Neither the all-token nor the special–patch mean variant consistently dominates, motivating selection based on measured calibration degradation.
   Trajectory ErrorsGeometric ErrorsSystem Efficiency
MethodFramesLayersAPE ↓\downarrowARE ↓\downarrowRPE t ↓\downarrowRPE r ↓\downarrowAcc. ↓\downarrowComp. ↓\downarrowCD ↓\downarrowParams ↓\downarrowLatency ↓\downarrowMemory ↓\downarrow
   (m)( ∘ )(m)( ∘ )(m)(m)(m)(M)(s)(GiB)
VGGT- Ω\Omega30480.319.990.180.931.151.081.116051.988.2
++ Ours30300.339.070.191.001.131.091.11378 ↓\downarrow 37.5%1.58 ↓\downarrow 20.4%6.9 ↓\downarrow 15.9%
VGGT- Ω\Omega100480.319.970.140.521.131.071.106056.5811.3
++ Ours100300.328.510.140.541.121.081.10378 ↓\downarrow 37.5%4.87 ↓\downarrow 26.1%10.0 ↓\downarrow 11.3%
Appendix
Table 2: Generalization across scenes and sequence lengths. Pruning configurations and recovery transformations are calibrated using only 30-frame sequences and applied unchanged to held-out test scenes with 30, 100, and 200 frames. The results demonstrate generalization to both unseen scenes and longer sequences. Parameter counts include only the aggregator, whereas latency and peak allocated GPU memory are measured over the complete forward pass.
Figure 14: A shared redundancy structure across the VGGT family. Two separated low-degradation regions consistently appear across dataset domains and evaluation metrics, with the dominant region occurring early in the aggregator. Scores are mirrored across the diagonal. This is the unnormalized, per-metric counterpart to Fig. 2 .
Figure 15: Pruning quality and same-dataset generalization. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the held-out evaluation sets for same-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Multi” allows multiple intervals according to each method’s search space; our method allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 4 .
Figure 16: Pruning quality and cross-dataset generalization. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the unseen datasets for cross-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Multi” allows multiple intervals according to each method’s search space; our method allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 4 .
Figure 17: Pruning quality and same-dataset generalization with representation-based proxies. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the held-out evaluation sets for same-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Double” allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 6 .
Figure 18: Pruning quality and cross-dataset generalization with representation-based proxies. For each method and budget, one pruning configuration is selected based only on the combined 100-scene calibration set from ScanNet and nuScenes and applied unchanged to the unseen datasets for cross-dataset generalization. “Single” restricts removal to one contiguous interval, whereas “Double” allows up to two separated intervals. This is the unnormalized, per-metric counterpart to Fig. 6 .
Figure 19: Calibration-based post-pruning recovery benchmark. Recovery transformations are fitted on the combined 100-scene calibration set from ScanNet and nuScenes and evaluated on their corresponding held-out evaluation sets. This is the unnormalized, per-metric counterpart to Fig. 7 .
Figure 20: Calibration-based post-pruning recovery benchmark. Recovery transformations are fitted on the combined 100-scene calibration set from ScanNet and nuScenes and evaluated on the unseen 7Scenes and Waymo datasets. This is the unnormalized, per-metric counterpart to Fig. 7 .

Explore similar work

CardsList

  1. RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer

    Jun 16, 2026Jinhao You, Shuo Lyu, Zhuohang Lyu +5Visual Geometry Grounded TransformerSelf-Supervised Vision Transformers

    University of Pennsylvania · University of California, Irvine · Nanyang Technological University +1

  2. PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers

    May 8, 2026Haotang Li, Zhenyu Qi, Shaohan Henry Wang +5Visual Geometry Grounded TransformerVisual Token Pruning

    University of Arizona · East Carolina University · Augusta University +1

  3. VGGT-ΩΩ

    May 14, 2026Jianyuan Wang, Minghao Chen, Shangzhan Zhang +7Visual Geometry Grounded TransformerScene Reconstruction

    1Visual Geometry Group, University of Oxford · 2Meta AI