Authors: Zunhai Su, Weihao Ye, Hansen Feng, Keyu Fan, Jing Zhang, Dahai Yu, Zhengwu Liu, Ngai Wong
Organizations: Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology · TCL Corporate Research (HK) Co., Ltd. · Department of Electrical and Electronic Engineering, The University of Hong Kong
Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42× and accelerating inference by 5.48×, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.
Figures & tables
Figure 1: Efficiency analysis on a single 80GB A100 GPU. As the number of input frames increases, StreamVGGT and VGGT exhibit significant FPS degradation and rapidly encounter out-of-memory (OOM) errors. In contrast, XStreamVGGT consistently delivers significantly higher frames per second (FPS) without encountering OOM issues.
Figure 2: Overview of XStreamVGGT. Upon receiving a new input frame (Step 1), Queries from the global attention layer are aggregated via average pooling to form a compact representation, which is then matched against the Key to estimate token importance (Step 2). Guided by these Key-derived importance scores, low-importance historical KV pairs are selectively pruned, while KVs from the first frame are explicitly retained to preserve geometric consistency. The remaining high-importance KVs are concatenated with the first-frame KVs and the newly generated KVs from the current frame (Step 3). Following pruning, the KV cache is further compressed using dimension-adaptive quantization, employing per-channel Key quantization and per-token Value quantization to reduce the impact of outlier channels on quantization accuracy (Step 4). This results in a compact KV cache for efficient subsequent updates (Step 5).
Figure 3: Attention sparsity analysis. The visualization shows attention heatmaps from Layer 14 of StreamVGGT. The visualization of attention heatmaps reveals that attention weights are predominantly concentrated on Query-relevant regions. In contrast, other areas exhibit significantly lower attention, indicating substantial redundancy in the feature representation.
Figure 4: Magnitude distributions of the Key and Value. The Key demonstrates significant channel-wise outliers, with a small subset of channels exhibiting magnitudes substantially larger than the others. In contrast, the distribution of the Value is more uniform, with no prominent outlier behavior.
Bits
Per-Token Quantization
Per-Channel Quantization
Keys
INT4
2.007×10−3
3.635×10−4
INT2
5.183×10−2
9.181×10−3
Values
INT4
5.035×10−4
4.704×10−4
INT2
2.641×10−2
2.190×10−2
Table 1: Analysis of quantization errors for per-token and per-channel schemes. The errors are computed based on the Mean Squared Error (MSE) metric with a group size of 64.
NRGBD
Acc ↓
Comp ↓
NC ↑
Method
Inference Type
Mean
Med.
Mean
Med.
Mean
Med.
VGGT
Offline
Unbounded Memory
0.079
0.020
0.083
0.023
0.909
0.989
StreamVGGT
Online
Unbounded Memory
0.085
0.044
0.079
0.038
0.862
0.986
XStreamVGGT (Ours)
Online
Bounded Memory
0.085
0.049
0.075
0.038
0.850
0.986
Table 2: 3D reconstruction comparison on NRGBD dataset.
7 Scenes
Acc ↓
Comp ↓
NC ↑
Method
Inference Type
Mean
Med.
Mean
Med.
Mean
Med.
VGGT
Offline
Unbounded Memory
0.085
0.038
0.089
0.039
0.786
0.889
StreamVGGT
Online
Unbounded Memory
0.132
0.058
0.116
0.042
0.749
0.863
XStreamVGGT (Ours)
Online
Bounded Memory
0.142
0.068
0.125
0.048
0.734
0.848
Table 3: 3D reconstruction comparison on 7-Scenes dataset.
TUM
ScanNet
Method
Inference Type
ATE ↓
RPE trans ↓
RPE rot ↓
ATE ↓
RPE trans ↓
RPE rot ↓
VGGT
Offline
Unbounded Memory
0.053
0.032
3.209
0.157
0.056
3.635
StreamVGGT
Online
Unbounded Memory
0.062
0.033
3.208
0.160
0.057
3.688
XStreamVGGT (Ours)
Online
Bounded Memory
0.068
0.035
3.184
0.171
0.061
3.837
Table 4: Camera pose estimation.
Sintel
Bonn
KITTI
Method
Inference Type
Abs Rel ↓
δ <1.25 ↑
Abs Rel ↓
δ <1.25 ↑
Abs Rel ↓
δ <1.25 ↑
VGGT
Offline
Unbounded Memory
0.274
67.4
0.054
97.2
0.071
94.1
StreamVGGT
Online
Unbounded Memory
0.254
68.5
0.052
97.1
0.072
94.7
XStreamVGGT
Online
Bounded Memory
0.254
68.5
0.052
97.1
0.072
94.7
Table 5: Monocular depth estimation.
Sintel
Bonn
KITTI
Method
Inference Type
Abs Rel ↓
δ <1.25 ↑
Abs Rel ↓
δ <1.25 ↑
Abs Rel ↓
δ <1.25 ↑
VGGT
Offline
Unbounded Memory
0.301
68.3
0.057
96.8
0.061
97.0
StreamVGGT
Online
Unbounded Memory
0.328
65.8
0.058
97.2
0.094
94.4
XStreamVGGT
Online
Bounded Memory
0.341
61.9
0.073
94.3
0.098
94.3
Table 6: Video depth evaluation.
Sintel
Bonn
KITTI
Method
Cache Length
Pruning
Quantization
Abs Rel ↓
δ <1.25 ↑
Abs Rel ↓
δ <1.25 ↑
Abs Rel ↓
δ <1.25 ↑
StreamVGGT
Unbounded
w/o
w/o
0.358
61.1
0.080
96.5
0.198
69.0
XStreamVGGT
2K
w/
w/o
0.343
61.4
0.067
94.9
0.224
58.0
XStreamVGGT
2K
w/
w/
0.341
61.9
0.073
94.3
0.206
63.9
Table 7: Ablation study of the pruning and quantization processes in video depth estimation.
Figure 5: Ablation study of cache length and analysis of memory with increasing frame length.
Figure 6: Qualitative reconstruction results comparing StreamVGGT and XStreamVGGT.
Figure 7: Qualitative depth estimation results comparing StreamVGGT and XStreamVGGT.
Reconstructing dense 3D geometry from continuous video streams requires stable inference under a constant memory budget. Existing O(1) frameworks primarily rely on a ``pure eviction'' paradigm, which suffers from significant information destruction due to binary token deletion and evaluation noise from localized, single-layer scoring. To address these bottlenecks, we propose StreamCacheVGGT, a training-free framework that reimagines cache management through two synergistic modules: Cross-Layer Consistency-Enhanced Scoring (CLCES) and Hybrid Cache Compression (HCC). CLCES mitigates activation noise by tracking token importance trajectories across the Transformer hierarchy, employing order-statistical analysis to identify sustained geometric salience. Leveraging these robust scores, HCC transcends simple eviction by introducing a three-tier triage strategy that merges moderately important tokens into retained anchors via nearest-neighbor assignment on the key-vector manifold. This approach preserves essential geometric context that would otherwise be lost. Extensive evaluations on five benchmarks (7-Scenes, NRGBD, ETH3D, Bonn, and KITTI) demonstrate that StreamCacheVGGT sets a new state-of-the-art, delivering superior reconstruction accuracy and long-term stability while strictly adhering to constant-cost constraints.
Xuanyi Liu, Chunan Yu, Deyi Ji +6
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception, but their KV-cache grows unbounded over long streams, limiting practical deployment. We study bounded-memory streaming geometry from the perspective of memory organization: unlike language modeling, where useful information can often be compressed at token level, geometry-driven inference relies on coherent and mutually compatible observations across views. Under fixed memory budgets, retaining history as isolated entries can progressively fragment the geometric context needed for stable long-horizon matching and fusion. We therefore propose FrameVGGT, a bounded-memory framework that maintains a fixed-capacity set of complementary memory units for streaming geometry. In our implementation, each unit is instantiated as a frame-wise KV segment summarized by a compact key-space prototype, together with a sparse anchor tier for persistent long-range references. Across long-sequence 3D reconstruction, video depth estimation, and camera pose estimation, FrameVGGT achieves favorable accuracy--memory trade-offs under bounded budgets while maintaining more stable geometry over long streams.
Zhisong Xu, Takeshi Oishi
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan
Reconstructing 3D geometry from streaming video requires continuous inference under bounded resources. Recent geometric foundation models achieve impressive reconstruction quality through all-to-all attention, yet their quadratic cost confines them to short, offline sequences. Causal-attention variants such as StreamVGGT enable single-pass streaming but accumulate an ever-growing KV cache, exhausting GPU memory within hundreds of frames and precluding the long-horizon deployment that motivates streaming inference in the first place. We present OVGGT, a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length. Our approach combines Self-Selective Caching, which leverages FFN residual magnitudes to compress the KV cache while remaining fully compatible with FlashAttention, with Dynamic Anchor Protection, which shields coordinate-critical tokens from eviction to suppress geometric drift over extended trajectories. Extensive experiments on indoor, outdoor, and ultra-long sequence benchmarks demonstrate that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy. Project page: https://vaisr.github.io/OVGGT/ Code: https://github.com/VAISR/OVGGT
Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu +3
National Taiwan University · National Taiwan University of Science and Technology