XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Zunhai Su; Weihao Ye; Hansen Feng; Keyu Fan; Jing Zhang; Dahai Yu; Zhengwu Liu; Ngai Wong

cs.CVFeb 25, 2026

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Authors: Zunhai Su, Weihao Ye, Hansen Feng, Keyu Fan, Jing Zhang, Dahai Yu, Zhengwu Liu, Ngai Wong

Organizations: Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology · TCL Corporate Research (HK) Co., Ltd. · Department of Electrical and Electronic Engineering, The University of Hong Kong

Abstract

Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42 $\times$ and accelerating inference by 5.48 $\times$ , enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

Figures & tables

Figure 1: Efficiency analysis on a single 80GB A100 GPU. As the number of input frames increases, StreamVGGT and VGGT exhibit significant FPS degradation and rapidly encounter out-of-memory (OOM) errors. In contrast, XStreamVGGT consistently delivers significantly higher frames per second (FPS) without encountering OOM issues.

Figure 2: Overview of XStreamVGGT. Upon receiving a new input frame (Step 1), Queries from the global attention layer are aggregated via average pooling to form a compact representation, which is then matched against the Key to estimate token importance (Step 2). Guided by these Key-derived importance scores, low-importance historical KV pairs are selectively pruned, while KVs from the first frame are explicitly retained to preserve geometric consistency. The remaining high-importance KVs are concatenated with the first-frame KVs and the newly generated KVs from the current frame (Step 3). Following pruning, the KV cache is further compressed using dimension-adaptive quantization, employing per-channel Key quantization and per-token Value quantization to reduce the impact of outlier channels on quantization accuracy (Step 4). This results in a compact KV cache for efficient subsequent updates (Step 5).

Panel of Figure 3 — Figure 3: Attention sparsity analysis. The visualization shows attention heatmaps from Layer 14 of StreamVGGT. The visualization of attention heatmaps reveals that attention weights are predominantly concentrated on Query-relevant regions. In contrast, other areas exhibit significantly lower attention, indicating substantial redundancy in the feature representation.

Panel of Figure 4 — Figure 4: Magnitude distributions of the Key and Value. The Key demonstrates significant channel-wise outliers, with a small subset of channels exhibiting magnitudes substantially larger than the others. In contrast, the distribution of the Value is more uniform, with no prominent outlier behavior.

	Bits	Per-Token Quantization	Per-Channel Quantization
Keys	INT4	$2.007\times 10^{-3}$	$3.635\times 10^{-4}$
Keys	INT2	$5.183\times 10^{-2}$	$9.181\times 10^{-3}$
Values	INT4	$5.035\times 10^{-4}$	$4.704\times 10^{-4}$
Values	INT2	$2.641\times 10^{-2}$	$2.190\times 10^{-2}$

Table 1: Analysis of quantization errors for per-token and per-channel schemes. The errors are computed based on the Mean Squared Error (MSE) metric with a group size of 64.

			NRGBD
			Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
Method	Inference Type		Mean	Med.	Mean	Med.	Mean	Med.
VGGT	Offline	Unbounded Memory	0.079	0.020	0.083	0.023	0.909	0.989
StreamVGGT	Online	Unbounded Memory	0.085	0.044	0.079	0.038	0.862	0.986
XStreamVGGT (Ours)	Online	Bounded Memory	0.085	0.049	0.075	0.038	0.850	0.986

Table 2: 3D reconstruction comparison on NRGBD dataset.

			7 Scenes
			Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
Method	Inference Type		Mean	Med.	Mean	Med.	Mean	Med.
VGGT	Offline	Unbounded Memory	0.085	0.038	0.089	0.039	0.786	0.889
StreamVGGT	Online	Unbounded Memory	0.132	0.058	0.116	0.042	0.749	0.863
XStreamVGGT (Ours)	Online	Bounded Memory	0.142	0.068	0.125	0.048	0.734	0.848

Table 3: 3D reconstruction comparison on 7-Scenes dataset.

			TUM			ScanNet
Method	Inference Type		ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
VGGT	Offline	Unbounded Memory	0.053	0.032	3.209	0.157	0.056	3.635
StreamVGGT	Online	Unbounded Memory	0.062	0.033	3.208	0.160	0.057	3.688
XStreamVGGT (Ours)	Online	Bounded Memory	0.068	0.035	3.184	0.171	0.061	3.837

Table 4: Camera pose estimation.

			Sintel		Bonn		KITTI
Method	Inference Type		Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$
VGGT	Offline	Unbounded Memory	0.274	67.4	0.054	97.2	0.071	94.1
StreamVGGT	Online	Unbounded Memory	0.254	68.5	0.052	97.1	0.072	94.7
XStreamVGGT	Online	Bounded Memory	0.254	68.5	0.052	97.1	0.072	94.7

Table 5: Monocular depth estimation.

			Sintel		Bonn		KITTI
Method	Inference Type		Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$
VGGT	Offline	Unbounded Memory	0.301	68.3	0.057	96.8	0.061	97.0
StreamVGGT	Online	Unbounded Memory	0.328	65.8	0.058	97.2	0.094	94.4
XStreamVGGT	Online	Bounded Memory	0.341	61.9	0.073	94.3	0.098	94.3

Table 6: Video depth evaluation.

				Sintel		Bonn		KITTI
Method	Cache Length	Pruning	Quantization	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$	Abs Rel $\downarrow$	${\delta}$ <1.25 $\uparrow$
StreamVGGT	Unbounded	w/o	w/o	0.358	61.1	0.080	96.5	0.198	69.0
XStreamVGGT	2K	w/	w/o	0.343	61.4	0.067	94.9	0.224	58.0
XStreamVGGT	2K	w/	w/	0.341	61.9	0.073	94.3	0.206	63.9

Table 7: Ablation study of the pruning and quantization processes in video depth estimation.

Panel of Figure 5 — Figure 5: Ablation study of cache length and analysis of memory with increasing frame length.

Panel of Figure 6 — Figure 6: Qualitative reconstruction results comparing StreamVGGT and XStreamVGGT.

Panel of Figure 7 — Figure 7: Qualitative depth estimation results comparing StreamVGGT and XStreamVGGT.

Explore similar work

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Apr 16, 2026Xuanyi Liu, Chunan Yu, Deyi Ji +63D ViTs Streaming 3D Reconstruction
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry
Mar 8, 2026Zhisong Xu, Takeshi Oishi3D ViTs Streaming 3D Reconstruction
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan
OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer
Mar 6, 2026Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu +3Streaming 3D Reconstruction 3D ViTs
National Taiwan University · National Taiwan University of Science and Technology

Apr 16, 2026J/K move · Enter open · S save

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

Xuanyi Liu, Chunan Yu, Deyi Ji, Qi Zhu+5

KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China+4

3D ViTs Streaming 3D Reconstruction KV Caching 3D Reconstruction Efficient ViTs Visual Token Compression+1

Open paper AskPDF arXiv

Reconstructing dense 3D geometry from continuous video streams requires stable inference under a constant memory budget. Existing

O(1)

frameworks primarily rely on a ``pure eviction'' paradigm, which suffers from significant information destruction due to binary token deletion and evaluation noise from localized, single-layer scoring. To address these bottlenecks, we propose StreamCacheVGGT, a training-free framework that reimagines cache management through two synergistic modules: Cross-Layer Consistency-Enhanced Scoring (CLCES) and Hybrid Cache Compression (HCC). CLCES mitigates activation noise by tracking token importance trajectories across the Transformer hierarchy, employing order-statistical analysis to identify sustained geometric salience. Leveraging these robust scores, HCC transcends simple eviction by introducing a three-tier triage strategy that merges moderately important tokens into retained anchors via nearest-neighbor assignment on the key-vector manifold. This approach preserves essential geometric context that would otherwise be lost. Extensive evaluations on five benchmarks (7-Scenes, NRGBD, ETH3D, Bonn, and KITTI) demonstrate that StreamCacheVGGT sets a new state-of-the-art, delivering superior reconstruction accuracy and long-term stability while strictly adhering to constant-cost constraints.

Abstract

Explore similar work

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry

OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer