OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer

Si-Yu Lu; Po-Ting Chen; Hui-Che Hsu; Sin-Ye Jhong; Wen-Huang Cheng; Yung-Yao Chen

cs.CVMar 6, 2026

OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer

Authors: Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu, Sin-Ye Jhong, Wen-Huang Cheng, Yung-Yao Chen

Organizations: National Taiwan University · National Taiwan University of Science and Technology

Abstract

Reconstructing 3D geometry from streaming video requires continuous inference under bounded resources. Recent geometric foundation models achieve impressive reconstruction quality through all-to-all attention, yet their quadratic cost confines them to short, offline sequences. Causal-attention variants such as StreamVGGT enable single-pass streaming but accumulate an ever-growing KV cache, exhausting GPU memory within hundreds of frames and precluding the long-horizon deployment that motivates streaming inference in the first place. We present OVGGT, a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length. Our approach combines Self-Selective Caching, which leverages FFN residual magnitudes to compress the KV cache while remaining fully compatible with FlashAttention, with Dynamic Anchor Protection, which shields coordinate-critical tokens from eviction to suppress geometric drift over extended trajectories. Extensive experiments on indoor, outdoor, and ultra-long sequence benchmarks demonstrate that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy. Project page: https://vaisr.github.io/OVGGT/ Code: https://github.com/VAISR/OVGGT

Figures & tables

Figure 2: Overview of OVGGT. At each time step, the input frame is encoded into tokens and processed by a spatial-temporal decoder that attends to a bounded KV cache. During inference, the Activation Value Rating module scores each token’s geometric salience, and the KV Cache Compression (KVCC) module evicts low-scoring tokens to maintain a fixed cache budget. Dynamic Anchor Protection (DAP) shields coordinate-critical tokens from eviction, ensuring long-range geometric stability.

Figure 3: Per-token FFN activation scores across layers, progressing from high-frequency textures (shallow) to geometric structures (mid) to semantic boundaries (deep).

Figure 4: Activation smoothing effectively improves reconstruction quality over vanilla token retention.

Figure 5: Qualitative comparison on indoor scene reconstruction (sequence length $=500$ ). Each row shows a different scene with close-up insets. Note that StreamVGGT is limited to a maximum of 200 input frames due to memory constraints.

Method	7-Scenes							NRGBD
	Seq. Len.	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Seq. Len.	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
	Seq. Len.	Mean	Med.	Mean	Med.	Mean	Med.	Seq. Len.	Mean	Med.	Mean	Med.	Mean	Med.
Spann3R [ 34 ]	200 0	$0.215$	$0.131$	$0.122$	$0.063$	$0.535$	$0.550$	100 0	$0.111$	$0.069$	$0.045$	$0.015$	$0.636$	$0.733$
CUT3R [ 37 ]		$0.087$	$0.048$	$0.045$	$0.014$	$0.566$	$0.601$		$0.039$	$0.024$	$0.013$	$0.004$	$0.645$	$0.748$
Point3R [ 39 ]		$0.041$	$0.019$	$0.023$	$0.006$	$0.579$	$0.622$		$0.046$	$0.028$	$0.016$	$0.004$	$0.662$	$0.775$
TTT3R [ 4 ]		$0.027$	$0.015$	$0.023$	$0.005$	$0.582$	$0.627$		$0.031$	$0.019$	$0.012$	$0.004$	$0.650$	$0.756$

Table 1: Quantitative comparison on the 7-Scenes [ 28 ] and NRGBD [ 1 ] datasets across different sequence lengths. Evict3R † denotes Evict3R with its pruning rate dynamically calibrated to match our constant budget. The best and second-best results are highlighted.

Method	ETH3D (Outdoor)						Long3D (Ultra-Long)
	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
CUT3R [ 37 ]	$0.940$	$0.607$	$0.709$	$0.374$	$0.718$	$0.812$	$6.189$	$2.405$	$1.921$	$1.535$	$0.501$	$0.497$
TTT3R [ 4 ]	$0.598$	$0.374$	$0.585$	$0.223$	$0.728$	$0.826$	$7.341$	$2.018$	$1.455$	$1.235$	$0.509$	$0.503$
StreamVGGT [ 45 ]	$0.601$	$0.369$	$0.442$	$0.169$	$0.791$	$0.933$	OOM
Evict3R † [ 17 ]	$0.605$	$0.375$	$0.442$	$0.163$	$0.792$	$0.934$	$4.928$	$2.710$	$0.715$	$0.204$	$0.507$	$0.504$

Table 2: Quantitative comparison on full sequences of ETH3D [ 27 ] and Long3D [ 43 ] datasets. Ours 200 and Ours 400 denote results with the default 200K and an increased 400K token budget, respectively. Best and second best results highlighted.

	Bonn						KITTI
Method	Abs Rel $\downarrow$			$\delta\!<\!1.25$ $\uparrow$			Abs Rel $\downarrow$			$\delta\!<\!1.25$ $\uparrow$
Method	100	300	500	100	300	500	100	300	500	100	300	500
StreamVGGT [ 45 ]	$0.055$	–	–	$0.974$	–	–	$0.166$	–	–	$0.740$	–	–
Evict3R † [ 17 ]	$0.063$	$0.072$	$0.072$	$0.963$	$0.951$	$0.957$	$0.192$	$0.213$	$0.198$	$0.693$	$0.700$	$0.705$
InfiniteVGGT [ 43 ]	$0.056$	$0.073$	$0.070$	$0.975$	$0.957$	$0.960$	$0.165$	$0.249$	$0.257$	$0.742$	$0.556$	$0.577$
Ours	$0.055$	$0.071$	$0.067$	$0.974$	$0.956$	$0.959$	$0.128$	$0.133$	$0.135$	$0.839$	$0.844$	$0.839$

Table 3: Video depth evaluation on Bonn [ 22 ] and KITTI [ 12 ] across different sequence lengths. Best results highlighted.

Figure 6: Efficiency comparison. FPS and VRAM vs. sequence length.

Budget	7-Scenes							NRGBD
	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		CD $\downarrow$	Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		CD $\downarrow$
	Mean	Med.	Mean	Med.	Mean	Med.	CD $\downarrow$	Mean	Med.	Mean	Med.	Mean	Med.	CD $\downarrow$
100K	$0.059$	$0.035$	$0.030$	$0.006$	$0.565$	$0.598$	$0.043$	$0.037$	$0.023$	$0.013$	$0.003$	$0.636$	$0.730$	$0.041$
200K	$0.026$	$0.009$	$0.020$	$0.004$	$0.571$	$0.609$	$0.032$	$0.037$	$0.022$	$0.015$	$0.003$	$0.642$	$0.740$	$0.042$
500K	$0.025$	$0.008$	$0.021$	$0.005$	$0.570$	$0.608$	$0.033$	$0.038$	$0.021$	$0.017$	$0.003$	$0.640$	$0.736$	$0.043$
750K	$0.027$	$0.009$	$0.022$	$0.005$	$0.571$	$0.609$	$0.035$	$0.041$	$0.023$	$0.018$	$0.004$	$0.639$	$0.736$	$0.046$

Table 4: Effect of cache budget. Reconstruction metrics on 7-Scenes and NRGBD at 300 frames. Best per column.

Table 10

$\mathcal{P}_{\mathrm{init}}$	$\mathcal{P}_{\mathrm{hist}}$	Far ( $>35$ units)			Near ( $15$ – $35$ units)
$\mathcal{P}_{\mathrm{init}}$	$\mathcal{P}_{\mathrm{hist}}$	F 1% $\uparrow$	F 5% $\uparrow$	$\boldsymbol{\delta}_{1.05}$ $\uparrow$	F 1% $\uparrow$	F 5% $\uparrow$	$\boldsymbol{\delta}_{1.05}$ $\uparrow$
		–	–	–	–	–	–
✓		$+5.43\%$	$+4.19\%$	$+4.22\%$	$+3.51\%$	$+2.81\%$	$+2.86\%$
✓	✓	$+10.15\%$	$+7.23\%$	$+7.23\%$	$+5.35\%$	$+4.69\%$	$+4.76\%$

Table 7: Effect of Dynamic Anchor Protection. Depth estimation improvement over the no-anchor baseline on KITTI [ 12 ] at 500 frames, split by depth range.

Figure S1 : Reconstruction quality vs. sequence length on 7-Scenes [ 28 ] . Mean Accuracy, Completeness, and Chamfer Distance are plotted from 25 to 200 frames. StreamVGGT (full cache) accumulates noise from redundant tokens as the sequence grows, causing progressive metric degradation. Random eviction under the same budget rapidly diverges, confirming that naive cache truncation is insufficient. OVGGT selectively retains geometrically salient tokens, achieving lower and more stable error throughout.

Method	TUM-Dynamic					ScanNet
Method	100	300	500	800	1k	100	300	500	800	1k
Evict3R † [ 17 ]	$0.017$	$0.032$	$0.045$	$0.074$	$0.083$	$0.060$	$0.184$	$0.285$	$0.430$	$0.491$
InfiniteVGGT [ 43 ]	$0.018$	$0.034$	$0.046$	$0.061$	$0.066$	$0.056$	$0.202$	$0.329$	$0.438$	$0.478$
Ours	$0.014$	$0.029$	$0.039$	$0.047$	$0.058$	$0.056$	$0.198$	$0.284$	$0.385$	$0.427$

Table S1 : Camera Pose evaluation (ATE

\downarrow

) on TUM-Dynamic [ 29 ] and ScanNet [ 6 ] across different sequence lengths. Best results highlighted.

		100 frames				300 frames
Strategy	FlashAttn	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	CD $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	CD $\downarrow$
Attention weight	×	0.024	0.020	0.595	0.035	0.023	0.022	0.573	0.032
Q $\cdot$ K dot product	✓	0.026	0.020	0.596	0.035	0.032	0.022	0.570	0.037
FFN residual (Ours)	✓	0.023	0.018	0.595	0.033	0.026	0.020	0.571	0.032

Table S2 : Probing experiment: eviction strategy comparison. 3D reconstruction on 7-Scenes [ 28 ] at 100 and 300 frames under identical budget (

B{=}200\text{K}

) with DAP disabled. All strategies share the same hybrid scoring framework.

Figure S2 : Failure case analysis. Reconstruction results of OVGGT at $t{=}100$ and $t{=}500$ frames compared with the ground truth. Red boxes highlight regions where cumulative drift becomes apparent in longer sequences: later frames exhibit progressively degraded geometric fidelity, particularly at structural boundaries and fine-grained details.

$K_{\max}$	Near		Far
$K_{\max}$	RMSE $\downarrow$	$\delta_{1.25}$ $\uparrow$	$\delta_{1.05}$ $\uparrow$	F 1% $\uparrow$
1	$1.553$	$0.861$	$0.091$	$0.018$
3	$1.546$	$0.863$	$0.092$	$0.019$
5	$1.552$	$0.862$	$0.091$	$0.018$
10	$1.548$	$0.863$	$0.091$	$0.018$

Table S3 : Effect of maximum historical anchor count

K_{\max}

. Video depth estimation on KITTI [ 12 ] at 500 frames under

B{=}200\text{K}

. Best results highlighted.

Explore similar work

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Apr 16, 2026Xuanyi Liu, Chunan Yu, Deyi Ji +63D ViTs Streaming 3D Reconstruction
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry
Mar 8, 2026Zhisong Xu, Takeshi Oishi3D ViTs Streaming 3D Reconstruction
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan
Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
May 10, 2026Zichen Zou, Xiaosong Jia, Zuxuan Wu +13D ViTs Streaming 3D Reconstruction
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI

Apr 16, 2026J/K move · Enter open · S save

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

Xuanyi Liu, Chunan Yu, Deyi Ji, Qi Zhu+5

KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China+4

3D ViTs Streaming 3D Reconstruction KV Caching 3D Reconstruction Efficient ViTs Visual Token Compression+1

Open paper AskPDF arXiv

Reconstructing dense 3D geometry from continuous video streams requires stable inference under a constant memory budget. Existing

O(1)

frameworks primarily rely on a ``pure eviction'' paradigm, which suffers from significant information destruction due to binary token deletion and evaluation noise from localized, single-layer scoring. To address these bottlenecks, we propose StreamCacheVGGT, a training-free framework that reimagines cache management through two synergistic modules: Cross-Layer Consistency-Enhanced Scoring (CLCES) and Hybrid Cache Compression (HCC). CLCES mitigates activation noise by tracking token importance trajectories across the Transformer hierarchy, employing order-statistical analysis to identify sustained geometric salience. Leveraging these robust scores, HCC transcends simple eviction by introducing a three-tier triage strategy that merges moderately important tokens into retained anchors via nearest-neighbor assignment on the key-vector manifold. This approach preserves essential geometric context that would otherwise be lost. Extensive evaluations on five benchmarks (7-Scenes, NRGBD, ETH3D, Bonn, and KITTI) demonstrate that StreamCacheVGGT sets a new state-of-the-art, delivering superior reconstruction accuracy and long-term stability while strictly adhering to constant-cost constraints.

Abstract

Explore similar work

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry

Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval