FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry

Zhisong Xu; Takeshi Oishi

cs.CVMar 8, 2026

FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry

Authors: Zhisong Xu, Takeshi Oishi

Organizations: Institute of Industrial Science, The University of Tokyo, Tokyo, Japan

Abstract

Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception, but their KV-cache grows unbounded over long streams, limiting practical deployment. We study bounded-memory streaming geometry from the perspective of memory organization: unlike language modeling, where useful information can often be compressed at token level, geometry-driven inference relies on coherent and mutually compatible observations across views. Under fixed memory budgets, retaining history as isolated entries can progressively fragment the geometric context needed for stable long-horizon matching and fusion. We therefore propose FrameVGGT, a bounded-memory framework that maintains a fixed-capacity set of complementary memory units for streaming geometry. In our implementation, each unit is instantiated as a frame-wise KV segment summarized by a compact key-space prototype, together with a sparse anchor tier for persistent long-range references. Across long-sequence 3D reconstruction, video depth estimation, and camera pose estimation, FrameVGGT achieves favorable accuracy--memory trade-offs under bounded budgets while maintaining more stable geometry over long streams.

Figures & tables

Figure 1: Memory organization strategies for bounded streaming geometry. Full memory grows unbounded over time, whereas token-level retention can fragment geometric context under fixed budgets. Frame-level retention preserves coherent frame-wise cache segments for downstream inference.

Figure 2: Pipeline of FrameVGGT. Previous inputs are encoded into per-layer KV blocks, which are managed by a mid-term pool with a distance-based greedy selection policy and an anchor pool gated by gap and geometry thresholds. The selected cache is then loaded to condition new inputs for streaming inference.

Method	7-Scenes						NRGBD
	Acc $\downarrow$	Acc ${}_{\text{med}}\downarrow$	Comp $\downarrow$	Comp ${}_{\text{med}}\downarrow$	NC $\uparrow$	NC ${}_{\text{med}}\uparrow$	Acc $\downarrow$	Acc ${}_{\text{med}}\downarrow$	Comp $\downarrow$	Comp ${}_{\text{med}}\downarrow$	NC $\uparrow$	NC ${}_{\text{med}}\uparrow$
CUT3R	0.181	0.127	0.095	0.0327	0.525	0.531	0.322	0.237	0.128	0.0334	0.553	0.566
Point3R	0.063	0.026	0.031	0.0150	0.559	0.589	0.113	0.048	0.037	0.0060	0.621	0.696
TTT3R	0.060	0.035	0.028	0.0050	0.560	0.588	0.161	0.082	0.093	0.0147	0.602	0.654
XStreamVGGT	0.105	0.052	0.048	0.0100	0.556	0.583	0.128	0.080	0.043	0.0069	0.621	0.703
InfiniteVGGT	0.041	0.016	0.024	0.0047	0.561	0.593	0.078	0.049	0.035	0.0069	0.647	0.757

Table 1: Reconstruction results on 7-Scenes and NRGBD. Best results are shown in bold.

Figure 3: Reconstruction visualization comparison on the 7scenes dataset. Our method ensures robust geometric consistency over extended sequences, maintaining a highly stable and reliable result.

Method	Acc $\downarrow$	Acc ${}_{\text{med}}\downarrow$	Comp $\downarrow$	Comp ${}_{\text{med}}\downarrow$	NC $\uparrow$	NC ${}_{\text{med}}\uparrow$
CUT3R	3.217	2.235	1.605	1.351	0.500	0.491
TTT3R	2.964	2.310	2.265	1.805	0.507	0.514
InfiniteVGGT	2.138	1.728	0.963	0.364	0.513	0.526
OVGGT*	2.148	1.687	1.594	0.574	0.526	0.530
Ours	1.698	1.331	1.032	0.433	0.536	0.540

Table 2: Reconstruction results on Long3D. Best results are shown in bold.

Table 6

Figure 4: Pose visualization comparison on the TUM dataset. Our method maintains a robustly stable trajectory and ensures geometric integrity even over exceptionally long-duration sequences.

Table 8

Figure 5: Visualization of memory-key heatmaps at different timesteps. Left: InfiniteVGGT. Right: FrameVGGT. Heatmaps visualize saved memory key snapshots at different checkpoints. Rows are retained memory tokens (internal memory order) and columns are key dimensions (one layer/head slice; batch=0 and head=0 when applicable). Color encodes the key value; the colorbar is auto-scaled per snapshot (ticks chosen automatically), so absolute intensities are not directly comparable across checkpoints. Banded patterns indicate groups of tokens with similar key profiles (higher row-wise correlation), while diffuse patterns indicate more heterogeneous keys.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Temporal evolution of $\Delta_{k}$ under bounded memory. Increasing $\Delta_{k}$ indicates stronger concentration and more brittle fusion. Token-level retention shows higher and growing $\Delta_{k}$ , while FrameVGGT remains more stable.

Figure 7: Additional reconstruction visualizations on 7-Scenes and NRGBD. We show representative renderings from FrameVGGT over extended streaming sequences. These examples complement the quantitative reconstruction results in the main paper by illustrating long-horizon structural consistency under bounded memory.

Figure 8: Additional reconstruction visualizations with recent-only memory on 7-Scenes and NRGBD.

Figure 9: Additional reconstruction visualizations with mid-term memory on 7-Scenes and NRGBD.

Figure 10: Additional reconstruction visualizations with the global anchor tier on 7-Scenes and NRGBD.

Method	7-Scenes						NRGBD
	Acc $\downarrow$	Acc ${}_{\text{med}}\downarrow$	Comp $\downarrow$	Comp ${}_{\text{med}}\downarrow$	NC $\uparrow$	NC ${}_{\text{med}}\uparrow$	Acc $\downarrow$	Acc ${}_{\text{med}}\downarrow$	Comp $\downarrow$	Comp ${}_{\text{med}}\downarrow$	NC $\uparrow$	NC ${}_{\text{med}}\uparrow$
Ours (Without Anchor)	0.028	0.009	0.019	0.0040	0.564	0.598	0.054	0.035	0.030	0.0060	0.670	0.782
Ours (With Anchor)	0.026	0.009	0.018	0.0040	0.571	0.607	0.053	0.034	0.031	0.0057	0.682	0.793

Appendix

Table 9: Reconstruction results on 7-Scenes and NRGBD. Best results are shown in bold.

Figure 11: Additional depth visualization results on BONN. We visualize predicted depth maps from FrameVGGT over extended streaming sequences.

Figure 12: Additional pose visualization results on the TUM dataset. The figure shows representative trajectory comparisons over longer sequences.

Explore similar work

OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer
Mar 6, 2026Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu +3Streaming 3D Reconstruction 3D ViTs
National Taiwan University · National Taiwan University of Science and Technology
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Apr 16, 2026Xuanyi Liu, Chunan Yu, Deyi Ji +63D ViTs Streaming 3D Reconstruction
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
Feb 25, 2026Zunhai Su, Weihao Ye, Hansen Feng +5Streaming 3D Reconstruction KV-Cache Compression
Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology +2

Mar 6, 2026J/K move · Enter open · S save

OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer

Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu, Sin-Ye Jhong+2

National Taiwan University · National Taiwan University of Science and Technology

Streaming 3D Reconstruction 3D ViTs Efficient ViTs KV-Cache Compression

Open paper AskPDF arXiv

Reconstructing 3D geometry from streaming video requires continuous inference under bounded resources. Recent geometric foundation models achieve impressive reconstruction quality through all-to-all attention, yet their quadratic cost confines them to short, offline sequences. Causal-attention variants such as StreamVGGT enable single-pass streaming but accumulate an ever-growing KV cache, exhausting GPU memory within hundreds of frames and precluding the long-horizon deployment that motivates streaming inference in the first place. We present OVGGT, a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length. Our approach combines Self-Selective Caching, which leverages FFN residual magnitudes to compress the KV cache while remaining fully compatible with FlashAttention, with Dynamic Anchor Protection, which shields coordinate-critical tokens from eviction to suppress geometric drift over extended trajectories. Extensive experiments on indoor, outdoor, and ultra-long sequence benchmarks demonstrate that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy. Project page: https://vaisr.github.io/OVGGT/ Code: https://github.com/VAISR/OVGGT

Abstract

Explore similar work

OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression