Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception, but their KV-cache grows unbounded over long streams, limiting practical deployment. We study bounded-memory streaming geometry from the perspective of memory organization: unlike language modeling, where useful information can often be compressed at token level, geometry-driven inference relies on coherent and mutually compatible observations across views. Under fixed memory budgets, retaining history as isolated entries can progressively fragment the geometric context needed for stable long-horizon matching and fusion. We therefore propose FrameVGGT, a bounded-memory framework that maintains a fixed-capacity set of complementary memory units for streaming geometry. In our implementation, each unit is instantiated as a frame-wise KV segment summarized by a compact key-space prototype, together with a sparse anchor tier for persistent long-range references. Across long-sequence 3D reconstruction, video depth estimation, and camera pose estimation, FrameVGGT achieves favorable accuracy--memory trade-offs under bounded budgets while maintaining more stable geometry over long streams.
Figures & tables
Figure 1: Memory organization strategies for bounded streaming geometry. Full memory grows unbounded over time, whereas token-level retention can fragment geometric context under fixed budgets. Frame-level retention preserves coherent frame-wise cache segments for downstream inference.
Figure 2: Pipeline of FrameVGGT. Previous inputs are encoded into per-layer KV blocks, which are managed by a mid-term pool with a distance-based greedy selection policy and an anchor pool gated by gap and geometry thresholds. The selected cache is then loaded to condition new inputs for streaming inference.
Method
7-Scenes
NRGBD
Acc ↓
Acc med↓
Comp ↓
Comp med↓
NC ↑
NC med↑
Acc ↓
Acc med↓
Comp ↓
Comp med↓
NC ↑
NC med↑
CUT3R
0.181
0.127
0.095
0.0327
0.525
0.531
0.322
0.237
0.128
0.0334
0.553
0.566
Point3R
0.063
0.026
0.031
0.0150
0.559
0.589
0.113
0.048
0.037
0.0060
0.621
0.696
TTT3R
0.060
0.035
0.028
0.0050
0.560
0.588
0.161
0.082
0.093
0.0147
0.602
0.654
XStreamVGGT
0.105
0.052
0.048
0.0100
0.556
0.583
0.128
0.080
0.043
0.0069
0.621
0.703
InfiniteVGGT
0.041
0.016
0.024
0.0047
0.561
0.593
0.078
0.049
0.035
0.0069
0.647
0.757
Table 1: Reconstruction results on 7-Scenes and NRGBD. Best results are shown in bold.
Figure 3: Reconstruction visualization comparison on the 7scenes dataset. Our method ensures robust geometric consistency over extended sequences, maintaining a highly stable and reliable result.
Method
Acc ↓
Acc med↓
Comp ↓
Comp med↓
NC ↑
NC med↑
CUT3R
3.217
2.235
1.605
1.351
0.500
0.491
TTT3R
2.964
2.310
2.265
1.805
0.507
0.514
InfiniteVGGT
2.138
1.728
0.963
0.364
0.513
0.526
OVGGT*
2.148
1.687
1.594
0.574
0.526
0.530
Ours
1.698
1.331
1.032
0.433
0.536
0.540
Table 2: Reconstruction results on Long3D. Best results are shown in bold.
Table 6
Figure 4: Pose visualization comparison on the TUM dataset. Our method maintains a robustly stable trajectory and ensures geometric integrity even over exceptionally long-duration sequences.
Table 8
Figure 5: Visualization of memory-key heatmaps at different timesteps. Left: InfiniteVGGT. Right: FrameVGGT. Heatmaps visualize saved memory key snapshots at different checkpoints. Rows are retained memory tokens (internal memory order) and columns are key dimensions (one layer/head slice; batch=0 and head=0 when applicable). Color encodes the key value; the colorbar is auto-scaled per snapshot (ticks chosen automatically), so absolute intensities are not directly comparable across checkpoints. Banded patterns indicate groups of tokens with similar key profiles (higher row-wise correlation), while diffuse patterns indicate more heterogeneous keys.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Temporal evolution of Δk under bounded memory. Increasing Δk indicates stronger concentration and more brittle fusion. Token-level retention shows higher and growing Δk , while FrameVGGT remains more stable.
Figure 7: Additional reconstruction visualizations on 7-Scenes and NRGBD. We show representative renderings from FrameVGGT over extended streaming sequences. These examples complement the quantitative reconstruction results in the main paper by illustrating long-horizon structural consistency under bounded memory.
Figure 8: Additional reconstruction visualizations with recent-only memory on 7-Scenes and NRGBD.
Figure 9: Additional reconstruction visualizations with mid-term memory on 7-Scenes and NRGBD.
Figure 10: Additional reconstruction visualizations with the global anchor tier on 7-Scenes and NRGBD.
Method
7-Scenes
NRGBD
Acc ↓
Acc med↓
Comp ↓
Comp med↓
NC ↑
NC med↑
Acc ↓
Acc med↓
Comp ↓
Comp med↓
NC ↑
NC med↑
Ours (Without Anchor)
0.028
0.009
0.019
0.0040
0.564
0.598
0.054
0.035
0.030
0.0060
0.670
0.782
Ours (With Anchor)
0.026
0.009
0.018
0.0040
0.571
0.607
0.053
0.034
0.031
0.0057
0.682
0.793
Appendix
Table 9: Reconstruction results on 7-Scenes and NRGBD. Best results are shown in bold.
Figure 11: Additional depth visualization results on BONN. We visualize predicted depth maps from FrameVGGT over extended streaming sequences.
Figure 12: Additional pose visualization results on the TUM dataset. The figure shows representative trajectory comparisons over longer sequences.
Reconstructing 3D geometry from streaming video requires continuous inference under bounded resources. Recent geometric foundation models achieve impressive reconstruction quality through all-to-all attention, yet their quadratic cost confines them to short, offline sequences. Causal-attention variants such as StreamVGGT enable single-pass streaming but accumulate an ever-growing KV cache, exhausting GPU memory within hundreds of frames and precluding the long-horizon deployment that motivates streaming inference in the first place. We present OVGGT, a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length. Our approach combines Self-Selective Caching, which leverages FFN residual magnitudes to compress the KV cache while remaining fully compatible with FlashAttention, with Dynamic Anchor Protection, which shields coordinate-critical tokens from eviction to suppress geometric drift over extended trajectories. Extensive experiments on indoor, outdoor, and ultra-long sequence benchmarks demonstrate that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy. Project page: https://vaisr.github.io/OVGGT/ Code: https://github.com/VAISR/OVGGT
Si-Yu Lu, Po-Ting Chen, Hui-Che Hsu +3
National Taiwan University · National Taiwan University of Science and Technology
Reconstructing dense 3D geometry from continuous video streams requires stable inference under a constant memory budget. Existing O(1) frameworks primarily rely on a ``pure eviction'' paradigm, which suffers from significant information destruction due to binary token deletion and evaluation noise from localized, single-layer scoring. To address these bottlenecks, we propose StreamCacheVGGT, a training-free framework that reimagines cache management through two synergistic modules: Cross-Layer Consistency-Enhanced Scoring (CLCES) and Hybrid Cache Compression (HCC). CLCES mitigates activation noise by tracking token importance trajectories across the Transformer hierarchy, employing order-statistical analysis to identify sustained geometric salience. Leveraging these robust scores, HCC transcends simple eviction by introducing a three-tier triage strategy that merges moderately important tokens into retained anchors via nearest-neighbor assignment on the key-vector manifold. This approach preserves essential geometric context that would otherwise be lost. Extensive evaluations on five benchmarks (7-Scenes, NRGBD, ETH3D, Bonn, and KITTI) demonstrate that StreamCacheVGGT sets a new state-of-the-art, delivering superior reconstruction accuracy and long-term stability while strictly adhering to constant-cost constraints.
Xuanyi Liu, Chunan Yu, Deyi Ji +6
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42× and accelerating inference by 5.48×, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.
Zunhai Su, Weihao Ye, Hansen Feng +5
Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology +2