3D reconstruction from a lengthy video stream input poses a dilemma for feed-forward reconstruction models (FFRMs), that a whole-stream inference context cannot be retained under limited GPU memory.Recent studies seek to resolve this problem via a trade-off between the integrity of inference context and GPU memory usage, which either suffer from a rapid memory inflation or degraded context integrity due to artificially capping memory usage.Driven by our key observation that the initial saliency of a token reliably dictates its long-term importance across the stream, we propose RegVGGT, a training-free token regulation method which aggressively regulates the tokens of incoming frames.By admitting at most 1% of tokens per frame to update the context memory, our method dramatically suppresses memory inflation as the stream progresses.Equipped with a FlashAttention-compatible token saliency estimation scheme, RegVGGT is capable of processing thousands of frames on a consumer-grade GPU with negligible compromise to reconstruction quality.Extensive experiments demonstrate that RegVGGT achieves state-of-the-art performance on long-horizon benchmarks across diverse FFRM prediction tasks, surpassing prior FFRM-based stream reconstruction baselines by a large margin.
Figures & tables
Figure 2: Visualization of temporal causal attention. We visualize attention maps to reveal the interaction between queries (rows) and keys (columns). Frames are sampled in a stride of 100 from ScanNet [ 3 ] dataset to depict the long-term behaviour of causal temporal attention. a) We depict layer-wise head-averaged attention map, visualizing Attn(Qt,Kt≤1) in Eq. 3 across frame index t . b) We sum up each column, i.e . the attention of different queries for the same key, showing a temporal non-increasing response of key token saliency. c) The head-wise attention maps within Layer 13.
Figure 3: Overview of RegVGGT. Instead of enforcing a fixed memory budget, RegVGGT regulates the per-frame information admitted into memory through a training-free token regulation strategy. We decouple layers into two types. Lqa entirely discard intermediate history, while Lqs strictly preserve the top 1% of salient tokens per frame. Coupled with LSE-Saliency, RegVGGT achieves long-term geometric consistency and true scene-agnostic scalability.
Model
Frames
TUM Dynamics
ScanNet
ATE (m) ↓
RPE trans ↓
RPE rot ↓
ATE (m) ↓
RPE trans ↓
RPE rot ↓
Evict3R* [ 17 ]
800
0.162
0.028
1.572
1.045
0.100
14.837
InfiniteVGGT* [ 35 ]
0.130
0.045
1.646
1.007
0.138
12.405
XStreamVGGT* [ 24 ]
0.121
0.020
1.354
1.000
0.089
10.225
RegVGGT* (ours)
0.109
0.016
1.288
0.949
0.071
6.662
Evict3R* [ 17 ]
900
0.160
0.027
1.632
1.073
0.100
15.159
Table 1: Camera Pose Estimation Results on TUM Dynamics [ 23 ] and ScanNet [ 3 ] .
Model
Frames
7-Scenes
NRGBD
Acc ↓
Comp ↓
NC ↑
Acc ↓
Comp ↓
NC ↑
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Evict3R* [ 17 ]
200
0.106
0.047
0.059
0.017
0.578
0.621
0.107
0.061
0.035
0.010
0.638
0.741
InfiniteVGGT* [ 35 ]
0.067
0.036
0.054
0.005
0.579
0.623
0.062
0.042
0.017
0.004
0.631
0.725
XStreamVGGT* [ 24 ]
0.075
0.038
0.029
0.005
0.581
0.627
0.076
0.051
0.023
0.005
0.633
0.733
RegVGGT* (ours)
0.036
0.020
0.024
0.004
0.591
0.643
0.043
0.031
0.016
0.004
0.641
0.745
Table 2: 3D Reconstruction Results on 7-Scenes [ 22 ] and NRGBD [ 1 ] .
Figure 4: Video Depth Estimation Results on Bonn [ 19 ] .
Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture, but the quadratic complexity of global attention prevents long context application. StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation. We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem. By retrieving a fixed number of relevant frames at each step, VGGT maintains a controllable memory budget, which is close to its training context length. Interestingly, we find that the similarity between current frame queries and cached history frame keys at the first global attention layer of VGGT is already a strong indicator of relevance, eliminating the need for additional learned scoring. To enhance information diversity similar to a recommender system, we propose Segment Sampling so that the retrieval spans distinct relevant segments rather than a single high-similarity region. We design a pose-aware spatial memory mechanism that organizes history frames according to their already estimated camera poses, enabling location-aware retrieval. Extensive experiments demonstrate that RetrieveVGGT achieves state-of-the-art performance, outperforming StreamVGGT, TTT3R, and InfiniteVGGT while maintaining constant memory usage regardless of sequence length. Code is available at https://github.com/zzctmd/RetrieveVGGT.
Zichen Zou, Xiaosong Jia, Zuxuan Wu +1
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI
Reconstructing dense 3D geometry from continuous video streams requires stable inference under a constant memory budget. Existing O(1) frameworks primarily rely on a ``pure eviction'' paradigm, which suffers from significant information destruction due to binary token deletion and evaluation noise from localized, single-layer scoring. To address these bottlenecks, we propose StreamCacheVGGT, a training-free framework that reimagines cache management through two synergistic modules: Cross-Layer Consistency-Enhanced Scoring (CLCES) and Hybrid Cache Compression (HCC). CLCES mitigates activation noise by tracking token importance trajectories across the Transformer hierarchy, employing order-statistical analysis to identify sustained geometric salience. Leveraging these robust scores, HCC transcends simple eviction by introducing a three-tier triage strategy that merges moderately important tokens into retained anchors via nearest-neighbor assignment on the key-vector manifold. This approach preserves essential geometric context that would otherwise be lost. Extensive evaluations on five benchmarks (7-Scenes, NRGBD, ETH3D, Bonn, and KITTI) demonstrate that StreamCacheVGGT sets a new state-of-the-art, delivering superior reconstruction accuracy and long-term stability while strictly adhering to constant-cost constraints.
Xuanyi Liu, Chunan Yu, Deyi Ji +6
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception, but their KV-cache grows unbounded over long streams, limiting practical deployment. We study bounded-memory streaming geometry from the perspective of memory organization: unlike language modeling, where useful information can often be compressed at token level, geometry-driven inference relies on coherent and mutually compatible observations across views. Under fixed memory budgets, retaining history as isolated entries can progressively fragment the geometric context needed for stable long-horizon matching and fusion. We therefore propose FrameVGGT, a bounded-memory framework that maintains a fixed-capacity set of complementary memory units for streaming geometry. In our implementation, each unit is instantiated as a frame-wise KV segment summarized by a compact key-space prototype, together with a sparse anchor tier for persistent long-range references. Across long-sequence 3D reconstruction, video depth estimation, and camera pose estimation, FrameVGGT achieves favorable accuracy--memory trade-offs under bounded budgets while maintaining more stable geometry over long streams.
Zhisong Xu, Takeshi Oishi
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan