RegVGGT: Sustainable Visual Geometry Grounding for Streaming via Regulated Memory

Hongbo Mao; Junjun Jiang; Youyu Chen; Jiaxin Zhang; Zhemeng Dong; Xianming Liu

cs.CVSep 20, 2026

RegVGGT: Sustainable Visual Geometry Grounding for Streaming via Regulated Memory

Authors: Hongbo Mao, Junjun Jiang, Youyu Chen, Jiaxin Zhang, Zhemeng Dong, Xianming Liu

Organizations: Harbin Institute of Technology · Harbin Institute of Technology, Harbin 150001, China

Abstract

3D reconstruction from a lengthy video stream input poses a dilemma for feed-forward reconstruction models (FFRMs), that a whole-stream inference context cannot be retained under limited GPU memory.Recent studies seek to resolve this problem via a trade-off between the integrity of inference context and GPU memory usage, which either suffer from a rapid memory inflation or degraded context integrity due to artificially capping memory usage.Driven by our key observation that the initial saliency of a token reliably dictates its long-term importance across the stream, we propose RegVGGT, a training-free token regulation method which aggressively regulates the tokens of incoming frames.By admitting at most 1% of tokens per frame to update the context memory, our method dramatically suppresses memory inflation as the stream progresses.Equipped with a FlashAttention-compatible token saliency estimation scheme, RegVGGT is capable of processing thousands of frames on a consumer-grade GPU with negligible compromise to reconstruction quality.Extensive experiments demonstrate that RegVGGT achieves state-of-the-art performance on long-horizon benchmarks across diverse FFRM prediction tasks, surpassing prior FFRM-based stream reconstruction baselines by a large margin.

Figures & tables

Figure 2: Visualization of temporal causal attention. We visualize attention maps to reveal the interaction between queries (rows) and keys (columns). Frames are sampled in a stride of 100 from ScanNet [ 3 ] dataset to depict the long-term behaviour of causal temporal attention. a) We depict layer-wise head-averaged attention map, visualizing $\mathrm{Attn}(\mathrm{Q}_{t},\mathrm{K}_{t\leq 1})$ in Eq. 3 across frame index $t$ . b) We sum up each column, i.e . the attention of different queries for the same key, showing a temporal non-increasing response of key token saliency. c) The head-wise attention maps within Layer 13.

Figure 3: Overview of RegVGGT. Instead of enforcing a fixed memory budget, RegVGGT regulates the per-frame information admitted into memory through a training-free token regulation strategy. We decouple layers into two types. $\mathcal{L}_{\text{qa}}$ entirely discard intermediate history, while $\mathcal{L}_{\text{qs}}$ strictly preserve the top 1% of salient tokens per frame. Coupled with LSE-Saliency, RegVGGT achieves long-term geometric consistency and true scene-agnostic scalability.

Model	Frames	TUM Dynamics			ScanNet
Model	Frames	ATE (m) $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE (m) $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
Evict3R* [ 17 ]	800	0.162	0.028	1.572	1.045	0.100	14.837
InfiniteVGGT* [ 35 ]		0.130	0.045	1.646	1.007	0.138	12.405
XStreamVGGT* [ 24 ]		0.121	0.020	1.354	1.000	0.089	10.225
RegVGGT* (ours)		0.109	0.016	1.288	0.949	0.071	6.662
Evict3R* [ 17 ]	900	0.160	0.027	1.632	1.073	0.100	15.159

Table 1: Camera Pose Estimation Results on TUM Dynamics [ 23 ] and ScanNet [ 3 ] .

Model	Frames	7-Scenes						NRGBD
		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$		Acc $\downarrow$		Comp $\downarrow$		NC $\uparrow$
		Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.	Mean	Med.
Evict3R* [ 17 ]	200	0.106	0.047	0.059	0.017	0.578	0.621	0.107	0.061	0.035	0.010	0.638	0.741
InfiniteVGGT* [ 35 ]		0.067	0.036	0.054	0.005	0.579	0.623	0.062	0.042	0.017	0.004	0.631	0.725
XStreamVGGT* [ 24 ]		0.075	0.038	0.029	0.005	0.581	0.627	0.076	0.051	0.023	0.005	0.633	0.733
RegVGGT* (ours)		0.036	0.020	0.024	0.004	0.591	0.643	0.043	0.031	0.016	0.004	0.641	0.745

Table 2: 3D Reconstruction Results on 7-Scenes [ 22 ] and NRGBD [ 1 ] .

Figure 4: Video Depth Estimation Results on Bonn [ 19 ] .

Table 6

Table 9

Explore similar work

Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
May 10, 2026Zichen Zou, Xiaosong Jia, Zuxuan Wu +13D ViTs Streaming 3D Reconstruction
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Apr 16, 2026Xuanyi Liu, Chunan Yu, Deyi Ji +63D ViTs Streaming 3D Reconstruction
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry
Mar 8, 2026Zhisong Xu, Takeshi Oishi3D ViTs Streaming 3D Reconstruction
Institute of Industrial Science, The University of Tokyo, Tokyo, Japan

May 10, 2026J/K move · Enter open · S save

Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval

Zichen Zou, Xiaosong Jia, Zuxuan Wu, Yu-Gang Jiang

Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI

3D ViTs Streaming 3D Reconstruction 3D Reconstruction Multi-View 3D Reconstruction Efficient Attention

Open paper AskPDF arXiv

Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture, but the quadratic complexity of global attention prevents long context application. StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation. We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem. By retrieving a fixed number of relevant frames at each step, VGGT maintains a controllable memory budget, which is close to its training context length. Interestingly, we find that the similarity between current frame queries and cached history frame keys at the first global attention layer of VGGT is already a strong indicator of relevance, eliminating the need for additional learned scoring. To enhance information diversity similar to a recommender system, we propose Segment Sampling so that the retrieval spans distinct relevant segments rather than a single high-similarity region. We design a pose-aware spatial memory mechanism that organizes history frames according to their already estimated camera poses, enabling location-aware retrieval. Extensive experiments demonstrate that RetrieveVGGT achieves state-of-the-art performance, outperforming StreamVGGT, TTT3R, and InfiniteVGGT while maintaining constant memory usage regardless of sequence length. Code is available at https://github.com/zzctmd/RetrieveVGGT.

Abstract

Explore similar work

Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

FrameVGGT: Coherence-Preserving Memory for Bounded Streaming Geometry