STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstruction

Runze Wang; Yuxuan Song; Youcheng Cai; Ligang Liu

cs.CVMar 18, 2026

STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstruction

Authors: Runze Wang, Yuxuan Song, Youcheng Cai, Ligang Liu

Organizations: University of Science and Technology of China

Abstract

Online 3D reconstruction from streaming inputs requires both long-term temporal consistency and efficient memory usage. Although causal variants of VGGT address this challenge through a key-value (KV) cache mechanism, the cache grows linearly with the stream length, creating a major memory bottleneck. Under limited memory budgets, early cache eviction significantly degrades reconstruction quality and temporal consistency. In this work, we observe that attention in causal transformers for 3D reconstruction exhibits intrinsic spatio-temporal sparsity. Based on this insight, we propose STAC, a Spatio-Temporally Aware Cache Compression framework for streaming 3D reconstruction with large causal transformers. STAC consists of three key components: (1) a Working Temporal Token Caching mechanism that preserves long-term informative tokens using decayed cumulative attention scores; (2) a Long-term Spatial Token Caching scheme that compresses spatially redundant tokens into voxel-aligned representations for memory-efficient storage; and (3) a Chunk-based Multi-frame Optimization strategy that jointly processes consecutive frames to improve temporal coherence and GPU efficiency. Extensive experiments show that STAC achieves state-of-the-art reconstruction quality while reducing memory consumption by nearly 10x and accelerating inference by 4x, substantially improving the scalability of real-time 3D reconstruction in streaming settings.

Figures & tables

Figure 1: Runtime–memory scaling in streaming 3D reconstruction. Bars show per-frame runtime (ms) and lines show KV cache memory (GB, log scale) as the stream grows. Compared with Causal-VGGT, STAC reduces KV cache growth and stabilizes per-frame runtime as the stream length increases.

Figure 2: Spatio-temporal attention sparsity. Representative attention patterns in the global causal attention map of Causal-VGGT, retaining the top 1024 keys per query for visualization. (a) Spatial attention aligned with camera motion; (b) Persistent focus on first-frame tokens as global references; (c) Temporal anchoring via tokens from semantically stable landmark frames; (d) Long-range attention to camera tokens encoding global context.

Figure 3: Overview of STAC. Our framework reconstructs 3D scenes online using spatio-temporal token caching and chunk-based causal inference. (a) The Causal-VGGT module processes ViT-tokenized frames in each chunk using causal attention over the working temporal cache $\mathcal{M}^{\text{temp}}$ and spatial cache $\mathcal{M}^{\text{spat}}$ retrieved from a 3D voxel grid. (b) During inference, Working Temporal Token Caching updates token scores after each KV cache access, retaining high-scoring anchor tokens $\mathcal{M}^{\text{anchor}}$ while preserving first-frame reference tokens $\mathcal{M}^{\text{refer}}$ and sliding-window tokens $\mathcal{M}^{\text{window}}$ , and evicting the rest. (c) Long-term Spatial Token Caching routes evicted tokens with 3D coordinates from the Head Decoder to many-to-one aggregation in $\mathcal{E}$ or one-to-one merging into $\mathcal{G}$ . When $\mathcal{G}$ is full, re-merging frees a slot for the incoming merged token, and the updated representations are stored in a 3D voxel grid for future retrieval.

Figure 4: Qualitative results on streaming inputs from the 7-scenes and NRGBD datasets.

Method	Type	NRGBD (stride 5)						7-Scenes (stride 5)						Mem (GB) $\downarrow$	FPS $\uparrow$
		ACC $\downarrow$		Comp $\downarrow$		NC $\uparrow$		ACC $\downarrow$		Comp $\downarrow$		NC $\uparrow$
		mean	med	mean	med	mean	med	mean	med	mean	med	mean	med
DUST3R-GA [ 42 ]	Optim	0.619	0.384	0.112	0.028	0.504	0.497	0.406	0.266	0.184	0.055	0.523	0.532	–	$<$ 1
MASt3R-GA [ 17 ]	Optim	0.599	0.311	0.306	0.096	0.552	0.570	0.459	0.266	0.187	0.040	0.544	0.568	–	$<$ 1
MonST3R-GA [ 50 ]	Optim	0.528	0.399	0.242	0.098	0.550	0.576	0.328	0.214	0.200	0.031	0.529	0.545	–	$<$ 1
CUT3R [ 41 ]	Online	0.219	0.142	0.098	0.036	0.612	0.686	0.097	0.050	0.042	0.010	0.585	0.632	0.18	19.79

Table 1: Quantitative results for point cloud reconstruction on the NRGBD and 7 Scenes datasets. Integrating STAC into STream3R and StreamVGGT substantially improves memory and runtime efficiency while preserving reconstruction quality. W8 denotes a sliding window size of 8. W22 and W26 indicate larger windows for fair total memory, though our method uses less memory at runtime. Mem: main values report total memory footprint for comparison; for STAC, this includes the spatial token cache storage, while parentheses show its runtime usage. FPS: end-to-end throughput measured over the full reconstruction pipeline, from image input to the final 3D outputs.

Method	Type	Sintel			TUM			ScanNet
Method	Type	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$	ATE $\downarrow$	RPE trans $\downarrow$	RPE rot $\downarrow$
DUST3R-GA [ 42 ]	Optim	0.417	0.250	5.796	0.083	0.017	3.567	0.081	0.028	0.784
MASt3R-GA [ 17 ]	Optim	0.185	0.060	1.496	0.038	0.012	0.448	0.078	0.020	0.475
MonST3R-GA [ 50 ]	Optim	0.111	0.044	0.869	0.098	0.019	0.935	0.077	0.018	0.529
CUT3R [ 41 ]	Online	0.213	0.066	0.621	0.046	0.015	0.473	0.099	0.022	0.600
Spann3R [ 38 ]	Online	0.329	0.110	4.471	0.056	0.021	0.591	0.096	0.023	0.661

Table 2: Quantitative camera pose estimation results on the Sintel, TUM Dynamics, and ScanNet datasets.

Method	Acc $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Mem (GB)	Runtime (ms)
Baseline	0.0776	0.0150	0.6865	0.858	92.56
w/o AC	0.0725	0.0209	0.6991	1.901	61.36
w/o SC	0.0713	0.0199	0.6939	0.572	39.08
w/o CB	0.0666	0.0175	0.6973	2.063	56.08
w/o CO	0.0673	0.0156	0.6948	1.805	138.12
Full	0.0648	0.0142	0.6995	2.210	71.18

Table 3: Ablation study of anchor tokens (AC), spatial token caching (SC), count-based bias (CB), and chunk-based optimization (CO), reporting reconstruction metrics, memory, and backbone runtime (excluding the image encoder and head decoders).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of a spatial attention head, where the top 1024 relevant key tokens are retained for each query. The axes are labeled at the frame level to facilitate identification of the corresponding frames.

Figure 6: Intra-voxel similarity distribution vs. cluster count. Distribution of intra-voxel similarity scores across voxels under varying cluster counts

n\in\{1,2,4,8\}

. Each score reflects the average cosine similarity of key/value tokens within voxel clusters.

Figure 7: Inter-voxel similarity vs. 3D distance. Mean and samples of $\mathrm{InterSim}(u_{a},u_{b})$ are shown as a function of Euclidean distance between voxel centers, averaged over sampled voxel pairs across scenes.

Figure 8: Layer-wise intra-voxel similarity in Causal-VGGT. Average intra-voxel similarity scores computed across all attention heads per layer.

Figure 9: Additional qualitative comparisons with baselines.

Method	Type	Sintel		Bonn		KITTI
Method	Type	Abs Rel $\downarrow$	${\delta<1.25}$ $\uparrow$	Abs Rel $\downarrow$	${\delta<1.25}$ $\uparrow$	Abs Rel $\downarrow$	${\delta<1.25}$ $\uparrow$
DUST3R-GA [ 42 ]	Optim	0.656	45.2	0.155	83.3	0.144	81.3
MASt3R-GA [ 17 ]	Optim	0.641	43.9	0.252	70.1	0.183	74.5
MonST3R-GA [ 50 ]	Optim	0.378	55.8	0.067	96.3	0.168	74.4
CUT3R [ 41 ]	Online	0.421	47.9	0.078	93.7	0.118	88.1
Spann3R [ 38 ]	Online	0.622	42.6	0.144	81.3	0.198	73.7

Appendix

Table 4: Comparison of scale-invariant depth estimation results (using per-sequence alignment) on the Sintel [ 5 ] , Bonn [ 22 ] , and KITTI [ 13 ] datasets. We evaluate accuracy (Abs Rel) and consistency (

\delta<1.25

) across different categories of methods. STream3R-W and StreamVGGT-W serve as sliding-window attention baselines, while our STream3R-STAC and StreamVGGT-STAC apply the cache compression strategy under the same runtime memory budget. Methods that rely on global alignment are marked as “GA”. The “Type” column indicates whether the method is optimization-based (“Optim”), streaming (“Online”), or offline (“Offline”).

Figure 10: Behavior of STAC on a long indoor stream from the whiteroom scene of the NRGBD dataset [ 2 ] . Left: backbone inference time per frame with a breakdown of major components. Right: KV-cache memory usage and the number of active voxels (dashed, right axis) over the stream.

$r$	$\lambda$	ACC $\downarrow$	Comp $\downarrow$	NC $\uparrow$	Mem(GB) $\downarrow$	Time(ms) $\downarrow$
0.025	0.8	0.064	0.014	0.695	4.861	78.85
0.10	0.8	0.065	0.015	0.698	1.119	73.52
0.05	0.6	0.065	0.015	0.696	1.755	68.78
0.05	0.9	0.064	0.014	0.700	2.251	74.70
0.05	0.8	0.065	0.014	0.700	2.210	71.18

Appendix

Table 5: Effect of voxel resolution

r

and merging threshold

\lambda

on reconstruction quality, memory, and runtime. Experiments are conducted on the NRGBD [ 2 ] dataset using Stream3R [ 16 ] with stride 5. Unless otherwise specified, STAC uses the default setting

r{=}0.05

and

\lambda{=}0.8

Policy	NRGBD			7-Scenes
Policy	ACC $\downarrow$	Comp $\downarrow$	NC $\uparrow$	ACC $\downarrow$	Comp $\downarrow$	NC $\uparrow$
Baseline	0.078	0.015	0.687	0.107	0.035	0.587
Random Selection	0.076	0.021	0.696	0.101	0.039	0.591
Uniform Merging	0.065	0.015	0.699	0.047	0.025	0.606
Ours(STAC)	0.065	0.014	0.700	0.047	0.024	0.606

Appendix

Table 6: Ablation study on 3D reconstruction performance using the NRGBD and 7-Scenes [ 31 ] datasets. Metrics include Accuracy (ACC), Completion (Comp), and Normal Consistency (NC). The baseline uses a sliding-window attention mechanism. “Random Selection” selects tokens without relevance estimation; “Uniform Merging” merges tokens uniformly; and “Ours” applies attention-guided selection and weighted merging.

Streaming 3D Reconstruction

Streaming 3D reconstruction incrementally builds geometrically consistent scene maps from incoming images or video under tight memory and latency limits.

See all 99 papers

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression
Feb 25, 2026Zunhai Su, Weihao Ye, Hansen Feng +5Streaming 3D Reconstruction KV-Cache Compression
Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology +2
Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
May 10, 2026Zichen Zou, Xiaosong Jia, Zuxuan Wu +1Memory-Efficient Attention 3D ViTs
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Apr 16, 2026Xuanyi Liu, Chunan Yu, Deyi Ji +63D ViTs Streaming 3D Reconstruction
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4

Feb 25, 2026J/K move · Enter open · S save

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Zunhai Su, Weihao Ye, Hansen Feng, Keyu Fan+4

Shenzhen International Graduate School, Tsinghua University · Institute of Artificial Intelligence, Xiamen University · China Star Optoelectronics Technology+2

Streaming 3D Reconstruction KV-Cache Compression 3D ViTs KV-Cache Quantization Visual Token Pruning

Open paper AskPDF arXiv

Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42

\times

and accelerating inference by 5.48

\times

, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

Abstract

Streaming 3D Reconstruction

Explore similar work

XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression