SpaceCache: Intermediate Feature Reuse for Efficient Semantic Segmentation on Point Cloud Streams

Dongheon Yang; Beomseok Kang; Jae-Joon Kim

cs.CVOct 15, 2026

SpaceCache: Intermediate Feature Reuse for Efficient Semantic Segmentation on Point Cloud Streams

Authors: Dongheon Yang, Beomseok Kang, Jae-Joon Kim

Organizations: Seoul National University

Abstract

Point-cloud streams are central to autonomous driving and robotics, yet existing segmentation networks process every frame from scratch, although consecutive frames largely overlap. We observe that the intermediate features of point-cloud transformers remain semantically consistent across frames for the same spatial region, especially in deep stages. Building on this, we present SpaceCache, a training-free feature cache that reuses these features across frames. Tokens of previously observed regions skip a sequence of transformer blocks and take their cached features, only the remaining tokens are computed, and the full token layout is restored for dense prediction. Lightweight cache management keeps reused features reliable by limiting each entry's age and evicting low-confidence entries. Unlike token merging, SpaceCache selects tokens at negligible cost and runs its cache operations once per encoder stage. On both outdoor and indoor benchmarks, such as nuScenes, Waymo, and ScanNet, with two transformer backbones of Utonia and LitePT-B, SpaceCache achieves up to 1.65x speedup with less than 1-point mIoU drop and consistently outperforms token-merging methods.

Figures & tables

Figure 1: Increasing temporal overlap across encoder stages. Visualization on the Waymo dataset using the stage-wise downsampling ratios of the Utonia and LitePT-B models. Both have five encoder stages (S0–S4). Each grid cell represents the voxel region corresponding to a single token, and its color indicates the ground-truth semantic class of that region. Magenta highlights voxels in frame $t{+}1$ that overlap with frame $t$ . As progressive downsampling increases the voxel size at deeper stages, tokens represent larger spatial regions and exhibit higher cross-frame overlap.

Figure 2: Temporal-reuse opportunity in point-cloud encoders. (a) A multi-stage encoder, where stage $l$ runs $B_{l}$ blocks over a fixed token set and downsampling between stages shrinks the token set while enlarging the region each token covers. (b) The median fraction of stage- $l$ overlapping tokens, whose region already appeared in the previous frame, grows substantially with depth. (c) The Reuse Agreement of stage- $l$ overlapping tokens also increases with depth, indicating that the overlapped previous-frame token and the corresponding current-frame token predict the same class (Appendix A.2 ).

Figure 3: Overview of SpaceCache. Inference pipeline with the Mask, Reuse, and Commit Gates. Cache-hit tokens bypass the selected transformer blocks, while cache-miss tokens are computed through the blocks. The Reuse Gate restores the full token layout, and the Commit Gate updates the cache after prediction. Each cached stage has its own SpaceCache.

Figure 4: Operation details of SpaceCache. Each entry stores a region index, cached feature, and age. Read: Current-frame tokens are mapped to global region indices and queried at the Mask Gate to identify cache hits; the corresponding cached features are then retrieved for reuse at the Reuse Gate. Write: Newly computed cache-miss features become candidate entries at the Reuse Gate; after prediction, the Commit Gate evicts invalid entries, commits valid candidates, and re-sorts the cache for the next-frame lookup.

Figure 5: Reuse Agreement of cache entries (Stage-3, Waymo with Utonia). (a) By region confidence $c(v)$ at $L{=}8$ for all, cached, and candidate entries; the dashed line marks $\theta{=}0.85$ . (b) By entry age with $L{=}\infty$ .

	Waymo				nuScenes
Method	mIoU (%) $\uparrow$	Latency $\downarrow$	GFLOPs $\downarrow$	Speedup $\uparrow$	mIoU (%) $\uparrow$	Latency $\downarrow$	GFLOPs $\downarrow$	Speedup $\uparrow$
Utonia	67.17	721	4438	1.00 $\times$	77.29	226	1269	1.00 $\times$
+ ToMe ∗	66.35	691	3794	1.04 $\times$	76.31	255	1024	0.89 $\times$
+ ALGM ∗	66.24	633	3751	1.14 $\times$	76.36	209	1060	1.08 $\times$
+ GitMerge3D †	66.38	706	3864	1.02 $\times$	76.32	237	1123	0.95 $\times$
+ SpaceCache $\,{}_{\text{Stage-}3}$	66.25	563	3177	1.28 $\times$	76.47	192	982	1.18 $\times$

Table 1: Comparison on outdoor streams. Each method is shown at its fewest-FLOPs setting within a 1-point mIoU drop from the backbone (rings in Figure 6 ). Latency (ms) and GFLOPs are measured per frame on representative scenes with dense attention. Dashes indicate that no setting satisfies the 1-point drop. ∗ : 2D method applied to point tokens. † : without the adaptive patch partition.

Figure 6: Accuracy-efficiency trade-off on outdoor streams. Curves sweep the merge ratio (ToMe, GitMerge3D), similarity threshold (ALGM), or confidence threshold $\theta$ (SpaceCache), plotting mIoU against FLOPs (top) and latency (bottom). Solid line and star–backbone; dashed lines–1- and 2-point mIoU drops; Rings–fewest-FLOPs point of each method within the 1-point drop.

	mIoU drop $\leq$ 1.0 pt				mIoU drop $\leq$ 2.0 pt
Method	mIoU (%) $\uparrow$	Kernel time $\downarrow$	GFLOPs $\downarrow$	Speedup $\uparrow$	mIoU (%) $\uparrow$	Kernel time $\downarrow$	GFLOPs $\downarrow$	Speedup $\uparrow$
LitePT-B	50.63	43.3	39.9	1.00 $\times$	50.63	43.3	39.9	1.00 $\times$
+ ToMe ∗	49.72	41.1	28.0	1.06 $\times$	48.80	40.5	26.6	1.07 $\times$
+ ALGM ∗	49.80	39.9	32.1	1.09 $\times$	48.75	37.2	29.7	1.16 $\times$
+ GitMerge3D †	49.64	47.2	37.3	0.92 $\times$	48.65	46.8	37.1	0.93 $\times$
+ SpaceCache $\,{}_{\text{Stage-}3}$	49.81	36.8	28.0	1.18 $\times$	48.68	35.5	25.6	1.22 $\times$

Table 2: Comparison on indoor streams. Each method is shown at its fewest-FLOPs setting within 1-point and 2-point mIoU drops. Kernel time (ms) is averaged over 50 representative frames. The notation follows Table 1 .

Table 3: Ablation study on Waymo with Utonia. (a) Lifetime control. (b) Stage selection. Gray: default setting;

L{=}1

reproduces the backbone. Hit: cache-hit fraction of each cached stage.

\Delta

Dyn./

\Delta

Sta.: accuracy change (points) on dynamic/static regions at stage-3 relative to

L{=}1

(83.7/94.8%; Appendix C.3 ).

\Delta

FLOPs: encoder FLOPs change relative to the backbone.

Figure 7: Latency and memory breakdown on the representative scenes of Waymo with Utonia, each method within a 2.0-point mIoU drop. (a) Latency per frame, split into the backbone layers and the policy overhead. (b) Policy overhead per frame, split into its components. (c) Peak memory with flash-attention, split into the model weights, the activation memory, and the cache.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

	Waymo		nuScenes		ScanNet
Method	mIoU (%) $\uparrow$	Speedup $\uparrow$	mIoU (%) $\uparrow$	Speedup $\uparrow$	mIoU (%) $\uparrow$	Speedup $\uparrow$
Utonia	67.17	1.00 $\times$	77.29	1.00 $\times$	–	–
+ GitMerge3D † ( $r{=}0.5$ )	65.64	1.05 $\times$	74.03	1.00 $\times$	–	–
+ GitMerge3D ( $r{=}0.5$ )	63.64	0.89 $\times$	73.35	0.86 $\times$	–	–
LitePT-B	72.82	1.00 $\times$	80.04	1.00 $\times$	50.63	1.00 $\times$
+ GitMerge3D † ( $r{=}0.5$ )	72.05	1.07 $\times$	74.92	1.06 $\times$	49.44	0.92 $\times$

Appendix

Table 4: GitMerge3D with and without its adaptive patch partition at the same merge ratio

r{=}0.5

. Speedup is measured in latency on Waymo and nuScenes and in kernel time on ScanNet. † : without the adaptive patch partition.

		Waymo		nuScenes		ScanNet
Method	Param.	Utonia	LitePT-B	Utonia	LitePT-B	LitePT-B
ToMe ∗	$r$	0.01, 0.015 , 0.025, 0.04, 0.045	0.01, 0.05, 0.095 , 0.12, 0.16, 0.2	0.01, 0.02 , 0.025, 0.035, 0.045, 0.05	0.02, 0.04, 0.06 , 0.09, 0.12	0.11 , 0.13
ALGM ∗	$\tau$	0.96, 0.97, 0.98, 0.995 , 1	0.6, 0.7, 0.75, 0.85, 0.95, 0.97	0.7, 0.95, 0.97, 0.995	0.9, 0.925, 0.95, 0.98, 1	0.9 , 0.8125
GitMerge3D †	$r$	0.1, 0.2, 0.3, 0.4 , 0.5, 0.6, 0.65, 0.72	0.5 , 0.645, 0.7, 0.72	0.1, 0.2, 0.3, 0.35 , 0.45, 0.5, 0.6	0.1, 0.2, 0.33 , 0.45, 0.6	0.55 , 0.645
SpaceCache Stage-3	$\theta$	off , 0.55, 0.7, 0.8 , 0.95, 0.99	off , 0.7, 0.8, 0.95, 0.99	0.8, 0.9, 0.97 , 0.99	off , 0.5, 0.65 , 0.8, 0.95, 0.99	0.85 , off
SpaceCache Stage-3,4	$\theta$	off , 0.5, 0.55, 0.8 , 0.95, 0.99	off , 0.6 , 0.8, 0.95, 0.99	0.7, 0.85, 0.95, 0.985 , 0.99	0.5, 0.7, 0.825, 0.95, 0.97 , 0.99	0.99 , 0.8

Appendix

Table 5: Swept parameter values. The outdoor columns list every point of the Figure 6 curves; the ScanNet column lists the two reported operating points of Table 2 . off disables the confidence gate. Bold marks the point picked within a 1.0-point mIoU drop.

	Waymo with Utonia
	Utonia	+ ToMe ∗	+ ALGM ∗	+ GitMerge3D †	+ SpaceCache Stage-3	+ SpaceCache Stage-3,4
Sweep parameter	–	$r{=}0.025$	$\tau{=}0.98$	$r{=}0.60$	$\theta{=}0.55$	$\theta{=}0.55$
Accuracy
mIoU (%) $\uparrow$	67.17	65.36	65.25	65.21	65.20	65.44
GFLOPs
Linear and SpConv	3321.5	2485.2	2622.8	3321.5	2178.0	1955.5

Appendix

Table 6: FLOPs, latency, and memory breakdown on the representative scenes, each method at its point within a 2.0-point mIoU drop. The latency and memory blocks are plotted in Figure 7 . FLOPs and latency are per frame and use dense attention, and memory uses flash-attention.

3D Point Cloud Segmentation

3D point cloud segmentation assigns semantic, instance, or part labels to points for scene understanding, mapping, robotics, and scientific measurement.

See all 70 papers

VaViT: Vanilla ViT for Automotive Point Cloud Semantic Segmentation
May 29, 2026Gilles Puy, Nermin Samet, Alexandre Boulch +33D Point Cloud Segmentation 3D ViTs
valeo.ai, Paris, France · LIGM, CNRS, Univ Gustave Eiffel, ENPC, IP Paris, France
StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression
Apr 16, 2026Xuanyi Liu, Chunan Yu, Deyi Ji +63D ViTs Streaming 3D Reconstruction
KOKONI 3D, Moxin Technology · Nanjing University of Science and Technology · University of Science and Technology of China +4
SCMerge: Spatial-Consistency-Aware Token Pair Merging for Efficient Semantic Segmentation
Oct 15, 2026Junfeng Ren, Yi Gong, Yuan ZengVisual Token Merging Efficient ViTs
Southern University of Science and Technology · Shenzhen Technology University

May 29, 2026J/K move · Enter open · S save

VaViT: Vanilla ViT for Automotive Point Cloud Semantic Segmentation

Gilles Puy, Nermin Samet, Alexandre Boulch, Spyros Gidaris+2

valeo.ai, Paris, France · LIGM, CNRS, Univ Gustave Eiffel, ENPC, IP Paris, France

3D Point Cloud Segmentation 3D ViTs 3D Semantic Segmentation

Open paper AskPDF arXiv

Plain Transformers have become the de-facto architecture for processing text, audio, image, and video, offering a unified backbone for multimodal learning. However, state-of-the-art architectures for point cloud semantic segmentation remain dominated by U-Nets architectures where convolutions are interleaved with local or windowed attentions. In this work, we show how to effectively leverage vanilla, non-hierarchical ViTs for the segmentation of large-scale automotive lidar scenes. We bridge the performance gap thanks to a carefully designed tokenizer, a lightweight decoder segmentation head, and tailored data augmentations. Our approach, VaViT, for Vanilla ViT, matches or exceeds the performance of state-of-the-art methods while maintaining the simplicity of ViT architecture. We provide extensive evaluations on nuScenes, SemanticKITTI, and Waymo Open Dataset to validate the efficiency of our method. Code and pre-trained models are available at https://github.com/valeoai/VaViT.

Abstract

3D Point Cloud Segmentation

Explore similar work

VaViT: Vanilla ViT for Automotive Point Cloud Semantic Segmentation

StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression

SCMerge: Spatial-Consistency-Aware Token Pair Merging for Efficient Semantic Segmentation