Stream4D: Streaming 4D Reconstruction with Training-free Motion Decoupling via Causal KV-Cache Attention

Wei Zhang; Ying Feng; Mengtong Wu; Shiqiang Gong; Huaqi Zhang; Qi Zhang

cs.CVOct 22, 2026

Stream4D: Streaming 4D Reconstruction with Training-free Motion Decoupling via Causal KV-Cache Attention

Authors: Wei Zhang, Ying Feng, Mengtong Wu, Shiqiang Gong, Huaqi Zhang, Qi Zhang

Organizations: School of Computer Science, Northwestern Polytechnical University, Xi’an, China · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., Hangzhou, China · Northwestern Polytechnical University, Xi’an, China

Abstract

4D scene reconstruction requires robust handling of dynamic objects, yet existing streaming methods commonly assume static scenes, leading to pose drift and ghosting artifacts over long sequences. We introduce Stream4D, a training-free, motion-decoupled streaming reconstruction framework that can be directly integrated into causal Transformer backbones (e.g., StreamVGGT, Stream3R). Our key observation is that causal attention implicitly encodes rich temporal motion cues within the KV cache. We adaptively aggregate these cues from multiple attention layers and across temporal windows, and fuse them with depth priors to handle challenging cases such as occlusions and textureless regions, producing precise dynamic masks that decouple static and dynamic regions. We then apply a soft key masking scheme that leverages these dynamic masks to suppress interference from dynamic regions in causal attention, thereby improving camera pose estimation accuracy. Stream4D inherits the efficiency of causal attention and remains compatible with existing streaming architectures. On diverse dynamic-scene benchmarks, Stream4D significantly improves camera pose estimation and static-scene reconstruction quality, yields reliable dynamic masks, and effectively mitigates long-sequence ghosting.

Figures & tables

Figure 1. We present Stream4D, a training-free framework that infers dynamic masks from causal-attention cues to decouple motion in streaming reconstruction. By masking dynamic regions during historical aggregation, Stream4D stabilizes online pose estimation and reduces long-sequence ghosting, yielding cleaner dynamic reconstructions.

Figure 2. Streaming reconstruction overview. Each incoming frame is encoded into tokens and decoded with temporal causal attention that attends only to past frames via a KV Cache. The cache is incrementally updated with $(\mathbf{K}_{t},\mathbf{V}_{t})$ , enabling efficient online inference.

Figure 3. Temporal saliency aggregation and fusion visualization. We compare attention statistics across frames and layers: per-token mean and variance heatmaps revealing low-mean/high-variance patterns on dynamic objects; cross-frame attention maps highlighting temporal inconsistency; multi-scale attention fusion; and final fused saliency and binary masks.

Figure 4. Streaming dynamic segmentation architecture. Left: standard causal streaming reconstruction with a KV Cache, where each frame attends only to past tokens. Right: our Stream4D augments the causal pipeline with motion-decoupled masking. Queries load cached keys/values to compute attention scores, aggregate temporal saliency across frames, and produce dynamic masks. These masks are then applied to modify the cached keys, effectively suppressing interference from dynamic regions during historical aggregation.

Method	Type	DAVIS-2016		DAVIS-2017		DAVIS-all
Method	Type	$\mathcal{J}_{M}$	$\mathcal{J}_{R}$	$\mathcal{J}_{M}$	$\mathcal{J}_{R}$	$\mathcal{J}_{M}$	$\mathcal{J}_{R}$
DUSt3R ( Wang et al., 2024a )	Offline	42.10	45.70	35.20	35.30	35.90	34.00
Easi3R_dust3r ( Chen et al., 2025 )	Offline	50.10	55.77	46.86	50.54	44.10	50.85
Easi3R_monst3r ( Chen et al., 2025 )	Offline	54.93	68.00	54.75	66.16	51.64	63.06
MonST3R ( Zhang et al., 2024a )	Offline	40.42	40.39	38.07	36.05	36.98	34.52
DAS3R ( Xu et al., 2024 )	Offline	41.13	38.67	44.51	43.95	43.33	38.93

Table 1. Dynamic object segmentation on DAVIS benchmarks. We report mean IoU (

\mathcal{J}_{M}

) and IoU recall (

\mathcal{J}_{R}

). “Type” indicates Offline vs. Streaming (single-pass, causal). ∗ indicates our adapted implementation with segmentation capability, as the original methods do not support segmentation.

Figure 5. Qualitative results on DAVIS dataset. Each panel shows the input frame, the Easi3R mask, and our Stream4D mask. Easi3R exhibits temporal jitter and boundary breaks under occlusions and non-rigid motions. Our streaming multi-level temporal aggregation with consistency constraints produces temporally smooth and spatially complete masks across slow motions (camel, goat, etc.) and fast motions (skateboarder, motorcyclist), demonstrating robust segmentation under causal KV-Cache attention.

Method	Type	ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$
Easi3R_monst3r ( Chen et al., 2025 )	Offline	0.032	0.008	1.075
Easi3R_dust3r ( Chen et al., 2025 )	Offline	0.022	0.009	0.806
MonST3R ( Zhang et al., 2024a )	Offline	0.038	0.010	1.172
POMATO ( Zhang et al., 2025a )	Offline	0.128	0.027	3.648
SpatialTrackerV2 ( Xiao et al., 2025a )	Offline	0.011	0.006	0.347
DAS3R ( Xu et al., 2024 )	Offline	0.052	0.012	1.560

Table 2. 4D reconstruction on DyCheck. We report pose metrics (ATE, RTE, RRE); lower is better. “Type” indicates Offline vs. Streaming (single-pass, causal).

Figure 6. Ghosting suppression in streaming 4D reconstruction. Naïve causal streaming methods integrate geometry from all past frames, causing moving objects to be repeatedly fused into the scene and leaving stacked replicas along their trajectories (ghosting artifacts), which grow with sequence length and can eventually corrupt nearby static surfaces. In contrast, Stream4D derives training-free dynamic masks from causal attention cues and applies soft key masking to suppress dynamic regions in the KV cache, so that dynamic objects appear only in the current frame while the background remains clean and sharp even in long sequences.

Figure 7. Qualitative comparison of estimated camera trajectories on three DyCheck sequences ( creeper , handwavy , mochi-high-five ). We plot the predicted trajectory (solid blue) against the ground truth (dashed gray) in the $x$ - $z$ plane. Left: Stream3R exhibits noticeable drift and erratic deviations, particularly in sequences with large dynamic motion ( handwavy , mochi-high-five ), where moving objects corrupt the pose estimation. Right: our Stream4D produces trajectories that more closely follow the ground truth, demonstrating that motion-decoupled masking effectively mitigates the interference of dynamic regions on camera pose estimation.

Figure 8. Inference time comparison across different sequence lengths. While Easi3R exhibits exponential time growth and suffers from Out-of-Memory (OOM) errors between 100 and 200 frames, Stream4D maintains linear scalability, performing comparably to the efficient StreamVGGT baseline.

4D Reconstruction

4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.

See all 140 papers

MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer
Mar 5, 2026Juntong Fang, Zequn Chen, Weiqi Zhang +44D Reconstruction Dynamic Scene Reconstruction
School of Software, Tsinghua University, Beijing, China · Li Auto
4DVGGT-D: 4D Visual Geometry Transformer with Improved Dynamic Depth Estimation
May 12, 2026Ying Zang, Xuanyi Liu, Yidong Han +94D Reconstruction Depth Estimation
4Huzhou University · 2Peking University · 1KOKONI 3D, Moxin Technology +4
UniQuery4R: Unified 4D Scene Reconstruction from a Single Query
Aug 18, 2026Tiancheng Chen, Sheng Tang, Wenhua Jin +4Scene Flow Estimation Dynamic Scene Reconstruction
Kosmo Research · Automotive Engineering Department, Jilin University · School of Software, Tsinghua University

Mar 5, 2026J/K move · Enter open · S save

MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer

Juntong Fang, Zequn Chen, Weiqi Zhang, Donglin Di+3

School of Software, Tsinghua University, Beijing, China · Li Auto

4D Reconstruction Dynamic Scene Reconstruction Monocular 3D Reconstruction 3D ViTs

Open paper AskPDF arXiv

Reconstructing dynamic 4D scenes remains challenging due to the presence of moving objects that corrupt camera pose estimation. Existing optimization methods alleviate this issue with additional supervision, but they are mostly computationally expensive and impractical in real-time applications. To address these limitations, we propose MoRe, a feedforward 4D reconstruction network that efficiently recovers dynamic 3D scenes from monocular videos. Built upon a strong static reconstruction backbone, MoRe employs an attention-forcing strategy to disentangle dynamic motion from static structure. To further enhance robustness, we fine-tune the model on large-scale, diverse datasets encompassing both dynamic and static scenes. Moreover, our grouped causal attention captures temporal dependencies and adapts to varying token lengths across frames, ensuring temporally coherent geometry reconstruction. Extensive experiments on multiple benchmarks demonstrate that MoRe achieves high-quality dynamic reconstructions with exceptional efficiency.

Abstract

4D Reconstruction

Explore similar work

MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer

4DVGGT-D: 4D Visual Geometry Transformer with Improved Dynamic Depth Estimation

UniQuery4R: Unified 4D Scene Reconstruction from a Single Query