SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes

Zhicheng Qiu; Jiarui Meng; Tong-an Luo; Yican Huang; Xuan Feng; Xuanfu Li; ZHan Xu

cs.CVMar 24, 2026

SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes

Authors: Zhicheng Qiu, Jiarui Meng, Tong-an Luo, Yican Huang, Xuan Feng, Xuanfu Li, ZHan Xu

Organizations: Huawei Technologies Ltd.

Abstract

We propose SLARM, a feed-forward model that unifies dynamic scene reconstruction, semantic understanding, and real-time streaming inference. SLARM captures complex, non-uniform motion through higher-order motion modeling, trained solely on differentiable renderings without any flow supervision. Besides, SLARM distills semantic features from LSeg to obtain language-aligned representations. This design enables semantic querying via natural language, and the tight coupling between semantics and geometry further enhances the accuracy and robustness of dynamic reconstruction. Moreover, SLARM processes image sequences using window-based causal attention, achieving stable, low-latency streaming inference without accumulating memory cost. Within this unified framework, SLARM achieves state-of-the-art results in dynamic estimation, rendering quality, and scene parsing, improving motion accuracy by 21%, reconstruction PSNR by 1.6 dB, and segmentation mIoU by 20% over existing methods.

Figures & tables

Figure 2: Pipeline of our SLARM model for streaming 4D Gaussian reconstruction. The SLARM model begins by extracting image tokens using a shared-weight Vision Transformer (ViT), and then appending special tokens to these image tokens. These tokens then undergo Frame Attention and Global Attention mechanisms, before being fed into the corresponding decoders to output the parameters.

Figure 3: Illustration of semantic supervision in SLARM. Left: Self-supervised feature learning via distillation from LSeg, where rendered semantic Gaussians are decoded. Right: Supervised training on labeled data by aligning predictions with class text embeddings.

Panel of Figure 4 — Figure 4: Illustration of motion handling for dynamic Gaussians under two modes. In offline inference, the target frame is synthesized by interpolating all input frames. In online inference, the target frame is reconstructed via backward warping from the subsequent frame.

Figure 5: (a) Qualitative comparison of dynamic scenes. (b) Influence of different semantic loss terms on model performance. (c) Impact of varying motion levels on model performance. (d) Comparison of inference speed and memory usage between online and offline modes.

Table 5

Figure 6: Qualitative comparison of semantic segmentation performance. Our method produces more accurate and coherent segmentations compared to previous 2D segmentation approaches.

Methods	EPE $(m)\downarrow$	$\text{Acc}_{5}(\%)$ $\uparrow$	$\text{Acc}_{10}(\%)\uparrow$	$\theta$ (rad) $\downarrow$
NSFP [ 30 ]	0.703	41.03	52.16	0.934
NSFP++ [ 39 ]	0.724	51.56	60.04	1.103
STORM [ 68 ]	0.304	79.01	83.74	0.667
Ours
SLARM-F	0.240	78.15	83.08	0.540
SLARM-W	0.337	81.07	84.26	0.725

Table 3: Comparison of scene flow estimation on the WOD.

Figure 7: Comparison of different language-aligned features. The first row is the input consists of a set of images from adjacent frames, the next three rows correspond to the three types of features for each frame, respectively.

Method	NuScenes		Argoverse2
Method	PSNR $\uparrow$	D-RMSE $\downarrow$	PSNR $\uparrow$	D-RMSE $\downarrow$
STORM	26.25	4.78	26.13	9.04
SLARM-F	26.71	4.24	26.49	8.65
SLARM-W	26.42	4.29	26.26	8.71

Table 4: Comparison on NuScenes and Argoverse2 Datasets.

Figure 8: Qualitative results on a simple outdoor scene: left shows rendered RGB, depth, 3D scene flow, and semantic map from predicted 4DGS; right displays a novel view of the 4DGS in a 3DGS visualizer.

Figure 9: Qualitative results on a complex outdoor scene: left shows rendered RGB, depth, 3D scene flow, and semantic map from predicted 4DGS; right displays a novel view of the 4DGS in a 3DGS visualizer.

Figure 10: Qualitative results on a complex outdoor scene: left shows rendered RGB, depth, 3D scene flow, and semantic map from predicted 4DGS; right displays a novel view of the 4DGS in a 3DGS visualizer.

Figure 11: Qualitative results on a complex outdoor scene: left shows rendered RGB, depth, 3D scene flow, and semantic map from predicted 4DGS; right displays a novel view of the 4DGS in a 3DGS visualizer.

Explore similar work

RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction
Mar 21, 2026Feiran Wang, Zezhou Shang, Gaowen Liu +1Dynamic Scene Reconstruction Streaming 3D Reconstruction
University of Illinois Chicago · Cisco Research
Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models
Oct 1, 2026Xinhao Xiang, Weiyang Li, Zhijie Zheng +2Multi-View Depth Estimation 4D Reconstruction
IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA
IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer
Jul 21, 2026Zhengyu Zou, Hao Li, Kuixuan Jiao +73D ViTs Streaming 3D Reconstruction
Horizon Robotics · S-Lab, Nanyang Technological University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

Mar 21, 2026J/K move · Enter open · S save

RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction

Feiran Wang, Zezhou Shang, Gaowen Liu, Yan Yan

University of Illinois Chicago · Cisco Research

Dynamic Scene Reconstruction Streaming 3D Reconstruction Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Streaming feed-forward 3D reconstruction enables real-time joint estimation of scene geometry and camera poses from RGB images. However, without explicit dynamic reasoning, streaming models can be affected by moving objects, causing artifacts and drift. In this work, we propose RayMap3R, a training-free streaming framework for dynamic scene reconstruction. We observe that RayMap-based predictions exhibit a static-scene bias, providing an internal cue for dynamic identification. Based on this observation, we construct a dual-branch inference scheme that identifies dynamic regions by contrasting RayMap and image predictions, suppressing their interference during memory updates. We further introduce reset metric alignment and state-aware smoothing to preserve metric consistency and stabilize predicted trajectories. Our method achieves state-of-the-art performance among streaming approaches on dynamic scene reconstruction across multiple benchmarks.

Abstract

Explore similar work

RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models

IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer