Dream-SLAM: Dreaming the Unseen for Active SLAM in Dynamic Environments
Authors: Xiangqi Meng, Pengxu Hou, Zhenjun Zhao, Javier Civera, Daniel Cremers, Hesheng Wang, Haoang Li
Organizations: Thrust of Robotics and Autonomous Systems, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · University of Zaragoza, Zaragoza, Spain. · School of Computation, Information and Technology, Technical University of Munich, Munich, Germany. · School of Automation and IntelligentSensing, Shanghai Jiao Tong University, Shanghai 200240, China.
In addition to the core tasks of simultaneous localization and mapping (SLAM), active SLAM additionally in- volves generating robot actions that enable effective and efficient exploration of unknown environments. However, existing active SLAM pipelines are limited by three main factors. First, they inherit the restrictions of the underlying SLAM modules that they may be using. Second, their motion planning strategies are typically shortsighted and lack long-term vision. Third, most approaches struggle to handle dynamic scenes. To address these limitations, we propose a novel monocular active SLAM method, Dream-SLAM, which is based on dreaming cross-spatio-temporal images and semantically plausible structures of partially observed dynamic environments. The generated cross-spatio-temporal im- ages are fused with real observations to mitigate noise and data incompleteness, leading to more accurate camera pose estimation and a more coherent 3D scene representation. Furthermore, we integrate dreamed and observed scene structures to enable long- horizon planning, producing farsighted trajectories that promote efficient and thorough exploration. Extensive experiments on both public and self-collected datasets demonstrate that Dream-SLAM outperforms state-of-the-art methods in localization accuracy, mapping quality, and exploration efficiency. Source code will be publicly available upon paper acceptance.
Figures & tables
Fig. 1: Dream-SLAM overview. Our pipeline consists of two main modules: localization and mapping, and exploration planning. (a) For localization, we propose to dream cross-spatio-temporal images, and use these images to construct additional 3D-2D foreground constraints that can effectively compensate for noise. For mapping, we propose a feedforward network to reconstruct per-pixel Gaussians of both static background and dynamic foreground. We further refine Gaussians based on multi-view constraints provided by cross-spatio-temporal and real images. (b) Our planning module dreams semantically plausible structures of unobserved areas. By integrating the dreamed and observed information, we plan a farsighted path, enabling an efficient and thorough exploration.
Fig. 2: Cross-spatio-temporal images for camera localization. (a) Traditional localization methods rely solely on the static background to estimate the camera pose. (b) In contrast, our method leverages both the dynamic foreground and static background by aligning the Gaussians’ rendering at time t with the dreamed cross-spatio-temporal image It+1t , which represents the scene at time t from the viewpoint of the (t+1) -th camera.
Fig. 3: Dreaming a cross-spatio-temporal image. Given the image It+1 , we segment the foreground and dilate the foreground mask to obtain the inpainting mask M . Then we feed the images It and It+1 , together with the mask M , into the diffusion model, which dreams the cross-spatio-temporal image It+1t .
Fig. 4: 3D Gaussian prediction and refinement. (a) Given images It+1 and It , we design a feedforward network to predict dynamic Gaussians at both time t+1 and time t . Here, we only visualize the predicted Gaussians at time t+1 . (b) We refine Gaussians at time t+1 based on the photometric loss regarding both dreamed cross-spatio-temporal images It−1t+1,Itt+1 and real image It+1 . These images depict the same scene content at time t+1 from the (t−1) -th, t -th, and (t+1) -th views, respectively.
Fig. 5: Dreaming semantically plausible structures of unexplored areas. At an unvisited waypoint, we place virtual cameras to render images from different views and select the suitable images. Then we inpaint the selected images, and use them to predict Gaussians. By integrating the dreamed Gaussians into the existing Gaussians, we obtain more complete structures of the environment.
Fig. 6: Comparison between global ordering strategies without and with the capability of structure dreaming. In this schematic diagram, the number below each image represents the accumulated length of the traveled path. They do not have a specific unit, but show relative magnitudes of values. (a) Strategies without dreaming capability result in suboptimal trajectories with unnecessary detours. (b) Our method leverages both observed and dreamed structures to plan a farsighted path.
TABLE I: Camera localization comparisons on the TUM [ 61 ] and Bonn [ 30 ] datasets.
Fig. 7: Qualitative comparison of dynamic scene mapping methods on Sequence wk_xyz in the TUM dataset [ 61 ] . We render the mapped scenes from novel views. (a) MonST3R [ 35 ] does not support a photo-realistic rendering due to the point cloud representation. (b) WildGS-SLAM [ 21 ] fails to map the dynamic foreground. (c) PG-SLAM [ 18 ] can reconstruct both static background and dynamic foreground, but some rendering regions remain low-quality. (d) Our Dream-SLAM can map both background and foreground, and also achieves the highest mapping quality.
Bonn dataset [ 30 ]
TUM dataset [ 61 ]
Seq. ps_track2
Seq. ps_track
Seq. mv_box2
Seq. wk_st
Seq. st_hf
Seq. wk_xyz
w/o
w
w/o
w
w/o
w
w/o
w
w/o
w
w/o
w
RoDyn-SLAM [ 17 ]
PSNR ↑
18.46
16.12
18.53
16.13
18.30
17.48
11.48
11.40
11.47
11.11
11.93
11.91
SSIM ↑
0.745
0.656
0.742
0.659
0.731
0.695
0.684
0.522
0.514
0.674
0.567
0.384
LPIPS ↓
0.545
0.649
0.506
0.597
0.571
0.612
0.502
0.663
0.472
0.511
0.570
0.745
WildGS-SLAM [ 21 ]
PSNR ↑
21.87
18.12
22,13
18.55
21.40
20.13
21.60
16.61
19.49
17.94
17.11
14.44
TABLE II: Rendering quality comparison on the Bonn [ 30 ] and TUM [ 61 ] datasets. “w/o” and “w” represent rendering evaluations without and with considering the dynamic foregrounds, respectively.
Time Cost Per Frame
WildGS-SLAM [ 21 ]
2.79 s
PG-SLAM [ 18 ]
1.93 s
Rodyn-SLAM [ 17 ]
1.42 s
Dream-SLAM (ours)
0.65 s
TABLE III: Efficiency comparison in terms of localization and mapping on Sequence mv_box in the Bonn dataset [ 30 ] .
TABLE IV: Exploration planning comparison on Gibson [ 67 ] and HM3D [ 68 ] datasets. “-” represents the failure of the full exploration.
Fig. 8: Comparison between ActiveSplat [ 6 ] and our Dream-SLAM in terms of exploration progress on Sequence T7nCRmufFNR of the HM3D dataset [ 68 ] . We provide a top-view visualization of the mapped scenes at some representative timestamps. A triplet of numbers below each image indicates CR, the length of the traversed path, and the proportion of the traversed path to the total path. The trajectory color reflects the time cost of exploration.
Fig. 9: Comparison between ActiveSplat [ 6 ] and our Dream-SLAM in terms of exploration progress on Sequence Cantwell of the Gibson dataset [ 67 ] . We provide a top-view visualization of the mapped scenes at some representative timestamps. A triplet of numbers below each image indicates CR, the length of the traversed path, and the proportion of the traversed path to the total path. The trajectory color reflects the time cost of exploration.
Sequence
Without Dreaming
Dreaming (ours)
RMSE ↓
SD ↓
RMSE ↓
SD ↓
wk_hf of TUM [ 61 ]
1.8
0.7
1.6
0.7
ps_track of Bonn [ 30 ]
1.7
0.7
1.5
0.5
TABLE V: Ablation study of localization using the dreamed cross-spatio-temporal images.
Fig. 10: Ablation study of mapping using the dreamed cross-spatio-temporal images. We compare the rendering results of Gaussian maps obtained without and with the dreamed images on (a) Sequence ps_track of the Bonn dataset [ 30 ] , and (b) Sequence wk_hf of the TUM dataset [ 61 ] .
Sequence
Without Dreaming
Dreaming (ours)
CR↑
PL↓
CR↑
PL↓
Cantwell of Gibson [ 67 ]
97.5
101.3
98.1
88.4
T7nCRmufFNR of HM3D [ 68 ]
95.5
85.1
96.8
75.8
TABLE VI: Ablation study of planning using the dreamed structures of unexplored areas.
Fig. 11: Setup of experiments on our self-collected data. (a) Mobile robot to collect data. It is built on a four-wheeled Mecanum chassis and equipped with a front-facing camera and a LiDAR. (b) Sample images of our self-collected data in a home environment containing dynamic humans.
Fig. 12: Comparison between ActiveSplat* [ 21 , 6 ] and our Dream-SLAM in terms of exploration progress on our self-collected data. We provide a top-view visualization of the mapped scenes at some representative timestamps. A pair of numbers below each image indicates the length of the traversed path, and the proportion of the traversed path to the total path. The trajectory color reflects the time cost of exploration.
Fig. 13: Dynamic scene mapping of our Dream-SLAM on our self-collected data. We render the reconstructed map composed of both static background and dynamic foreground from a novel view. The red and green dotted lines denote the trajectories of human and camera, respectively.
WildGS-SLAM [ 21 ]
Dream-SLAM (ours)
RMSE ↓
SD ↓
RMSE ↓
SD ↓
16.6
7.3
10.8
5.6
TABLE VII: Camera localization comparison between WildGS-SLAM [ 21 ] and our Dream-SLAM on our self-collected data.
Robust visual Simultaneous Localization and Mapping (vSLAM) in dynamic environments faces challenges due to the presence of moving objects that may introduce erroneous observations, corrupt the map reconstruction and overall degrade localization accuracy. Existing vSLAM approaches often rely on predefined semantic classes or costly pixel-wise classification, limiting their robustness or applicability in real-world scenarios. This paper presents CAOS-SLAM, a dynamic vSLAM framework that extends ORB-SLAM3 with a hybrid perception pipeline to detect, track, and filter both semantically known and previously unseen moving objects in real-time. On-demand panoptic segmentation is performed for object discovery, with candidate unknown objects instantiated through depth-based clustering of unclassified regions. Between panoptic updates, persistent object tracks are updated using lightweight detection when available or temporal mask propagation based on world-frame Kalman prediction, depth consistency, and appearance similarity. The estimated object motion is then used to reject dynamic features from the SLAM pipeline. Experimental results on multiple dynamic RGB-D datasets and real-world experiments demonstrate improved localization accuracy over state-of-the-art dynamic vSLAM approaches, achieving up to 59% lower localization error in scenes characterized by unknown dynamic objects and up to 88% lower error in heavily occluded scenes, while achieving a median processing rate of 30 FPS on a laptop. Furthermore, the tracking module provides estimates of object trajectories and velocities, supporting potential future applications in perception, navigation, and motion-planning tasks in dynamic environments. Code and dataset will be released upon acceptance.
Gaia Borghi, João Carlos Virgolino Soares, Angelo Bratta +2
Dynamic Legged Systems Lab (DLS), Istituto Italiano di Tecnologia, Genoa, Italy · Section of Automation & Control, Department of Electronic Systems, Aalborg University, Denmark
We present a robust, real-time RGB SLAM system that handles dynamic environments by leveraging differentiable Uncertainty-aware Bundle Adjustment. Traditional SLAM methods typically assume static scenes, leading to tracking failures in the presence of motion. Recent dynamic SLAM approaches attempt to address this challenge using predefined dynamic priors or uncertainty-aware mapping, but they remain limited when confronted with unknown dynamic objects or highly cluttered scenes where geometric mapping becomes unreliable. In contrast, our method estimates per-pixel uncertainty by exploiting multi-view visual feature inconsistency, enabling robust tracking and reconstruction even in real-world environments. The proposed system achieves state-of-the-art camera poses and scene geometry in cluttered dynamic scenarios while running in real time at around 10 FPS. Code and datasets are available at https://github.com/MoyangLi00/DROID-W.git.
In dynamic and unstructured environments, conventional SLAM systems generally suffer from significant accuracy degeneration due to their static assumptions. In this work, we propose Robust Semantic-aware Gaussian Splatting SLAM (RoSe-SLAM), to address the dynamic challenge by a holistic semantic scene understanding from uncalibrated monocular inputs, achieving accurate camera tracking and high-quality geometry reconstruction. Unlike conventional semantic SLAM using handcrafted semantic labels, our RoSe-SLAM exploits the semantic feature from 2D foundation model to enhance the dynamic tracking and mapping performance. By distilling the rich semantic features to our Gaussian fields, our method effectively identifies dynamic distractors and achieves semantic-aware multi-view consistency, significantly enhancing the geometric reconstruction and scene inpainting. Specifically, we propose a spatial-temporal motion mask generation module, enabling both long-term motion monitoring and short-term transient dynamics capturing, achieving robust and effective disentanglement of dynamic objects and static backgrounds. During global bundle adjustment, we propose an occlusion-aware keyframe selection mechanism to prioritize the occlusion as metric to pick the keyframes, and a multi-view semantic consistency module to improve the mapping quality in dynamic environments. By combining geometric motion cues with semantic priors, our system dynamically filters unreliable observations and reconstructs accurate static scene geometry. Extensive experiments conducted on benchmark datasets including dynamic TUM, Bonn and Wild-Mocap datasets, demonstrate that our method achieves superior performance in both trajectory estimation and static scene mapping, outperforming existing dynamic RGB SLAM baselines in long-term dynamic indoor environments.
Wenting Wang, Jiaxin Guo, Wenzhen Dong +3
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR. · Department of Mechanical, Aerospace and Civil Engineering, The University of Manchester, Manchester, UK. · Centre for Perceptual and Interactive Intelligence (CPII) Limited, Hong Kong SAR.