ScanDreamer: A Map-Grounded World Model for Quadrotor Navigation
Authors: Yunyi Gao, Yixing Shi, Yize Liu, Hai-Duong Tran, Nhat-Hoang P. Nguyen, Shuai Wang, Yi Deng, Shengming Liu, Sim Kuan Goh, Duc-Thinh Pham, Sameer Alam
Organizations: School of Artificial Intelligence and Robotics, Xiamen University Malaysia, Malaysia · Center for AI Research, VinUniversity, Hanoi, Vietnam · Air Traffic Management Research Institute, Nanyang Technological University, Singapore
Quadrotors inspecting buildings and tunnels must navigate unknown clutter with a forward-facing depth camera whose limited field of view constrains scene understanding and collision-avoiding navigation. Recent world-model agents learn navigation through imagined experience (e.g., AirDreamer), but remain data-intensive and often lack explicit geometric memory and awareness of what the camera can observe. We propose ScanDreamer, a map-grounded world model that constructs an online 2D scan-memory map from depth images and vehicle pose. Our world model, actor, and critic are conditioned on this map, which also defines a geodesic progress reward. Moreover, a forward-cone penalty discourages horizontal motion outside the camera's viewing direction. During imagination, ScanDreamer propagates the quadrotor pose over a frozen map and derives progress and wall-contact signals from the mapped geometry. We train ScanDreamer for 0.8 million environment steps (5.5 h on one GPU) and evaluate it in NVIDIA Isaac Sim over 1560 episodes spanning unseen layouts and parameter ranges. Across three seeds (mean ± SD), ScanDreamer achieves 96.3 ± 0.3% success with 3.6 ± 0.3% collision, compared with 59.6 ± 3.0% success and 28.8 ± 2.8% collision for AirDreamer at the same online-interaction budget. Removing the forward-cone penalty increases collisions to 9.9 ± 3.1%. These results show that combining explicit map memory, map-grounded imagination, and perception-aware motion substantially improves navigation performance and data efficiency. The source code is available at https://github.com/WANGSHUAIzzzz/ScanDreamer.
Figures & tables
Fig. 1: Overview of ScanDreamer. Top (flight): each depth image ot updates the scan-memory map Mt (black, occupied; gray, unobserved; shading, cost-to-go Tt , darker is lower), and crops and Tt at the pose ξt give the map feature mt and the plan vector pt . Bottom (imagination): the calibrated pose prior converts action sequences from this state into poses ξ^k on the frozen map, and the crops and Tt are read again at each ξ^k . The blue sequence replays the next 15 logged actions. Of the two others, the one toward the goal hits a mapped wall ( × ), and the sideways one incurs the forward-cone penalty (halo).
Training family
Scenes
Key parameters (training intervals)
Pillar forest
1,440
density 0.02–0.10 m -2 , gap 1.0–2.0 m
Wall field
1,440
1–8 walls of 3–10 m, slits 1.2–1.6 m
Traps
2,880
1–3 U, C, V, or L traps, depth 2.0–4.0 m
Combination
4,800
pillar forest, 1–2 traps, 0–3 walls
Corridor tree
6,240
2–3 branches, arms 3–9 m, width 2.1–2.5 m
Grid maze
2,880
5 × 5 to 7 × 7 cells of 2.6–3.5 m
TABLE I: Training Set and Test Suites
Fig. 2: Representative top-down test scenes and successful ScanDreamer trajectories (blue) from the start (circle) to the goal (star). Columns denote the five test suites and rows denote scene families. In-dist. and Param. cover all eight training families, whereas Struct. and Topo. show four of six and four of ten families, respectively. Mixed combines forests, walls, and traps; Fork is a T-junction with a misleading goal. For each family, we show the successfully solved scene whose detour ratio Rg is closest to the family median. Because Rg≈1 throughout open space, those rows instead show increasing pillar counts.
SR per suite (%) ↑
All suites pooled
Method
Steps
Seeds
In-dist.
Param.
Topo.
Struct.
Open
SR ↑
CR ↓
TO ↓
SPL ↑
NavRL [ 6 ] , released
–
–
28.5
27.9
8.6
47.3
100.0
33.8
11.3
54.5
0.324
NavRL, retrained
0.8M
1
5.3
4.3
0.5
12.3
45.0
8.4
37.1
54.5
0.077
NavRL, retrained
2.0M
1
15.1
12.1
0.5
5.7
88.0
15.6
16.7
67.6
0.144
TD-MPC2 [ 15 ]
0.8M
1
7.3
3.6
2.3
15.3
55.0
10.8
52.1
37.0
0.096
TD-MPC2
1.0M
1
5.6
3.6
2.7
11.7
48.0
8.9
51.9
39.0
0.076
TABLE II: Comparison with Baselines on the Five Test Suites
Variant
Map
Geo.
Cone
Imag.
Arr.
SR ↑
CR ↓
TO ↓
SPL ↑
Fwd. (%) ↑
Off-axis (%)
Appr. (s) ↓
Ours
✓
✓
✓
✓
✓
96.3 ± 0.3
3.6 ± 0.3
0.1 ± 0.0
0.831 ± 0.008
92.4 ± 0.9
35.8 ± 10.8
0.38 ± 0.01
w/o forward cone
✓
✓
✓
✓
89.9 ± 3.1
9.9 ± 3.1
0.1 ± 0.1
0.770 ± 0.021
40.1 ± 7.0
77.0 ± 3.2
0.40 ± 0.04
w/o arrival handling
✓
✓
✓
✓
96.2 ± 1.6
3.4 ± 1.6
0.3 ± 0.1
0.815 ± 0.027
86.4 ± 2.3
40.2 ± 9.2
1.39 ± 0.30
w/o cone and arrival
✓
✓
✓
90.3 ± 3.3
9.2 ± 3.1
0.5 ± 0.2
0.757 ± 0.029
36.4 ± 7.4
74.2 ± 13.6
1.02 ± 0.27
Geodesic reward only
✓
51.6 ± 2.5
32.5 ± 8.8
14.4 ± 9.7
0.431 ± 0.034
48.7 ± 3.5
83.6 ± 4.1
0.99 ± 0.38
TABLE III: Ablation of ScanDreamer (0.8M Environment Steps)
Fig. 3: Outcomes of the 1560 test episodes, binned by detour ratio Rg (770, 390, and 400 per bin). Segments below 15% are not labeled.
Fig. 4: Success rate of training episodes in windows of 50k environment steps, shown as the mean (line) and range (band) over training runs. Black-edged markers are the checkpoints tested in Table II .
Fig. 5: A test flight of ScanDreamer through a 16-room building of the in-distribution suite ( Rg=3.49 ). The rows show the depth image ot , the scan-memory map Mt built so far, and the ground truth with the camera’s field of view. The map Mt is colored by the cost-to-go Tt and gray where not yet observed, and the dashed line follows the steepest descent of Tt . At 7.7, 12.0, and 16.5 s, a newly mapped wall has just closed the route. The right column shows the full paths of the three runs of ScanDreamer, two of them pale, and of one AirDreamer run after 1.0M steps, whose end is marked by a square.
Fig. 6: Direction of horizontal motion relative to the camera axis at 0.8M steps, as shares of the distance flown in successful test episodes. The right column gives the off-axis collisions among all collisions.
Agile quadrotor flight in cluttered scenes requires more than a reactive mapping from a depth image to a control command: the vehicle must remember which regions have been observed, infer nearby occupied space, and act under partial visibility and tight latency. In this paper, we present Mapping-Aware Dreamer (MAD), a geometry-aware world model for vision-based quadrotor flight. Instead of using raw-image reconstruction as the main self-supervised objective, MAD learns recurrent latent dynamics that reconstruct robocentric occupancy and visibility grid maps together with proprioceptive states. This design forces the latent state to encode local geometry, visibility history, and ego-motion in a form that is directly relevant to collision avoidance. MAD is trained in DiffAero using a GPU-parallel map-construction module that provides high-throughput supervision for occupancy and visibility. The learned representation is used in three policy-learning modes: imagination-based MAD-Dreamer and feature-extractor variants based on PPO and SHAC. Across visual navigation and racing tasks, MAD-based agents achieve higher success rates, faster flight, and better cross-task transfer than corresponding vision-only baselines. The model also produces interpretable map predictions and accurate ego-motion estimates from depth observations. We further deploy the learned policy on a physical quadrotor with an Intel RealSense D435i and demonstrate safe indoor and outdoor flight under limited sensing, reaching 9.66 m/s in simulation and 5.05 m/s in real-world forest experiments. These results show that mapping-aware world models provide a practical middle ground between modular aerial navigation and end-to-end learning.
Xinhong Zhang, Runqing Wang, Yunfan Ren +6
State Key Lab of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China · Zhongguancun Academy, Beijing 100094, China · School of Computer Science and Technology, Tongji University +2
Navigating a drone in unseen and cluttered environments requires reliable generalization to unseen scene layouts and understanding of environmental structure relative to the robot's capabilities. Previous methods, which assume the same environment configuration, often rely heavily on human-designed perception pipelines and predefined rules to guide the robot toward the target. This process is environment-dependent and generalizes poorly across environments. Inspired by animal navigation behavior, we design a navigation framework that navigates with a reinforcement-learning-based policy on top of a world-model-based environment understanding to overcome these issues. In addition, a sparse reward function without hand-crafted shaping terms is designed to avoid local minima traps and encourage yaw control behaviors. In simulation and on real drones, our method exhibits emergent capabilities for navigating complex, unseen environments and escaping local optima where other methods fail. In challenging maps, it achieves a 5.3% higher navigation success rate than best baseline. Furthermore, the proposed framework achieves effective sim-to-real transfer without any tuning during deployment. The code will be publicly available.
Zian Liu, Andong Yang, Chunkai Yang +3
Institute for AI Industry Research, Tsinghua University, Beijing, China. · Department of Electronic Engineering, Tsinghua University, Beijing, China. · School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, China.
DreamerV3-style reconstructive world models offer a route to vision-based navigation for aerial robots through learned latent dynamics. Yet their behavior under shifts in the training environment and their ability to support navigation using imagined observations, predictions generated by rolling the latent dynamics forward without new sensory input, remain insufficiently characterized. We study four training-environment regimes for depth-based quadrotor navigation, spanning a fixed layout, a finite set of layouts, low-discrepancy continuous obstacle sampling, and i.i.d. uniform sampling. We cross-evaluate reconstruction before and after Reinforcement Learning fine-tuning, separately evaluate the resulting policies in simulation, and transfer them to a real quadrotor without real-world fine-tuning. We demonstrate onboard flight driven by imagined state and depth: a 24.7m mission in which 66% of flight time and 76% of traveled distance occur during repeated 5s sensor-free imagination windows, with prediction quality degrading over each open-loop interval and recovering after sensory re-grounding. Additionally, in closed-loop hardware transfer, all policies succeed in a setting close to the simulated geometry, whereas the policy trained from a single fixed layout fails under a corridor-topology and obstacle-type shift, while the other three succeed.