cs.AIApr 9, 2026
SaveHow Far Are Large Multimodal Models from Human-Level Spatial Action? A Benchmark for Goal-Oriented Embodied Navigation in Urban Airspace
Organizations: Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Northeastern University, Qinhuangdao, China · National University of Defense Technology, Changsha, China · Shandong University, Jinan, China ·
Abstract
Large multimodal models (LMMs) show strong visual-linguistic reasoning but their capacity for spatial decision-making and action remains unclear. In this work, we investigate whether LMMs can achieve embodied spatial action like human through a challenging scenario: goal-oriented navigation in urban 3D spaces. We first spend over 500 hours constructing a dataset comprising 5,037 high-quality goal-oriented navigation samples, with an emphasis on 3D vertical actions and rich urban semantic information. Then, we comprehensively assess 17 representative models, including non-reasoning LMMs, reasoning LMMs, agent-based methods, and vision-language-action models. Experiments show that current LMMs exhibit emerging action capabilities, yet remain far from human-level performance. Furthermore, we reveal an intriguing phenomenon: navigation errors do not accumulate linearly but instead diverge rapidly from the destination after a critical decision bifurcation. The limitations of LMMs are investigated by analyzing their behavior at these critical decision bifurcations. Finally, we experimentally explore four promising directions for improvement: geometric perception, cross-view understanding, spatial imagination, and long-term memory. The project is available at: https://github.com/serenditipy-AC/Embodied-Navigation-Bench.
Figures & tables
Figure 1. Overview of the proposed benchmark. Goal-oriented embodied navigation in urban airspace is defined as: given linguistic instructions, the agent takes progressive actions based on continuous embodied observations to approach the target location. Current LMMs still exhibit a substantial gap compared to humans in spatial action.
| Benchmark | Simulator | Dataset Task Type | Action Distribution | LMMs |
| AVDN ( Fan et al., 2023b ) | Satellite Images | Dialog Navigation | Pure Horizontal | ✗ |
| AerialVLN ( Liu et al., 2023b ) | UE + Airsim | Route-Oriented Navigation | Horizontal-Dominant | ✗ |
| AeroVerse ( Yao et al., 2024 ) | UE + Airsim | Route-Oriented Navigation | Horizontal-Dominant | ✓ |
| EmbodiedCity ( Gao et al., 2024b ) | UE + Airsim | Route-Oriented Navigation | Horizontal-Dominant | ✓ |
| CityNav ( Lee et al., 2024 ) | WebGL-Based | Goal-Oriented Navigation | Horizontal-Dominant | ✗ |
| TRAVEL ( Wang et al., 2024 ) | UE + Airsim | Goal-Oriented Navigation | Horizontal-Dominant | ✗ |
Table 1. Benchmarks (simulators and datasets) for aerial embodied navigation in cities.
Figure 2. a. Dataset Construction Pipeline. b. The length distribution of navigation trajectories. c. Proportion of various types of actions. d. The relative position of trajectories to the origin. e. Word cloud of goal instructions.
Table 2. Goal-oriented embodied navigation results of the popular LMMs and agent-based approaches. The short, middle, and long groups correspond to ground truth trajectories of less than 118.2 meters, between 118.2 meters and 223.6 meters, and greater than 223.6 meters, respectively.
Figure 3. The change in navigation completion progress (%) as a function of navigation steps. Navigation completion progress is defined as the ratio between the distance from the current step to the goal and the distance from the initial position to the goal. The star-marked point is defined as the Critical Decision Bifurcation (CDB), which characterizes the spiral of divergence in LMM’s embodied navigation.
Figure 4. A goal-oriented embodied navigation case: GPT-5.1’s trajectory, along with its embodied observations and thought processes at key moments, demonstrated human-like reasoning and action output.
Figure 5. The gaps between LMMs and humans in spatial actions can be summarized into four aspects: a. Insufficient ability in urban semantic perception. b. Limited cross-view scene understanding. c. Lack of common sense about the consequences of actions. d. Deficiency in long-term memory and planning. A third-person perspective view comparing the ground truth and the LMM-navigated trajectories. The green line represents the ground truth path, while the red line depicts the actual route taken by the LMM. Stars denote critical decision bifurcations (CDBs).
Figure 6. Experimental design to enhance spatial action capability of LMMs: a. Geometric Perception Enhancement. b. Cross-View Understanding Enhancement. c. Spatial Imagination. d. Sparse Memory.
| Method | Average | ||
| SR | SPL | DTG | |
| Backbone | 14.7 | 12.8 | 120.6 |
| + Geometric Perception | 24.2^{\color[rgb]{0,0.7,0}\uparrow 9.5} | 22.9^{\color[rgb]{0,0.7,0}\uparrow 10.1} | 138.9^{\color[rgb]{1,0,0}\uparrow 18.3} |
| + Cross-View Understanding | 25.0^{\color[rgb]{0,0.7,0}\uparrow 10.3} | 16.7^{\color[rgb]{0,0.7,0}\uparrow 3.9} | 118.6^{\color[rgb]{0,0.7,0}\downarrow 2.0} |
| + Spatial Imagination | 19.0^{\color[rgb]{0,0.7,0}\uparrow 4.3} | 16.7^{\color[rgb]{0,0.7,0}\uparrow 3.9} | 115.9^{\color[rgb]{0,0.7,0}\downarrow 4.7} |
| + Sparse Memory | 17.0^{\color[rgb]{0,0.7,0}\uparrow 2.3} | 16.3^{\color[rgb]{0,0.7,0}\uparrow 3.5} | 134.3^{\color[rgb]{1,0,0}\uparrow 13.7} |
Table 3. Further exploration for LMM improvement.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Navigation dataset examples.
| Model | Input Cost ($) | Output Cost ($) | Total Cost ($) |
| Gemini-2.0-Flash | 157.70 | 41.03 | 198.73 |
| Gemini-2.5-Flash | 315.41 | 170.94 | 486.35 |
| Gemini-2.5-Pro | 1314.20 | 683.75 | 1997.95 |
| GLM-4.6V | 150.62 | 29.39 | 180.01 |
| OpenAI o4-mini | 1156.49 | 300.85 | 1457.35 |
| GPT-4o | 2628.40 | 683.75 | 3312.15 |
Table 4. Evaluation Cost of Different Models
Figure 8. Navigation prompt details.
| Step | Obs | Thinking | Action |
| 0 | We appear to be in or near a residential/commercial complex with tall apartment buildings and a low-rise row of shops ahead. The pavilion is said to be “inside the residential area ahead,” so it’s likely further forward among or behind the front row of buildings. Adjusting the gimbal down will give better situational awareness before advancing. | angle down | |
| 1 | The drone is in a residential/commercial plaza area. The camera is tilted down at -45°, showing mostly the ground near the drone. The pavilion is said to be inside the residential area ahead. Advancing forward will bring us closer to the inner area where a pavilion is likely located. | move forth | |
| 2 | We are moving across an open paved area toward the row of shops and trees that mark the edge of the residential zone. The pavilion should be further inside this area. The camera at -45° gives a good view of the ground and the base of the buildings ahead. We should continue flying straight forward. | move forth | |
| 3 | We are now close to the edge of the open plaza, with a walkway, planting strip, and trees ahead. The pavilion should be further inside the residential area, likely beyond this tree line. Continuing straight forward will bring us into the landscaped zone where a pavilion is more likely to appear. | move forth | |
| 4 | We are now just in front of the tree-lined planting strip and walkway near the shopfronts. The pavilion should be further into the residential interior. We should continue flying straight ahead over the walkway and closer to the buildings, maintaining the current gimbal angle for good ground visibility. | move forth | |
| 5 | We have reached the sidewalk and green belt directly in front of the shops; trees are now close beneath us. To avoid getting too close to tree canopies while continuing to advance, it’s safer to gain a bit more altitude first, then proceed forward afterward. | move up |
Table 5. Action Process of GPT-5.1
| Step | Obs | Thinking | Action |
| 17 | The pavilion is now clearly visible ahead and slightly to the right, standing on a paved area near the path. We are at a good altitude with no high obstacles between us and the pavilion. Continuing to fly straight forward will bring us directly closer to the pavilion. | move forth | |
| 18 | The pavilion is now slightly ahead and to the right, clearly visible on the paved area. We are nearly above the grassy area in front of it. Moving straight will bring us closer while still maintaining safe clearance. We should continue advancing forward. | move forth | |
| 19 | We are now very close to the pavilion, which is ahead-right on the paved area. To position the drone more directly over or beside the pavilion as the final navigation target, we should move slightly closer along our current heading. | move forth | |
| 20 | The pavilion is now partially visible at the bottom-right of the image; we have flown slightly past and to the left of it. To better align with and be above the pavilion, we should shift our position laterally to the right rather than continue forward. | move right |
Table 6. Action Process of GPT 5.1
| Step | Obs | Thinking | Action |
| 0 | The drone is hovering over a large open plaza with a row of shops at the base of tall residential towers directly ahead. The customer’s designated pavilion is likely located within the residential area beyond the shops, so it’s currently out of view at the horizontal angle. Lowering the gimbal will reveal ground-level features and help identify pathways or the pavilion structure itself before advancing further. | adjust camera gimbal downwards | |
| 1 | The camera is currently pitched at –45°, looking mostly at the ground and the row of shops. To identify and frame the pavilion farther ahead in the residential complex, I need to restore the camera to a horizontal view. Once leveled, I can then spot the pavilion and advance directly toward it. | adjust camera gimbal upwards | |
| 2 | With the gimbal leveled at 0°, I can see the front row of shops at the base of the residential towers. The pavilion is further inside the complex and not yet visible, so I need to advance the drone toward the buildings to close the distance before I can spot and align with the pavilion. | move forth | |
| 3 | The drone is now closer to the front row of shops but the pavilion is still hidden behind. Continuing forward will bring the drone nearer to the residential complex entrance, allowing the pavilion to come into view. | move forth | |
| 4 | The row of shops still blocks the pavilion from view even after advancing twice. To bring the pavilion into view and reach the residential area entrance, I should continue moving forward. | move forth | |
| 5 | The camera is currently pitched at –45°, looking mostly at the ground and the row of shops. To identify and frame the pavilion farther ahead in the residential complex, I need to restore the camera to a horizontal view. Once leveled, I can then spot the pavilion and advance directly toward it. | adjust camera gimbal upwards |
Table 7. Action Process of o4-mini
| Step | Obs | Thinking | Action |
| 17 | I’m now just left of the pavilion hidden behind the white box structure. Shifting to the right will bring the drone directly over the pavilion so I can center it before closing in further. | move right | |
| 18 | The pavilion is now visible to the right of the white utility building and I’m aligned to enter the courtyard. Advancing forward will close the distance and position the drone directly over the pavilion. | move forth | |
| 19 | The pavilion is still partially obscured by the white utility building in the center. Shifting right will bring the drone directly over the pavilion area and center it in frame before advancing further. | move right | |
| 20 | The small pavilion is still partially hidden behind the white utility building. Shifting the drone to the right will center the pavilion in view and clear the obstruction before the final forward approach. | move right | |
| 21 | The pavilion is now fully visible to the right of the utility building and well within my view. Advancing forward will close the remaining distance and position the drone directly above the pavilion for a close inspection. | move forth | |
| 22 | The pavilion is now directly ahead and clearly visible at ground level. Advancing forward one more time will place the drone directly above the pavilion for a close-up view. | move forth |
Table 8. Action Process of o4-mini
| Step | Obs | Thinking | Action |
| 0 | I am currently positioned in front of tall residential buildings with a commercial strip at their base and a wide empty paved area. The camera gimbal is horizontal (0.0°). I cannot see any pavilion yet. The most logical first step is to move forward to get closer to the residential area and potentially reveal the pavilion. | move forth | |
| 1 | The current image is identical to the previous one. The goal is still “a pavilion inside the residential area ahead” and I still cannot see it. To find the pavilion, I need to move into the residential area. The most direct path is forward. | move forth | |
| 2 | I have moved forward since the last image. The commercial strip now appears closer. However, I still cannot see the pavilion, which is expected to be “inside the residential area ahead.” To reach the residential area, I need to continue moving past the commercial establishments. | move forth | |
| 3 | The drone has moved forward again. The commercial strip is now even closer. I still cannot see any pavilion. It’s likely located past this commercial strip, within the residential complex. Moving forward is still the best option to continue approaching the residential area. | move forth | |
| 4 | The drone has moved closer to the commercial strip. The street and the grassy median are now prominent. I am still at the external boundary of the residential area. Continuing to move forward is the most direct way to get past the shops and into the residential complex. | move forth | |
| 5 | I have repeatedly moved forward and am now very close to the commercial strip. The previous strategy of simply moving forward is no longer viable as it would lead to collision. Gaining elevation (move_up) appears to be the most promising action to survey the area and locate the pavilion from a higher vantage point. | move up |
Table 9. Action Process of Gemini-2.5-Flash
| Step | Obs | Thinking | Action |
| 17 | I have successfully moved up, bringing the drone above the obstructing foliage, and now have a clear overhead view of the glass pavilion. Since I am currently directly above it, the next logical step is to descend to its level. | move down | |
| 18 | I have descended by 10 meters and my view is now once again obstructed by dense foliage. To get a clear view of the pavilion and its surroundings again, I need to move the drone upwards above the foliage. | move up | |
| 19 | I have successfully moved the drone up again, placing it above the obstructing foliage, and now have a clear overhead view of the glass pavilion. Since the drone is now directly above the target, the most direct action to reach the pavilion is to descend. | move down |
Table 10. Action Process of Gemini-2.5-Flash
| Step | Obs | Thinking | Action |
| 0 | The current scene shows an urban setting with tall buildings on either side. To navigate towards the pavilion located within this residential area, we need to adjust our heading slightly away from these structures. | adjust camera gimbal downwards | |
| 1 | Given the current camera angle at ‘-45‘ degrees, which tilts downwards significantly, my next step would involve adjusting both the camera orientation upwards while also preparing to move forward toward potential pathways leading into the residential area. This upward movement helps clear obstructed views before proceeding further. | adjust camera gimbal upwards | |
| 2 | Currently, there seems no immediate obstacle blocking direct progress; however, given the layout visible here—wide open spaces flanked by high-rise buildings—it might make sense initially to move straight ahead if possible without turning right immediately due to potentially narrow paths or tight corners nearby not clearly seen yet. | move forth | |
| 3 | From the image description, it appears like there’s some kind of pathway or road extending directly ahead between two rows of buildings. Given the goal is to reach a pavilion likely situated somewhere along this path, continuing straight could lead us closer to achieving that objective efficiently since other directions seem less promising based solely on visual cues available now. | move forth | |
| 4 | Based on the current perspective shown above, it looks like we’re positioned near but still outside a row of shops or commercial establishments. There doesn’t appear to be another obvious route branching off just beyond them unless hidden behind shadows or obscured elements out-of-frame. Since the goal involves reaching something specifically labeled "pavilion," let me consider whether advancing straight may reveal more details about its exact position relative to those buildings. | move forth | |
| 5 | Looking closely at the image again, despite being close enough visually to see part of the building structure resembling a pavilion-like design among others, it isn’t entirely certain how far back they extend nor their precise positioning compared against surrounding areas. However, considering proximity suggests continued advancement might help clarify surroundings further. | move forth |
Table 11. Action Process of Qwen2.5-VL-7B
| Step | Obs | Thinking | Action |
| 18 | Progression forward didn’t yield any noticeable signs of a pavilion. Considering the dense foliage and lack thereof, it might require altering trajectory to explore other possibilities. | turn left | |
| 19 | The current view shows a well-maintained pedestrian path alongside buildings, but no evident signs of a pavilion. Turning left might reveal more options or confirm the absence altogether. | turn right | |
| 20 | Since the last turn-right maneuver showed no signs of a pavilion, it means we might want to return to the original path or explore further along parallel ways. Given the consistent pattern observed throughout explorations, maintaining steady forward movement aligns best with systematic navigation strategies. | move forth | |
| 21 | This view presents an empty indoor corridor, which deviates sharply from expectations when seeking a pavilion outdoors. Returning to outdoor navigation ensures adherence strictly to planned goals. | move forth | |
| 22 | The current view depicts an interior space, which contradicts the goal of navigating to a pavilion within the residential area. It signifies deviation from expected external navigation patterns. | move forth | |
| 23 | The current view displays a floor texture, indicating that the drone has landed or entered an enclosed space unexpectedly. This situation disrupts ongoing navigation efforts directed towards exiting the residential area and reaching the pavilion. | move up |
Table 12. Action Process of Qwen2.5-VL-7B
Table 13. Action Process of Openfly
More in this topic
Embodied Navigation
Embodied navigation studies how robots perceive, map, and move through environments to reach goals and follow instructions.
See all 134 papers