Vision-Language Navigation for Aerial Robots: Towards the Era of Large Language Models
Authors: Xingyu Xia, Lekai Zhou, Yujie Tang, Xiaozhou Zhu, Hai Zhu, Wen Yao
Organizations: Defense Innovation Institute, Chinese Academy of Military Sciences, Beijing 10071, China · Intelligent Game and Decision Laboratory, Beijing 100071, China · Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China
Aerial vision-and-language navigation (Aerial VLN) aims to enable unmanned aerial vehicles (UAVs) to interpret natural language instructions and autonomously navigate complex three-dimensional environments by grounding language in visual perception. This survey provides a critical and analytical review of the Aerial VLN field, with particular attention to the recent integration of large language models (LLMs) and vision-language models (VLMs). We first formally introduce the Aerial VLN problem and define two interaction paradigms: single-instruction and dialog-based, as foundational axes. We then organize the body of Aerial VLN methods into a taxonomy of five architectural categories: sequence-to-sequence and attention-based methods, end-to-end LLM/VLM methods, hierarchical methods, multi-agent methods, and dialog-based navigation methods. For each category, we systematically analyze design rationales, technical trade-offs, and reported performance. We critically assess the evaluation infrastructure for Aerial VLN, including datasets, simulation platforms, and metrics, and identify their gaps in scale, environmental diversity, real-world grounding, and metric coverage. We consolidate cross-method comparisons on shared benchmarks and analyze key architectural trade-offs, including discrete versus continuous actions, end-to-end versus hierarchical designs, and the simulation-to-reality gap. Finally, we synthesize seven concrete open problems: long-horizon instruction grounding, viewpoint robustness, scalable spatial representation, continuous 6-DoF action execution, onboard deployment, benchmark standardization, and multi-UAV swarm navigation, with specific research directions grounded in the evidence presented throughout the survey.
Figures & tables
Fig. 1 : The evolution from UAV Navigation to Aerial VLN. Classical UAV navigation relies on the perception-planning-control loop to execute flight commands. Triggered by human-robot interaction (HRI), specifically natural-language instructions, Aerial VLN aligns visual perception with instructions, enabling UAVs to perform semantic action execution rather than solely following planned trajectories.
Fig. 2 : Structure and organization of the survey paper.
Fig. 3 : The problem definition of Aerial VLN.
Fig. 4 : The evolution of VLN. It progresses from the foundation phase, characterized by discrete indoor navigation and LSTM/CNN architectures, to the expansion phase involving continuous outdoor trajectories processed by Transformer. For future frontier phase, LLM-driven active reasoning empowers embodied and aerial agents to achieve universal adaptability in complex environments.
TABLE I : Comparative characteristics of VLN across indoor, outdoor ground, and aerial settings. Representative benchmarks are listed for each setting. Values are approximate and based on dataset statistics reported in the cited works.
Fig. 5 : Interaction paradigms of Aerial VLN: Aerial vision-and-instruction navigation (AVIN) vs. aerial vision-and-dialog navigation (AVDN).
Fig. 6 : The taxonomy of Aerial VLN methods.
Method
Input Repr.
Model Role
Action Format
Benchmarks
Huang et al. [ 73 ]
RGB + parsed NL
Rule-based parser
Discrete
Custom
Misra et al. [ 40 ]
RGB + NL
Goal predictor + actor
Discrete
LANI
Blukis et al. [ 41 , 74 ]
Semantic map
Visitation predictor
Continuous ( v , ω )
Custom (simplified)
AerialVLN [ 13 ]
RGB-D + NL (sub-instr.)
Cross-modal decoder
Discrete (4-DoF)
AerialVLN, AerialVLN-S
DBDP [ 80 ]
RGB seq. + NL
Dual-branch encoder–decoder
Discrete
AerialVLN
Zhao et al. [ 43 ]
Multi-view skybox + NL
BEV selector + BERT
Discrete
AerialVLN
TABLE II : Summary of sequence-to-sequence and attention-based Aerial VLN methods.
Method
Input Repr.
Model Role
Action Format
Training
Benchmarks
NavAgent [ 31 ]
Multi-scale RGB + NL
VLM action predictor
Discrete
SFT
AerialVLN
FlightGPT [ 30 ]
RGB + NL + CoT prompt
VLM with CoT
Discrete
SFT + RL
AerialVLN
Gao et al. [ 44 ]
STMR matrix + NL
LLM with CoT
Discrete
Zero-shot
AerialVLN-S, OpenFly
GeoNav [ 92 ]
Cognitive map + RGB + NL
MLLM with CoT
Discrete
SFT
CityNav
LogisticsVLN [ 17 ]
RGB + NL
Multiple VLMs
Discrete
SFT
Custom (delivery)
UAV-ON [ 93 ]
4-dir RGB + NL + state
LLM action predictor
Discrete
SFT
UAV-ON
TABLE III : Summary of end-to-end LLM/VLM Aerial VLN methods.
Method
High-Level Planner
Low-Level Controller
Interface
Action Format
SkyVLN [ 32 ]
LLM + WPO
NMPC
Motion commands (text)
Continuous (via NMPC)
VLFly [ 106 ]
LLM encoder + VLM
ViNT waypoint planner
Visual target (image region)
Continuous ( (v,w) )
AirStar [ 34 ]
LLM task planner
A* + Ego-Planner
Task spec (mode + skill)
Continuous (trajectory)
CityNavAgent [ 46 ]
LLM + global memory
Graph search + local controller
Topological graph nodes
Discrete (graph hops)
OpenUAV [ 45 ]
MLLM
Vision-based local planner
Target pose ∈SE(3)
Continuous (6-DoF)
TABLE IV : Summary of hierarchical Aerial VLN methods. The interface column describes the intermediate representation between the high-level planner and the low-level controller.
UAV-CodeAgents [ 35 ]
MMCNav [ 36 ]
Agent roles
Airspace manager + UAV agent
Observation + Planning + Execution + Feedback
Organization
Vertical (manager–executor)
Horizontal (process pipeline)
Communication
ReAct loop (text-based)
Sequential pipeline + feedback loop
Model backbone
LLM + VLM (both agents)
MLLM (all agents)
Action format
Trajectory waypoints
Discrete actions
Training
Zero-shot (ReAct prompting)
SFT
TABLE V : Summary of multi-agent Aerial VLN methods.
Method
Input Repr.
Model Role
Action Format
Training
HAA-Trans. [ 14 ]
Satellite RGB + dialog hist.
Cross-attn. encoder–decoder
Waypoints
Supervised
Su et al. [ 70 ]
Satellite RGB + dialog hist.
Graph-aware grounding + decoder
Waypoints
Supervised
Su et al. [ 71 ]
Satellite RGB + dialog segments
Fine-grained aligner + decoder
Waypoints
Supervised
Qiao et al. [ 62 ]
Satellite RGB + dialog hist.
Rotated detector + multi-stage pre-train
Waypoints
Multi-stage pre-train
TABLE VI : Summary of dialog-based Aerial VLN methods. All methods operate under the AVDN paradigm on the dataset of Fan et al. [ 14 ] .
Dataset
Category
Year
View
Scale
Key Characteristics
Dedicated Aerial VLN Datasets
LANI [ 40 ]
Training
2018
UAV
6K instr.
First aerial VLN corpus; 2D only
AerialVLN [ 13 ]
Training+Eval
2023
UAV
8.4K traj.
25 UE scenes; 4-DoF discrete
CityNav [ 47 ]
Training+Eval
2024
UAV
32K traj.
Real point clouds; language-goal nav
AeroVerse [ 122 ]
Pre-train+Eval
2024
UAV
10K–500K
Virtual pre-train to real fine-tune pipeline
EmbodiedCity [ 115 ]
Training+Eval
2024
UAV (multi)
–
Beijing CBD digital twin; dynamic traffic
TABLE VII : Unified overview of Aerial VLN datasets. Dedicated datasets pair flight trajectories with navigation instructions. Underexploited datasets contain UAV-perspective imagery with domain-specific annotations and potential for Aerial VLN applications. DoF: degrees of freedom of the action space. Modalities listed are those provided for each trajectory or image.
TABLE VIII : Comparative assessment of simulation platforms for Aerial VLN. Ratings ( ∙∙∙ = strong, ∙∙∘ = moderate, ∙∘∘ = weak) reflect suitability for Aerial VLN specifically, not general robotics simulation. “Aerial VLN adoption” indicates whether the platform has been used in published Aerial VLN works.
Category
Metric
Adoption
Nav. Success & Path
SR (Success Rate)
Universal
SPL (Success × Path Length)
Common
nDTW (Norm. Dyn. Time Warp)
Moderate
RMSE / MAE (Path Deviation)
Moderate
LLM Reasoning
ICR (Instruction Completion)
Rare
ZGSR (Zero-shot Generalization)
Rare
TABLE IX : Taxonomy of evaluation metrics for Aerial VLN. Metrics are grouped by the dimension of performance they capture. “Adoption” indicates how widely each metric is reported in current Aerial VLN publications.
Method
Validation Seen
Validation Unseen
SR (%) ↑
OSR (%) ↑
SDTW (%) ↑
NE (m) ↓
SR (%) ↑
OSR (%) ↑
SDTW (%) ↑
NE (m) ↓
Random [ 13 ]
0.0
0.0
0.0
109.6
0.0
0.0
0.0
149.7
Seq2Seq [ 13 ]
4.8
19.8
1.6
146.0
2.3
11.7
0.7
218.9
CMA [ 13 ]
3.0
23.2
0.6
121.0
3.2
16.0
1.1
172.1
LAG [ 13 ]
7.2
15.7
2.4
90.2
5.1
10.5
1.4
127.9
Grid-based View [ 43 ]
20.8
33.4
10.2
70.3
7.4
16.1
2.5
121.3
TABLE X : Performance comparison of selected methods on the AerialVLN-S dataset.
Method
Validation Seen
Validation Unseen
Test Unseen
SPL (%) ↑
SR (%) ↑
GP (%) ↑
SPL (%) ↑
SR (%) ↑
GP (%) ↑
SPL (%) ↑
SR (%) ↑
GP (%) ↑
Random [ 14 ]
0.5
1.6
-84.1
0.2
1.0
-81.4
0.5
1.1
-86.6
HAA-LSTM [ 14 ]
11.6
13.0
50.3
18.3
20.0
54.4
12.6
14.1
50.8
HAA-Transformer [ 14 ]
14.7
17.3
56.3
16.5
20.4
55.2
12.9
15.7
54.2
TG-GAT [ 70 ]
12.9
16.0
56.9
18.8
23.3
54.3
15.1
18.7
56.5
DBDP [ 80 ]
15.6
18.3
58.7
18.6
22.6
55.8
13.7
16.3
55.8
TABLE XI : Performance comparison of selected methods on the AVDN dataset.
Benchmark
Method
Metrics
NE (m) ↓
SR (%) ↑
OSR (%) ↑
SPL (%) ↑
OpenUAV Test Seen
Random
222.20
0.14
0.21
0.07
CMA (100%) [ 48 ]
135.73
8.37
18.72
7.90
TravelUAV (25%) [ 45 ]
132.59
11.59
24.50
10.45
OpenVLN (25%) [ 95 ]
125.97
14.39
28.03
12.94
OpenUAV Unseen Map
Random
199.42
0.00
0.00
0.00
TABLE XII : Performance comparison of selected methods on the OpenUAV benchmark.
Method l
Validation Seen (Easy)
Validation Unseen (Medium)
Test Unseen (Hard)
NE (m) ↓
SR (%) ↑
OSR (%) ↑
SPL (%) ↑
NE (m) ↓
SR (%) ↑
OSR (%) ↑
SPL (%) ↑
NE (m) ↓
SR (%) ↑
OSR (%) ↑
SPL (%) ↑
Seq2Seq [ 47 ]
58.50
8.43
17.31
7.28
78.60
5.13
10.90
4.65
98.10
3.81
13.92
2.79
CMA [ 47 ]
68.00
6.25
13.28
5.40
75.90
4.38
9.29
3.90
94.60
4.68
12.01
4.05
AerialVLN [ 47 ]
56.60
10.16
22.20
7.89
72.70
6.35
15.24
5.06
85.10
6.72
18.21
5.16
FlightGPT [ 30 ]
66.10
17.57
30.26
15.78
68.10
14.69
29.33
13.24
76.20
21.20
35.38
19.24
SA-GCS [ 30 ]
59.68
18.69
31.98
17.26
63.76
16.88
31.48
15.89
68.42
24.55
37.36
22.86
TABLE XIII : Performance comparison of selected methods on the CityNav benchmark.
Vision-and-Language Navigation for Unmanned Aerial Vehicles (UAV-VLN) represents a pivotal challenge in embodied artificial intelligence, focused on enabling UAVs to interpret high-level human commands and execute long-horizon tasks in complex 3D environments. This paper provides a comprehensive and structured survey of the field, from its formal task definition to the current state of the art. We establish a methodological taxonomy that charts the technological evolution from early modular and deep learning approaches to contemporary agentic systems driven by large foundation models, including Vision-Language Models (VLMs), Vision-Language-Action (VLA) models, and the emerging integration of generative world models with VLA architectures for physically-grounded reasoning. The survey systematically reviews the ecosystem of essential resources simulators, datasets, and evaluation metrics that facilitates standardized research. Furthermore, we conduct a critical analysis of the primary challenges impeding real-world deployment: the simulation-to-reality gap, robust perception in dynamic outdoor settings, reasoning with linguistic ambiguity, and the efficient deployment of large models on resource-constrained hardware. By synthesizing current benchmarks and limitations, this survey concludes by proposing a forward-looking research roadmap to guide future inquiry into key frontiers such as multi-agent swarm coordination and air-ground collaborative robotics.
Hanxuan Chen, Jie Zheng, Siqi Yang +9
Autel Robotics, Shenzhen, China · School of Intelligent Software and Engineering, Nanjing University, Nanjing, China · School of Computer, Data & Information Sciences, University of Wisconsin-Madison, Madison, WI, USA +3
Aerial vision-and-language navigation (VLN) enables unmanned aerial vehicles to execute long-horizon natural-language instructions from visual observations in complex three-dimensional environments. However, recent aerial VLN models often rely on large-scale vision-language backbones and dense visual histories, imposing substantial computation and memory costs that hinder onboard deployment. We propose LightVLN, a lightweight history-aware aerial VLN framework that combines a compact 0.5B language backbone with compact representations of both historical and current observations. LightVLN compresses each historical frame into a single token using visual features already computed by the policy. It further introduces history- and instruction-conditioned local aggregation to reduce the current observation from 256 to 32 visual tokens while preserving navigation-relevant spatial information. With up to 16 historical frames, the policy uses at most 48 observation-derived tokens. On the public OpenFly dataset, LightVLN achieves 50.93% Test-Seen and 36.14% Test-Unseen success rates (SR), outperforming the evaluated 7B language-backbone baselines on most reported metrics. It also achieves 25.83% SR on AerialVLN-S Val-Seen. In a reconstructed unseen campus, we deploy LightVLN on a DJI M350 RTK with an external Jetson Orin NX 16 GB for closed-loop onboard-compute real-to-sim hardware-in-the-loop (HIL) evaluation, achieving 14.61 Hz model inference and 11.13 Hz end-to-end decision updates. These results demonstrate the effectiveness and efficiency of LightVLN for aerial navigation.
Yiming Zhao, Tianshun Li, Jingle He +2
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
Aerial Vision-and-Language Navigation (Aerial VLN) enables unmanned aerial vehicles (UAVs) to follow natural language instructions and navigate complex urban environments. While recent advances have achieved progress through large-scale memory graphs and lookahead path planning, they remain limited by shallow instruction understanding and high computational cost. In particular, existing methods rely primarily on landmark descriptions, overlooking directional cues "a key source of spatial context in human navigation". In this work, we propose LookasideVLN, a new paradigm that exploits directional cues in natural language to achieve both more accurate spatial reasoning and greater computational efficiency. LookasideVLN comprises three core components: (1) an Egocentric Lookaside Graph (ELG) that dynamically encodes instruction-relevant landmarks and their directional relationships, (2) a Spatial Landmark Knowledge Base (SLKB) that provides lightweight memory retrieval from prior navigation experiences, and (3) a Lookaside MLLM Navigation Agent that aligns multimodal information from user instructions, visual observations, and landmark-direction information from ELG for path planning. Extensive experiments show that LookasideVLN significantly outperforms the state-of-the-art CityNavAgent, even with a single-level lookahead, demonstrating that leveraging directional cues is a powerful yet efficient strategy for Aerial VLN.
Yuwei Ning, Ganlong Zhao, Yipeng Qin +4
Sun Yat-sen University · Peng Cheng Laboratory · The Chinese University of Hong Kong +4