Vision-Language Navigation for Aerial Robots: Towards the Era of Large Language Models

Xingyu Xia; Lekai Zhou; Yujie Tang; Xiaozhou Zhu; Hai Zhu; Wen Yao

cs.ROApr 9, 2026

Vision-Language Navigation for Aerial Robots: Towards the Era of Large Language Models

Authors: Xingyu Xia, Lekai Zhou, Yujie Tang, Xiaozhou Zhu, Hai Zhu, Wen Yao

Organizations: Defense Innovation Institute, Chinese Academy of Military Sciences, Beijing 10071, China · Intelligent Game and Decision Laboratory, Beijing 100071, China · Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China

Abstract

Aerial vision-and-language navigation (Aerial VLN) aims to enable unmanned aerial vehicles (UAVs) to interpret natural language instructions and autonomously navigate complex three-dimensional environments by grounding language in visual perception. This survey provides a critical and analytical review of the Aerial VLN field, with particular attention to the recent integration of large language models (LLMs) and vision-language models (VLMs). We first formally introduce the Aerial VLN problem and define two interaction paradigms: single-instruction and dialog-based, as foundational axes. We then organize the body of Aerial VLN methods into a taxonomy of five architectural categories: sequence-to-sequence and attention-based methods, end-to-end LLM/VLM methods, hierarchical methods, multi-agent methods, and dialog-based navigation methods. For each category, we systematically analyze design rationales, technical trade-offs, and reported performance. We critically assess the evaluation infrastructure for Aerial VLN, including datasets, simulation platforms, and metrics, and identify their gaps in scale, environmental diversity, real-world grounding, and metric coverage. We consolidate cross-method comparisons on shared benchmarks and analyze key architectural trade-offs, including discrete versus continuous actions, end-to-end versus hierarchical designs, and the simulation-to-reality gap. Finally, we synthesize seven concrete open problems: long-horizon instruction grounding, viewpoint robustness, scalable spatial representation, continuous 6-DoF action execution, onboard deployment, benchmark standardization, and multi-UAV swarm navigation, with specific research directions grounded in the evidence presented throughout the survey.

Figures & tables

Fig. 1 : The evolution from UAV Navigation to Aerial VLN. Classical UAV navigation relies on the perception-planning-control loop to execute flight commands. Triggered by human-robot interaction (HRI), specifically natural-language instructions, Aerial VLN aligns visual perception with instructions, enabling UAVs to perform semantic action execution rather than solely following planned trajectories.

Fig. 2 : Structure and organization of the survey paper.

Fig. 3 : The problem definition of Aerial VLN.

Fig. 4 : The evolution of VLN. It progresses from the foundation phase, characterized by discrete indoor navigation and LSTM/CNN architectures, to the expansion phase involving continuous outdoor trajectories processed by Transformer. For future frontier phase, LLM-driven active reasoning empowers embodied and aerial agents to achieve universal adaptability in complex environments.

Dimension	Indoor VLN	Outdoor Ground VLN	Aerial VLN
Representative benchmarks	R2R [ 48 ] , VLN-CE [ 49 ]	Touchdown [ 50 ] , StreetLearn [ 51 ]	AerialVLN [ 13 ] , CityNav [ 47 ] OpenFly [ 33 ] , AVDN [ 14 ]
Configuration space	$SE(2)$ : 3-DoF	$SE(2)$ : 3-DoF	$SE(3)$ : 6-DoF
Action space	Discrete node-based movement or	Discrete panoramic turns or	Discrete 6-direction primitives or
	low-level continuous ( $v$ , $\omega$ )	street-level steps	continuous 6-DoF ( $v$ , $\omega$ )
Typical trajectory length	5–15 m	100–1000 m	100–500+ m
Avg. instruction length	$\sim$ 29 words (R2R)	$\sim$ 80 words (Touchdown)	$\sim$ 80–180 words (AerialVLN, OpenFly)

TABLE I : Comparative characteristics of VLN across indoor, outdoor ground, and aerial settings. Representative benchmarks are listed for each setting. Values are approximate and based on dataset statistics reported in the cited works.

Fig. 5 : Interaction paradigms of Aerial VLN: Aerial vision-and-instruction navigation (AVIN) vs. aerial vision-and-dialog navigation (AVDN).

Fig. 6 : The taxonomy of Aerial VLN methods.

Method	Input Repr.	Model Role	Action Format	Benchmarks
Huang et al. [ 73 ]	RGB + parsed NL	Rule-based parser	Discrete	Custom
Misra et al. [ 40 ]	RGB + NL	Goal predictor + actor	Discrete	LANI
Blukis et al. [ 41 , 74 ]	Semantic map	Visitation predictor	Continuous ( $v$ , $\omega$ )	Custom (simplified)
AerialVLN [ 13 ]	RGB-D + NL (sub-instr.)	Cross-modal decoder	Discrete (4-DoF)	AerialVLN, AerialVLN-S
DBDP [ 80 ]	RGB seq. + NL	Dual-branch encoder–decoder	Discrete	AerialVLN
Zhao et al. [ 43 ]	Multi-view skybox + NL	BEV selector + BERT	Discrete	AerialVLN

TABLE II : Summary of sequence-to-sequence and attention-based Aerial VLN methods.

Method	Input Repr.	Model Role	Action Format	Training	Benchmarks
NavAgent [ 31 ]	Multi-scale RGB + NL	VLM action predictor	Discrete	SFT	AerialVLN
FlightGPT [ 30 ]	RGB + NL + CoT prompt	VLM with CoT	Discrete	SFT + RL	AerialVLN
Gao et al. [ 44 ]	STMR matrix + NL	LLM with CoT	Discrete	Zero-shot	AerialVLN-S, OpenFly
GeoNav [ 92 ]	Cognitive map + RGB + NL	MLLM with CoT	Discrete	SFT	CityNav
LogisticsVLN [ 17 ]	RGB + NL	Multiple VLMs	Discrete	SFT	Custom (delivery)
UAV-ON [ 93 ]	4-dir RGB + NL + state	LLM action predictor	Discrete	SFT	UAV-ON

TABLE III : Summary of end-to-end LLM/VLM Aerial VLN methods.

Method	High-Level Planner	Low-Level Controller	Interface	Action Format
SkyVLN [ 32 ]	LLM + WPO	NMPC	Motion commands (text)	Continuous (via NMPC)
VLFly [ 106 ]	LLM encoder + VLM	ViNT waypoint planner	Visual target (image region)	Continuous ( $(v,w)$ )
AirStar [ 34 ]	LLM task planner	A* + Ego-Planner	Task spec (mode + skill)	Continuous (trajectory)
CityNavAgent [ 46 ]	LLM + global memory	Graph search + local controller	Topological graph nodes	Discrete (graph hops)
OpenUAV [ 45 ]	MLLM	Vision-based local planner	Target pose $\in SE(3)$	Continuous (6-DoF)

TABLE IV : Summary of hierarchical Aerial VLN methods. The interface column describes the intermediate representation between the high-level planner and the low-level controller.

	UAV-CodeAgents [ 35 ]	MMCNav [ 36 ]
Agent roles	Airspace manager + UAV agent	Observation + Planning + Execution + Feedback
Organization	Vertical (manager–executor)	Horizontal (process pipeline)
Communication	ReAct loop (text-based)	Sequential pipeline + feedback loop
Model backbone	LLM + VLM (both agents)	MLLM (all agents)
Action format	Trajectory waypoints	Discrete actions
Training	Zero-shot (ReAct prompting)	SFT

TABLE V : Summary of multi-agent Aerial VLN methods.

Method	Input Repr.	Model Role	Action Format	Training
HAA-Trans. [ 14 ]	Satellite RGB + dialog hist.	Cross-attn. encoder–decoder	Waypoints	Supervised
Su et al. [ 70 ]	Satellite RGB + dialog hist.	Graph-aware grounding + decoder	Waypoints	Supervised
Su et al. [ 71 ]	Satellite RGB + dialog segments	Fine-grained aligner + decoder	Waypoints	Supervised
Qiao et al. [ 62 ]	Satellite RGB + dialog hist.	Rotated detector + multi-stage pre-train	Waypoints	Multi-stage pre-train

TABLE VI : Summary of dialog-based Aerial VLN methods. All methods operate under the AVDN paradigm on the dataset of Fan et al. [ 14 ] .

Dataset	Category	Year	View	Scale	Key Characteristics
Dedicated Aerial VLN Datasets
LANI [ 40 ]	Training	2018	UAV	6K instr.	First aerial VLN corpus; 2D only
AerialVLN [ 13 ]	Training+Eval	2023	UAV	8.4K traj.	25 UE scenes; 4-DoF discrete
CityNav [ 47 ]	Training+Eval	2024	UAV	32K traj.	Real point clouds; language-goal nav
AeroVerse [ 122 ]	Pre-train+Eval	2024	UAV	10K–500K	Virtual pre-train to real fine-tune pipeline
EmbodiedCity [ 115 ]	Training+Eval	2024	UAV (multi)	–	Beijing CBD digital twin; dynamic traffic

TABLE VII : Unified overview of Aerial VLN datasets. Dedicated datasets pair flight trajectories with navigation instructions. Underexploited datasets contain UAV-perspective imagery with domain-specific annotations and potential for Aerial VLN applications. DoF: degrees of freedom of the action space. Modalities listed are those provided for each trajectory or image.

Fig. 7 : Typical simulation platforms: Gazebo [ 132 , 133 ] , Habitat [ 134 ] , AirSim [ 45 ] , Issac Sim [ 135 , 136 ] , Unity [ 137 , 138 ] , GTA-V [ 139 , 140 ]

Criterion	Gazebo	Habitat	AirSim	Isaac Sim	Unity	GTA-V
Visual realism	$\bullet\circ\circ$	$\bullet\bullet\circ$	$\bullet\bullet\bullet$	$\bullet\bullet\bullet$	$\bullet\bullet\circ$	$\bullet\bullet\bullet$
UAV flight dynamics	$\bullet\bullet\circ$	$\circ\circ\circ$	$\bullet\bullet\bullet$	$\bullet\bullet\circ$	$\bullet\circ\circ$	$\circ\circ\circ$
6-DoF continuous control	$\bullet\bullet\bullet$	$\circ\circ\circ$	$\bullet\bullet\bullet$	$\bullet\bullet\bullet$	$\bullet\bullet\circ$	$\circ\circ\circ$
Outdoor/city-scale env.	$\bullet\circ\circ$	$\circ\circ\circ$	$\bullet\bullet\bullet$	$\bullet\bullet\circ$	$\bullet\bullet\circ$	$\bullet\bullet\bullet$
Parallel training support	$\bullet\circ\circ$	$\bullet\bullet\bullet$	$\bullet\circ\circ$	$\bullet\bullet\bullet$	$\bullet\bullet\circ$	$\circ\circ\circ$
Ease of env. customization	$\bullet\bullet\circ$	$\bullet\circ\circ$	$\bullet\circ\circ$	$\bullet\circ\circ$	$\bullet\bullet\bullet$	$\circ\circ\circ$

TABLE VIII : Comparative assessment of simulation platforms for Aerial VLN. Ratings (

\bullet\bullet\bullet

= strong,

\bullet\bullet\circ

= moderate,

\bullet\circ\circ

= weak) reflect suitability for Aerial VLN specifically, not general robotics simulation. “Aerial VLN adoption” indicates whether the platform has been used in published Aerial VLN works.

Category	Metric	Adoption
Nav. Success & Path	SR (Success Rate)	Universal
	SPL (Success $\times$ Path Length)	Common
	nDTW (Norm. Dyn. Time Warp)	Moderate
	RMSE / MAE (Path Deviation)	Moderate
LLM Reasoning	ICR (Instruction Completion)	Rare
	ZGSR (Zero-shot Generalization)	Rare

TABLE IX : Taxonomy of evaluation metrics for Aerial VLN. Metrics are grouped by the dimension of performance they capture. “Adoption” indicates how widely each metric is reported in current Aerial VLN publications.

Method	Validation Seen				Validation Unseen
Method	SR (%) $\uparrow$	OSR (%) $\uparrow$	SDTW (%) $\uparrow$	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SDTW (%) $\uparrow$	NE (m) $\downarrow$
Random [ 13 ]	0.0	0.0	0.0	109.6	0.0	0.0	0.0	149.7
Seq2Seq [ 13 ]	4.8	19.8	1.6	146.0	2.3	11.7	0.7	218.9
CMA [ 13 ]	3.0	23.2	0.6	121.0	3.2	16.0	1.1	172.1
LAG [ 13 ]	7.2	15.7	2.4	90.2	5.1	10.5	1.4	127.9
Grid-based View [ 43 ]	20.8	33.4	10.2	70.3	7.4	16.1	2.5	121.3

TABLE X : Performance comparison of selected methods on the AerialVLN-S dataset.

Method	Validation Seen			Validation Unseen			Test Unseen
Method	SPL (%) $\uparrow$	SR (%) $\uparrow$	GP (%) $\uparrow$	SPL (%) $\uparrow$	SR (%) $\uparrow$	GP (%) $\uparrow$	SPL (%) $\uparrow$	SR (%) $\uparrow$	GP (%) $\uparrow$
Random [ 14 ]	0.5	1.6	-84.1	0.2	1.0	-81.4	0.5	1.1	-86.6
HAA-LSTM [ 14 ]	11.6	13.0	50.3	18.3	20.0	54.4	12.6	14.1	50.8
HAA-Transformer [ 14 ]	14.7	17.3	56.3	16.5	20.4	55.2	12.9	15.7	54.2
TG-GAT [ 70 ]	12.9	16.0	56.9	18.8	23.3	54.3	15.1	18.7	56.5
DBDP [ 80 ]	15.6	18.3	58.7	18.6	22.6	55.8	13.7	16.3	55.8

TABLE XI : Performance comparison of selected methods on the AVDN dataset.

Benchmark	Method	Metrics
Benchmark	Method	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$
OpenUAV Test Seen	Random	222.20	0.14	0.21	0.07
	CMA (100%) [ 48 ]	135.73	8.37	18.72	7.90
	TravelUAV (25%) [ 45 ]	132.59	11.59	24.50	10.45
	OpenVLN (25%) [ 95 ]	125.97	14.39	28.03	12.94
OpenUAV Unseen Map	Random	199.42	0.00	0.00	0.00

TABLE XII : Performance comparison of selected methods on the OpenUAV benchmark.

Method l	Validation Seen (Easy)				Validation Unseen (Medium)				Test Unseen (Hard)
Method l	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$
Seq2Seq [ 47 ]	58.50	8.43	17.31	7.28	78.60	5.13	10.90	4.65	98.10	3.81	13.92	2.79
CMA [ 47 ]	68.00	6.25	13.28	5.40	75.90	4.38	9.29	3.90	94.60	4.68	12.01	4.05
AerialVLN [ 47 ]	56.60	10.16	22.20	7.89	72.70	6.35	15.24	5.06	85.10	6.72	18.21	5.16
FlightGPT [ 30 ]	66.10	17.57	30.26	15.78	68.10	14.69	29.33	13.24	76.20	21.20	35.38	19.24
SA-GCS [ 30 ]	59.68	18.69	31.98	17.26	63.76	16.88	31.48	15.89	68.42	24.55	37.36	22.86

TABLE XIII : Performance comparison of selected methods on the CityNav benchmark.

Explore similar work

Vision-and-Language Navigation for UAVs: Progress, Challenges, and a Research Roadmap
Apr 15, 2026Hanxuan Chen, Jie Zheng, Siqi Yang +9Vision-Language Navigation Aerial VLN
Autel Robotics, Shenzhen, China · School of Intelligent Software and Engineering, Nanjing University, Nanjing, China · School of Computer, Data & Information Sciences, University of Wisconsin-Madison, Madison, WI, USA +3
LightVLN: Efficient Aerial Vision-and-Language Navigation with Compact Memory and History-Guided Local Aggregation
Oct 4, 2026Yiming Zhao, Tianshun Li, Jingle He +2Memory-Augmented VLMs Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigation
Apr 19, 2026Yuwei Ning, Ganlong Zhao, Yipeng Qin +4Vision-Language Navigation Aerial VLN
Sun Yat-sen University · Peng Cheng Laboratory · The Chinese University of Hong Kong +4

Apr 15, 2026J/K move · Enter open · S save

Vision-and-Language Navigation for UAVs: Progress, Challenges, and a Research Roadmap

Hanxuan Chen, Jie Zheng, Siqi Yang, Tianle Zeng+8

Autel Robotics, Shenzhen, China · School of Intelligent Software and Engineering, Nanjing University, Nanjing, China · School of Computer, Data & Information Sciences, University of Wisconsin-Madison, Madison, WI, USA+3

Vision-Language Navigation Aerial VLN UAV Navigation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-and-Language Navigation for Unmanned Aerial Vehicles (UAV-VLN) represents a pivotal challenge in embodied artificial intelligence, focused on enabling UAVs to interpret high-level human commands and execute long-horizon tasks in complex 3D environments. This paper provides a comprehensive and structured survey of the field, from its formal task definition to the current state of the art. We establish a methodological taxonomy that charts the technological evolution from early modular and deep learning approaches to contemporary agentic systems driven by large foundation models, including Vision-Language Models (VLMs), Vision-Language-Action (VLA) models, and the emerging integration of generative world models with VLA architectures for physically-grounded reasoning. The survey systematically reviews the ecosystem of essential resources simulators, datasets, and evaluation metrics that facilitates standardized research. Furthermore, we conduct a critical analysis of the primary challenges impeding real-world deployment: the simulation-to-reality gap, robust perception in dynamic outdoor settings, reasoning with linguistic ambiguity, and the efficient deployment of large models on resource-constrained hardware. By synthesizing current benchmarks and limitations, this survey concludes by proposing a forward-looking research roadmap to guide future inquiry into key frontiers such as multi-agent swarm coordination and air-ground collaborative robotics.

Abstract

Explore similar work

Vision-and-Language Navigation for UAVs: Progress, Challenges, and a Research Roadmap

LightVLN: Efficient Aerial Vision-and-Language Navigation with Compact Memory and History-Guided Local Aggregation

LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigation