cs.ROApr 15, 2026
SaveVision-and-Language Navigation for UAVs: Progress, Challenges, and a Research Roadmap
Organizations: Autel Robotics, Shenzhen, China · School of Intelligent Software and Engineering, Nanjing University, Nanjing, China · School of Computer, Data & Information Sciences, University of Wisconsin-Madison, Madison, WI, USA · Southern University of Science and Technology, Shenzhen, China · The University of Hong Kong, Hong Kong SAR, China · School of Software and Microelectronics, Peking University, Beijing, China
Abstract
Vision-and-Language Navigation for Unmanned Aerial Vehicles (UAV-VLN) represents a pivotal challenge in embodied artificial intelligence, focused on enabling UAVs to interpret high-level human commands and execute long-horizon tasks in complex 3D environments. This paper provides a comprehensive and structured survey of the field, from its formal task definition to the current state of the art. We establish a methodological taxonomy that charts the technological evolution from early modular and deep learning approaches to contemporary agentic systems driven by large foundation models, including Vision-Language Models (VLMs), Vision-Language-Action (VLA) models, and the emerging integration of generative world models with VLA architectures for physically-grounded reasoning. The survey systematically reviews the ecosystem of essential resources simulators, datasets, and evaluation metrics that facilitates standardized research. Furthermore, we conduct a critical analysis of the primary challenges impeding real-world deployment: the simulation-to-reality gap, robust perception in dynamic outdoor settings, reasoning with linguistic ambiguity, and the efficient deployment of large models on resource-constrained hardware. By synthesizing current benchmarks and limitations, this survey concludes by proposing a forward-looking research roadmap to guide future inquiry into key frontiers such as multi-agent swarm coordination and air-ground collaborative robotics.
Figures & tables
Fig. 1: An overview of the UAV-based Vision-and-Language Navigation (UAV-VLN) research landscape, illustrating the core components and methodological evolution. The process begins with a natural language command that directs an autonomous agent. This figure contrasts the traditional modular pipeline, which separates perception, reasoning, and control, with the modern integrated approach centered on Embodied Multimodal Large Models (EMLMs). The UAV executes the resulting action in a complex 3D world, navigating key challenges such as the simulation-to-reality gap. The bottom panel outlines the structure of this survey, mapping the progression from foundational definitions to a forward-looking research roadmap.
Fig. 2: The UAV-VLN task modeled as a Partially Observable Markov Decision Process (POMDP). The agent receives an observation and reward from the environment, updates its internal belief state, and selects an action based on its policy to influence the environment’s next state.
Fig. 3: A comparison of action space formulations for UAV navigation. Low-level continuous control offers fine-grained maneuverability at the cost of planning complexity, whereas high-level discretized actions simplify decision-making but can result in less efficient paths.
| Feature | Classical Decomposed Robotics Pipelines | Modular Deep Learning Architectures | Emergence of End-to-End Learning Policies |
|---|---|---|---|
| Architectural Approach | Decomposed Algorithmic Stages | Modular Neural Components | Direct Sensor-to-action Mapping |
| Core Component | Classical Planner/controller | Cross-modal Fusion Module | Unified Visuomotor Policy |
| Generated Output | Explicit Geometric Trajectory | Fused Feature Representation | Direct Flight Commands |
TABLE I: A comparative analysis of foundational architectural paradigms in early UAV-VLN systems. The table illustrates the evolution from classical decomposed robotics pipelines to modular deep learning architectures and end-to-end policies, contrasting their core architectural approach, primary components, and generated outputs.
| Paradigm | Method | Year | Model Type | Core Technique | Key Contribution |
|---|---|---|---|---|---|
| Modular & Early Learning | CMA / LAG baselines [ 29 ] | 2023 | CNN-RNN | Cross-modal attention | First large-scale UAV-VLN benchmark baselines |
| RRT + NLU pipeline [ 73 ] | 2023 | Hybrid | RNN parser + RRT planner | NLU-guided classical planning for UAVs | |
| SLAM + Visual Servoing [ 74 ] | 2025 | Classical | SLAM + IBVS control | GPS-denied navigation with decoupled perception-control | |
| DRL Obstacle Avoidance [ 75 ] | 2021 | DRL | DQN / A2C policies | End-to-end visuomotor learning for collision avoidance | |
| Memory-based DRL [ 54 ] | 2018 | DRQN | CNN-RNN + RL | Recurrent memory for sequential navigation at 60 Hz | |
| DAGGer + Diffusion [ 60 ] | 2021 | IL + Diffusion | Imitation + diffusion policy | Multimodal action distributions for VLN-CE |
TABLE II: A comprehensive summary of representative UAV-VLN methods organized by methodological paradigm. For each method, we detail the core technical approach, the model type, and the key contribution to the field. VLM = Vision-Language Model; VLA = Vision-Language-Action Model; LLM = Large Language Model; RL = Reinforcement Learning; IL = Imitation Learning.
Fig. 4: A timeline of the methodological evolution in UAV Vision-and-Language Navigation (UAV-VLN). The figure charts the progression from early modular deep learning systems to VLM-powered world models, the emergence of VLA models, the development of open-source VLA foundations, the rise of flow-based VLA and generative world models, and culminating in the current era of world model and VLA integration for physically-grounded embodied reasoning.
| Paradigm | Sub-Category | Core Approach | Characteristics & Limitations |
|---|---|---|---|
| Modular & Early Learning Approaches | Classical Decomposed Robotics Pipelines | Decomposes navigation into discrete stages: perception (SLAM), planning (A*, RRT), and control (sense-think-act). | High interpretability; suffers from brittleness and cascading error propagation. |
| Modular Deep Learning Architectures | Replaces classical modules with CNNs (vision) and RNNs (language); uses cross-modal attention for fusion. | More robust perception; creates information bottlenecks and struggles with complex reasoning. | |
| Emergence of End-to-End Learning Policies | Learns direct sensor-to-action mapping via DRL or Imitation Learning, bypassing explicit intermediate representations. | Conceptually simpler; challenged by sample inefficiency and long-term credit assignment. | |
| Architectures for Long-Horizon Spatiotemporal Understanding | Temporal Transformers | Treats navigation as a sequence modeling problem; uses transformer self-attention to encode full trajectory history. | Enables direct access to past events and improves long-term memory over recurrent models. |
| Visual Language Maps | Reframes navigation as spatial reasoning; fuses VLM features with geometric reconstructions for queryable maps. | Grounds language in the physical environment; high computational cost and sensitivity to odometry errors. | |
| Cognitive Maps & Scene Graphs | Employs multi-scale spatial memories with global cognitive maps and local scene graphs for coarse-to-fine reasoning. | Mimics human navigation strategies; computationally complex in large-scale environments. |
TABLE III: A methodological taxonomy of UAV Vision-and-Language Navigation (UAV-VLN) agents, organized into three main paradigms: Modular and Early Learning Approaches, Architectures for Long-Horizon Spatiotemporal Understanding, and Foundation Model-Driven Agentic Systems. Each paradigm is further decomposed into its constituent sub-categories with their core approaches and key characteristics.
Fig. 5: This figure illustrates the evolution of early learning approaches in UAV-VLN, categorized into three primary paradigms. It begins with Classical Decomposed Robotics Pipelines that separate tasks into discrete stages, progresses to Modular Deep Learning Architectures where classical components are replaced by neural networks, and culminates in the Emergence of End-to-End Learning Policies that learn direct sensor-to-action mappings.
Fig. 6: Two primary architectural paradigms for long-horizon navigation. Temporal history encoding (left) treats the task as a sequence modeling problem, using transformers to reason over past observations and actions, while explicit world modeling (right) reframes it as a spatial reasoning task by constructing a queryable semantic map of the environment.
Fig. 7: Architectural paradigms for foundation model-driven agents. VLMs can act as a deliberative cognitive core to generate high-level plans (left), VLAs can serve as end-to-end policies mapping perception directly to action (center), or they can be combined in a hierarchical system that balances reasoning and reactive control (right).
| Platform | Rendering Engine | Physics | ROS | Multi-Engine | 3DGS | Primary Use in UAV-VLN |
|---|---|---|---|---|---|---|
| AirSim [ 204 ] | Unreal Engine 4 | Fast Physics | ✓ | ✗ | ✗ | High-fidelity visual training; AerialVLN testbed |
| Gazebo + ROS [ 205 ] | OGRE / SDF | ODE / Bullet | ✓ | ✗ | ✗ | Digital twins; physics-accurate robotics integration |
| CARLA [ 13 ] | Unreal Engine 4 | PhysX | ✓ | ✗ | ✗ | Urban driving scenes; logistics VLN |
| Unity ML-Agents [ 129 ] | Unity HDRP | PhysX | ✓ | ✗ | ✗ | Morphing vehicle simulation; indoor scenes |
| AerialVLN Sim [ 29 ] | Unreal Engine 4 | AirSim | ✗ | ✗ | ✗ | First dedicated city-scale UAV-VLN benchmark |
| OpenFly [ 35 ] | UE + GTA V + Google Earth | Mixed | ✗ | ✓ | ✓ | Large-scale diverse data generation (100K trajectories) |
TABLE IV: Comparison of high-fidelity simulation platforms used in UAV-VLN research. Platforms are categorized by their rendering engine, physics fidelity, and primary contribution to the field. 3DGS = 3D Gaussian Splatting; UE = Unreal Engine; ROS = Robot Operating System. CARLA-Air uniquely integrates CARLA and AirSim within a single UE4 process, providing both ground and aerial native APIs with shared physics tick and rendering pipeline.
Fig. 8: The evolution from single-engine simulators to multi-engine data generation pipelines. Platforms like OpenFly integrate diverse sources like game engines and geospatial data to create large-scale, varied datasets, mitigating model overfitting and improving generalization.
| Pillar | Evolution Stage | Key Developments |
|---|---|---|
| High-Fidelity Simulation Environments | Foundational Simulators | Shifted from ground-based platforms (Matterport3D) to dedicated aerial environments. Utilized UE4 + AirSim and Gazebo + ROS for photorealistic rendering and physics. The AerialVLN simulator established the first large-scale benchmark and quantified the sim-to-real challenge. |
| Multi-Engine Platforms | Developed to overcome data diversity bottlenecks of single-engine simulators. OpenFly integrates UE, GTA V, Google Earth, and 3DGS through automated pipelines, enabling creation of 100K+ diverse training trajectories. | |
| Photorealistic & Differentiable Worlds | Leverages 3D Gaussian Splatting (3DGS) and NVIDIA Isaac Sim for photorealistic, differentiable environments. Enables direct end-to-end training of low-level control policies and closes both perceptual and dynamics portions of the sim-to-real gap. | |
| Benchmark Datasets and Task Variations | City-Scale Navigation | Pioneering benchmarks (AerialVLN: 8,446 trajectories, 25K instructions) with long-horizon trajectories and continuous 4-DoF action spaces. Revealed a 5.1% (model) vs. 80.8% (human) success rate gap. |
| Interactive & Fine-Grained Tasks | AVDN introduced multi-turn human-human dialog for ambiguity resolution. Expanded to delivery logistics, social navigation, and fine-grained control evaluation (UAV-Flow: 30K+ real-world trajectories). | |
| Large-Scale for Generalization | CityNav (32,637 real-world trajectories), OpenFly (100K+), and SAGE-Bench (2M pairs in 3DGS worlds). Specialized domains: agriculture (A2A), disaster response (DisasterM3), indoor (IndoorUAV), and remote sensing. |
TABLE V: A hierarchical overview of the essential resources for UAV-based Vision-and-Language Navigation research, organized into three pillars: High-Fidelity Simulation Environments, Benchmark Datasets and Task Variations, and Evaluation Protocols and Embodied Skill Assessment. Each pillar traces the evolution from foundational tools to advanced systems designed to improve model generalization and bridge the sim-to-real gap.
| Dataset | Year | #Traj. | #Instr. | Environment | Interaction | Action | Data Source | Distinguishing Feature |
|---|---|---|---|---|---|---|---|---|
| AerialVLN [ 29 ] | 2023 | 8,446 | 25,024 | Outdoor urban/rural | Single instruction | C, 4-DoF | UE4 + AirSim | First city-scale UAV-VLN benchmark |
| AVDN [ 55 ] | 2023 | 3,064 | Multi-turn | Outdoor urban | Human-human dialog | C | xView imagery | Multi-turn dialog for ambiguity resolution |
| CityNav [ 33 ] | 2025 | 32,637 | 32,637 | Real-world cities | Single instruction | C, 4-DoF | Google Earth | Largest real-world aerial navigation dataset |
| OpenFly [ 35 ] | 2025 | 100,000+ | 100,000+ | Diverse outdoor | Single instruction | C | UE + GTA V + 3DGS | Multi-engine data generation; 18 scenes |
| UAV-Flow [ 62 ] | 2025 | 30,000+ | — | Real-world flight | Trajectory replay | C, 6-DoF | Real flight logs | Fine-grained continuous control evaluation |
| IndoorUAV [ 36 ] | 2025 | — | — | Indoor environments | Single instruction | C | Simulation | Complex indoor aerial navigation |
TABLE VI: Comparison of major benchmark datasets for UAV-based Vision-and-Language Navigation. Datasets are organized chronologically and characterized by their scale, environment type, interaction modality, and action space. Traj. = Trajectories; Instr. = Instructions; DoF = Degrees of Freedom; C = Continuous; D = Discrete.
| Category | Metric | Abbreviation | Measures | Representative Benchmarks |
| Holistic Task-Completion | Success Rate | SR | Binary goal arrival within threshold | AerialVLN, CityNav, OpenFly |
| Oracle Success Rate | OSR | Any waypoint within goal radius | AerialVLN, AeroDuo | |
| Navigation Error | NE | Euclidean distance to goal at stop | AerialVLN, CityNav | |
| Success weighted by Path Length | SPL | Goal success penalized by path efficiency | AerialVLN, CityNav, OpenFly | |
| Normalized Dynamic Time Warping | nDTW | Path alignment with reference trajectory | AerialVLN, UAV-Flow | |
| Success weighted by nDTW | SDTW | Path fidelity combined with task success | AerialVLN |
TABLE VII: Taxonomy of evaluation metrics for UAV-VLN, categorized into holistic task-completion metrics and granular decomposed-skill assessments. Holistic metrics evaluate overall navigation performance, while decomposed metrics diagnose specific sub-skill competencies to guide targeted model improvement.
Fig. 9: A comparison of evaluation paradigms. Holistic metrics like SPL provide a single score for task completion, while decomposed frameworks like BEDI offer a granular assessment of core sub-skills, enabling precise diagnosis of agent failures.
| Aspect | Challenge / Solution | Description | Key Issues / Techniques |
|---|---|---|---|
| Characterizing the Reality Gap (The Problem) | |||
| Visual Domain Shift | Rendering & lighting discrepancies | Differences in rendering fidelity, textures, and lighting between simulators and real-world sensor data cause perception failures. | Real-world variations (motion blur, viewpoint, scale); partial observability from limited FoV; failures on reflective surfaces and adverse lighting. |
| Semantic understanding errors | Flawed environmental understanding from noisy depth, segmentation errors, and sensor noise profiles. | High cost of real-time semantic map maintenance; geographical and sensor resolution biases in datasets. | |
| Dynamics Mismatch | Action space abstraction | Simplified discrete actions in simulation fail to capture continuous real-world physics (aerodynamics, actuator latency, wind). | Overestimation of safe velocities; mismatch between high-level decisions and low-level controller execution. |
| State estimation drift | Imperfect localization in GPS-denied environments using visual-inertial odometry or LiDAR; accumulated EKF and SLAM errors. | Sensor noise at high speed; processing latency; inaccurate calibration; single-policy controllers failing beyond training distribution. | |
| Environmental Complexity Gap | Dynamic elements & long-horizon | Simulators lack semantic richness and unpredictable real-world dynamics; cascading errors over long trajectories. | High collision rates; navigation deadlocks; 5.1% (AI) vs. 80.8% (human) SR in AerialVLN. |
TABLE VIII: A structured breakdown of the simulation-to-reality (sim-to-real) gap in UAV vision-language navigation. The upper section deconstructs the problem into three core components: visual domain shift, dynamics mismatch, and environmental complexity gap. The lower section outlines the corresponding solution strategies: photorealistic simulation via neural rendering, domain adaptation techniques, and systematic deployment frameworks.
Fig. 10: A conceptual breakdown of the sim-to-real gap into its three constituent challenges: disparities in visual perception, physical dynamics, and environmental complexity.
Fig. 11: The typical multi-stage pipeline for sim-to-real transfer, progressing from pure simulation to Software-in-the-Loop (SIL) and Hardware-in-the-Loop (HIL) validation before final deployment on a physical UAV.
Fig. 12: A comparison of three primary strategies for reasoning under uncertainty. Procedural reasoning decomposes instructions into steps, spatial grounding uses explicit world models for disambiguation, and interactive/implicit methods either query a human for clarification or rely on end-to-end foundation models.
Fig. 13: Three architectural paradigms for safety assurance in autonomous flight. Formal methods use explicit safety filters, learned approaches embed safety into an end-to-end policy, and verifiable systems use logic-based oversight to validate high-level plans before execution.
Fig. 14: A comparison of coordination architectures for UAV swarms. (a) Centralized models rely on a single planner, offering global optimality at the cost of a single point of failure. (b) Decentralized models enhance robustness through peer-to-peer communication. (c) Hybrid models, such as Centralized Training with Decentralized Execution (CTDE), balance these trade-offs.
Fig. 15: Illustration of a foundation model acting as a collaborative intelligence layer for an air-ground team. The model fuses multimodal sensory data from both the UAV and UGV with high-level human commands to generate synchronized, executable plans for the entire system.