Unmanned Aerial Vehicles (UAVs) are expected to achieve semantic reasoning, intelligent decision-making, and long-horizon navigation under strict onboard constraints. While recent foundation-model-driven approaches have shown promising progress toward this goal, existing methods remain limited by challenges in 3D data availability, trajectory feasibility, long-horizon reasoning, and computational efficiency in complex environments. To bridge these gaps, we propose VLN-AR, a hierarchical vision-language-navigation (VLN) framework for aerial robots. VLN-AR decouples multimodal reasoning and low-level control, enabling structured prediction of 3D pose commands and navigation intentions that are translated into dynamically feasible actions. To mitigate data scarcity and domain gaps, we introduce a fully automated 3D Gaussian Splatting (3DGS)-based pipeline to generate scalable photorealistic UAV navigation data. A progressive three-stage training strategy further enhances grounding, scene reasoning, and complex navigation. In addition, a lightweight cognitive agent improves long-horizon reasoning via memory-aware task decomposition. Finally, we design onboard optimization techniques, including token sparsification and model lightweighting, achieving 7.1x-9.4x speedup while maintaining high navigation performance on resource-limited UAV platforms. Experiments show great improvements over existing methods in both simulation and real-world UAV navigation. Project website: https://wyz000.github.io/VLN-AR/
Figures & tables
Fig. 2 : Different UAV navigation foundation models.
Fig. 3 : The model architecture of the proposed VLN-AR model.
Fig. 4 : The improved ViT architecture and token sparsification .
Fig. 5 : Automated UAV navigation data generation and collection process.
Fig. 6 : Cognitive-driven UAV navigation agent pipeline for VLN-AR.
Training Stage
Hyperparameter
Value
Stage I (Grounding- Reasoning- enhanced SFT)
Batch size
32
Learning rate
5×10−5
Epochs
1
Weight decay
0.01
λvqa
1.0
λbbox
1.0
TABLE I: Hyperparameters for Multi-stage Training.
Types
Methods
AerialVLN Dataset [ 32 ]
OpenFly Dataset [ 31 ]
Our Simulation Dataset
SR(%) ↑
NE(m) ↓
TT(s) ↓
SR ↑
NE ↓
TT ↓
SR ↑
NE ↓
TT ↓
End-to-end VLN Methods
OpenUAV [ 33 ]
23.02
60.11
55.76
20.36
76.02
61.43
16.41
14.88
25.51
AerialVLN [ 32 ]
33.17
59.48
46.91
35.73
66.44
52.26
36.21
16.93
21.58
OpenFly [ 31 ]
34.03
64.21
47.02
30.94
77.54
48.11
26.73
14.14
20.02
UAV-Flow [ 34 ]
39.66
54.42
51.37
38.38
72.63
52.94
34.18
12.77
25.11
MLLM-based Navigation Agent
CityNavAgent [ 35 ]
31.84
63.02
113.57
21.75
74.93
128.20
34.96
14.60
26.58
TABLE II: Benchmark comparison of different methods on UAV navigation datasets
Fig. 7 : Task distribution of test data.
Fig. 8 : Simulation tests of VLN-AR across different navigation datasets using both visual and language inputs.
Fig. 9 : Hardware and software architectures of two different experimental drones.
Fig. 10 : Real-world experiments for navigation scenarios utilizing both visual and language inputs.
Model
Floating Point Operations
Baseline Decode Speed (s/token)
Our Decode Speed (s/token)
7B-AWQ
∼1815.8 GFLOPs
1.034
0.110
4B-AWQ
∼946.7 GFLOPs
0.508
0.062
2B-AWQ
∼563.3 GFLOPs
0.227
0.032
TABLE III : Onboard inference performance on different models
Fig. 11 : Robustness tests on 3D floating scenes, large height variations, dim lighting, dynamic disturbance, and zero-shot deployment.
Test scenes
Results
3D floating scene
18 / 20
Large height variation
19 / 20
Dim lighting environment
17 / 20
Dynamic obstacle interference
17 / 20
Zero-shot deployment
18 / 20
TABLE IV: Robustness test of different navigation scenes
Methods
Grounding-reasoning -enhanced SFT
Navigation -specific SFT
RFT for Reasoning -aware Navigation
Scene Understanding
Spatial Reasoning
Precise Navigation
Long-horizon Navigation
Baseline
×
×
×
77.4
68.2
21.3
17.8
Stage I
✓
×
×
87.2
94.5
28.1
22.5
Stage II
×
✓
×
75.6
64.3
68.5
64.4
Stage I + Stage II
✓
✓
×
80.2
88.3
60.6
61.5
Stage III
×
×
✓
82.6
76.1
45.4
36.3
Ours
✓
✓
✓
85.5
94.6
85.7
88.0
TABLE V: Ablation study of the multi-stage training strategy (Success rates %)
Fig. 12 : Ablation study of different training navigation datasets.
Fig. 13 : Impact of different token retention ratios for ViT.
Datasets
Tasks
Object Navigation
Spatial Reasoning
Long-horizon Navigation
Ours
98.1
94.6
88.0
w/o Agent
95.7
92.4
57.4
TABLE VI: Ablation study of the aerial navigation agent
Fig. 14 : Ablation study of the inference optimization and the few-shot transfer.
Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability. While Vision-Language-Action (VLA) models excel at semantic reasoning, they suffer from brittleness due to geometric inconsistency and dynamics mismatch. To address this, we propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling. Instead of direct regression, ImagineUAV employs a latent video diffusion model to generate instruction-conditioned future observations, explicitly imagining environmental evolution, from which 6-DoF motions are inferred via an action extractor. A kinodynamic planner then refines these estimates into collision-free trajectories. Additionally, a step-distilled inference pipeline ensures real-time execution. With only 1.3B parameters, ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights, validating the practicality of imagination-driven aerial navigation.
Xuchen Liu, Jiawei Huang, Shihao Xia +3
Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China
Vision-Language Navigation (VLN) enables UAV autonomous navigation in unknown environments by mapping language instructions to real-time visual inputs. Compared with GPS-dependent or pre-programmed navigation, VLN supports intuitive human-machine interaction and stronger environmental adaptability, requiring tight integration of high-level semantic reasoning and low-latency flight control.Existing methods suffer from structural misalignment between global multimodal understanding and sequential action generation, causing jittery trajectories and severe decision latency for long-horizon aerial navigation. To solve this issue, we propose FSD-VLN, a fast-slow dual-system architecture disentangling semantic reasoning and low-latency flight command generation.The framework has two asynchronous branches: a slow stream extracting stable semantic priors from pre-trained vision-language models, and a Diffusion Transformer (DiT) fast stream modeling cross-temporal action distributions to produce consistent flight outputs. We further introduce a time-aware adaptive optimizer to stabilize long-sequence training and reduce gradient oscillation.Large-scale low-altitude simulation experiments show FSD-VLN achieves up to 2X higher navigation success rates on unseen scenes than SOTA methods, while cutting single-action inference delay and total task runtime by over 50%. Our work validates the benefit of decoupled semantic-control modeling and provides a practical paradigm for long-horizon aerial VLN.
Xueke Zhu, Qingyan Meng, Liutao Yu +4
Pengcheng Laboratory · Shenzhen Institutes of Advanced Technology · Peking University
UAV vision-language navigation (UAV-VLN) focuses on enabling an aerial agent to follow natural-language instructions in open 3D environments from egocentric visual observations. Current approaches suffer from three coupled issues: weak grounding of instruction-relevant landmarks in visual observations, insufficient exploitation of long-horizon history, and unstable decisions under local traps or repeated exploration. To address these issues, we propose a unified semantic-to-decision framework. First, we present an instruction-grounded semantic enhancement module that injects object-level semantics and relative spatial cues into the current observation state. Subsequently, we develop a relevance-aware dynamic temporal aggregation strategy that reweights the full history buffer while converting a few high-relevance frames into structured landmark prompts for the decoder. Finally, we devise a topology-aware decision method that combines local-optimum cognition with group-relative policy optimization under progress, goal, semantic, and path-compliance rewards. Experiments on the widely used AerialVLN and OpenFly benchmarks clearly demonstrate that our method achieves state-of-the-art performance.
Zeyuan Ma, Jiaxin Chen, Di Huang
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China · School of Computer Science and Engineering, Beihang University, Beijing, China