The integration of Vision-Language Models (VLMs) in autonomous Unmanned Aerial Vehicles (UAVs) offers unprecedented semantic reasoning capabilities. However, real-time closed-loop navigation requires not only low inference latency but also obedience to structured flight commands. This paper proposes a hybrid FSM-VLM control architecture for UAVs in GPS-free environments. The system combines a deterministic Finite State Machine (FSM) for low-level physical control with an asynchronous VLM copilot for high-level semantic pathfinding. We evaluate three models with different parameter scales in a Software-In-The-Loop (SITL) simulation. The framework isolates and measures syntax errors at the format level versus semantic hallucinations at the logic level in a normal and degraded scenario. This study demonstrates that parameter scaling, and not pure latency, remains the primary bottleneck for the safe and compatible integration of VLM into autonomous flights.
Figures & tables
Figure 1: Simulation environment for the autonomous vehicle in an indoor area of the Gazebo, with the panels displaying RViz debugging overlays.
Figure 2: Proposed hybrid FSM-VLM architecture. The FSM ensures real-time control, while the asynchronous VLM provides semantic guidance.
Figure 3: Overview of Missions used for benchmarking, on the Gazebo environment of tests.
Figure 4: Autonomous Vehicle Components in a Frame in the Simulation Environment.
Figure 5: Visual examples of QR markers in the comparison configuration between Normal Environment vs. Degraded Environment.
Figure 6: Trajectory comparison across different VLM backends in the Normal Environment of Mission All.
Figure 7: Overlay of trajectories in a degraded environment of Mission 1. The baseline model fails at the intersection, whereas the VLM backend attempts to use backtracking to recover the mission path.
Environment
Model
Avg Latency (s)
Mean Err (m)
Max Err (m)
Format Int.
Success Rate
Normal
FSM - CV
0.001
0.199
0.802
N/A
100.0%
Moondream 2
0.702
0.161
0.798
14.5%
0.0%
Llava-Llama3
2.686
0.202
0.798
99.5%
80.0%
Llama 3.2 (11B)
5.580
0.167
0.804
100.0%
100.0%
Degraded
FSM - CV
0.001
0.225
0.798
N/A
0.0%
Moondream 2
0.703
2.744
28.30
1.4%
16.7%
Table 1: Comprehensive comparison of VLM-based autonomous navigation under Normal and Degraded (texture-damaged) environments. Bold values indicate the best result within each environment block. Success Rate represents mission completion without manual intervention.
Figure 8: Correlation between VLM inference latency and tracking error.
Vision-Language Navigation (VLN) enables UAV autonomous navigation in unknown environments by mapping language instructions to real-time visual inputs. Compared with GPS-dependent or pre-programmed navigation, VLN supports intuitive human-machine interaction and stronger environmental adaptability, requiring tight integration of high-level semantic reasoning and low-latency flight control.Existing methods suffer from structural misalignment between global multimodal understanding and sequential action generation, causing jittery trajectories and severe decision latency for long-horizon aerial navigation. To solve this issue, we propose FSD-VLN, a fast-slow dual-system architecture disentangling semantic reasoning and low-latency flight command generation.The framework has two asynchronous branches: a slow stream extracting stable semantic priors from pre-trained vision-language models, and a Diffusion Transformer (DiT) fast stream modeling cross-temporal action distributions to produce consistent flight outputs. We further introduce a time-aware adaptive optimizer to stabilize long-sequence training and reduce gradient oscillation.Large-scale low-altitude simulation experiments show FSD-VLN achieves up to 2X higher navigation success rates on unseen scenes than SOTA methods, while cutting single-action inference delay and total task runtime by over 50%. Our work validates the benefit of decoupled semantic-control modeling and provides a practical paradigm for long-horizon aerial VLN.
Xueke Zhu, Qingyan Meng, Liutao Yu +4
Pengcheng Laboratory · Shenzhen Institutes of Advanced Technology · Peking University
Test-time scaling offers a promising method to improve the inference performance of Vision-Language Models (VLMs) without additional training. Existing approaches to vision-language navigation (VLN) for Unmanned Aerial Vehicle (UAV) typically relies on a single inference pass, which can falter in complex environments by producing suboptimal or unsafe trajectories. In this paper, we explore a simple and effective approach to apply test-time scaling to VLN for UAV. We enhance navigation reasoning through an iterative refinement process that requires no extra model training, guiding the model to re-evaluate its initial navigation plan for better accuracy and safety. Our method first prompts the model to generate multiple parallel candidates and then performs a self-correction step, achieving deeper and more robust planning without changing the underlying model. To further strengthen decision-making, we design a multi-criteria scoring function to evaluate the refined candidates based on safety, goal alignment, and forward-progress. This simple yet powerful combination enables a frozen UAV navigation VLMs to self-correct and generate more accurate and reliable flight plans, achieving SOTA performance in this task.
Feinan Cheng, Dongliang Xu, Wenli Nong +4
Shandong University Weihai, China · China University of Petroleum (East China) Qingdao, China · MBZUAI Abu Dhabi, United Arab Emirates
Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers. To address this gap in UAV task settings, we introduce FLIGHT, a Fine-grained Long-horizon Instruction-Guided benchmark for Hybrid UAV navigation and reasoning Tasks, which combines multi-stage instructions with dense 6-DoF trajectory annotations across two dataset splits: Fine-grained VLN and Long-horizon Flow. To endow the UAV agent with the capability of real-time in-flight reasoning over task execution status and mission planning, while simultaneously accommodating high-frequency, real-time precise control, we further propose FLIGHT VLA, an asynchronous architecture that decouples a low-frequency Streaming Pilot Vision-Language Model (VLM) for task-state reasoning from a high-frequency diffusion action model for continuous control, supervised by explicit Pilot Reasoning texts that summarize the current flight state and anticipate the next subgoal. In closed-loop evaluation, FLIGHT VLA consistently surpasses representative VLN and VLA baselines on our FLIGHT benchmarks, achieving stronger multi-stage completion, subgoal adherence, and terminal control. Its trained Streaming Pilot Reasoning VLM further improves UAV video reasoning, validating the effectiveness of our design.
Xiangyi Zheng, Xiangyu Wang, Qinan Liao +6
Colab, Beihang University · National University of Singapore · Meituan