The integration of Vision-Language Models (VLMs) in autonomous Unmanned Aerial Vehicles (UAVs) offers unprecedented semantic reasoning capabilities. However, real-time closed-loop navigation requires not only low inference latency but also obedience to structured flight commands. This paper proposes a hybrid FSM-VLM control architecture for UAVs in GPS-free environments. The system combines a deterministic Finite State Machine (FSM) for low-level physical control with an asynchronous VLM copilot for high-level semantic pathfinding. We evaluate three models with different parameter scales in a Software-In-The-Loop (SITL) simulation. The framework isolates and measures syntax errors at the format level versus semantic hallucinations at the logic level in a normal and degraded scenario. This study demonstrates that parameter scaling, and not pure latency, remains the primary bottleneck for the safe and compatible integration of VLM into autonomous flights.
Figures & tables
Figure 1: Simulation environment for the autonomous vehicle in an indoor area of the Gazebo, with the panels displaying RViz debugging overlays.
Figure 2: Proposed hybrid FSM-VLM architecture. The FSM ensures real-time control, while the asynchronous VLM provides semantic guidance.
Figure 3: Overview of Missions used for benchmarking, on the Gazebo environment of tests.
Figure 4: Autonomous Vehicle Components in a Frame in the Simulation Environment.
Figure 5: Visual examples of QR markers in the comparison configuration between Normal Environment vs. Degraded Environment.
Figure 6: Trajectory comparison across different VLM backends in the Normal Environment of Mission All.
Figure 7: Overlay of trajectories in a degraded environment of Mission 1. The baseline model fails at the intersection, whereas the VLM backend attempts to use backtracking to recover the mission path.
Environment
Model
Avg Latency (s)
Mean Err (m)
Max Err (m)
Format Int.
Success Rate
Normal
FSM - CV
0.001
0.199
0.802
N/A
100.0%
Moondream 2
0.702
0.161
0.798
14.5%
0.0%
Llava-Llama3
2.686
0.202
0.798
99.5%
80.0%
Llama 3.2 (11B)
5.580
0.167
0.804
100.0%
100.0%
Degraded
FSM - CV
0.001
0.225
0.798
N/A
0.0%
Moondream 2
0.703
2.744
28.30
1.4%
16.7%
Table 1: Comprehensive comparison of VLM-based autonomous navigation under Normal and Degraded (texture-damaged) environments. Bold values indicate the best result within each environment block. Success Rate represents mission completion without manual intervention.
Figure 8: Correlation between VLM inference latency and tracking error.