VLN on the Fly: An Onboard Vision-Language Navigation Stack for Aerial Robots

Marco S. Tayar; Felipe Tommaselli; Gianluca Capezzuto; Pedro Antonio Rabelo Saraiva; Pedro H. V. de Freitas; Lucas Kido; Guilherme Sonego; Ricardo V. Godoy

cs.ROAug 7, 2026

VLN on the Fly: An Onboard Vision-Language Navigation Stack for Aerial Robots

Authors: Marco S. Tayar, Felipe Tommaselli, Gianluca Capezzuto, Pedro Antonio Rabelo Saraiva, Pedro H. V. de Freitas, Lucas Kido, Guilherme Sonego, Ricardo V. Godoy,

Organizations: University of S˜ao Paulo (USP), Brazil

Abstract

Running vision-language navigation fully onboard an aerial robot is hard, since grounding, planning, and control must share limited compute and a single-stage error is difficult to isolate in flight. End-to-end aerial policies fuse these stages into one network, giving up the observability and safety checks a modular stack keeps available. We propose VLN on the Fly, an onboard stack that keeps grounding, planning, and control as separate, inspectable stages. A quantized VLM grounds an instruction to a coarse image cell, depth lifts it to a 3D goal, a fast B-spline planner returns a feasible trajectory, and a pretrained reinforcement learning policy tracks it to motor commands across quadrotors. Across 15 onboard flights over three everyday referents in a controlled indoor volume, the stack reaches the target in 13 of 15 trials with 5.72 cm mean goal error and 39.3% average GPU utilization. In 6 additional cluttered-environment trials, the stack tracks collision-free trajectories under onboard perception gating.

Figures & tables

Figure 1: VLN on the Fly , an onboard vision-language navigation stack that grounds language instructions and plans and executes trajectories toward named referents, here a fire extinguisher, a trash bin, or a chair.

Figure 2: Overview of the VLN on the Fly. The RGB-D camera and language instruction are processed onboard by Qwen3.5 to project a 3D goal, EGO-Planner generates a feasible B-spline trajectory, and RAPTOR tracks the resulting setpoints as motor commands. OptiTrack pose is fused by EKF2 for localization in the experimental setup, while the safety layer gates motion.

Panel of Figure 3 — Figure 3: Experimental Setup. (a) Controlled indoor flight volume for the open-vocabulary object-goal trials, with the quadrotor, marked operating bounds, and everyday referents. (b) Cluttered setup for the path-planning trials (Sec. 4.3 ), with a table, chair, tool case, and a freestanding vertical panel placed between the take-off pose and the trash-bin referent.

Figure 4: Qualitative Grounding and Planning Result. A single onboard frame from a representative successful run for the instruction “go to the trash bin” , with the fire extinguisher and chair present as distractors. (a) VLM grounding over the image grid: the selected center cell (blue) localizes the referent. (b) The same frame as a navigation command: the back-projected 3D goal at the selected cell’s ray, and the executed onboard trajectory projected into the camera view.

Referent	Success rate $\uparrow$	Goal err. (cm) $\downarrow$	Track err. (cm) $\downarrow$	Avg. GPU (%)
Trash bin	1.00	0.00	34.96	42.2
Chair	0.80	10.07	26.13	37.3
Fire extinguisher	0.80	7.08	37.64	38.4
Overall	0.87	5.72	32.91	39.3

Table 1: Open-Vocabulary Object-Goal Navigation

Stage	Median	95th pct.	Max
Time-to-first-token	0.60 s	0.66 s	0.72 s
Total inference	0.79 s	0.88 s	1.12 s

Table 2: Onboard VLM Inference Timing

Region grid	Valid-goal rate $\uparrow$	Selection jitter (m) $\downarrow$
$1\times 1$	0.94	0.07
$3\times 3$ (ours)	1.00	0.70
$5\times 5$	1.00	1.27
Single pixel	0.15	1.16

Table 3: Grounding Resolution Ablation

Prompted referent	Not visible	Modal region	Share
Chair (left)	53%	Top-left	86%
Trash bin (center)	44%	Center	90%
Fire extinguisher (absent)	66%	Center	79%

Table 4: VLM Region Selection by Prompted Referent

Figure 5: Planned vs. Flown Trajectory in Clutter. EGO-Planner’s collision-free B-spline (blue) runs from take-off (green) to the goal object (star), and RAPTOR tracks it (orange). Altitude is held near the commanded band, faint floor shadows show the horizontal path. The window spans trajectory generation to the first goal-reach.

Run	Goal err. (cm) $\downarrow$	Mean track. err. (cm) $\downarrow$	p90 track. err. (cm) $\downarrow$
1	0.00	14.5	18.6
2	18.3	15.1	26.4
3	20.7	29.9	79.1
4	0.00	10.9	21.2
Mean	9.75	17.6	36.3

Table 5: Goal and Tracking Errors for Cluttered Flights

Aerial VLN

Aerial vision-and-language navigation (VLN) enables UAVs to follow natural-language goals using visual perception and flight control.

See all 61 papers

LightVLN: Efficient Aerial Vision-and-Language Navigation with Compact Memory and History-Guided Local Aggregation
Oct 4, 2026Yiming Zhao, Tianshun Li, Jingle He +2Memory-Augmented VLMs Vision-Language Models
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency
Mar 11, 2026Guiyong Zheng, Yueting Ban, Mingjie Zhang +2Vision-Language Navigation Aerial VLN
School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China. · Southern University of Science and Technology, Shenzhen, China · The Hong Kong University of Science and Technology, Guangzhou, China.
DreamFly: Causal Memory and Receding-Horizon Diffusion Planning for Aerial Vision-Language Navigation
Aug 12, 2026Yan Deng, Fei XuMemory-Augmented VLMs Diffusion-Based Planning
School of Electronic Information Engineering, Xi’an Technological University, Xi’an, China · School of Computer Science and Engineering, Xi’an Technological University, Xi’an, China

Oct 4, 2026J/K move · Enter open · S save

LightVLN: Efficient Aerial Vision-and-Language Navigation with Compact Memory and History-Guided Local Aggregation

Yiming Zhao, Tianshun Li, Jingle He, Ruonan Chai+1

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

Memory-Augmented VLMs Vision-Language Models UAV Navigation Vision-Language Navigation Efficient VLM Training Aerial VLN

Open paper AskPDF arXiv

Aerial vision-and-language navigation (VLN) enables unmanned aerial vehicles to execute long-horizon natural-language instructions from visual observations in complex three-dimensional environments. However, recent aerial VLN models often rely on large-scale vision-language backbones and dense visual histories, imposing substantial computation and memory costs that hinder onboard deployment. We propose LightVLN, a lightweight history-aware aerial VLN framework that combines a compact 0.5B language backbone with compact representations of both historical and current observations. LightVLN compresses each historical frame into a single token using visual features already computed by the policy. It further introduces history- and instruction-conditioned local aggregation to reduce the current observation from 256 to 32 visual tokens while preserving navigation-relevant spatial information. With up to 16 historical frames, the policy uses at most 48 observation-derived tokens. On the public OpenFly dataset, LightVLN achieves 50.93% Test-Seen and 36.14% Test-Unseen success rates (SR), outperforming the evaluated 7B language-backbone baselines on most reported metrics. It also achieves 25.83% SR on AerialVLN-S Val-Seen. In a reconstructed unseen campus, we deploy LightVLN on a DJI M350 RTK with an external Jetson Orin NX 16 GB for closed-loop onboard-compute real-to-sim hardware-in-the-loop (HIL) evaluation, achieving 14.61 Hz model inference and 11.13 Hz end-to-end decision updates. These results demonstrate the effectiveness and efficiency of LightVLN for aerial navigation.

Abstract

Aerial VLN

Explore similar work

LightVLN: Efficient Aerial Vision-and-Language Navigation with Compact Memory and History-Guided Local Aggregation

OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency

DreamFly: Causal Memory and Receding-Horizon Diffusion Planning for Aerial Vision-Language Navigation