ProtoVis-Nav: Prototypical Visual Imagination for Visual-Spatial Aligned UAV Navigation

Guangyuan Sui; Yunzhong Hou; Chi Harold Liu

cs.RONov 12, 2026

ProtoVis-Nav: Prototypical Visual Imagination for Visual-Spatial Aligned UAV Navigation

Authors: Guangyuan Sui, Yunzhong Hou, Chi Harold Liu

Organizations: Beijing Institute of Technology

Abstract

Vision language navigation (VLN) is very challenging for Unmanned Aerial Vehicles (UAVs) since small errors in 3D motion prediction can lead to navigation errors and collisions. Unable to fathom execution outcomes in 3D environments, existing methods often struggle to predict accurate navigation waypoints from language instructions. To solve this, we introduce the Prototypical Visual Imagination framework, shortened as ProtoVis-Nav, which forces the model to imagine an outcome-associated local visual representation for the intended next UAV movement. Specifically, we predict feature prototypes and use their linear combination to estimate the visual feature of the target region. In this manner, the VLN model first imagines the local visual representation of the next intended location, and then uses predicted prototypes as context for generating the final action, promoting alignment between visual context and waypoint outputs. Extensive experiments demonstrate the effectiveness of the ProtoVis-Nav framework and show substantial improvements over existing baselines.

Figures & tables

Figure 1: Overall pipeline of ProtoVis-Nav.

Figure 2: Illustration of target-region visual feature selection during training. The left panel shows our motion-dominance analysis for selecting the front or down view. Given the selected view, the next intended UAV location is projected onto the current image plane, and a local target patch is cropped around the projected center to supervise prototypical visual imagination.

Figure 3: Illustration of the prototypical visual imagination head with $K$ <pvi> tokens and one output visual token. The hidden states of multiple <pvi> tokens are combined by softmax-normalized weights into a predicted target-region visual feature, which is supervised by the target crop feature extracted from the visual encoder.

Method	Full				Easy				Hard
Method	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$
Random Action	222.20	0.14	0.21	0.07	142.07	0.26	0.39	0.13	320.12	0.00	0.00	0.00
CMA	135.73	8.37	18.72	7.90	84.89	11.48	24.52	10.68	197.77	4.57	11.65	4.51
TravelUAV	106.28	16.10	44.26	14.30	68.78	18.84	47.61	16.39	152.04	12.76	40.16	11.76
NavFoM	93.05	29.17	49.24	25.03	58.98	32.91	53.16	27.87	143.83	23.58	43.40	20.80
LongFly	60.02	36.39	65.87	31.07	38.10	38.52	71.90	31.24	85.20	33.94	58.94	30.88

Table 1: Comparison with state-of-the-art methods on the OpenUAV benchmark.

Table 5

Figure 8: Qualitative example of ProtoVis-Nav on OpenUAV. The UAV follows the instruction and gradually approaches the target region. The green check mark indicates successful navigation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Illustration of the trajectory completion module. The predicted coarse waypoint and the current front-view RGB image are fused to generate a fine-grained local trajectory for UAV execution.

Model	Relation	$\leq 16$ px	$\leq 24$ px	$\leq 32$ px
Test Seen
ProtoVis-Nav	PVI–GT	55.10 [52.00, 58.16]	81.90 [79.39, 84.16]	87.20 [84.99, 89.13]
	PVI–WP	53.93 [50.80, 57.03]	80.29 [77.68, 82.66]	86.11 [83.80, 88.13]
	WP–GT	82.12 [79.60, 84.40]	87.54 [85.32, 89.46]	92.54 [90.73, 94.03]
Base	WP–GT	81.70 [79.15, 84.00]	87.12 [84.87, 89.07]	92.23 [90.38, 93.75]
Test Unseen

Appendix

Table 4: Spatial alignment on OpenUAV. Values are percentages with Wilson 95% confidence intervals in brackets.

Model	Mean waypoint error (m) $\downarrow$
Base	0.4630
ProtoVis-Nav	0.4576

Appendix

Table 5: Mean waypoint prediction error on 1,000 sampled OpenUAV test frames.

Metric	Test Seen	Test Unseen
PVI–GT cosine similarity	0.8440 [0.8402, 0.8477]	0.8449 [0.8415, 0.8482]
PVI–random cosine similarity	0.7666 [0.7604, 0.7728]	0.7535 [0.7464, 0.7605]
Mean first-relevant rank ( $<32$ px) $\downarrow$	11.19 [9.22, 13.57]	15.56 [13.38, 18.05]
Recall@1 ( $<32$ px)	27.00% [24.34%, 29.84%]	21.10% [18.68%, 23.74%]
Recall@5 ( $<32$ px)	63.50% [60.47%, 66.43%]	54.90% [51.80%, 57.96%]
Recall@10 ( $<32$ px)	80.40% [77.83%, 82.74%]	69.30% [66.37%, 72.08%]

Appendix

Table 6: Direct PVI feature similarity and spatial retrieval accuracy. Values in brackets are 95% confidence intervals.

Variant	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$
ProtoVis-Nav	51.77	50.78	66.50	44.68
+ Camera image noise	52.33	49.51	64.80	44.17
+ Position and yaw noise	56.24	46.42	63.11	41.25
+ Execution mismatch	59.92	47.69	62.41	42.02

Appendix

Table 7: Robustness of ProtoVis-Nav to observation and execution perturbations on OpenUAV Test Seen.

Model	Trajectories	nDTW $\uparrow$	Successes	SR (%) $\uparrow$
Base	273	0.2971	185	67.77
ProtoVis-Nav	273	0.3104	197	72.16

Appendix

Table 8: Closed-loop results on UAV-Flow.

Backbone	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$	Latency (ms) $\downarrow$
Wan2.1-VACE	111.46	17.25	22.30	14.79	333.83
Qwen3-VL	51.77	50.78	66.50	44.68	185.48

Appendix

Table 9: Navigation performance and inference latency of different backbones on OpenUAV Test Seen.

Model	PVI setting	NE (m) $\downarrow$	SR (%) $\uparrow$	OSR (%) $\uparrow$	SPL (%) $\uparrow$
Base	–	55.67	46.47	62.69	41.14
ProtoVis-Nav	Intact	51.77	50.78	66.50	44.68
	Shuffled	57.36	41.09	54.56	36.88
	Masked	70.99	37.87	49.65	34.74
	Random	77.18	36.75	47.69	33.38

Appendix

Table 10: Effect of inference-time PVI interventions on OpenUAV Test Seen Full.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 321 papers

ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning
May 31, 2026Xuchen Liu, Jiawei Huang, Shihao Xia +3UAV Navigation Vision-Language Navigation
Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China
FreeFly-Thinking : Aligning Chain-of-Thought Reasoning with Continuous UAV Navigation
Mar 7, 2026Jiaxu Zhou, Shaobo Wang, Zhiyuan Yang +2Aerial VLN UAV Navigation
SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University, China · School of Physics and Astronomy, Shanghai Jiao Tong University, China · Tongji University, China
AirForesight: Current-to-Future Spatial Map Imagination with Cross-Space Planning Consistency for UAV-VLN
Aug 13, 2026Yutong Liu, Xiaojie Li, Mingzhu Xu +1Visual Spatial Reasoning UAV Navigation
Harbin Institute of Technology, Shenzhen Shenzhen, China · Harbin Institute of Technology, Shenzhen Shenzhen Loop Area Institute Shenzhen, China · Shandong University Jinan, China

May 31, 2026J/K move · Enter open · S save

ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning

Xuchen Liu, Jiawei Huang, Shihao Xia, Bingxi Liu+2

Pengcheng Laboratory, Shenzhen, Guangdong, China · School of Computer Science and Cyber Engineering, Guangzhou University, Guangzhou, Guangdong, China · Southern University of Science and Technology, Shenzhen, Guangdong, China

UAV Navigation Vision-Language Navigation World Action Models Motion Planning Aerial VLN

Open paper AskPDF arXiv

Vision-language navigation (VLN) for UAVs demands grounding free-form instructions into 6-DoF flight under partial observability. While Vision-Language-Action (VLA) models excel at semantic reasoning, they suffer from brittleness due to geometric inconsistency and dynamics mismatch. To address this, we propose ImagineUAV, an imagination-driven framework leveraging cascaded world-action modeling. Instead of direct regression, ImagineUAV employs a latent video diffusion model to generate instruction-conditioned future observations, explicitly imagining environmental evolution, from which 6-DoF motions are inferred via an action extractor. A kinodynamic planner then refines these estimates into collision-free trajectories. Additionally, a step-distilled inference pipeline ensures real-time execution. With only 1.3B parameters, ImagineUAV outperforms prior VLN and VLA baselines on benchmarks and real-world flights, validating the practicality of imagination-driven aerial navigation.

Abstract

Vision-Language Navigation

Explore similar work

ImagineUAV: Aerial Vision-Language Navigation via World-Action Modeling and Kinodynamic Planning

FreeFly-Thinking : Aligning Chain-of-Thought Reasoning with Continuous UAV Navigation

AirForesight: Current-to-Future Spatial Map Imagination with Cross-Space Planning Consistency for UAV-VLN