UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models

Qiyao Zhang; Shuhua Zheng; Jianli Sun; Chengxiang Li; Xianke Wu; Zihan Song; Zhiyong Cui; Yisheng Lv

cs.CVApr 2, 2026

UAV-Track VLA: Embodied Aerial Tracking via Vision-Language-Action Models

Authors: Qiyao Zhang, Shuhua Zheng, Jianli Sun, Chengxiang Li, Xianke Wu, Zihan Song, Zhiyong Cui, Yisheng Lv, +1 more

Organizations: Beijing Institute of Technology, Beijing, China · Flying Intelligence Team (Virtual Research Community) · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Sanya, Sanya, Hainan, China · Beijing University of Posts and Telecommunications, Beijing, China · Hunan University, Changsha, Hunan, China · Beihang University, Beijing, China

Abstract

Embodied visual tracking is crucial for Unmanned Aerial Vehicles (UAVs) executing complex real-world tasks. In dynamic urban scenarios with complex semantic requirements, Vision-Language-Action (VLA) models show great promise due to their cross-modal fusion and continuous action generation capabilities. To benchmark multimodal tracking in such environments, we construct a dedicated evaluation benchmark and a large-scale dataset encompassing over 890K frames, 176 tasks, and 85 diverse objects. Furthermore, to address temporal feature redundancy and the lack of spatial geometric priors in existing VLA models, we propose an improved VLA tracking model, UAV-Track VLA. Built upon the $π_{0.5}$ architecture, our model introduces a temporal compression net to efficiently capture inter-frame dynamics. Additionally, a parallel dual-branch decoder comprising a spatial-aware auxiliary grounding head and a flow matching action expert is designed to decouple cross-modal features and generate fine-grained continuous actions. Systematic experiments in the CARLA simulator validate the superior end-to-end performance of our method. Notably, in challenging long-distance pedestrian tracking tasks, UAV-Track VLA achieves a 61.76% success rate and 269.65 average tracking frames, significantly outperforming existing baselines. Furthermore, it demonstrates robust zero-shot generalization in unseen environments and reduces single-step inference latency by 33.4% (to 0.0571s) compared to the original $π_{0.5}$ , enabling highly efficient, real-time UAV control. Data samples and demonstration videos are available at: https://github.com/Hub-Tian/UAV-Track_VLA.

Figures & tables

Figure 1: The UAV-Track VLA model follows human instructions to simultaneously predict the target position and continuous UAV flight actions.

Figure 2: A comprehensive overview of the multidimensional diversity of our benchmark dataset. Top: Visualization of environment diversity. Middle: Task paradigms and overall dataset scale of the UAV-Track benchmark. Bottom: Quantitative distribution of target categories, kinematic parameters, and tracking distances.

Benchmark	Emboied Tracking	Kinematic Diversity	Language	Continuous Motion	UAV Dedicated
DAT [ 8 ]	✓	✓	✗	✓	✓
AD-AOT [ 7 ]	✓	✗	✗	✗	✓
RefDrone [ 19 ]	✗	✓	✓	✗	✓
IndoorUAV [ 14 ]	✗	✗	✓	✓	✓
UAV-Flow/UAV-Flow-Sim [ 37 ]	✗	✗	✓	✓	✓
VLA-AN [ 16 ]	✓	✗	✓	✓	✓

Table 1: Comparison of Existing UAV Tracking and Embodied AI Benchmarks

Figure 3: Overall architecture of our proposed UAV-Track VLA model. The model processes heterogeneous inputs through a spatiotemporal encoder and parallel decoders. Within the encoder, historical frames are temporally compressed and concatenated with the current frame to capture dynamic evolution, then fused with the instruction via an LLM. The shared multimodal features are subsequently dispatched to two decoupled branches: a spatial-aware auxiliary grounding head that predicts the target’s relative pose $P_{t}$ to inject precise geometric priors for robust high-frequency control

Model	Target	Close		Suitable		Far
Model	Target	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT [ 47 ]	Veh. / Ped.	36.00 / 18.50	0.00 / 0.00	30.17 / 20.70	0.00 / 0.00	33.24 / 21.52	0.00 / 0.00
WALL-OSS [ 48 ]	Veh. / Ped.	25.67 / 44.51	0.00 / 0.00	47.80 / 55.88	0.00 / 2.44	40.29 / 41.62	0.00 / 0.00
$\pi_{0}$ [ 12 ]	Veh. / Ped.	67.14 / 86.27	1.43 / 0.00	94.53 / 142.78	7.35 / 18.75	89.14 / 133.53	3.39 / 17.65
$\pi_{0.5}$ [ 34 ]	Veh. / Ped.	81.07 / 110.44	5.56 / 7.41	138.55 / 191.97	22.54 / 28.57	174.81 / 214.28	33.90 / 44.44
UAV-Track VLA (Ours)	Veh. / Ped.	98.47 / 203.93	12.73 / 31.82	164.79 / 263.61	29.61 / 60.20	194.91 / 269.65	37.88 / 61.76

Table 2: Quantitative results on Seen Maps

Model	Target	Close		Suitable		Far
Model	Target	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT [ 47 ]	Veh. / Ped.	31.21 / 18.35	0.00 / 0.00	31.61 / 18.65	0.00 / 0.00	39.14 / 19.71	0.00 / 0.00
WALL-OSS [ 48 ]	Veh. / Ped.	29.63 / 39.30	0.00 / 0.00	46.59 / 52.56	0.00 / 0.00	34.78 / 41.82	0.00 / 0.00
$\pi_{0}$ [ 12 ]	Veh. / Ped.	76.05 / 84.83	2.63 / 4.35	95.22 / 148.20	10.00 / 20.00	76.91 / 146.42	2.22 / 21.05
$\pi_{0.5}$ [ 34 ]	Veh. / Ped.	83.49 / 130.86	4.88 / 23.81	201.72 / 135.38	33.33 / 23.81	128.20 / 139.18	17.07 / 5.88
UAV-Track VLA (Ours)	Veh. / Ped.	75.56 / 148.94	6.67 / 29.41	150.00 / 210.24	29.79 / 36.84	159.84 / 226.90	27.91 / 55.00

Table 3: Zero-shot performance on Unseen Maps

Map Type	Setting	Target	Close		Suitable		Far
Map Type	Setting	Target	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
Seen	w/o Aux Head	Veh. / Ped.	80.45 / 125.43	10.45 / 10.00	176.13 / 166.67	29.87 / 23.33	161.76 / 183.96	30.88 / 39.29
Seen	UAV-Track VLA	Veh. / Ped.	98.47 / 203.93	12.73 / 31.82	164.79 / 263.61	29.61 / 60.20	194.91 / 269.65	37.88 / 61.76
Unseen	w/o Aux Head	Veh. / Ped.	70.22 / 96.96	3.12 / 12.50	174.24 / 137.48	26.83 / 8.70	120.98 / 159.22	16.67 / 22.22
Unseen	UAV-Track VLA	Veh. / Ped.	75.56 / 148.94	6.67 / 29.41	150.00 / 210.24	29.79 / 36.84	159.84 / 226.90	27.91 / 55.00

Table 4: Ablation Study: Impact of the Spatial Auxiliary Head

Model	Average Latency (s)
WALL-OSS [ 48 ]	0.4524
$\pi_{0.5}$ [ 34 ]	0.0857
$\pi_{0}$ [ 12 ]	0.0691
UAV-Track VLA (Ours)	0.0571

Table 5: Inference Efficiency Comparison

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Target	Distance	X-axis Range (m)	Y-axis Range (m)	Z-offset Range (m)	Yaw Perturbation Range (Rear/Right-Rear/Left-Rear)
Vehicle	Near	$[-2.5,-1.75]$	$[-3.0,3.0]$	$[0.2,1.5]$	$[-10^{\circ},10^{\circ}]$ / $[-60^{\circ},-10^{\circ}]$ / $[10^{\circ},60^{\circ}]$
	Close	$[-4.0,-1.0]$	$[-2.0,2.0]$	$[0.3,2.5]$	$[-10^{\circ},10^{\circ}]$ / $[-60^{\circ},-10^{\circ}]$ / $[10^{\circ},60^{\circ}]$
	Far	$[-8.0,-2.0]$	$[-1.5,1.5]$	$[0.6,5.0]$	$[-10^{\circ},10^{\circ}]$ / $[-60^{\circ},-10^{\circ}]$ / $[10^{\circ},60^{\circ}]$
Pedestrian	Near	$[-1.5,-0.5]$	$[-0.5,0.5]$	$[0.1,0.7]$	$[-8^{\circ},8^{\circ}]$ / $[-50^{\circ},-8^{\circ}]$ / $[8^{\circ},50^{\circ}]$
	Close	$[-2.0,-0.8]$	$[-1.0,1.0]$	$[0.2,1.5]$	$[-8^{\circ},8^{\circ}]$ / $[-50^{\circ},-8^{\circ}]$ / $[8^{\circ},50^{\circ}]$
	Far	$[-4.0,-1.6]$	$[-0.8,0.8]$	$[0.4,3.0]$	$[-8^{\circ},8^{\circ}]$ / $[-50^{\circ},-8^{\circ}]$ / $[8^{\circ},50^{\circ}]$

Appendix

Table 6: Spatial Configurations for Automated Data Collection.

Figure 4: Exemplar Gallery of Heterogeneous Tracking Targets. This collage provides a visual overview of the diverse candidate assets utilized as tracking targets within the UAV-Track benchmark. The targets encompass standard vehicles, two-wheelers (bicycles, motorcycles), and pedestrians with varying demographics (adults, children). These assets form the basis for the fine-grained semantic attribute mapping (e.g., color, age) used in the instructions described in Section A.2.

Substitution Type	Original Base Prompt (Seen)	Unseen Test Prompt (Zero-Shot)
Verb Changed	Track the child male pedestrian at a long distance.	Focus on the child male pedestrian at a long distance.
	Track the red bicycle at a long distance.	Keep an eye on the red bicycle at a long distance.
	Track the vehicle at a long distance.	Focus on the vehicle at a long distance.
	Track the child female pedestrian at a long distance.	Focus on the child female pedestrian at a long distance.
	Track the black motorcycle at a suitable distance.	Pursue the black motorcycle at a suitable distance.
	Track the orange motorcycle at a long distance.	Pursue the orange motorcycle at a long distance.

Appendix

Table 7: Detailed Mapping of Unseen Zero-Shot Prompts against Seen Training Prompts

Panel of Figure 5 — Figure 5: Visualization of Cross-Modal Attention Maps. We visualize the average attention weights from language tokens to the current frame’s visual tokens across the last four layers of the LLM backbone. Top rows: The full UAV-Track VLA model exhibits highly concentrated and accurate attention on the specific target. Bottom rows: The ablated model without the Spatial-Aware Auxiliary Grounding Head shows diffuse and scattered attention, struggling to ground the semantic instruction in the visual space.

Substituted Component	Vehicle Tracking		Pedestrian Tracking
Substituted Component	Norm. ATF	Norm. SR	Norm. ATF	Norm. SR
Object	1.46	1.60	1.04	0.79
Verb	1.30	1.38	2.53	1.46
Distance	1.96	2.18	2.39	4.76

Appendix

Table 8: Normalized Performance Impact of Single-Variable Prompt Substitution. Lower scores denote higher sensitivity (i.e., more severe performance drop when the component is altered).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	34.62	0	32.33	0	34.15	0	19.30	0	19.20	0	18.25	0
WALL-OSS	23.33	0	44.59	0	28.54	0	50.14	0	52.86	0	31.60	0
$\pi_{0}$	75.50	10	98.50	11.11	91.60	0	70.67	0	108.10	20	132.33	33.33
w/o Aux. Head	106.88	12.50	127.20	13.33	123.67	8.33	127.40	20	130	25	157.14	28.57

Appendix

Table 9: Detailed Tracking Performance on Town02 (Seen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	54.31	0	50.31	0	47.38	0	18.29	0	18.29	0	18.71	0
WALL-OSS	27	0	44.15	0	45.69	0	36.86	0	57.14	0	40.71	0
$\pi_{0}$	64.56	0	116.50	8.33	95.92	0	92.17	0	167.71	28.57	146.80	20
w/o Aux. Head	110.15	7.69	221.77	38.46	231.05	75	109	0	186.17	0	240	40

Appendix

Table 10: Detailed Tracking Performance on Town05 (Seen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	34.29	0	28.53	0	33	0	19.80	0	24.32	0	24.40	0
WALL-OSS	25.14	0	42.26	0	35.85	0	43	0	62.48	0	43.16	0
$\pi_{0}$	67.64	0	66.69	0	94.80	10	107.17	0	222.33	33.33	137	21.43
w/o Aux. Head	65.50	16.67	129.50	20	120.30	0	180.40	10	187.36	28	168.16	32

Appendix

Table 11: Detailed Tracking Performance on Town06 (Seen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	36.09	0	32.55	0	34.64	0	18.50	0	20.89	0	18.58	0
WALL-OSS	23	0	44.10	0	32.73	0	50.17	0	56.67	0	68.68	0
$\pi_{0}$	71.87	0	119.50	16.67	102.50	6.25	83	0	149	16.67	140.80	20
w/o Aux. Head	65.25	0	124.62	15.38	128.92	30.77	172.67	16.67	182.26	26.32	196.21	36.84

Appendix

Table 12: Detailed Tracking Performance on Town07 (Seen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	23.57	0	20.29	0	14.50	0	14	0	19	0	17	0
WALL-OSS	23.43	0	73	0	23	0	36	0	50.12	0	28	0
$\pi_{0}$	60.06	0	75.60	0	54.91	0	86.20	0	125.50	0	112.43	0
w/o Aux. Head	61.85	7.69	147.60	26.67	158.17	41.67	108.50	0	168.62	25	186.75	37.50

Appendix

Table 13: Detailed Tracking Performance on Town10 (Seen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	22.80	0	28.88	0	34.64	0	17.44	0	16.57	0	17.71	0
WALL-OSS	27.77	0	48	0	29.45	0	46.62	0	41.33	0	43	0
$\pi_{0}$	85.60	0	72.17	0	97.53	6.67	99.67	0	130.44	11.11	118.40	20
w/o Aux. Head	74.25	0	148.07	14.29	132.50	12.50	92.33	11.11	139	12.50	148.20	20

Appendix

Table 14: Detailed Tracking Performance on Town01 (Unseen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	38	0	43.83	0	43.25	0	19.50	0	21	0	22.29	0
WALL-OSS	21.78	0	57.29	0	33.53	0	32.75	0	45.33	0	36.43	0
$\pi_{0}$	68.67	0	92.57	14.29	60.53	0	85.29	14.29	136.67	0	155.67	33.33
w/o Aux. Head	61.82	9.09	207.43	50	105.69	7.69	86.89	11.11	125.86	0	241.17	50

Appendix

Table 15: Detailed Tracking Performance on Town03 (Unseen Map).

Model	Vehicle						Pedestrian
	Close		Suitable		Far		Close		Suitable		Far
	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)	ATF	SR (%)
ACT	33.25	0	28.33	0	39.15	0	19.25	0	18.75	0	19.14	0
WALL-OSS	36.92	0	32.58	0	39.93	0	42.14	0	65.29	0	46.33	0
$\pi_{0}$	75.62	6.25	117.64	14.29	72.67	0	65.29	0	213	66.67	157.80	0
w/o Aux. Head	74.85	0	166.69	15.38	122.08	30.77	119	16.67	146.12	12.50	96.86	0

Appendix

Table 16: Detailed Tracking Performance on Town04 (Unseen Map).

Explore similar work

CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking
Jul 16, 2026Ruilong Ren, Songsheng Cheng, Yunpeng Zhou +9Aerial Robotics Visual Object Tracking
1Autel Robotics · 2Northeast Normal University · 3Southern University of Science and Technology +2
VL-UniTrack: A Unified Framework with Visual-Language Prompts for UAV-Ground Visual Tracking
May 6, 2026Boyue Xu, Ruichao Hou, Tongwei Ren +1Visual Object Tracking Multi-View Learning
State Key Laboratory for Novel Software Technology, Nanjing University
CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization
May 18, 2026Xiangyue Wang, Hanxuan Chen, Songsheng Cheng +7Aerial Robotics Trajectory Optimization
Autel Robotics · Nanjing University · Peking University +2

Jul 16, 2026J/K move · Enter open · S save

CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking

Ruilong Ren, Songsheng Cheng, Yunpeng Zhou, Hanxuan Chen+8

1Autel Robotics · 2Northeast Normal University · 3Southern University of Science and Technology+2

Aerial Robotics Visual Object Tracking Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Dynamic target tracking is essential for Unmanned Aerial Vehicles (UAVs) operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing Vision-Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. To this end, we present CosFly-VLA, a spatially aware VLA model that jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. To train this policy, we use a large-scale recipe over diverse data sources. Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool injects UAV-view depth, distance, and 3-D spatial reasoning. A three-stage Curriculum-based Supervised Fine-Tuning (SFT) process then specializes the tracker through multi-head warm-up followed by two-stage curriculum learning over natural and hard / long-occlusion data. Chain-of-Thought (CoT) training subsequently teaches recovery-oriented reasoning traces before structured answers. Finally, a closed-loop Reinforcement Learning (RL) stage optimizes tracking behavior with a multi-component reward covering stand-off tracking, grounding quality, collision avoidance, and task success. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test. Closed-loop optimization improves Success Rate (SR) by 29.8% and 2.5%, respectively. These results demonstrate progress from visible-frame imitation toward spatially grounded action-closed-loop control, evaluated under a shared oracle state history.

Abstract

Explore similar work

CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking

VL-UniTrack: A Unified Framework with Visual-Language Prompts for UAV-Ground Visual Tracking

CosFly-Track: A Large-Scale Multi-Modal Dataset for UAV Visual Tracking via Multi-Constraint Trajectory Optimization