UAVBench and UAVIT-1M: Benchmarking and Enhancing MLLMs for Low-Altitude UAV Vision-Language Understanding

Yang Zhan; Yuan Yuan

cs.CVMar 15, 2026

UAVBench and UAVIT-1M: Benchmarking and Enhancing MLLMs for Low-Altitude UAV Vision-Language Understanding

Authors: Yang Zhan, Yuan Yuan

Organizations: School of Artificial Intelligence, Optics, and Electronics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China.

Abstract

Multimodal Large Language Models (MLLMs) have made significant strides in natural images and satellite remote sensing images. However, understanding low-altitude drone scenarios remains a challenge. Existing datasets primarily focus on a few specific low-altitude visual tasks, which cannot fully assess the ability of MLLMs in real-world low-altitude UAV applications. Therefore, we introduce UAVBench, a comprehensive benchmark, and UAVIT-1M, a large-scale instruction tuning dataset, designed to evaluate and improve MLLMs' abilities in low-altitude vision-language tasks. UAVBench comprises 43 test units and 966k high-quality data samples across 10 tasks at the image-level and region-level. UAVIT-1M consists of approximately 1.24 million diverse instructions, covering 789k multi-scene images and about 2,000 types of spatial resolutions with 11 distinct tasks. UAVBench and UAVIT-1M feature pure real-world visual images and rich weather conditions, and involve manual verification to ensure high quality. Our in-depth analysis of 11 state-of-the-art MLLMs using UAVBench reveals that open-source MLLMs cannot generate accurate conversations about low-altitude visual content, lagging behind closed-source MLLMs. Extensive experiments demonstrate that fine-tuning open-source MLLMs on UAVIT-1M significantly addresses this gap. Our contributions pave the way for bridging the gap between current MLLMs and low-altitude UAV real-world application demands. (Project page: https://UAVBench.github.io/)

Figures & tables

Fig. 1: Sampled UAVIT-1M examples from our designed 11 tasks. We present the images, the corresponding questions, and the answers. These tasks are divided into two levels, namely, Image-level: Image Classification, Detailed Classification, Target Counting, Image Captioning, Detailed Image Captioning, and Image VQA; Region-level: Region VQA, Region Captioning, Region Classification, Region Detection, and Visual Grounding.

Fig. 2: An overview of (a) data processing for UAVBench and (b) instruction construction pipeline for UAVIT-1M.

Dataset	Official	Dataset	Number of Images in	Spatial Resolution	#Class	Supported	Visual Data Sources
Dataset	Link	Size	UAVIT-1M (UAVBench)	Spatial Resolution	#Class	Tasks	Visual Data Sources
AeroScapes [ 28 ]	Link	752M	2,621 (648)	1280 $\times$ 720	12	SS	commercial drone
AU-AIR [ 6 ]	Link	2.2GB	30,000 (2,823)	1920 $\times$ 1080	8	OD	quadrotor (Parrot Bebop 2)
CapERA [ 4 ]	Link	1.46GB	2,568 (296)	640 $\times$ 640	25	VC	ERA dataset
DroneVehicle [ 35 ]	Link	13.05GB	18,920 (8,734)	840 $\times$ 712	5	OD	DJI M200 aircraft
ERA [ 26 ]	Link	6.29GB	22,521 (2,607)	640 $\times$ 640	25	ER	from Youtube ${\dagger}$

TABLE I: The 21 selected low-altitude visual or vision-language datasets for the construction of UAVBench and UAVIT-1M. Supported tasks include Semantic Segmentation (SS), Object Detection (OD), Video Captioning (VC), Event Recognition (ER), Natural Language-guided Geolocalization (NLG), Action Recognition (AR), Referring Expression Comprehension (REC), Object Tracking (OT), Multiple Object Tracking (MOT), and Image-Text Retrieval (ITR).

{\dagger}

indicates that there might be visual data in this source that does not originate from the real world.

ID	Category	Mapped	ID	Category	Mapped	ID	Category	Mapped	ID	Category	Mapped
1	agricultural machinery		2	aircraft		3	animal		4	ar-marker
5	artifact		6	awning-tricycle	89/94	7	backpacking		8	bald tree	65/88/93
9	baseball		10	basketball		11	bicycle	12	12	bike	11
13	bird	3	14	boat	95	15	boating		16	building	25
17	bus	70/94	18	car	94	19	car racing		20	chainsawing trees
21	cliff jumping		22	concert		23	conflict		24	constructing

TABLE II: Semantically rich mapped category dictionary. The object categories are all from UAVBench and UAVIT-1M. The mapping ID is manually annotated.

Raw Task	New Task	Data Generation Method
Recognition	Image Classification	Set the category of the image as the answer.
SS	Detailed Classification	Set the category of all objects contained in the image as the answer.
OD	Counting Target	Set a category as the question and the total number of the corresponding boxes as the answer.
VC	Image Caption	Given the raw description sentence, use the LLM to create captions as the answer.
ITR	Image Caption	Set the raw description sentence as the answer.
NLG	Detailed Image Caption	Set a description paragraph as the answer.

TABLE III: The specific explanation of data generation for the 11 new low-altitude vision-language tasks shown in Figure 1 .

Spatial hierarchy	Tasks	#images	#Annotations	Data sources
UAVBench		261k	966k
Image Level	- Cls	17,920	17,920	ERA, MOD20
	- D.Cls	723	723	AeroScapes, VDD, UDD
	- Cap	2,398	11,990	CapERA, UDV DIT
	- VQA	51,609	52,912	VDD, UDD, AeroScapes, HazyDet, UAVDT-M, AU-AIR, UAVDT-S
	- Count	34,181	62,537	AU-AIR, DroneVehicle, HazyDet, VisDrone2019-DET, RDDTS, UAVDT-M

TABLE IV: Image and annotation statistics on UAVBench and UAVIT-1M.

Fig. 3: Category distribution in each task. Zoom in to view the specific categories and corresponding quantities.

Fig. 4: Image resolution and target position distributions. Best viewed by zooming in.

Fig. 5: Some typical statistics of UAVBench and UAVIT-1M. (a) Distribution of question types in image and region-level VQA tasks. (b) Distribution of object sizes in all region-level tasks. (c) Distribution of difficulty in the target counting task.

Model	Vision Modality Encoder	V-L Alignment Layer	LLM Backbone
MiniGPTv2-UAV	Eva-CLIP ViT @448 ◊	Linear Projector ⧫	LLaMA-2-Chat-7B ⧫
LLaVA1.5-UAV	CLIP ViT-L/14 @336 ◊	MLP ⧫	Vicuna-v1.5-7B ⧫
GeoChat-UAV	CLIP ViT-L/14 @336 ◊	MLP ◊	Vicuna-v1.5-7B ⧫

TABLE V: Details of the baseline model structure and weight fine-tuning (

\lozenge

: frozen weights;

\blacklozenge

: fine-tuning with LoRA techniques).

Fig. 6: Typical architecture of Multimodal Large Language Models (MLLMs), composed of a vision modality encoder, a vision-language alignment layer, and a large language model. Three types of alignment layers are mainly used, namely the linear layer or MLP, the Q-Former model, and the additional cross-attention layer.

Model	Size	Cls (Acc(%))		VQA (Acc(%))
		ERA	MOD $\dagger$	Pre.	Com.	Wea.	Wea. $\dagger$	Flying Altitude
		ERA	MOD $\dagger$	Pre.	Com.	Wea.	UAVDT-S	Rec.	$\text{Est}_{int.}$	$\text{Est}_{dec.}$
Open-source MLLMs (unfine-tuned)
MiniGPT-v2 [ 8 ]		7B	60.34	83.13	74.82	61.04	78.62	89.48	16.91	2.05	0.00
LLaVA-1.5 [ 24 ]	7B	64.37	82.84	81.22	72.21	88.98	95.24	19.86	0.00	0.00
Qwen-VL-Chat [ 2 ]	7B	73.46	85.38	76.89	55.57	88.03	91.95	35.81	3.93	0.32

TABLE VI: Evaluation results of different MLLMs on image-level Cls and VQA tasks.

\dagger

denotes results on out-of-domain data. This setting is also termed zero-shot evaluation in many articles, as samples from this domain are unseen during training. ”Pre.” represents Presence, ”Com.” represents Comparison, ”Wea.” represents Weather condition recognition, ”Rec.” represents Recognition, ”Est” represents Estimation, ”int.” represents integer, and ”dec.” represents decimal.

Model	Size	D.Cls						Count (Acc(%))
		AeroScapes		VDD		UDD		AU-AIR	Drone Vehicle	VisDrone	HazyDet		UAVDT-M	RDDTS $\dagger$
		F1(%)	mAP(%)	F1(%)	mAP(%)	F1(%)	mAP(%)	AU-AIR	Drone Vehicle	VisDrone	normal	hazy	UAVDT-M	RDDTS $\dagger$
Open-source MLLMs (unfine-tuned)
MiniGPT-v2 [ 8 ]		7B	10.18	9.29	1.79	27.55	0.98	34.69	44.52	22.43	13.20	16.01	15.77	18.75	20.64
LLaVA-1.5 [ 24 ]	7B	20.02	7.87	18.18	44.36	13.57	27.38	45.36	22.24	14.44	16.48	15.53	25.90	17.74
Qwen-VL-Chat [ 2 ]	7B	14.82	1.18	22.39	9.07	24.12	14.85	50.88	27.35	17.33	21.14	20.30	26.44	22.66

TABLE VII: Evaluation results of different MLLMs on image-level D.Cls, and Count tasks.

\dagger

denotes results on out-of-domain data. This setting is also termed zero-shot evaluation in many articles, as samples from this domain are unseen during training.

Model	Size	capERA								UDV DIT
Model	Size	B-1	B-2	B-3	B-4	M.	R.	C.	S.	B-1	B-2	B-3	B-4	M.	R.	C.	S.
Open-source MLLMs (unfine-tuned)
MiniGPT-v2 [ 8 ]	7B	28.7	15.0	8.0	4.3	14.7	25.8	15.4	11.5	24.8	10.7	4.9	2.3	14.0	22.1	6.2	9.2
LLaVA-1.5 [ 24 ]	7B	21.4	12.6	7.6	4.6	16.8	26.8	20.3	13.9	20.1	10.1	4.9	2.3	14.8	22.7	11.0	12.7
Qwen-VL-Chat [ 2 ]	7B	31.9	17.1	9.6	5.6	12.1	26.8	28.7	10.7	22.8	10.6	4.9	2.3	11.2	22.7	12.0	7.7
InternLM-XC2 [ 14 ]	7B	18.3	8.4	5.7	1.9	11.8	23.6	13.1	4.2	14.5	7.2	2.8	0.6	10.6	16.3	2.9	2.6

TABLE VIII: Evaluation results of different MLLMs on the image-level Cap task. ”B” represents BLUE, ”M.” represents METEOR, ”R.” represents ROUGE, ”C.” represents CIDER, and ”S.” represents SPICE.

Fig. 7: (a) Radar charts illustrating performance comparison between 11 open-source MLLMs and closed-source MLLMs across 10 tasks. (b) A comprehensive comparison including the low-altitude MLLMs fine-tuned on our UAVIT-1M. (c) Performance gain analysis of the fine-tuned low-altitude MLLMs in both in-domain and out-of-domain tests.

Model	Size	R.VQA	R.Cls (Acc(%))
		UAVDT-S $\dagger$	AU-AIR	WebUAV-3M	DroneVehicle	UAVDT-M	HazyDet		UAVDT-S $\dagger$	RefDrone $\dagger$	RDDTS $\dagger$
		Acc(%)	AU-AIR	WebUAV-3M	DroneVehicle	UAVDT-M	normal	hazy	UAVDT-S $\dagger$	RefDrone $\dagger$	RDDTS $\dagger$
Open-source MLLMs (unfine-tuned)
MiniGPT-v2 [ 8 ]		7B	47.88	0.00	0.00	0.13	0.04	0.00	0.00	0.02	1.56	0.01
LLaVA-1.5 [ 24 ]	7B	48.41	31.44	15.64	76.78	55.80	51.96	50.43	52.65	35.87	64.08
Qwen-VL-Chat [ 2 ]	7B	63.34	75.16	18.43	72.12	80.39	80.60	80.80	82.56	43.82	85.93

TABLE IX: Evaluation results of different MLLMs on region-level R.VQA and R.Cls tasks.

\dagger

denotes results on out-of-domain data. This setting is also termed zero-shot evaluation in many articles, as samples from this domain are unseen during training.

Model	Size	Det (AP@IoU=0.5(%))								VG ([email protected](%))
		AU-AIR	DroneVehicle	UAVDT-M	VisDrone	HazyDet		RefDrone $\dagger$	RDDTS $\dagger$	RefDrone $\dagger$	UAVDT-S $\dagger$
		AU-AIR	DroneVehicle	UAVDT-M	VisDrone	normal	hazy	RefDrone $\dagger$	RDDTS $\dagger$	RefDrone $\dagger$	UAVDT-S $\dagger$
Open-source MLLMs (unfine-tuned)
MiniGPT-v2 [ 8 ]		7B	4.21	0.35	0.47	0.24	0.52	0.48	0.02	0.72	12.46	42.18
LLaVA-1.5 [ 24 ]	7B	0.46	0.04	0.00	0.00	0.04	0.03	0.03	0.02	2.09	2.90
Qwen-VL-Chat [ 2 ]	7B	13.60	3.04	3.90	0.35	1.24	1.34	1.03	1.84	29.53	51.50

TABLE X: Evaluation results of different MLLMs on region-level Det and VG tasks.

\dagger

denotes results on out-of-domain data. This setting is also termed zero-shot evaluation in many articles, as samples from this domain are unseen during training.

Model	Size	WebUAV-3M								UAVDT-S $\dagger$
Model	Size	B-1	B-2	B-3	B-4	M.	R.	C.	S.	B-1	B-2	B-3	B-4	M.	R.	C.	S.
Open-source MLLMs (unfine-tuned)
MiniGPT-v2 [ 8 ]	7B	0.1	0.1	0.0	0.0	2.6	7.0	10.0	7.0	54.2	42.5	43.6	0.0	24.6	62.8	281.1	59.1
LLaVA-1.5 [ 24 ]	7B	5.4	2.8	1.3	0.6	5.0	13.9	12.4	6.7	33.6	21.2	15.5	0.0	18.0	42.9	39.5	27.0
Qwen-VL-Chat [ 2 ]	7B	11.2	5.9	3.1	1.8	6.6	17.8	16.9	15.3	39.1	23.0	11.7	0.0	24.0	45.6	29.8	40.6
InternLM-XC2 [ 14 ]	7B	1.8	1.0	0.3	0.0	2.5	5.6	8.3	4.5	20.6	13.4	5.6	0.0	10.7	22.0	10.8	15.7

TABLE XI: Evaluation results of different MLLMs on the region-level R.Cap task.

\dagger

denotes results on out-of-domain data. This setting is also termed zero-shot evaluation in many articles, as samples from this domain are unseen during training. ”B” represents BLUE, ”M.” represents METEOR, ”R.” represents ROUGE, ”C.” represents CIDER, and ”S.” represents SPICE.

Dimension	Level	Criteria
Object	A	The caption has the correct types of objects and rich attribute information.
	B	The caption has objects and attribute information, with an accuracy higher than 50%.
	C	The caption has objects and attribute information, but with an accuracy less than 50%.
	D	The caption has no object descriptions.
Position	A	The caption has correct position descriptions for objects.
	B	The caption has position descriptions for objects with an accuracy higher than 50%.

TABLE XII: The scoring criteria of the four-level rating system for the generated region-level captions.

Fig. 8: The comparison among LLaVA-1.5, LLaVA1.5-UAV, MiniGPT-v2, and MiniGPTv2-UAV on the UAVDT-S test unit of the region caption task. The scores are manually evaluated from three dimensions by 10 volunteers, as shown in (a), (b), and (c).

Fig. 9: Performance comparisons of some representative or SOTA models at different image resolutions and on the Count, R.Cls, and VG tasks. The resolutions r1 through r5 correspond to mainstream video resolutions, namely 480P, 720P, 960P, 1080P, and 4K, respectively. Obviously, there is a positive correlation between image resolution and model performance.

Fig. 10: Performance comparisons of some representative or SOTA models at different flight altitudes and on the Count, VQA, and R.Cls tasks. Flying altitude indicates the flying height of UAVs when the images are captured, i.e. , low ( $<$ 30m), mid ( $\geq$ 30m and $\leq$ 70m), and high ( $>$ 70m).

Fig. 11: Performance comparisons of some representative or SOTA models at different target quantities and on the Count and R.Cls tasks.

Fréchet Inception Distance			Cosine Similarity
UAV $\leftrightarrow$ RS	UAV $\leftrightarrow$ Natural	RS $\leftrightarrow$ Natural	UAV $\leftrightarrow$ RS	UAV $\leftrightarrow$ Natural	RS $\leftrightarrow$ Natural
0.60 $\pm$ 0.002	0.57 $\pm$ 0.004	0.73 $\pm$ 0.007	0.73 $\pm$ 0.002	0.81 $\pm$ 0.001	0.72 $\pm$ 0.002

TABLE XIII: Quantitative analysis of the domain gap between UAV, Natural, and Satellite RS images. The metrics reported represent the mean values and variance across 20 independent trials.

Fig. 12: Visualization of the domain gap between low-altitude UAV data and natural/satellite RS data using t-SNE clustering on visual features. Here, only the results of 10 sampling are presented.

Task identifier	Data scale	Task level		Image-level $\dagger$			Region-level $\dagger$
		Image	Region	Cls	VQA	Count	R.VQA	R.Cls	VG (%)
		level	level	(Acc(%))	(Acc(%))	(Acc(%))	(Acc(%))	(Acc(%))	([email protected])
LLaVA1.5-UAV baseline
✗	100%	✓	✓	81.82	94.20	23.51	46.78	93.71	3.04
✓	100%	✓	✓	83.00	98.94	24.59	48.48	96.32	3.73
✓	50%	✓	✓	81.47	93.48	23.33	45.37	90.97	2.22

TABLE XIV: Ablation studies on the task identifier, data scale, and task level. The result of each task corresponds to the average of the out-of-domain tests

\dagger

in Tables VI - XI .

Fig. 13: The zero-shot performance of the first 200k training steps during fine-tuning of MiniGPT-v2 on UAVIT-1M.

Fig. 14: Qualitative comparisons in the task of image classification.

Fig. 15: Qualitative comparisons in the task of detailed classification.

Fig. 16: Qualitative comparisons in the task of target counting.

Fig. 17: Qualitative comparisons in the task of image captioning.

Fig. 18: Qualitative comparisons in the task of visual question answering.

Fig. 19: Qualitative comparisons in the task of region visual question answering.

Fig. 20: Qualitative comparisons in the task of region caption.

Fig. 21: Qualitative comparisons in the task of region classification.

Fig. 22: Qualitative comparisons in the task of region detection.

Fig. 23: Qualitative comparisons in the task of region detection.

Fig. 24: Qualitative comparisons in the task of visual grounding.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion
Jun 26, 2026Haoyu Zhang, Meng Liu, Qianlong Xiang +3Visual Spatial Reasoning UAV Navigation
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Pengcheng Laboratory, Shenzhen 518000, China · School of Computer Science and Technology, Shandong Jianzhu University, Jinan 250101, China +4
Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System
Aug 12, 2026Haoyu Zhang, Shuoxun Zhang, Peng Ye +5Visual Question Answering Multi-Agent LLM Systems
Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China · Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China, and also with Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China, and also with The Chinese University of Hong Kong, Hong Kong · College of Future Information Technology, Fudan University, Shanghai 200433, China +1
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Jul 17, 2026Like Liu, Zhengzheng Xu, Haitao He +3Multimodal Grounding Aerial Robotics
Northwestern Polytechnical University Xi’an, Shaanxi, China

Jun 26, 2026J/K move · Enter open · S save

SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion

Haoyu Zhang, Meng Liu, Qianlong Xiang, Kun Wang+2

School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Pengcheng Laboratory, Shenzhen 518000, China · School of Computer Science and Technology, Shandong Jianzhu University, Jinan 250101, China+4

Visual Spatial Reasoning UAV Navigation Cooperative Perception

Open paper AskPDF arXiv

Spatial intelligence is essential for low-altitude unmanned aerial vehicle (UAV) perception, collaboration, and navigation. However, existing UAV benchmarks often emphasize image-level recognition, single-view understanding, or narrow answer formats, leaving 3D spatial inference, multi-view collaboration, scene dynamics, and diverse task formulations insufficiently evaluated. To address these gaps, we introduce SpatialUAV, a real low-altitude UAV benchmark comprising 4,331 curated instances across 14 fine-grained task types, covering semantic discrimination, spatial relation, aerial--aerial collaboration, aerial--ground collaboration, and motion understanding. SpatialUAV organizes all samples into a unified visual-input--question--answer schema, while supporting seven input configurations and nine answer formats, including option labels, region identifiers, geometric values, cross-view correspondences, and free-form motion descriptions. To ensure reliable and grounded evaluation, our data construction pipeline integrates detector-assisted regions, depth supervision, metadata-derived rules, extensive manual annotation, blind filtering, and multi-turn human validation, together with task-specific metrics for heterogeneous outputs. Evaluating representative vision-language models across three categories, we show that current models remain far from human-level performance, with pronounced bottlenecks in cross-view association, structured grounding, geometric reasoning, and temporal viewpoint understanding. These results offer empirical guidance for advancing low-altitude UAV spatial intelligence. Code and data are available at https://github.com/Hyu-Zhang/SpatialUAV.

Abstract

Vision-Language Models

Explore similar work

SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System

Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs