Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Prompts
Organizations: School of Geosciences and InfoPhysics, Central South University, Changsha 410083, China.
Abstract
UAV image perception with VLMs aims to identify and localize targets based on semantic and spatial task prompts. However, small, densely distributed objects and cluttered backgrounds hinder these tasks, particularly when concise prompts provide insufficient guidance. Although task-specific fine-tuning improves image-text alignment, it requires large-scale annotations and model-specific updates, increasing adaptation costs across models and tasks. To address this, we propose AerialVP, a task-adaptive, training-free agent that adapts VLMs through prompt enhancement rather than parameter optimization. AerialVP dynamically adapts its enhancement strategy to the original prompt's specificity, selectively incorporating image-derived semantics, spatial positions, and object relationships. These cues provide discriminative descriptions and spatial constraints, leveraging VLMs' pretrained alignment and reasoning capabilities to improve image-text alignment and reduce target ambiguity. Specifically, an LLM-based Task Engine coordinates modular tools for prompt analysis, visual information extraction, and prompt fusion. The enhanced prompt and original image are then provided to the unchanged backend VLM for prediction. This prompt-centric strategy enables training-free adaptation across diverse open-source and proprietary VLMs. For systematic evaluation, we introduce AerialSense, a unified benchmark covering Aerial Visual Grounding, Aerial Visual Reasoning, and Aerial Visual Question Answering across diverse UAV scenes and imaging conditions. Experiments demonstrate consistent improvements across all three tasks and evaluated VLMs. AerialVP also achieves competitive performance against LoRA-based instruction fine-tuning, validating prompt-centric adaptation. Our work is available at https://github.com/lostwolves/AerialVP.
Figures & tables
| Dataset Name | Task Types | Number of UAV Images | Number of Task Samples | Number of Target Categories | Average Instruction Length |
|---|---|---|---|---|---|
| HurMic-VQA [ 57 ] | VQA | 3,197 | 9,591 | 4 | 7.11 |
| FloodNet-VQA [ 58 ] | VQA | 2,188 | 7,355 | 9 | 7.89 |
| FloodNet-VQA V2.0 [ 59 ] | VQA | 2,348 | 10,480 | 9 | 7.93 |
| refGeo [ 60 ] | VG | 623 | 26,482 | 1 | 5.58 |
| AerialVG [ 61 ] | VG | 4,211 | 4,723 | 26 | 19.75 |
| AerialSense | VG, VQA, VR | 7,119 | 53,374 | 40 | 20.76 |
| Task Model | Semantic Enhancement | Spatial Position Enhancement | Spatial Relationship Enhancement | Acc | Mean IoU |
| Proprietary Models | |||||
| GPT-4o | 2.04% | 3.69% | |||
| 45.05% | 41.26% | ||||
| GPT-4.1 | 3.60% | 11.91% | |||
| 49.28% | 44.87% | ||||
| Gemini-2.0-Flash | 10.17% | 14.99% |
| Task Model | Semantic Enhancement | Spatial Relationship Enhancement | Acc(VR) | Acc(VQA) |
| Proprietary Models | ||||
| GPT-4o | 50.72% | 60.03% | ||
| 80.71% | 80.53% | |||
| GPT-4.1 | 70.40% | 82.94% | ||
| 90.32% | 96.73% | |||
| Gemini-2.0-Flash | 75.65% | 73.82% |
| Task Model | Enhancement Strategy | VG | VQA | VR | |
|---|---|---|---|---|---|
| Acc | Mean IoU | Acc | Acc | ||
| InternVL3-8B | / | 2.40% | 7.38% | 74.91% | 56.60% |
| Fine-tuning | 6.04% | 12.52% | 74.84% | 84.12% | |
| Prompt | 14.21% | 14.50% | 95.40% | 78.85% | |
| InternVL3-14B | / | 11.62% | 18.36% | 74.93% | 63.22% |
| Fine-tuning | 20.11% | 25.97% | 74.78% | 70.14% | |
| Task Model | Semantic Enhancement | Spatial Position Enhancement | Spatial Relationship Enhancement | Acc | Mean IoU |
|---|---|---|---|---|---|
| Proprietary Models | |||||
| Qwen2.5-Max | 1.22% | 7.17% | |||
| 7.18% | 6.34% | ||||
| 22.40% | 19.85% | ||||
| 0.07% | 0.14% | ||||
| 38.33% | 33.88% |
| Task Model | Semantic Enhancement | Spatial Relationship Enhancement | Acc(VR) | Acc(VQA) |
|---|---|---|---|---|
| Proprietary Models | ||||
| Qwen2.5-Max | 77.03% | 65.95% | ||
| 87.29% | 73.66% | |||
| 82.71% | 70.66% | |||
| 90.48% | 77.54% | |||
| Open-Source Models |
| Model | Acc (VG) | Mean IoU (VG) | Acc (VR) | Acc (VQA) |
| Proprietary Models | ||||
| GPT-4o | +43.01% | +37.57% | +29.99% | +20.50% |
| GPT-4.1 | +45.68% | +32.96% | +19.92% | +13.79% |
| Gemini-2.0-Flash | +40.92% | +31.23% | +14.75% | +21.53% |
| Claude-3.7-Sonnet | +36.13% | +28.14% | +22.70% | +18.68% |
| Qwen2.5-Max | +42.65% | +32.70% | +13.45% | +11.59% |
| Task Model | Prompt Enhancement Tool | Acc | Mean IoU | ||
| Semantic | Spatial Position | Spatial Relationship | |||
| Proprietary Models | |||||
| GPT-4.1 | DAM | YoloWorld-X | SpaceOm | 49.28% | 44.87% |
| LLaVA-1.5-7B | YoloWorld-X | SpaceOm | 44.00% | 40.56% | |
| LLaVA-OneVision-7B | YoloWorld-X | SpaceOm | 47.58% | 43.39% | |
| Gemini-2.0-Flash | DAM | YoloWorld-X | SpaceOm | 51.09% | 46.22% |
| Task Model | Prompt Enhancement Tool | Acc | Mean IoU | ||
| Semantic | Spatial Position | Spatial Relationship | |||
| Proprietary Models | |||||
| GPT-4.1 | DAM | YoloWorld-X | SpaceOm | 49.28% | 44.87% |
| DAM | Grounding-DINO | SpaceOm | 40.21% | 37.21% | |
| Gemini-2.0-Flash | DAM | YoloWorld-X | SpaceOm | 51.09% | 46.22% |
| DAM | Grounding-DINO | SpaceOm | 42.65% | 38.90% | |
| Task Model | Prompt Enhancement Tool | Acc | Mean IoU | ||
| Semantic | Spatial Position | Spatial Relationship | |||
| Proprietary Models | |||||
| GPT-4.1 | DAM | YoloWorld-X | SpaceOm | 49.28% | 44.87% |
| DAM | YoloWorld-X | LLaVA-1.5-7B | 62.00% | 55.53% | |
| DAM | YoloWorld-X | LLaVA-OneVision-7B | 58.32% | 52.64% | |
| Gemini-2.0-Flash | DAM | YoloWorld-X | SpaceOm | 51.09% | 46.22% |
VLM Adaptation
Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.
See all 559 papers