Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generation

Jintao Sun; Gangyi Ding; Donglin Di; Hu Zhang; Zhedong Zheng

cs.CVApr 7, 2026

Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generation

Authors: Jintao Sun, Gangyi Ding, Donglin Di, Hu Zhang, Zhedong Zheng

Organizations: Beijing Institute of Technology · Harbin Institute of Technology · CSIRO Data61 · University of Macau

Abstract

Vision-Language Models have achieved strong progress in ground-view visual understanding, yet they remain brittle in high-altitude Unmanned Aerial Vehicle scenes, where objects are tiny and densely packed, textures are repetitive, and top-down orientations are ambiguous. We introduce UAVReason, a large-scale UAV-native dataset and evaluation suite for studying unified aerial reasoning and generation under this nadir-view domain shift. UAVReason aligns RGB imagery, depth maps, semantic segmentation masks, captions, and question-answer pairs within a consistent aerial domain. It contains 23.6K captioned frames, 273K VQA pairs including 68.2K two-frame temporal questions, and 188.8K cross-modal generation samples across RGB, depth, and segmentation modalities. We further adapt UAVReason-Bagel as a unified understanding-and-generation baseline that jointly optimizes language reasoning and dense visual generation objectives. Experiments show that general-purpose VLMs and off-the-shelf unified generators struggle with UAV-native grounding, while UAVReason-Bagel substantially improves over its pretrained counterpart, increasing VQA-1F F1 from 0.394 to 0.711, VQA-2F F1 from 0.427 to 0.822, and heading-aware VQA F1 from 0.798 to 0.973. For generation, it improves segmentation mIoU to 0.143 and reduces KID from 0.078 to 0.048 for depth-segmentation-text-conditioned RGB synthesis. More importantly, our ablations reveal a bidirectional synergy between synthesis and reasoning. Dense generation objectives improve temporal semantic consistency, while language-level reasoning regularizes sparse-condition image synthesis. These results suggest that unified reasoning and generation provide effective geometry-aware structural priors for physically grounded aerial intelligence. All data, code, and evaluation tools will be released.

Figures & tables

Figure 1: UAVReason , a unified benchmark for nadir-view spatio-temporal reasoning and cross-modal generation. 1Frame-VQA (Left): Single-frame reasoning with orientation cues ( e.g. , north), requiring object counting, scene captioning, spatial reasoning, and comparison ability. 2Frame-VQA (Middle): Two-frame temporal reasoning over aligned viewpoints, probing motion direction and temporal changes in relative distance and scene changes. Generation (Right): Pixel-level conditional synthesis with diverse inputs, covering semantic segmentation (RGB $\rightarrow$ Seg), depth estimation (RGB $\rightarrow$ Depth), and conditional RGB reconstruction from segmentation and text prompts (Seg+T $\rightarrow$ RGB).

Category	Dataset	Meta		Dense Geom.		Reasoning			Gen.	Scale & Stats
Category	Dataset	Typ	Vw	Dep	Seg	VQA	Tmp	Cap	CMG	Scale & Stats
Detection / Segmentation	VisDrone [ 41 ]	R	M	✗	✗	✗	✗	✗	✗	262k frames, 10k imgs
	UAVDT [ 6 ]	R	M	✗	✗	✗	✗	✗	✗	$\sim$ 80k frames
	UAVid [ 18 ]	R	O	✗	✓	✗	✗	✗	✗	300 labeled 4K seqs
	AeroScapes [ 20 ]	R	M	✗	✓	✗	✗	✗	✗	3.3k images
	DOTA/iSAID [ 31 ]	R	M	✗	✓	✗	✗	✗	✗	188k/655k instances

Table 1: Comparison with existing UAV, aerial, and remote sensing benchmarks. We summarize representative datasets by category and by supported supervision and tasks, including meta properties (Typ, Vw), dense geometry signals (Dep, Seg), reasoning capabilities (VQA, Tmp, Cap), and generation support (CMG), together with dataset scale and statistics. Legend: Typ denotes real (R) or synthetic (S); Vw denotes nadir (N), oblique (O), or mixed (M) views; Dep and Seg indicate dense depth and segmentation annotations; VQA , Tmp , and Cap indicate visual question answering, temporal reasoning, and captioning; CMG indicates cross-modal generation.

Figure 2: Overview of UAVReason benchmark tasks and I/O protocols. Top (Reasoning): Language-centric reasoning on nadir-view UAV imagery, including single-frame spatial queries ( e.g. , counting/comparison and referring questions with orientation cues), global scene captioning, and two-frame temporal reasoning (motion direction and cross-time relational verification). Middle (Geometry prediction): Dense pixel-level supervision aligned to the same UAV frames, covering RGB $\rightarrow$ Depth (grayscale depth visualization, near-to-far) and RGB $\rightarrow$ Seg (palette PNG with crisp boundaries). Bottom (Conditional generation): Controllable RGB synthesis conditioned on geometric modalities and text, including Depth+Text $\rightarrow$ RGB, Seg+Text $\rightarrow$ RGB, and Depth+Seg+Text $\rightarrow$ RGB. Together, these tasks unify semantic reasoning with geometric/structural constraints, supporting unified training and evaluation of grounding fidelity, geometric consistency, and controllable synthesis in the UAV domain.

Figure 3: UAVReason construction pipeline. We apply the off-the-shelf tools to generate dense, physically grounded annotations for the nadir-view UAV imagery, including depth rendering, 3D instance lifting and refinement, and hierarchical question and caption generation.

Figure 4: Task and Data Distribution in UAVReason.

Figure 5: Unified multi-task baseline for UAVReason. We adopt a shared multi-modal transformer backbone with two task-specialized experts: a Reasoning Expert (top) optimized for next-token prediction over language outputs, and a Generation Expert (bottom) optimized for velocity prediction under diffusion-style latent denoising. Text is tokenized and fused with visual features via multi-modal self-attention, while expert-specific QKV projections and FFNs specialize computation for reasoning versus generation. The reasoning branch consumes UAV RGB frames with questions (and captioning prompts) to produce textual answers, whereas the generation branch conditions on RGB/depth/segmentation (and optional text) to synthesize target modalities. This design enables a single backbone to jointly learn language reasoning and pixel-level generation under a consistent UAV domain.

Method	VQA-1F			VQA-2F			VQA-2F ${}_{\text{ihead}}$			Caption
Method	EM $\uparrow$	F1 $\uparrow$	LLM-J $\uparrow$	EM $\uparrow$	F1 $\uparrow$	LLM-J $\uparrow$	EM $\uparrow$	F1 $\uparrow$	LLM-J $\uparrow$	B4 $\uparrow$	C $\uparrow$	LLM-J $\uparrow$
Group A: Off-the-shelf General VLMs / MLLMs
Qwen2.5-VL-7B	0.041	0.217	0.355	0.000	0.138	0.454	0.000	0.088	0.000	0.015	0.321	7.02
Qwen2.5-VL-3B	0.052	0.163	0.325	0.000	0.236	0.323	0.000	0.086	0.050	0.019	0.319	6.33
Qwen3-VL-8B	0.031	0.113	0.325	0.000	0.207	0.475	0.000	0.088	0.080	0.435	0.041	7.01
Qwen3-VL-4B	0.031	0.132	0.340	0.000	0.279	0.414	0.000	0.041	0.070	0.037	0.513	6.99

Table 2: Results on UAVReason for spatio-temporal VQA and scene captioning. We evaluate 3 reasoning settings: single-frame VQA (VQA-1F), two-frame VQA (VQA-2F), and two-frame VQA with implicit heading cues (VQA-2F

{}_{\text{ihead}}

), alongside scene captioning. For VQA, we report Exact Match (EM), token-level F1, and an LLM-based judge score (LLM-J) to measure semantic correctness beyond lexical overlap. Methods are grouped into general-purpose VLMs, UAV/remote-sensing VLMs, unified generation models, and our unified framework. UAVReason-Bagel achieves state-of-the-art performance across all tasks. Compared with the ablation row ( Reasoning only ), joint training with generation improves semantic consistency in temporal reasoning (higher LLM-J).

Method	Perc.	Conditional Generation Quality
Method	Seg	dSeg+T $\to$ RGB			Seg+T $\to$ RGB			d+T $\to$ RGB
	mIoU $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$
Off-the-shelf Unified Generators
OmniGen2	0.037	0.094	25.46	0.317	0.093	24.29	0.507	0.094	24.40	0.378
Show-o2-7B	0.023	–	–	–	0.109	25.67	0.537	0.118	23.81	0.507
Our Unified Framework

Table 3: Dense perception and conditional cross-modal generation on UAVReason. We report 18-class mIoU for semantic segmentation ( Perc. ). For conditional image generation, we evaluate three settings: depth+segmentation+text to RGB (dSeg+T

\to

RGB), segmentation+text to RGB (Seg+T

\to

RGB), and depth+text to RGB (d+T

\to

RGB). Quality is measured by KID (

\downarrow

), CLIP score (

\uparrow

), and DINO similarity (

\uparrow

). UAVReason-Bagel significantly outperforms off-the-shelf unified generators. The w/o Understanding ablation shows that reasoning tasks contribute to better generation quality (lower KID, higher DINO) in sparse-conditioning settings (d+T

\to

RGB).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Resolution	VQA-1F		VQA-2F		VQA-2F ${}_{\text{ihead}}$		Caption
Resolution	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	B4 $\uparrow$	CIDEr $\uparrow$
256	0.196	0.711	0.186	0.822	0.657	0.973	0.216	1.530
384	0.186	0.727	0.197	0.850	0.793	0.980	0.254	1.800
512	0.189	0.725	0.200	0.847	0.868	0.990	0.236	1.663

Appendix

Table 4: Effect of input image resolution on UAVReason understanding tasks. Resolution 384 yields the best overall trade-off, achieving the strongest captioning scores and competitive VQA performance. Higher resolution (512) specifically benefits heading-aware VQA, likely due to better visibility of subtle orientation cues.

Resolution	Perc.	Conditional Generation Quality
Resolution	Seg	dSeg+T $\to$ RGB			Seg+T $\to$ RGB			d+T $\to$ RGB
	mIoU $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$
256	0.143	0.048	29.52	0.648	0.050	29.41	0.524	0.036	29.23	0.609
384	0.194	0.041	29.14	0.554	0.032	30.23	0.696	0.028	29.20	0.582
512	0.144	0.024	28.02	0.697	0.024	29.52	0.701	0.018	29.38	0.724

Appendix

Table 5: Effect of input resolution on dense perception and conditional generation. Higher resolution consistently improves generation fidelity (lowest KID at 512) and structural consistency (highest DINO at 512). However, segmentation performance peaks at 384.

Learning Rate	VQA-1F		VQA-2F		VQA-2F ${}_{\text{ihead}}$		Caption
Learning Rate	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	B4 $\uparrow$	CIDEr $\uparrow$
8e-6	0.196	0.711	0.186	0.822	0.657	0.973	0.216	1.530
1e-5	0.186	0.726	0.159	0.839	0.725	0.725	0.225	1.540
5e-6	0.189	0.725	0.200	0.847	0.868	0.990	0.236	1.663

Appendix

Table 6: Impact of learning rate on understanding tasks. Our default setting (

8\mathrm{e}{-6}

) achieves the highest accuracy (EM) on single-frame VQA, which is the foundational capability for the model. While a lower rate (

5\mathrm{e}{-6}

) benefits long-text captioning,

8\mathrm{e}{-6}

provides a robust baseline across different question types.

Learning Rate	Perc.	Conditional Generation Quality
Learning Rate	Seg	dSeg+T $\to$ RGB			Seg+T $\to$ RGB			d+T $\to$ RGB
	mIoU $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$
8e-6	0.143	0.048	29.52	0.648	0.050	29.41	0.524	0.036	29.23	0.609
1e-5	0.093	0.070	29.73	0.563	0.073	28.91	0.443	0.028	28.23	0.486
5e-6	0.135	0.067	28.85	0.582	0.071	29.49	0.514	0.062	28.84	0.507

Appendix

Table 7: Effect of learning rate on dense perception and generation. Our selected learning rate (

8\mathrm{e}{-6}

) significantly outperforms others in segmentation (mIoU) and image generation fidelity (KID/DINO), indicating that pixel-level tasks require this specific optimization magnitude to converge effectively.

Figure 6: Qualitative comparison on fine-grained spatio-temporal reasoning. We evaluate UAVReason-Bagel (Ours) against leading general-domain VLMs ( e.g. , Qwen2.5-VL [ 2 ] , InternVL [ 4 ] ) across three challenging aerial tasks. Left (Counting): Our model accurately counts tiny, densely packed instances (traffic barriers), whereas baselines suffer from severe miss-detection or count hallucinations. Middle (Navigation): Guided by explicit orientation cues ( e.g. , “North is right”), our model correctly aligns the heading to infer the “West” trajectory, while general VLMs fail to adapt compass directions to the nadir view. Right (Temporal Dynamics): Our model successfully detects fine-grained status changes (separation) and quantifies distance shifts, whereas baselines often hallucinate the opposite motion trend ( e.g. , “approaching”).

Figure 7: Qualitative comparison on fine-grained scene description. On a nadir-view airstrip, UAVReason-Bagel (Ours) achieves superior semantic granularity , correctly identifying the context (“airstrip”) and resolving specific instances ( e.g. , “sedans”). In contrast, general-domain VLMs exhibit severe semantic drift, often hallucinating non-existent scenes ( e.g. , “construction site”) or resorting to coarse abstractions (generic “vehicles”).

Figure 8: Qualitative comparison of dense perception and conditional synthesis. We visualize predictions from UAVReason-Bagel versus general-domain baselines (BAGEL, OmniGen2). In perception tasks (Top rows), our model recovers fine-grained object boundaries and smooth depth gradients, whereas baselines exhibit significant noise and semantic errors. In conditional synthesis (Bottom rows), our model generates photorealistic textures that strictly adhere to the input geometric layouts (depth/segmentation). In contrast, baselines suffer from severe domain shift, resulting in structural hallucinations and failure to follow the nadir-view control signals.

Figure 9: Qualitative comparison of dense perception and conditional synthesis. We visualize predictions from UAVReason-Bagel versus general-domain baselines (BAGEL, OmniGen2). In perception tasks (Top rows), our model recovers fine-grained object boundaries and smooth depth gradients, whereas baselines exhibit significant noise and semantic errors. In conditional synthesis (Bottom rows), our model generates photorealistic textures that strictly adhere to the input geometric layouts (depth/segmentation). In contrast, baselines suffer from severe domain shift, resulting in structural hallucinations and failure to follow the nadir-view control signals.

Panel of Figure 10 — Figure 10: Prompts for the Image Enrichment phase. Step-1 verifies global layout, while Step-2 fills fine-grained attributes and spatial relations.

Panel of Figure 11 — Figure 11: Prompts for Question Generation. The top block instructs the model to cover diverse cognitive tasks (e.g., counting, spatial reasoning) on single frames. The bottom block enforces cross-frame tracking and dynamic state analysis for temporal understanding.

Panel of Figure 12 — Figure 12: Prompts for the Program-Aided Answer Generation. The pipeline consists of three stages: the Planner parses the natural language question into a structured JSON query; the Writer synthesizes a fluent answer based on ground-truth facts computed by the deterministic executor; and the Validator performs a final sanity check to ensure no numerical hallucinations are included.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

Apr 17, 2026

PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation

Reasoning segmentation has recently expanded from ground-level scenes to remote-sensing imagery, yet UAV data poses distinct challenges, including oblique viewpoints, ultra-high resolutions, and extreme scale variations. To address these issues, we formally define the UAV Reasoning Segmentation task and organize its semantic requirements into three dimensions: Spatial, Attribute, and Scene-level reasoning. Based on this formulation, we construct DRSeg, a large-scale benchmark for UAV reasoning segmentation, containing 10k high-resolution aerial images paired with Chain-of-Thought QA supervision across all three reasoning types. As a benchmark companion, we introduce PixDLM, a simple yet effective pixel-level multimodal language model that serves as a unified baseline for this task. Experiments on DRSeg establish strong baseline results and highlight the unique challenges of UAV reasoning segmentation, providing a solid foundation for future research.

Shuyan Ke, Yifan Mei, Changli Wu +4

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute

Image Segmentation Remote Sensing Image Segmentation Multimodal Reasoning VLM Reasoning Reasoning Segmentation UAV Remote Sensing

Ask PDF

Aug 12, 2026

AeroGround: A Comprehensive Benchmark for Aerial-Ground Collaborative Reasoning

Vision-language models (VLMs) have been widely employed in understanding and reasoning tasks for unmanned aerial vehicles (UAVs). Existing UAV benchmarks primarily focus on aerial-view scenarios. However, whether current VLMs can perform well on understanding and reasoning tasks in aerial-ground collaborative scenarios which are practical in real-world applications like rescue and infrastructure inspection remains underexplored. To address this gap, we introduce AeroGround, a comprehensive benchmark for evaluating VLMs in aerial-ground collaborative reasoning. AeroGround is built upon a simulated aerial-ground dataset containing approximately 29,000 multimodal observation groups from diverse open environments, and provides 2,250 high-quality question-answering instances covering cross-view correspondence, spatial understanding, and reasoning. Experiments on 16 pretrained VLMs, together with two domain-adapted variants, reveal a substantial gap between current models and human performance: the best model achieves an average accuracy of 54.4%, whereas humans reach 93.3%. By systematically revealing the strengths and limitations of existing models in aerial-ground collaborative reasoning, AeroGround provides a foundation for developing more capable aerial-ground collaborative embodied intelligence systems.

Shenghong Yi, Lin Zhang, Muzian Li +6

Shanghai Innovation Institute · College of Future Information Technology, Fudan University · The Chinese University of Hong Kong +2

VLM Evaluation VLM Reasoning Aerial Robotics

Ask PDF

Aug 12, 2026

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System

Multimodal Large Language Model (MLLM)-based UAV aerial image understanding and reasoning is essential for aerial intelligence yet poses distinct challenges arising from extreme scale variation, arbitrary camera orientations, and high object density. Despite growing interest, existing evaluations remain fragmented across individual datasets and narrow tasks, leaving a critical gap in unified assessment of UAV understanding and reasoning capabilities. To fill this gap, we construct UAVQA-Bench, a benchmark of 1,500 human-annotated QA pairs drawn from 13 public UAV datasets, covering 6 capability dimensions and 16 tasks in both multiple-choice and visual grounding formats. Systematic evaluation of a broad range of open-source and closed-source MLLMs as well as agent-based systems on UAVQA-Bench identifies three key failure modes: domain-toolset mismatch, unchecked error propagation, and static reasoning. Motivated by these findings, we propose UAV-MAS, a training-free multi-agent system for MLLM-based UAV aerial image understanding and reasoning, comprising a Domain-Specific Perception Engine (DSPE) that routes queries to task-appropriate visual tools, a Context-Aware Iterative Refinement module (CAIR) that validates intermediate reasoning to curb error accumulation, and a Difficulty-Aware Adaptive Search mechanism (DAAS) that adjusts search depth to question difficulty. UAV-MAS with a 32B open-source MLLM achieves 77.0% overall accuracy on UAVQA-Bench, surpassing Gemini 3 Pro by 4.0%, while the 8B variant improves 8.7% over its base model.

Haoyu Zhang, Shuoxun Zhang, Peng Ye +5

Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China · Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China, and also with Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China, and also with The Chinese University of Hong Kong, Hong Kong · College of Future Information Technology, Fudan University, Shanghai 200433, China +1

Visual Question Answering Multi-Agent LLM Systems Multimodal Large Language Models Multimodal QA Multimodal Model Evaluation

Ask PDF

Abstract

VLM Reasoning

Explore similar work

PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation

AeroGround: A Comprehensive Benchmark for Aerial-Ground Collaborative Reasoning

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System