Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generation

Jintao Sun; Gangyi Ding; Donglin Di; Hu Zhang; Zhedong Zheng

cs.CVApr 7, 2026

Can Vision-Language Models Think from the Sky? Unifying UAV Reasoning and Generation

Authors: Jintao Sun, Gangyi Ding, Donglin Di, Hu Zhang, Zhedong Zheng

Organizations: Beijing Institute of Technology · Harbin Institute of Technology · CSIRO Data61 · University of Macau

Abstract

Vision-Language Models have achieved strong progress in ground-view visual understanding, yet they remain brittle in high-altitude Unmanned Aerial Vehicle scenes, where objects are tiny and densely packed, textures are repetitive, and top-down orientations are ambiguous. We introduce UAVReason, a large-scale UAV-native dataset and evaluation suite for studying unified aerial reasoning and generation under this nadir-view domain shift. UAVReason aligns RGB imagery, depth maps, semantic segmentation masks, captions, and question-answer pairs within a consistent aerial domain. It contains 23.6K captioned frames, 273K VQA pairs including 68.2K two-frame temporal questions, and 188.8K cross-modal generation samples across RGB, depth, and segmentation modalities. We further adapt UAVReason-Bagel as a unified understanding-and-generation baseline that jointly optimizes language reasoning and dense visual generation objectives. Experiments show that general-purpose VLMs and off-the-shelf unified generators struggle with UAV-native grounding, while UAVReason-Bagel substantially improves over its pretrained counterpart, increasing VQA-1F F1 from 0.394 to 0.711, VQA-2F F1 from 0.427 to 0.822, and heading-aware VQA F1 from 0.798 to 0.973. For generation, it improves segmentation mIoU to 0.143 and reduces KID from 0.078 to 0.048 for depth-segmentation-text-conditioned RGB synthesis. More importantly, our ablations reveal a bidirectional synergy between synthesis and reasoning. Dense generation objectives improve temporal semantic consistency, while language-level reasoning regularizes sparse-condition image synthesis. These results suggest that unified reasoning and generation provide effective geometry-aware structural priors for physically grounded aerial intelligence. All data, code, and evaluation tools will be released.

Figures & tables

Figure 1: UAVReason , a unified benchmark for nadir-view spatio-temporal reasoning and cross-modal generation. 1Frame-VQA (Left): Single-frame reasoning with orientation cues ( e.g. , north), requiring object counting, scene captioning, spatial reasoning, and comparison ability. 2Frame-VQA (Middle): Two-frame temporal reasoning over aligned viewpoints, probing motion direction and temporal changes in relative distance and scene changes. Generation (Right): Pixel-level conditional synthesis with diverse inputs, covering semantic segmentation (RGB $\rightarrow$ Seg), depth estimation (RGB $\rightarrow$ Depth), and conditional RGB reconstruction from segmentation and text prompts (Seg+T $\rightarrow$ RGB).

Category	Dataset	Meta		Dense Geom.		Reasoning			Gen.	Scale & Stats
Category	Dataset	Typ	Vw	Dep	Seg	VQA	Tmp	Cap	CMG	Scale & Stats
Detection / Segmentation	VisDrone [ 41 ]	R	M	✗	✗	✗	✗	✗	✗	262k frames, 10k imgs
	UAVDT [ 6 ]	R	M	✗	✗	✗	✗	✗	✗	$\sim$ 80k frames
	UAVid [ 18 ]	R	O	✗	✓	✗	✗	✗	✗	300 labeled 4K seqs
	AeroScapes [ 20 ]	R	M	✗	✓	✗	✗	✗	✗	3.3k images
	DOTA/iSAID [ 31 ]	R	M	✗	✓	✗	✗	✗	✗	188k/655k instances

Table 1: Comparison with existing UAV, aerial, and remote sensing benchmarks. We summarize representative datasets by category and by supported supervision and tasks, including meta properties (Typ, Vw), dense geometry signals (Dep, Seg), reasoning capabilities (VQA, Tmp, Cap), and generation support (CMG), together with dataset scale and statistics. Legend: Typ denotes real (R) or synthetic (S); Vw denotes nadir (N), oblique (O), or mixed (M) views; Dep and Seg indicate dense depth and segmentation annotations; VQA , Tmp , and Cap indicate visual question answering, temporal reasoning, and captioning; CMG indicates cross-modal generation.

Figure 2: Overview of UAVReason benchmark tasks and I/O protocols. Top (Reasoning): Language-centric reasoning on nadir-view UAV imagery, including single-frame spatial queries ( e.g. , counting/comparison and referring questions with orientation cues), global scene captioning, and two-frame temporal reasoning (motion direction and cross-time relational verification). Middle (Geometry prediction): Dense pixel-level supervision aligned to the same UAV frames, covering RGB $\rightarrow$ Depth (grayscale depth visualization, near-to-far) and RGB $\rightarrow$ Seg (palette PNG with crisp boundaries). Bottom (Conditional generation): Controllable RGB synthesis conditioned on geometric modalities and text, including Depth+Text $\rightarrow$ RGB, Seg+Text $\rightarrow$ RGB, and Depth+Seg+Text $\rightarrow$ RGB. Together, these tasks unify semantic reasoning with geometric/structural constraints, supporting unified training and evaluation of grounding fidelity, geometric consistency, and controllable synthesis in the UAV domain.

Figure 3: UAVReason construction pipeline. We apply the off-the-shelf tools to generate dense, physically grounded annotations for the nadir-view UAV imagery, including depth rendering, 3D instance lifting and refinement, and hierarchical question and caption generation.

Figure 4: Task and Data Distribution in UAVReason.

Figure 5: Unified multi-task baseline for UAVReason. We adopt a shared multi-modal transformer backbone with two task-specialized experts: a Reasoning Expert (top) optimized for next-token prediction over language outputs, and a Generation Expert (bottom) optimized for velocity prediction under diffusion-style latent denoising. Text is tokenized and fused with visual features via multi-modal self-attention, while expert-specific QKV projections and FFNs specialize computation for reasoning versus generation. The reasoning branch consumes UAV RGB frames with questions (and captioning prompts) to produce textual answers, whereas the generation branch conditions on RGB/depth/segmentation (and optional text) to synthesize target modalities. This design enables a single backbone to jointly learn language reasoning and pixel-level generation under a consistent UAV domain.

Method	VQA-1F			VQA-2F			VQA-2F ${}_{\text{ihead}}$			Caption
Method	EM $\uparrow$	F1 $\uparrow$	LLM-J $\uparrow$	EM $\uparrow$	F1 $\uparrow$	LLM-J $\uparrow$	EM $\uparrow$	F1 $\uparrow$	LLM-J $\uparrow$	B4 $\uparrow$	C $\uparrow$	LLM-J $\uparrow$
Group A: Off-the-shelf General VLMs / MLLMs
Qwen2.5-VL-7B	0.041	0.217	0.355	0.000	0.138	0.454	0.000	0.088	0.000	0.015	0.321	7.02
Qwen2.5-VL-3B	0.052	0.163	0.325	0.000	0.236	0.323	0.000	0.086	0.050	0.019	0.319	6.33
Qwen3-VL-8B	0.031	0.113	0.325	0.000	0.207	0.475	0.000	0.088	0.080	0.435	0.041	7.01
Qwen3-VL-4B	0.031	0.132	0.340	0.000	0.279	0.414	0.000	0.041	0.070	0.037	0.513	6.99

Table 2: Results on UAVReason for spatio-temporal VQA and scene captioning. We evaluate 3 reasoning settings: single-frame VQA (VQA-1F), two-frame VQA (VQA-2F), and two-frame VQA with implicit heading cues (VQA-2F

{}_{\text{ihead}}

), alongside scene captioning. For VQA, we report Exact Match (EM), token-level F1, and an LLM-based judge score (LLM-J) to measure semantic correctness beyond lexical overlap. Methods are grouped into general-purpose VLMs, UAV/remote-sensing VLMs, unified generation models, and our unified framework. UAVReason-Bagel achieves state-of-the-art performance across all tasks. Compared with the ablation row ( Reasoning only ), joint training with generation improves semantic consistency in temporal reasoning (higher LLM-J).

Method	Perc.	Conditional Generation Quality
Method	Seg	dSeg+T $\to$ RGB			Seg+T $\to$ RGB			d+T $\to$ RGB
	mIoU $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$
Off-the-shelf Unified Generators
OmniGen2	0.037	0.094	25.46	0.317	0.093	24.29	0.507	0.094	24.40	0.378
Show-o2-7B	0.023	–	–	–	0.109	25.67	0.537	0.118	23.81	0.507
Our Unified Framework

Table 3: Dense perception and conditional cross-modal generation on UAVReason. We report 18-class mIoU for semantic segmentation ( Perc. ). For conditional image generation, we evaluate three settings: depth+segmentation+text to RGB (dSeg+T

\to

RGB), segmentation+text to RGB (Seg+T

\to

RGB), and depth+text to RGB (d+T

\to

RGB). Quality is measured by KID (

\downarrow

), CLIP score (

\uparrow

), and DINO similarity (

\uparrow

). UAVReason-Bagel significantly outperforms off-the-shelf unified generators. The w/o Understanding ablation shows that reasoning tasks contribute to better generation quality (lower KID, higher DINO) in sparse-conditioning settings (d+T

\to

RGB).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Resolution	VQA-1F		VQA-2F		VQA-2F ${}_{\text{ihead}}$		Caption
Resolution	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	B4 $\uparrow$	CIDEr $\uparrow$
256	0.196	0.711	0.186	0.822	0.657	0.973	0.216	1.530
384	0.186	0.727	0.197	0.850	0.793	0.980	0.254	1.800
512	0.189	0.725	0.200	0.847	0.868	0.990	0.236	1.663

Appendix

Table 4: Effect of input image resolution on UAVReason understanding tasks. Resolution 384 yields the best overall trade-off, achieving the strongest captioning scores and competitive VQA performance. Higher resolution (512) specifically benefits heading-aware VQA, likely due to better visibility of subtle orientation cues.

Resolution	Perc.	Conditional Generation Quality
Resolution	Seg	dSeg+T $\to$ RGB			Seg+T $\to$ RGB			d+T $\to$ RGB
	mIoU $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$
256	0.143	0.048	29.52	0.648	0.050	29.41	0.524	0.036	29.23	0.609
384	0.194	0.041	29.14	0.554	0.032	30.23	0.696	0.028	29.20	0.582
512	0.144	0.024	28.02	0.697	0.024	29.52	0.701	0.018	29.38	0.724

Appendix

Table 5: Effect of input resolution on dense perception and conditional generation. Higher resolution consistently improves generation fidelity (lowest KID at 512) and structural consistency (highest DINO at 512). However, segmentation performance peaks at 384.

Learning Rate	VQA-1F		VQA-2F		VQA-2F ${}_{\text{ihead}}$		Caption
Learning Rate	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	EM $\uparrow$	F1 $\uparrow$	B4 $\uparrow$	CIDEr $\uparrow$
8e-6	0.196	0.711	0.186	0.822	0.657	0.973	0.216	1.530
1e-5	0.186	0.726	0.159	0.839	0.725	0.725	0.225	1.540
5e-6	0.189	0.725	0.200	0.847	0.868	0.990	0.236	1.663

Appendix

Table 6: Impact of learning rate on understanding tasks. Our default setting (

8\mathrm{e}{-6}

) achieves the highest accuracy (EM) on single-frame VQA, which is the foundational capability for the model. While a lower rate (

5\mathrm{e}{-6}

) benefits long-text captioning,

8\mathrm{e}{-6}

provides a robust baseline across different question types.

Learning Rate	Perc.	Conditional Generation Quality
Learning Rate	Seg	dSeg+T $\to$ RGB			Seg+T $\to$ RGB			d+T $\to$ RGB
	mIoU $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$	KID $\downarrow$	CLIP $\uparrow$	DINO $\uparrow$
8e-6	0.143	0.048	29.52	0.648	0.050	29.41	0.524	0.036	29.23	0.609
1e-5	0.093	0.070	29.73	0.563	0.073	28.91	0.443	0.028	28.23	0.486
5e-6	0.135	0.067	28.85	0.582	0.071	29.49	0.514	0.062	28.84	0.507

Appendix

Table 7: Effect of learning rate on dense perception and generation. Our selected learning rate (

8\mathrm{e}{-6}

) significantly outperforms others in segmentation (mIoU) and image generation fidelity (KID/DINO), indicating that pixel-level tasks require this specific optimization magnitude to converge effectively.

Figure 6: Qualitative comparison on fine-grained spatio-temporal reasoning. We evaluate UAVReason-Bagel (Ours) against leading general-domain VLMs ( e.g. , Qwen2.5-VL [ 2 ] , InternVL [ 4 ] ) across three challenging aerial tasks. Left (Counting): Our model accurately counts tiny, densely packed instances (traffic barriers), whereas baselines suffer from severe miss-detection or count hallucinations. Middle (Navigation): Guided by explicit orientation cues ( e.g. , “North is right”), our model correctly aligns the heading to infer the “West” trajectory, while general VLMs fail to adapt compass directions to the nadir view. Right (Temporal Dynamics): Our model successfully detects fine-grained status changes (separation) and quantifies distance shifts, whereas baselines often hallucinate the opposite motion trend ( e.g. , “approaching”).

Figure 7: Qualitative comparison on fine-grained scene description. On a nadir-view airstrip, UAVReason-Bagel (Ours) achieves superior semantic granularity , correctly identifying the context (“airstrip”) and resolving specific instances ( e.g. , “sedans”). In contrast, general-domain VLMs exhibit severe semantic drift, often hallucinating non-existent scenes ( e.g. , “construction site”) or resorting to coarse abstractions (generic “vehicles”).

Figure 8: Qualitative comparison of dense perception and conditional synthesis. We visualize predictions from UAVReason-Bagel versus general-domain baselines (BAGEL, OmniGen2). In perception tasks (Top rows), our model recovers fine-grained object boundaries and smooth depth gradients, whereas baselines exhibit significant noise and semantic errors. In conditional synthesis (Bottom rows), our model generates photorealistic textures that strictly adhere to the input geometric layouts (depth/segmentation). In contrast, baselines suffer from severe domain shift, resulting in structural hallucinations and failure to follow the nadir-view control signals.

Figure 9: Qualitative comparison of dense perception and conditional synthesis. We visualize predictions from UAVReason-Bagel versus general-domain baselines (BAGEL, OmniGen2). In perception tasks (Top rows), our model recovers fine-grained object boundaries and smooth depth gradients, whereas baselines exhibit significant noise and semantic errors. In conditional synthesis (Bottom rows), our model generates photorealistic textures that strictly adhere to the input geometric layouts (depth/segmentation). In contrast, baselines suffer from severe domain shift, resulting in structural hallucinations and failure to follow the nadir-view control signals.

Panel of Figure 10 — Figure 10: Prompts for the Image Enrichment phase. Step-1 verifies global layout, while Step-2 fills fine-grained attributes and spatial relations.

Panel of Figure 11 — Figure 11: Prompts for Question Generation. The top block instructs the model to cover diverse cognitive tasks (e.g., counting, spatial reasoning) on single frames. The bottom block enforces cross-frame tracking and dynamic state analysis for temporal understanding.

Panel of Figure 12 — Figure 12: Prompts for the Program-Aided Answer Generation. The pipeline consists of three stages: the Planner parses the natural language question into a structured JSON query; the Writer synthesizes a fluent answer based on ground-truth facts computed by the deterministic executor; and the Validator performs a final sanity check to ensure no numerical hallucinations are included.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation
Apr 17, 2026Shuyan Ke, Yifan Mei, Changli Wu +4Image Segmentation Remote Sensing Image Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute
AeroGround: A Comprehensive Benchmark for Aerial-Ground Collaborative Reasoning
Aug 12, 2026Shenghong Yi, Lin Zhang, Muzian Li +6VLM Evaluation VLM Reasoning
Shanghai Innovation Institute · College of Future Information Technology, Fudan University · The Chinese University of Hong Kong +2
Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System
Aug 12, 2026Haoyu Zhang, Shuoxun Zhang, Peng Ye +5Visual Question Answering Multi-Agent LLM Systems
Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China · Embedded Deep Learning and Visual Analysis Laboratory, College of Future Information Technology, Fudan University, Shanghai 200433, China, and also with Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China, and also with The Chinese University of Hong Kong, Hong Kong · College of Future Information Technology, Fudan University, Shanghai 200433, China +1

Apr 17, 2026J/K move · Enter open · S save

PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation

Shuyan Ke, Yifan Mei, Changli Wu, Yonghan Zheng+3

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · Shanghai Innovation Institute

Image Segmentation Remote Sensing Image Segmentation Multimodal Reasoning VLM Reasoning Reasoning Segmentation UAV Remote Sensing

Open paper AskPDF arXiv

Reasoning segmentation has recently expanded from ground-level scenes to remote-sensing imagery, yet UAV data poses distinct challenges, including oblique viewpoints, ultra-high resolutions, and extreme scale variations. To address these issues, we formally define the UAV Reasoning Segmentation task and organize its semantic requirements into three dimensions: Spatial, Attribute, and Scene-level reasoning. Based on this formulation, we construct DRSeg, a large-scale benchmark for UAV reasoning segmentation, containing 10k high-resolution aerial images paired with Chain-of-Thought QA supervision across all three reasoning types. As a benchmark companion, we introduce PixDLM, a simple yet effective pixel-level multimodal language model that serves as a unified baseline for this task. Experiments on DRSeg establish strong baseline results and highlight the unique challenges of UAV reasoning segmentation, providing a solid foundation for future research.

Abstract

VLM Reasoning

Explore similar work

PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation

AeroGround: A Comprehensive Benchmark for Aerial-Ground Collaborative Reasoning

Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System