AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network

Yu Hu; Jianyang Gu; Hao Liu; Yue Cao; Jozsef Hamari; Zheng Liu; Mohsen Zardadi

cs.CVMar 13, 2026

AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network

Authors: Yu Hu, Jianyang Gu, Hao Liu, Yue Cao, Jozsef Hamari, Zheng Liu, Mohsen Zardadi

Organizations: The University of British Columbia, Okanagan, Kelowna, BC, Canada · The Ohio State University, Columbus, OH, USA · TerraSense Analytics, Kelowna, BC, Canada

Abstract

Adapting vision-language models to remote sensing imagery remains challenging due to two key factors: limited semantic coverage in textual representations and insufficient adaptability of visual features. These issues are particularly significant in aerial scenes, which involve various visual appearances and fine-grained object distinctions. We propose AVION, a knowledge distillation framework tailored for remote sensing adaptation of vision-language models. The teacher module constructs semantically rich textual prototypes by collecting descriptions from a large language model and verifying validity using remote sensing image features. The student module integrates lightweight and learnable prompts into both vision and language encoders, guided by the teacher to align embeddings and their cross-modal relationships. Once trained, the student operates independently during inference. Experiments on six optical remote sensing benchmarks show that AVION improves few-shot classification and base-class accuracy without degrading generalization to novel categories. It also enhances mean recall for cross-modal retrieval, with minimal additional trainable parameters.

Figures & tables

Figure 1: The t-SNE visualization of visual embeddings from different datasets. The same class demonstrates large variations across datasets. However, only class names are provided in the datasets, which limits multimodal alignment.

Figure 2: Overview of AVION. Upper-left (Training Student). Learnable prompt tokens are injected into the text and vision encoders. The student outputs embeddings: ${\mathbf{t}}^{S}_{k}$ (student text embedding for class $k$ ) and ${\mathbf{v}}^{S}_{i}$ (visual embedding for image $i$ ). Bottom-left (Offline Teacher). LLM-based Domain Prompting generates multiple class-aware descriptions, which are encoded into $\mathbf{t}^{T}_{k,j}$ . Selective Prototype Aggregation verifies these candidates using teacher visual embeddings ${\mathbf{v}}^{T}_{i}$ and aggregates them into an RS-aware text prototype ${\mathbf{t}}^{T*}_{k}$ . Right (Training Objectives). Tri-aspect alignment: (i) textual alignment $\mathcal{L}_{\text{text}}$ maximizes the cosine similarity between ${\mathbf{t}}^{S}_{k}$ and ${\mathbf{t}}^{T*}_{k}$ ; (ii) visual alignment $\mathcal{L}_{\text{img}}$ aligns ${\mathbf{v}}^{S}_{i}$ with ${\mathbf{v}}^{T}_{i}$ ; (iii) similarity logit alignment $\mathcal{L}_{\text{logit}}$ matches teacher logits $s^{T}_{i,k}=({\mathbf{v}}^{T}_{i})^{\top}{\mathbf{t}}^{T*}_{k}$ and student logits $s^{S}_{i,k}=({\mathbf{v}}^{S}_{i})^{\top}{\mathbf{t}}^{S}_{k}$ via temperature-scaled KL. A standard task loss $\mathcal{L}_{\text{task}}$ (cross-entropy) is applied on the student.

Figure 3: LLM-based Domain Prompting. Given a class $k$ , an RS-aware query asks the LLM to produce aerial-view descriptions. RS-Flag is used to examine whether the description contains RS-related tokens.

Figure 4: Selective Prototype Aggregation. For each class $k$ , teacher image embeddings $\{\mathbf{v}^{T}_{k,i}\}_{i\in\mathcal{B}_{k}}$ are averaged to form the visual prototype ${\widehat{\mathbf{v}}}^{T}_{k}$ . Each LLM-generated description is encoded to a teacher text embedding $\mathbf{t}_{k,j}$ and scored by cosine similarity $s_{k,j}=({\widehat{\mathbf{v}}}^{T}_{k})^{\top}\mathbf{t}^{T}_{k,j}$ . A median/MAD threshold $\pm\zeta_{s}$ removes outlier candidates. The remaining embeddings are combined with softmax-normalized weights (over kept $j$ ) to obtain the class textual prototype ${\mathbf{t}}^{T*}_{k}$ .

Method	1-shot	2-shot	4-shot	8-shot	16-shot
Zero-shot Methods
CLIP [ 27 ]	$63.24$
RemoteCLIP [ 25 ]	$72.92$
GeoRSCLIP [ 41 ]	$\mathbf{72.95}$
PEFT Methods
CoOp [ 43 ]	$69.98$	$78.95$	$84.52$	$87.57$	$90.24$

Table 1: Few-shot classification against state-of-the-art methods. The table presents the average classification accuracy (%) over 6 datasets. The best-performing result for each

K

-shot setting is in bold , and the runner-up is underlined .

Method	Base	Novel	HM
GeoRSCLIP [ 41 ]	$78.10$	$79.75$	$78.81$
CoOp [ 43 ]	$91.17$	$69.52$	$78.88$
CoCoOp [ 42 ]	$91.85$	$70.69$	$80.06$
MMRL [ 13 ]	$93.75$	$73.69$	$82.76$
MaPLe [ 20 ]	$92.87$	$74.66$	$82.90$
PromptKD [ 24 ]	$93.67$	$76.58$	$84.20$

Table 2: Average base-to-novel results (%) across all 6 datasets (ViT-B/32 student). We report Base, Novel, and HM.

Figure 5: t-SNE of base-to-novel visual and text embeddings on RESISC-45. (a) GeoRSCLIP [ 41 ] (b) AVION (ours) (c) APPLeNet [ 28 ] (d) MMRL [ 13 ] . Gray dots represent the visual embedding of base classes . Colored dots represent the visual embedding of novel classes with each color standing for a class. Colored crosses represent the text embedding of novel classes for their class names. A, B, C, and D are four classes.

Testing Dataset	Training Dataset	Method	Image $\to$ Text			Text $\to$ Image			mR
Testing Dataset	Training Dataset	Method	R@1	R@5	R@10	R@1	R@5	R@10	mR
RSITMD [ 37 ]	RSITMD	ML Retrieval [ 1 ]	$19.09$	$40.26$	$54.42$	$17.61$	$49.73$	$60.59$	$41.38$
	RET - 3	CLIP [ 27 ]	$23.67$	$50.00$	$63.27$	$22.61$	$55.27$	$69.87$	$47.63$
	RET - 3 + DET - 10 + SEG - 4	RemoteCLIP [ 25 ]	$27.88$	$50.66$	$65.71$	$22.17$	$56.46$	$67.41$	$49.38$
	RSITMD + RSICD	GeoRSCLIP-FT [ 41 ]	$32.30$	$53.32$	$67.92$	$25.04$	$57.88$	$74.38$	$51.81$
	RSITMD + RSICD	AVION (Ours)	$\mathbf{33.17}$	$\mathbf{54.46}$	$\mathbf{69.18}$	$\mathbf{26.09}$	$\mathbf{59.22}$	$\mathbf{75.41}$	$\mathbf{52.92}$

Table 3: Cross-modal retrieval ( R@K , %) on RSITMD [ 37 ] and RSICD [ 26 ] (ViT-B/32 student; protocol details in Sec. 4.1 ). mR is the macro-average over

\{\text{T}\rightarrow\text{I},\text{I}\rightarrow\text{T}\}\times\{1,5,10\}

\Delta

is relative to GeoRSCLIP. Our method (AVION) is lightweight, updating only prompts/heads; see Appendix F for a full efficiency analysis.

ID	Configuration Description	ID	Configuration Description
B0	Shallow text-only prompt (CoOp-style), $\mathcal{L}_{\mathrm{task}}$ only.	B4	Replace manual prototypes with LLM-generated ones.
B1	+ Deep prompt tuning.	B5	+ Selective prototype aggregation (vision-guided).
B2	+ Representation alignment ( $\mathcal{L}_{\mathrm{img}}$ ).	B6	+ Behavioral alignment ( $\mathcal{L}_{\mathrm{logit}}$ ) in a single-stage schedule.
B3	+ Semantic alignment ( $\mathcal{L}_{\mathrm{text}}$ ) with manual textual prototypes.	B7	Final Model : 30% warm-up (Sec. 4.1 ).

Table 4: Configurations for our incremental ablation study. Each configuration builds upon the previous one and culminates in the full system (B7).

ID	Base	Novel	HM	K-shot
ID	Base	Novel	HM	1	2	4	8	16
B0	$91.17$	$69.52$	$78.88$	$69.98$	$78.95$	$84.52$	$87.57$	$90.24$
B1	$93.52$	$53.51$	$66.71$	$66.95$	$84.68$	$84.61$	$87.34$	$91.25$
B2	$89.74$	$61.15$	$72.74$	$70.21$	$82.99$	$85.31$	$89.65$	$91.51$
B3	$91.21$	$68.63$	$78.34$	$71.98$	$83.72$	$85.90$	$90.43$	$92.18$
B4	$91.03$	$74.08$	$81.67$	$72.15$	$84.71$	$87.43$	$91.89$	$92.14$

Table 5: Main ablation results for classification. We report Base, Novel, and Harmonic Mean (HM) accuracy. Metrics are averaged over six datasets.

ID	Prototype Source	Aggregation	RS-Flag	HM (%)
P0	Manual (single template)	none	–	$79.52$
P1	Manual (multi-templates)	mean	–	$79.91$
P2	LLM (unconstrained)	mean	no	$80.85$
P3	LLM (domain-constrained)	mean	yes	$81.70$
P4	LLM (domain-constrained)	text-only selection	yes	$82.05$
P5	LLM (random subset)	mean	no	$80.31$

Table 6: Teacher text-prototype ablation (P0–P6). All settings use the same objectives, with different prototype sources and aggregation. HM averaged over the same datasets/seeds.

Figure 6: Few-shot classification accuracy of AVION with different CLIP-based backbones in the few-shot setting.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Few-shot classification performance (top-1 accuracy, %) across six remote sensing datasets (AID [ 31 ] , EuroSAT [ 14 ] , RESISC-45 [ 7 ] , UCMerced [ 33 ] , WHU-RS19 [ 32 ] , and PatternNet [ 44 ] ) for different numbers of shots $K\!\in\!\{1,2,4,8,16\}$ .

Dataset	Method	1	2	4	8	16
AID	CoCoOp	74.91	83.23	85.90	90.53	93.28
	CoOp	77.22	82.09	85.67	89.77	92.79
	MMRL	76.30	80.35	87.23	91.28	92.89
	MaPLe	75.24	82.26	86.89	91.13	93.18
	PromptKD	74.85	80.72	86.41	90.67	92.95
	APPLeNet	81.79	83.24	87.11	91.24	93.20

Appendix

Table 7: Few-shot classification (top-1, %) on six remote sensing datasets. Winners are bold ; runners-up are underlined .

AID [ 31 ]	Base	Novel	HM
GeoRSCLIP [ 41 ]	71.96	76.90	74.35
CoOp [ 43 ]	95.53	66.41	78.35
CoCoOp [ 42 ]	94.83	68.03	79.22
MMRL [ 13 ]	95.27	70.40	80.97
MaPLe [ 20 ]	95.10	70.60	81.04
PromptKD [ 24 ]	94.90	70.00	80.57

Appendix

Table 8: Detailed base-to-novel results (%) on six remote sensing datasets (ViT-B/32 student). Baselines: GeoRSCLIP [ 41 ] , CoOp [ 43 ] , CoCoOp [ 42 ] , MMRL [ 13 ] , MaPLe [ 20 ] , PromptKD [ 24 ] , and APPLeNet [ 28 ] . We report Base, Novel, and HM;

\Delta

denotes the difference between Ours and the best-performing baseline for each metric.

Figure 8: Base-to-novel generalization (HM, %) across six remote sensing datasets (ViT-B/32 student). The radar chart plots the harmonic mean between base and novel accuracies for AVION and competing baselines; a larger enclosed area indicates better overall base-to-novel performance.

Symbol	Meaning	Default / Search
$N$	# training images	–
$C$	# classes	–
$D$	image/text embedding dimension	inherited from the VLM
$f_{T},g_{T}$	teacher image/text encoders	frozen
$f_{S},g_{S}$	student image/text encoders	backbones frozen
$\mathbf{v}^{T}_{i}$	$\ell_{2}$ -normalized teacher image feature for $x_{i}$	unit $\ell_{2}$ -norm

Appendix

Table 9: Symbols and default/search ranges used in AVION. In all experiments we fix

K_{p}{=}30

\beta{=}10

\gamma{=}2

\zeta_{s}{=}3.0

Class Name	Simple Template	LLM-Generated Candidates
airport	a photo of an airport	- Overhead airport imagery reveals aprons (with parked aircraft at stands) adjacent to terminals, connected to the airfield via taxilanes and the taxiway system. - An aerial view of an airport showing multiple long, paved runways and taxiways. - Satellite imagery capturing a large airfield complex with terminal buildings and hangars.
industrial area	a photo of an industrial area	- A top-down view of an industrial zone characterized by large factory buildings and smokestacks. - Aerial imagery of a manufacturing plant complex with large warehouses and shipping depots. - A satellite photo showing a dense cluster of large-scale industrial structures and storage tanks.
commercial area	a photo of a commercial area	- An overhead view of a commercial district featuring multi-story office buildings and retail centers. - Nadir-view imagery of a dense downtown business area with high-rise buildings and streets. - An aerial photo of a shopping center or mall surrounded by a large, organized parking lot.
forest	a photo of a forest	- A vast expanse of dense, green forest canopy as seen from a satellite. - An aerial view looking straight down at a dense woodland of coniferous or deciduous trees. - A top-down remote sensing image of a large, contiguous area covered by trees.

Appendix

Table 10: Examples of LLM-generated candidate descriptions from our LLM-based domain prompting module, compared to standard simple templates. The generated candidates provide richer semantic and visual cues specific to remote sensing imagery.

Setting	$\lambda_{\mathrm{img}}$	$\lambda_{\mathrm{text}}$	$\lambda_{\mathrm{logit}}$	Avg. HM
Only $\mathcal{L}_{\mathrm{logit}}$	0	0	1.0	83.14
All 1.0 (2-stage)	1.0	1.0	1.0	85.47
All 1.0 (Warm-up)	1.0	1.0	1.0	86.91
Ours (Warm-up)	0.5	0.5	1.0	87.05

Appendix

Table 12: Ablation on loss weights (

\lambda

) and scheduling strategies. We report the average harmonic mean (HM, %) for base-to-novel generalization.

Figure 9: Qualitative analysis of the RS-Flag mechanism using the “stadium” class. Candidates are ranked by their initial similarity ( $s_{k,j}$ ). The visualization compares the baseline softmax weights with our calibrated weights. Dashed circles highlight candidates (e.g., indices 1–3, 8–9, 11–12, 14, 26) that lack explicit remote sensing context and are consequently down-weighted to emphasize true RS-aware descriptions.

Method	Source	Target
Method	PatternNet	RESISC-45	UCMerced	AID
GeoRSCLIP (ZS)	60.17	71.89	79.72	73.72
APPLeNet	86.63	73.25	80.18	76.40
AVION (Ours)	92.09	76.13	82.05	77.99

Appendix

Table 13: Cross-dataset evaluation (%). Models are trained on PatternNet (16-shot) and evaluated zero-shot on targets. All methods utilize the GeoRSCLIP (ViT-B/32) backbone. ZS: Zero-shot.

Offline LLM	Few-Shot		Base-to-Novel
Offline LLM	4-shot	16-shot	Base	Novel	HM
GPT-5	88.45	92.85	95.15	78.45	86.00
Llama-3.1-70B-Inst.	87.12	93.65	95.72	79.55	86.89
Gemini 2.5 Flash (Ours)	88.31	93.69	95.64	79.94	87.05

Appendix

Table 14: Ablation on offline LLM choices. We report average few-shot accuracy and base-to-novel generalization (Base, Novel, HM) in % across six RS datasets. The framework remains stable regardless of the LLM generator used.

Method	Base	Novel	HM
MMRL	77.90	71.30	74.45
AVION (Teacher: ViT-L/14)	77.27	74.02	75.63
AVION (Teacher: ViT-H/14)	79.15	74.79	76.94

Appendix

Table 15: Base-to-novel generalization (%) on the ImageNet benchmark. The student model is CLIP (ViT-B/16). AVION demonstrates competitive transferability on general-domain images without the RS-Flag constraint.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

Jul 17, 2026

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is sufficient even without architectural novelty.

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2

INSAIT, Sofia University "St. Kliment Ohridski"

Remote Sensing Image Understanding Remote Sensing VQA Remote Sensing VLMs Large Vision-Language Models

Ask PDF

Mar 12, 2026

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce. Prevailing pseudo-labeling pipelines address this gap by distilling knowledge from large frontier models, but this dependence on large teachers is costly, limits scalability, and caps achievable performance at the ceiling of the teacher. We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency. Our key insight is that a capable base VLM can serve as its own annotation engine: by pairing aerial images with rendered OpenStreetMap (OSM) tiles, we leverage optical character recognition and chart comprehension capabilities of the model to generate captions enriched by OSM's vast auxiliary metadata. The model is then fine-tuned on the resulting corpus with satellite imagery alone, yielding OSMDA-VLM, a domain-adapted VLM that requires no manual labeling and no stronger external VLM teacher. We conduct exhaustive evaluations spanning six zero-shot and five in-distribution benchmarks across vision-language tasks, where OSMDA leads to substantial improvement. We further compare against nine competitive baselines, demonstrating that our method achieves superior overall performance, while being substantially cheaper to train than teacher-dependent alternatives. These results suggest that, given a strong foundation model, alignment with crowd-sourced geographic data is a practical and scalable path towards remote sensing domain adaptation. Dataset and model weights will be made publicly available upon acceptance.

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +3

INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”

Remote Sensing VLMs VLM Adaptation Vision-Language Models Unsupervised Domain Adaptation

Ask PDF

May 18, 2026

SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning

Remote sensing vision-language models commonly rely on pretrained visual encoders to convert images into semantic features before language-model reasoning. While effective for scene-level understanding, this pipeline may prematurely compress local visual evidence, making fine-grained spatial reasoning vulnerable to language priors, especially in ultra-high-resolution remote sensing imagery. We present SkyNative, a native multimodal framework for remote sensing that adopts an encoder-free architecture, removing the pretrained visual backbone to directly represent images as raw patch tokens in the language-model token space. To reconcile low-level visual patches with textual tokens, SkyNative introduces a modality-aware decoupling mechanism that uses modality-specific parameters within a unified autoregressive backbone. We further introduce a visual reliance benchmark that diagnoses whether models ground their answers in image evidence through progressive visual degradation and misleading textual prompts. Across standard remote sensing understanding tasks and large-format spatial reasoning evaluations, SkyNative shows stronger image-grounded perception and improved robustness against prompt-induced language priors. These results suggest that native patch-level multimodal modeling is a promising direction for reliable remote sensing vision-language reasoning.

Xiao Yang, Ronghao Fu, Zhiwen Lin +10

College of Computer Science and Technology, Jilin University, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education

VLM Evaluation VLM Robustness Visual Reasoning Vision-Language Grounding Multimodal Learning Remote Sensing VLMs

Ask PDF

Abstract

VLM Adaptation

Explore similar work

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning