AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network

Yu Hu; Jianyang Gu; Hao Liu; Yue Cao; Jozsef Hamari; Zheng Liu; Mohsen Zardadi

cs.CVMar 13, 2026

AVION: Aerial Vision-Language Instruction from Offline Teacher to Prompt-Tuned Network

Authors: Yu Hu, Jianyang Gu, Hao Liu, Yue Cao, Jozsef Hamari, Zheng Liu, Mohsen Zardadi

Organizations: The University of British Columbia, Okanagan, Kelowna, BC, Canada · The Ohio State University, Columbus, OH, USA · TerraSense Analytics, Kelowna, BC, Canada

Abstract

Adapting vision-language models to remote sensing imagery remains challenging due to two key factors: limited semantic coverage in textual representations and insufficient adaptability of visual features. These issues are particularly significant in aerial scenes, which involve various visual appearances and fine-grained object distinctions. We propose AVION, a knowledge distillation framework tailored for remote sensing adaptation of vision-language models. The teacher module constructs semantically rich textual prototypes by collecting descriptions from a large language model and verifying validity using remote sensing image features. The student module integrates lightweight and learnable prompts into both vision and language encoders, guided by the teacher to align embeddings and their cross-modal relationships. Once trained, the student operates independently during inference. Experiments on six optical remote sensing benchmarks show that AVION improves few-shot classification and base-class accuracy without degrading generalization to novel categories. It also enhances mean recall for cross-modal retrieval, with minimal additional trainable parameters.

Figures & tables

Figure 1: The t-SNE visualization of visual embeddings from different datasets. The same class demonstrates large variations across datasets. However, only class names are provided in the datasets, which limits multimodal alignment.

Figure 2: Overview of AVION. Upper-left (Training Student). Learnable prompt tokens are injected into the text and vision encoders. The student outputs embeddings: ${\mathbf{t}}^{S}_{k}$ (student text embedding for class $k$ ) and ${\mathbf{v}}^{S}_{i}$ (visual embedding for image $i$ ). Bottom-left (Offline Teacher). LLM-based Domain Prompting generates multiple class-aware descriptions, which are encoded into $\mathbf{t}^{T}_{k,j}$ . Selective Prototype Aggregation verifies these candidates using teacher visual embeddings ${\mathbf{v}}^{T}_{i}$ and aggregates them into an RS-aware text prototype ${\mathbf{t}}^{T*}_{k}$ . Right (Training Objectives). Tri-aspect alignment: (i) textual alignment $\mathcal{L}_{\text{text}}$ maximizes the cosine similarity between ${\mathbf{t}}^{S}_{k}$ and ${\mathbf{t}}^{T*}_{k}$ ; (ii) visual alignment $\mathcal{L}_{\text{img}}$ aligns ${\mathbf{v}}^{S}_{i}$ with ${\mathbf{v}}^{T}_{i}$ ; (iii) similarity logit alignment $\mathcal{L}_{\text{logit}}$ matches teacher logits $s^{T}_{i,k}=({\mathbf{v}}^{T}_{i})^{\top}{\mathbf{t}}^{T*}_{k}$ and student logits $s^{S}_{i,k}=({\mathbf{v}}^{S}_{i})^{\top}{\mathbf{t}}^{S}_{k}$ via temperature-scaled KL. A standard task loss $\mathcal{L}_{\text{task}}$ (cross-entropy) is applied on the student.

Figure 3: LLM-based Domain Prompting. Given a class $k$ , an RS-aware query asks the LLM to produce aerial-view descriptions. RS-Flag is used to examine whether the description contains RS-related tokens.

Figure 4: Selective Prototype Aggregation. For each class $k$ , teacher image embeddings $\{\mathbf{v}^{T}_{k,i}\}_{i\in\mathcal{B}_{k}}$ are averaged to form the visual prototype ${\widehat{\mathbf{v}}}^{T}_{k}$ . Each LLM-generated description is encoded to a teacher text embedding $\mathbf{t}_{k,j}$ and scored by cosine similarity $s_{k,j}=({\widehat{\mathbf{v}}}^{T}_{k})^{\top}\mathbf{t}^{T}_{k,j}$ . A median/MAD threshold $\pm\zeta_{s}$ removes outlier candidates. The remaining embeddings are combined with softmax-normalized weights (over kept $j$ ) to obtain the class textual prototype ${\mathbf{t}}^{T*}_{k}$ .

Method	1-shot	2-shot	4-shot	8-shot	16-shot
Zero-shot Methods
CLIP [ 27 ]	$63.24$
RemoteCLIP [ 25 ]	$72.92$
GeoRSCLIP [ 41 ]	$\mathbf{72.95}$
PEFT Methods
CoOp [ 43 ]	$69.98$	$78.95$	$84.52$	$87.57$	$90.24$

Table 1: Few-shot classification against state-of-the-art methods. The table presents the average classification accuracy (%) over 6 datasets. The best-performing result for each

K

-shot setting is in bold , and the runner-up is underlined .

Method	Base	Novel	HM
GeoRSCLIP [ 41 ]	$78.10$	$79.75$	$78.81$
CoOp [ 43 ]	$91.17$	$69.52$	$78.88$
CoCoOp [ 42 ]	$91.85$	$70.69$	$80.06$
MMRL [ 13 ]	$93.75$	$73.69$	$82.76$
MaPLe [ 20 ]	$92.87$	$74.66$	$82.90$
PromptKD [ 24 ]	$93.67$	$76.58$	$84.20$

Table 2: Average base-to-novel results (%) across all 6 datasets (ViT-B/32 student). We report Base, Novel, and HM.

Figure 5: t-SNE of base-to-novel visual and text embeddings on RESISC-45. (a) GeoRSCLIP [ 41 ] (b) AVION (ours) (c) APPLeNet [ 28 ] (d) MMRL [ 13 ] . Gray dots represent the visual embedding of base classes . Colored dots represent the visual embedding of novel classes with each color standing for a class. Colored crosses represent the text embedding of novel classes for their class names. A, B, C, and D are four classes.

Testing Dataset	Training Dataset	Method	Image $\to$ Text			Text $\to$ Image			mR
Testing Dataset	Training Dataset	Method	R@1	R@5	R@10	R@1	R@5	R@10	mR
RSITMD [ 37 ]	RSITMD	ML Retrieval [ 1 ]	$19.09$	$40.26$	$54.42$	$17.61$	$49.73$	$60.59$	$41.38$
	RET - 3	CLIP [ 27 ]	$23.67$	$50.00$	$63.27$	$22.61$	$55.27$	$69.87$	$47.63$
	RET - 3 + DET - 10 + SEG - 4	RemoteCLIP [ 25 ]	$27.88$	$50.66$	$65.71$	$22.17$	$56.46$	$67.41$	$49.38$
	RSITMD + RSICD	GeoRSCLIP-FT [ 41 ]	$32.30$	$53.32$	$67.92$	$25.04$	$57.88$	$74.38$	$51.81$
	RSITMD + RSICD	AVION (Ours)	$\mathbf{33.17}$	$\mathbf{54.46}$	$\mathbf{69.18}$	$\mathbf{26.09}$	$\mathbf{59.22}$	$\mathbf{75.41}$	$\mathbf{52.92}$

Table 3: Cross-modal retrieval ( R@K , %) on RSITMD [ 37 ] and RSICD [ 26 ] (ViT-B/32 student; protocol details in Sec. 4.1 ). mR is the macro-average over

\{\text{T}\rightarrow\text{I},\text{I}\rightarrow\text{T}\}\times\{1,5,10\}

\Delta

is relative to GeoRSCLIP. Our method (AVION) is lightweight, updating only prompts/heads; see Appendix F for a full efficiency analysis.

ID	Configuration Description	ID	Configuration Description
B0	Shallow text-only prompt (CoOp-style), $\mathcal{L}_{\mathrm{task}}$ only.	B4	Replace manual prototypes with LLM-generated ones.
B1	+ Deep prompt tuning.	B5	+ Selective prototype aggregation (vision-guided).
B2	+ Representation alignment ( $\mathcal{L}_{\mathrm{img}}$ ).	B6	+ Behavioral alignment ( $\mathcal{L}_{\mathrm{logit}}$ ) in a single-stage schedule.
B3	+ Semantic alignment ( $\mathcal{L}_{\mathrm{text}}$ ) with manual textual prototypes.	B7	Final Model : 30% warm-up (Sec. 4.1 ).

Table 4: Configurations for our incremental ablation study. Each configuration builds upon the previous one and culminates in the full system (B7).

ID	Base	Novel	HM	K-shot
ID	Base	Novel	HM	1	2	4	8	16
B0	$91.17$	$69.52$	$78.88$	$69.98$	$78.95$	$84.52$	$87.57$	$90.24$
B1	$93.52$	$53.51$	$66.71$	$66.95$	$84.68$	$84.61$	$87.34$	$91.25$
B2	$89.74$	$61.15$	$72.74$	$70.21$	$82.99$	$85.31$	$89.65$	$91.51$
B3	$91.21$	$68.63$	$78.34$	$71.98$	$83.72$	$85.90$	$90.43$	$92.18$
B4	$91.03$	$74.08$	$81.67$	$72.15$	$84.71$	$87.43$	$91.89$	$92.14$

Table 5: Main ablation results for classification. We report Base, Novel, and Harmonic Mean (HM) accuracy. Metrics are averaged over six datasets.

ID	Prototype Source	Aggregation	RS-Flag	HM (%)
P0	Manual (single template)	none	–	$79.52$
P1	Manual (multi-templates)	mean	–	$79.91$
P2	LLM (unconstrained)	mean	no	$80.85$
P3	LLM (domain-constrained)	mean	yes	$81.70$
P4	LLM (domain-constrained)	text-only selection	yes	$82.05$
P5	LLM (random subset)	mean	no	$80.31$

Table 6: Teacher text-prototype ablation (P0–P6). All settings use the same objectives, with different prototype sources and aggregation. HM averaged over the same datasets/seeds.

Figure 6: Few-shot classification accuracy of AVION with different CLIP-based backbones in the few-shot setting.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Few-shot classification performance (top-1 accuracy, %) across six remote sensing datasets (AID [ 31 ] , EuroSAT [ 14 ] , RESISC-45 [ 7 ] , UCMerced [ 33 ] , WHU-RS19 [ 32 ] , and PatternNet [ 44 ] ) for different numbers of shots $K\!\in\!\{1,2,4,8,16\}$ .

Dataset	Method	1	2	4	8	16
AID	CoCoOp	74.91	83.23	85.90	90.53	93.28
	CoOp	77.22	82.09	85.67	89.77	92.79
	MMRL	76.30	80.35	87.23	91.28	92.89
	MaPLe	75.24	82.26	86.89	91.13	93.18
	PromptKD	74.85	80.72	86.41	90.67	92.95
	APPLeNet	81.79	83.24	87.11	91.24	93.20

Appendix

Table 7: Few-shot classification (top-1, %) on six remote sensing datasets. Winners are bold ; runners-up are underlined .

AID [ 31 ]	Base	Novel	HM
GeoRSCLIP [ 41 ]	71.96	76.90	74.35
CoOp [ 43 ]	95.53	66.41	78.35
CoCoOp [ 42 ]	94.83	68.03	79.22
MMRL [ 13 ]	95.27	70.40	80.97
MaPLe [ 20 ]	95.10	70.60	81.04
PromptKD [ 24 ]	94.90	70.00	80.57

Appendix

Table 8: Detailed base-to-novel results (%) on six remote sensing datasets (ViT-B/32 student). Baselines: GeoRSCLIP [ 41 ] , CoOp [ 43 ] , CoCoOp [ 42 ] , MMRL [ 13 ] , MaPLe [ 20 ] , PromptKD [ 24 ] , and APPLeNet [ 28 ] . We report Base, Novel, and HM;

\Delta

denotes the difference between Ours and the best-performing baseline for each metric.

Figure 8: Base-to-novel generalization (HM, %) across six remote sensing datasets (ViT-B/32 student). The radar chart plots the harmonic mean between base and novel accuracies for AVION and competing baselines; a larger enclosed area indicates better overall base-to-novel performance.

Symbol	Meaning	Default / Search
$N$	# training images	–
$C$	# classes	–
$D$	image/text embedding dimension	inherited from the VLM
$f_{T},g_{T}$	teacher image/text encoders	frozen
$f_{S},g_{S}$	student image/text encoders	backbones frozen
$\mathbf{v}^{T}_{i}$	$\ell_{2}$ -normalized teacher image feature for $x_{i}$	unit $\ell_{2}$ -norm

Appendix

Table 9: Symbols and default/search ranges used in AVION. In all experiments we fix

K_{p}{=}30

\beta{=}10

\gamma{=}2

\zeta_{s}{=}3.0

Class Name	Simple Template	LLM-Generated Candidates
airport	a photo of an airport	- Overhead airport imagery reveals aprons (with parked aircraft at stands) adjacent to terminals, connected to the airfield via taxilanes and the taxiway system. - An aerial view of an airport showing multiple long, paved runways and taxiways. - Satellite imagery capturing a large airfield complex with terminal buildings and hangars.
industrial area	a photo of an industrial area	- A top-down view of an industrial zone characterized by large factory buildings and smokestacks. - Aerial imagery of a manufacturing plant complex with large warehouses and shipping depots. - A satellite photo showing a dense cluster of large-scale industrial structures and storage tanks.
commercial area	a photo of a commercial area	- An overhead view of a commercial district featuring multi-story office buildings and retail centers. - Nadir-view imagery of a dense downtown business area with high-rise buildings and streets. - An aerial photo of a shopping center or mall surrounded by a large, organized parking lot.
forest	a photo of a forest	- A vast expanse of dense, green forest canopy as seen from a satellite. - An aerial view looking straight down at a dense woodland of coniferous or deciduous trees. - A top-down remote sensing image of a large, contiguous area covered by trees.

Appendix

Table 10: Examples of LLM-generated candidate descriptions from our LLM-based domain prompting module, compared to standard simple templates. The generated candidates provide richer semantic and visual cues specific to remote sensing imagery.

Setting	$\lambda_{\mathrm{img}}$	$\lambda_{\mathrm{text}}$	$\lambda_{\mathrm{logit}}$	Avg. HM
Only $\mathcal{L}_{\mathrm{logit}}$	0	0	1.0	83.14
All 1.0 (2-stage)	1.0	1.0	1.0	85.47
All 1.0 (Warm-up)	1.0	1.0	1.0	86.91
Ours (Warm-up)	0.5	0.5	1.0	87.05

Appendix

Table 12: Ablation on loss weights (

\lambda

) and scheduling strategies. We report the average harmonic mean (HM, %) for base-to-novel generalization.

Figure 9: Qualitative analysis of the RS-Flag mechanism using the “stadium” class. Candidates are ranked by their initial similarity ( $s_{k,j}$ ). The visualization compares the baseline softmax weights with our calibrated weights. Dashed circles highlight candidates (e.g., indices 1–3, 8–9, 11–12, 14, 26) that lack explicit remote sensing context and are consequently down-weighted to emphasize true RS-aware descriptions.

Method	Source	Target
Method	PatternNet	RESISC-45	UCMerced	AID
GeoRSCLIP (ZS)	60.17	71.89	79.72	73.72
APPLeNet	86.63	73.25	80.18	76.40
AVION (Ours)	92.09	76.13	82.05	77.99

Appendix

Table 13: Cross-dataset evaluation (%). Models are trained on PatternNet (16-shot) and evaluated zero-shot on targets. All methods utilize the GeoRSCLIP (ViT-B/32) backbone. ZS: Zero-shot.

Offline LLM	Few-Shot		Base-to-Novel
Offline LLM	4-shot	16-shot	Base	Novel	HM
GPT-5	88.45	92.85	95.15	78.45	86.00
Llama-3.1-70B-Inst.	87.12	93.65	95.72	79.55	86.89
Gemini 2.5 Flash (Ours)	88.31	93.69	95.64	79.94	87.05

Appendix

Table 14: Ablation on offline LLM choices. We report average few-shot accuracy and base-to-novel generalization (Base, Novel, HM) in % across six RS datasets. The framework remains stable regardless of the LLM generator used.

Method	Base	Novel	HM
MMRL	77.90	71.30	74.45
AVION (Teacher: ViT-L/14)	77.27	74.02	75.63
AVION (Teacher: ViT-H/14)	79.15	74.79	76.94

Appendix

Table 15: Base-to-novel generalization (%) on the ImageNet benchmark. The student model is CLIP (ViT-B/16). AVION demonstrates competitive transferability on general-domain images without the RS-Flag constraint.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 538 papers

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
Jul 17, 2026Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +2Remote Sensing Image Understanding Remote Sensing VQA
INSAIT, Sofia University "St. Kliment Ohridski"
OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs
Mar 12, 2026Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi +3Remote Sensing VLMs VLM Adaptation
INSAIT, Sofia University "St. Kliment Ohridski" · INSAIT, Sofia University “St. Kliment Ohridski”
SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning
May 18, 2026Xiao Yang, Ronghao Fu, Zhiwen Lin +10VLM Evaluation VLM Robustness
College of Computer Science and Technology, Jilin University, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education

Jul 17, 2026J/K move · Enter open · S save

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Luc Van Gool+1

INSAIT, Sofia University "St. Kliment Ohridski"

Remote Sensing Image Understanding Remote Sensing VQA Remote Sensing VLMs Large Vision-Language Models

Open paper AskPDF arXiv

Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is sufficient even without architectural novelty.

Abstract

VLM Adaptation

Explore similar work

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning