Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery

Minh Kha Do; Wei Xiang; Kang Han; Di Wu; Khoa Phan; Yi-Ping Phoebe Chen; Gaowen Liu; Ramana Rao Kompella

cs.CVFeb 26, 2026

Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery

Authors: Minh Kha Do, Wei Xiang, Kang Han, Di Wu, Khoa Phan, Yi-Ping Phoebe Chen, Gaowen Liu, Ramana Rao Kompella

Organizations: La Trobe University, Melbourne, VIC 3086, Australia · Cisco Research, San Jose, CA, USA

Abstract

Vision-language foundation models (VLFMs) promise zero-shot and retrieval understanding for Earth observation. While operational satellite systems often lack full multi-spectral coverage, making RGB-only inference highly desirable for scalable deployment, the adoption of VLFMs for satellite imagery remains hindered by two factors: (1) multi-spectral inputs are informative but difficult to exploit consistently due to band redundancy and misalignment; and (2) CLIP-style text encoders limit semantic expressiveness and weaken fine-grained alignment. We present SATtxt, a spectrum-aware VLFM that operates with RGB inputs only at inference while retaining spectral cues learned during training. Our framework comprises two stages. First, Spectral Representation Distillation transfers spectral priors from a frozen multi-spectral teacher to an RGB student via a lightweight projector. Second, Spectrally Grounded Alignment with Instruction-Augmented LLMs bridges the distilled visual space and an expressive LLM embedding space. Across EuroSAT, BigEarthNet, and ForestNet, SATtxt improves zero-shot classification on average by 4.2%, retrieval by 5.9%, and linear probing by 2.7% over baselines, showing an efficient path toward spectrum-aware vision-language learning for Earth observation. Project page: https://ikhado.github.io/sattxt/

Figures & tables

Figure 1: SATtxt outperforms existing VLFMs across three satellite benchmarks while requiring only RGB inputs. By contrast, multi-spectral VLFMs ( e.g . , DOFA-CLIP [ 44 ] ) exhibit inconsistent gains from multi-spectral (MS) inputs.

Figure 2: Patch-wise similarity between images and label prompts for models pre-trained on the same dataset. FT- DINOv3txt denotes DINOv3txt further pre-trained on this dataset, consistent with Llama3-MS-CLIP and SATtxt . Using an instruction-augmented LLM as the text encoder, our method produces sharper object-level focus ( e.g . , river) and clearer contextual relations ( e.g . , residential), supporting strong zero-shot predictions.

Figure 3: Pre-training strategies for VLFMs on satellite imagery. (a) CLIP-style continued pre-training ( e.g . , RemoteCLIP), (b) LiT: Locked-image Tuning ( e.g . , DINOv3txt ), (c) LTT: Locked-text Tuning ( e.g . , DOFA-CLIP), and (d) SATtxt : bridges two strong, frozen encoders via instruction-augmented text from an LLM, improving training efficiency as well as zero-shot and linear-probe performance.

Figure 4: Two-stage pre-training pipeline for SATtxt . Stage 1 ( SRD ) - dashed lines: a vision projector is trained to reconstruct multi-spectral representations from an RGB encoder by distilling a frozen MS teacher, transferring spectral knowledge so MS inputs are unnecessary in Stage 2 and at inference. Stage 2 ( SGI-LLM ) - solid lines: with vision and text encoders frozen, distilled vision features are aligned with LLM-based text embeddings using instruction-augmented prompts, enhancing cross-modal representations while preserving pretrained capabilities.

Training Approach	Model	Input	Locked?		Zero-shot Classification			Retrieval
	Model	Input	Text	Vision	EuroSAT	BigEarthNet	ForestNet	EuroSAT	BigEarthNet	ForestNet
Generative	GeoChat [ 18 ]	RGB	LoRA	✗	$35.00$	$24.16$	$6.31$	-	-	-
	EarthDial [ 37 ]	MS	LoRA	✗	$63.28$	$52.60$	$8.22$	-	-	-
Contrastive	CLIP [ 32 ]	RGB	✗	✗	$46.90$	$54.85$	$8.30$	$56.92$	$28.57$	$11.78$
	RemoteCLIP [ 22 ]	RGB	✗	✗	$34.02$	$52.28$	$8.46$	$34.34$	$26.62$	$10.35$
	GeoRSCLIP [ 50 ]	RGB	✗	✗	$52.92$	$58.80$	$8.33$	$51.36$	$32.80$	$15.84$

Table 1: Zero-shot classification and text-to-image retrieval on three satellite benchmarks. The “Locked? ” column indicates whether the text/vision encoders are frozen or adapter-tuned ( ✓ /LoRA) vs. fully unfrozen (✗). “-” denotes an unsupported task or an inapplicable dataset.

Figure 5: Patch-wise image-text similarity maps. For each image, we compute cosine similarity between patch-level vision embeddings and the text embedding produced from the prompt “a satellite image of { class } ” . Compared to baselines, SATtxt yields sharper and more contiguous responses that trace class-consistent structures ( e.g . , linear rivers and highways) and reduce spurious activations in the background.

Figure 6: UMAP ( $n_{\text{neighbors}}=50$ , $\text{minDist}=0$ , cosine distance) visualization of feature embeddings for (a) Llama3-MS-CLIP with MS inputs and (b) our SATtxt with RGB inputs. Points are colored by class. SATtxt yields more compact intra-class clusters and clearer inter-class separation than Llama3-MS-CLIP.

Training Approach	Model	Input	EuroSAT	BigEarthNet		ForestNet
Training Approach	Model	Input		10%	100%
MIM	SpectralGPT [ 13 ]	MS	$92.04$	$74.85$	$79.16$	-
MIM	Terramind [ 16 ]	MS	$96.13$	$75.84$	84.68	$48.64$
Contrastive	CLIP [ 32 ]	RGB	$92.00$	$64.93$	$75.56$	$41.99$
	RemoteCLIP [ 22 ]	RGB	$96.19$	$64.06$	$75.58$	$42.09$
	GeoRSCLIP [ 50 ]	RGB	$95.76$	$77.45$	$81.19$	$46.53$

Table 2: Linear-probe on EuroSAT, BigEarthNet (using 10% and 100% of training samples), and ForestNet. “-” denotes an inapplicable dataset.

Model	EuroSAT		BigEarthNet		ForestNet
Model	CLS	Retrieval	CLS	Retrieval	CLS	Retrieval
Baseline (FT- DINOv3txt )	58.6	70.6	58.1	50.9	16.7	16.3
$\text{\lx@glossaries@gls@link{main}{method}{{{}}SATtxt} }_{\text{SRD + CLIPtext}}$	65.4	74.9	58.3	51.3	17.2	19.7
$\text{\lx@glossaries@gls@link{main}{method}{{{}}SATtxt} }_{\text{SRD + Mistral-7B}}$	68.2	76.2	59.1	52.3	17.5	20.1
$\text{\lx@glossaries@gls@link{main}{method}{{{}}SATtxt} }_{\text{SRD + Llama-3.1-8B}}$	70.1	76.7	59.9	52.6	17.4	22.2
$\text{\lx@glossaries@gls@link{main}{method}{{{}}SATtxt} }_{\text{Llama-3.1-8B}}$ + Inst.	65.3	74.7	58.3	50.2	16.9	22.0

Table 3: Ablation of model components on three benchmarks. We incrementally add (i) SRD , (ii) frozen LLM text encoders, and (iii) instruction-augmented prompting. Every component yields a consistent gain, and the full model ( SRD + LLM + instruction prompts) achieves the best results on all three benchmarks.

Figure 7: Ablation on text-side pooling types. Across the datasets and metrics considered, mean pooling yields the most stable and highest performance.

		EuroSAT		BigEarthNet		ForestNet
Model	Input	CLS	Retrieval	CLS	Retrieval	CLS	Retrieval
Llama3-MS-CLIP	MS	67.86	75.26	59.63	52.42	-	-
DOFA-CLIP	MS	59.04	71.54	56.58	44.52	17.02	19.55
$\text{SATtxt}_{\text{SatMAE}}$	RGB	71.65	78.41	59.41	53.20	17.15	20.14
$\text{SATtxt}_{\text{SpectralGPT}}$	RGB	73.40	78.97	60.18	53.10	17.61	22.59

Table 4: Ablation on MS teacher choice. SATtxt exhibits limited sensitivity to the choice of MS teacher.

Training Approach	Model	Input	Zero-shot Classification
	Model	Input	EuroSAT	BigEarthNet	ForestNet
Generative	GeoChat	RGB	$30.82$	$14.62$	$6.15$
	EarthDial	MS	$57.26$	$31.80$	$8.11$
Contrastive	CLIP	RGB	$40.35$	$54.85$	$8.30$
	RemoteCLIP	RGB	$27.34$	$52.28$	$8.50$
	GeoRSCLIP	RGB	$47.04$	$58.80$	$8.27$

Table 5: Zero-shot classification (F1

\uparrow

) on three satellite benchmarks. “-” denotes an unsupported task or an inapplicable dataset.

Training Approach	Model	Input	EuroSAT	BigEarthNet		ForestNet
Training Approach	Model	Input		10%	100%
MIM	SpectralGPT	MS	$95.96$	$60.70$	$69.93$	-
MIM	Terramind	MS	$91.65$	$65.49$	74.08	$37.30$
Contrastive	CLIP	RGB	$91.83$	$45.32$	$64.12$	$24.02$
	RemoteCLIP	RGB	$95.02$	$45.40$	$64.24$	$18.35$
	GeoRSCLIP	RGB	$95.76$	$60.71$	$70.65$	$34.74$

Table 6: Linear-probe (F1

\uparrow

) on EuroSAT, BigEarthNet (10% and 100%), and ForestNet. “-” denotes an inapplicable dataset.

Figure 8: PCA-based feature map visualizations and patch-wise image-text similarity maps with labels.

Model	Backbone
CLIP	ViT-L-14
RemoteCLIP	ViT-L-14
GeoRSCLIP	ViT-B-32
SkyCLIP	ViT-L-14
Llama3-MS-CLIP	ViT-B-16
DOFA-CLIP	ViT-L-14

Table 7: Model configurations used in our experiments.

Details	EuroSAT	BigEarthNet-10%	BigEarthNet-100%	ForestNet
# Training Samples	16,200	25,000	269,695	6,464
# Validation Samples	5,400	10,000	123,723	989
# Test Samples	5,400	125,866	125,866	993
# Spectral Bands	13	12	12	5
# Classes	10	19	19	12

Table 8: Downstream datasets used in this study.

Figure 9: Stage 1 ( SRD ) configuration details.

Figure 10: Stage 2 ( SGI-LLM ) configuration details.

Panel of Table 9 — Table 9: The list of instructions for vision-language alignment.

Remote Sensing VLMs

See all 41 papers

Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting
Jun 15, 2026Eirini Baltzi, Dionysis Christopoulos, Sotiris Spanos +2Remote Sensing Image Understanding Zero-Shot Image Classification
Remote Sensing Lab, National Technical University of Athens Athens, Greece · Department of Engineering and Sciences, Universitas Mercatorum Rome, Italy
AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models
Aug 16, 2026Md Aminur Hossain, Omkumar Vaghasiya, Rajeev Ranjan Dwivedi +2Remote Sensing VLMs Vision-Language Alignment
Space Applications Centre, ISRO, Ahmedabad, India · Indian Institute of Science Education and Research (IISER) Bhopal, India · CSRE, Indian Institute of Technology Bombay, India
BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation
Mar 31, 2026Johann-Ludwig Herzog, Mathis Jürgen Adler, Leonard Hackel +5Remote Sensing Remote Sensing Benchmarks
BIFOLD and Technische Universität Berlin · University of Trento · National Technical University and National Observatory of Athens

Jun 15, 2026J/K move · Enter open · S save

Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting

Eirini Baltzi, Dionysis Christopoulos, Sotiris Spanos, Valsamis Ntouskos+1

Remote Sensing Lab, National Technical University of Athens Athens, Greece · Department of Engineering and Sciences, Universitas Mercatorum Rome, Italy

Remote Sensing Image Understanding Zero-Shot Image Classification Remote Sensing Benchmarks Remote Sensing VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) have sparked growing interest in zero-shot Earth Observation (EO) downstream tasks, with further gains enabled by remote-sensing-adapted models. We examine this setting across 17 VLM variants and 12 remote sensing (RS) datasets under Meta-Prompting for Visual Recognition (MPVR), and show that zero-shot performance remains highly sensitive to textual design choices, from the meta-prompts used to guide the LLM in generating class descriptions to the descriptions themselves. We explore why semantically rich LLM-generated class descriptions do not translate into consistent gains over simple domain-adapted CLIP-style descriptions. While LLM descriptions are more semantically expressive, they can also introduce noise in the text embedding space, reducing robustness in downstream tasks. We support this observation through a text log-likelihood analysis in the whitened CLIP feature space, comparing LLM-generated and template-based descriptions. Building on this finding, we study query embedding calibration and show that lightweight calibration of the query space consistently yields strong improvements in zero-shot classification and retrieval. Overall, our results provide practical insight into the trade-off between semantic richness and robustness, and identify embedding calibration as a simple and effective tool for improving zero-shot remote sensing performance.

Abstract

Remote Sensing VLMs

Explore similar work

Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting

AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models

BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation