Reuse or Relearn? A Spectral View of Earth Observation Foundation Models

Mehmet Ozgur Turkoglu; Valerio Marsocci; Dominik J. Mühlematter; Dominik Senti; Konrad Schindler; Helge Aasen

cs.LGSep 26, 2026

Reuse or Relearn? A Spectral View of Earth Observation Foundation Models

Authors: Mehmet Ozgur Turkoglu, Valerio Marsocci, Dominik J. Mühlematter, Dominik Senti, Konrad Schindler, Helge Aasen

Organizations: Agroscope, EO of Agroecosystems · ESA, Φ-lab · ETH Zurich

Abstract

Foundation models are rarely used as generic, frozen feature extractors; instead, they are fine-tuned for the target downstream application. This practice is particularly prevalent in Earth observation (EO), and it raises a question that downstream accuracy alone cannot answer: does fine-tuning reuse the pretrained representation, or does it relearn a new one? We study this with spectral diagnostics that compare a model before and after adaptation, quantifying how well its dominant singular subspaces are preserved, how broadly the weight update is distributed, and how large it is. Using natural image models such as CLIP and DINO as a reference, we find that, under the evaluated fine-tuning settings, EO models undergo far larger, higher-rank updates and retain much less of their pretrained structure, so their downstream performance is often obtained with substantial changes to the pretrained weight structure. The diagnostics further provide insight into how cheaply a model can be adapted: where the pretrained subspaces are preserved, adapting a small fraction of the parameters can match full fine-tuning, and where they are not, it can fall behind. More broadly, foundation models, and EO foundation models in particular, should be assessed not only by benchmark accuracy, but also by how reusable their pretrained representation is.

Figures & tables

Figure 1: Left: One attention output projection $W\in\mathbb{R}^{n\times n}$ per panel, pretrained and after full fine-tuning on MADOS (Prithvi-2 block 12, $n=768$ and CROMA block 7, $n=1024$ ). Points are 260 sampled columns of $W$ , arrows are the two leading left singular vectors, scaled by their singular values ( $U_{2}\Sigma_{2}$ ). Both are projected onto the 3-D subspace that best fits the four arrows. SSI k measures how well the pretrained and fine-tuned top- $k$ subspaces align, independently of singular-value magnitude. Right: DINOv3 exhibits higher SSI than DINOv2 on both datasets, suggesting that stronger pretraining is associated with greater preservation of pretrained subspaces during fine-tuning. Both models exhibit higher ERA on PASTIS than on MADOS, suggesting that adaptation to the more complex task involves a broader range of update directions.

		Full-FT spectral diagnostics						mIoU (%) $\uparrow$
Dataset	Backbone	$\mathrm{SSI}_{32}$		ERA		$\rho$		Full	LoRA	SVF
		Attn	MLP	Attn	MLP	Attn	MLP
MADOS	CROMA	.545	.695	.466	.542	.452	.462	63.6	63.9	64.4
	DOFA	.673	.762	.513	.674	.345	.332	58.5	55.9	58.9
	Prithvi-2	.786	.787	.558	.718	.292	.261	50.9	62.1	61.0
	TerraMind	.890	.909	.459	.611	.195	.187	65.2	63.9	62.3

Table 2: Full-FT spectral diagnostics and test mIoU on three EO benchmarks. Attn and MLP denote layer groups, and

\rho

denotes relative weight change. Dataset means average the five backbones; the overall mean weights all 15 model–dataset pairs equally. Bold performance values identify the best adaptation method within each backbone, and dataset mean.

		Full-FT spectral diagnostics						mIoU (%) $\uparrow$
Dataset	Backbone	$\mathrm{SSI}_{32}$		ERA		$\rho$		Full	LoRA	SVF
		Attn	MLP	Attn	MLP	Attn	MLP
BraDD	CROMA	.287	.357	.672	.743	.914	1.090	54.1	55.8	57.4
	DOFA	.637	.715	.609	.737	.422	.427	48.1	47.3	47.7
	TerraMind	.449	.457	.721	.838	.638	.645	59.1	58.7	60.3
	SSL4EO-MAE	.745	.694	.685	.796	.349	.400	52.3	51.8	53.7

Table 3: Full-FT spectral diagnostics and test mIoU on BraDD. Attn and MLP denote layer groups;

\rho

is relative weight change. The mean row weights all four backbones equally. Bold marks the best performance per row.

		Full-FT spectral diagnostics						Accuracy (%) $\uparrow$
Dataset	Backbone	$\mathrm{SSI}_{32}$		ERA		$\rho$		Full	LoRA	SVF
		Attn	MLP	Attn	MLP	Attn	MLP
TimeMatch	CROMA	.116	.074	.589	.628	1.330	1.445	85.6	85.4	73.7
	DOFA	.188	.182	.587	.701	.851	.883	86.2	81.5	70.0
	Prithvi-2	.327	.175	.597	.707	.654	.726	86.6	84.7	72.5
	SSL4EO-MAE	.437	.310	.696	.712	.639	.834	84.3	85.5	73.0

Table 4: Full-FT spectral diagnostics and test accuracy on the canonical single-target TimeMatch experiment, with FR/31TCJ/2017 held out. Attn and MLP denote layer groups;

\rho

is relative weight change. The mean row weights all seven backbones equally. Bold marks best performance per row.

		Full-FT spectral diagnostics						Performance (%) $\uparrow$
Dataset	Backbone	$\mathrm{SSI}_{32}$		ERA		$\rho$		Full	LoRA	SVF
		Attn	MLP	Attn	MLP	Attn	MLP
Flowers-102	CLIP	.982	.992	.160	.246	.051	.051	91.2	90.7	93.7
	DINOv2	.994	.990	.160	.286	.045	.046	97.0	98.7	98.1
	DINOv3	.998	.999	.069	.102	.021	.020	98.1	97.4	94.7
CIFAR-10	CLIP	.976	.984	.237	.337	.065	.061	98.1	97.2	97.6

Table 5: Full-FT spectral diagnostics and downstream accuracy (mIoU for VOC). Attn and MLP denote layer groups;

\rho

is relative weight change. Spectral means weight datasets and backbones equally. Bold marks the best performance per row.

Figure 3: (a) Depth-resolved $\mathrm{SSI}_{32}$ under full-FT for matched-modality transfer (mean $\pm$ 1 std across encoders and benchmarks) and CROMA cross-modal transfer on BraDD. Annotations compare cross-modal and modality-matched mIoU. (b) Full-FT mIoU versus relative weight change $\rho$ on MADOS and PASTIS. Dashed lines are least-squares fits excluding TerraMind. (c) Test mIoU gains over architecture-matched random initialization, in pp, for full-FT, SVF, and LoRA: five EO backbones on MADOS/PASTIS and three CV backbones on VOC, under the evaluated training budgets.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

				SSI		ERA		Rel. Change
Downstream Model	Base Model	Res.	Task	Attn	MLP	Attn	MLP	Attn	MLP
Aurora 0.25° Fine-Tuned	Aurora 0.25° Pretrained	0.25°	HRES T0 Forecast	0.992	0.997	0.150	0.141	0.055	0.060
Aurora 0.1° Fine-Tuned	Aurora 0.25° Pretrained	0.1°	High-Res Forecast	0.902	0.953	0.528	0.762	0.218	0.235
Aurora 0.25° Wave	Aurora 0.25° Pretrained	0.25°	Wave Prediction	0.918	0.945	0.563	0.779	0.220	0.277
Aurora 0.4° Air Pollution	Aurora 0.25° 12h Pretrained	0.4°	Air Pollution	0.865	0.917	0.683	0.864	0.319	0.356

Appendix

Table 6: Spectral diagnostics for Aurora (1.3B) across downstream Earth-system tasks.

Figure 4: Sensitivity of SSI to the subspace rank

k

on (a) CIFAR-10 and (b) VOC. SSI remains largely stable across the evaluated ranks, showing that high subspace preservation is robust to the choice of

k

for the runs shown. Legends report accuracy and mIoU, respectively.

Figure 5: Sensitivity of SSI to rank

k

on (a) PASTIS and (b) BraDD. Legends report mIoU. Each curve varies

k

with trained weights fixed.

Panel of Figure 6 — Figure 6: Learning-rate sensitivity of DINOv3 under full fine-tuning on (a) Flowers-102 and (b) VOC. Spectral diagnostics use the left axis and test performance the right axis; the dashed line marks the default learning rate. At low learning rates, the encoder changes very little and SSI remains close to one despite suboptimal performance. Performance peaks at $10^{-4}$ to $2\times 10^{-4}$ , where SSI 32 remains above 0.99. Larger learning rates degrade performance and broaden the update (higher ERA), but relative weight change stays small ( $\rho\leq 0.15$ ) and SSI 32 stays above 0.93. Overall, subspace preservation is stable across the sweep, unlike for the EO encoders (Fig. 7 ).

Panel of Figure 7 — Figure 7: Learning rate sensitivity of (a) AnySat and (b) Presto under full fine-tuning on TimeMatch. Spectral diagnostics use the left axis and test accuracy the right axis; the dashed line marks the default learning rate used in Tab. 4 . Unlike DINOv3 (Fig. 6 ), neither encoder reaches its best accuracy while preserving its dominant subspaces. AnySat peaks at the default with SSI 32 of 0.24; lower learning rates raise SSI at a cost in accuracy, and learning rates of $3\times 10^{-4}$ and above diverge (24.3%, below the plotted range). Presto improves monotonically with the learning rate while SSI decreases and $\rho$ increases, so its best accuracy coincides with the lowest SSI in the sweep. Overall, the EO diagnostics are far less stable across learning rates than their CV counterparts.

Figure 8: Extended full fine-tuning of DINOv3 on Flowers-102 and PASCAL VOC, using $10\times$ more optimization steps and twice the default initial learning rate. Curves show validation accuracy or mIoU as fractions, alongside SSI, ERA, and relative weight change $\rho$ , aggregated over the analyzed encoder matrices. The dashed line marks epoch 10 within each extended run. SSI remains high and relative weight change remains small, while ERA increases more noticeably on VOC but still stays relatively low.

Figure 9: Decoder comparison across PASTIS, BraDD, MADOS, and FLAIR-2. Blue bars show the mean absolute change in $\mathrm{SSI}_{32}$ ; red bars show the mean mIoU difference between UPerNet and the linear decoder.

Figure 10: Difference in test mIoU between SVF and full fine-tuning across encoders. Diamonds indicate means over datasets. MMEarth exhibits the largest mean performance deficit.

Figure 11: Depth profile of subspace preservation under full fine-tuning, per benchmark. Grey lines are individual encoders; red is the mean with

\pm 1

std.

Encoder	Pretraining objective	Modality	$\min(m,n)$
CROMA	cross-modal contrastive + MAE	optical, SAR	1024
DOFA	wavelength-conditioned	multi-sensor	768
Prithvi-2	masked autoencoding, multitemporal	optical	768
TerraMind	multimodal generative	multi-sensor	768
SSL4EO-MAE	masked autoencoding	optical, SAR	384

Appendix

Table 7: Main EO encoders, their pretraining objectives, and input modalities. Matrix dimensions are summarized by the median

\min(m,n)

across analyzed attention and MLP weight matrices.

Benchmark	Modality	Input	Bands	$T$	Classes	Train / Val / Test
MADOS	optical S2	$240^{2}$	11	1	15	1,433 / 642 / 728
PASTIS	optical S2	$128^{2}$	10	6	20	folds 1-3 / 4 / 5
FLAIR-2	aerial RGB	$512^{2}$	0 3	1	13	4,049 / 1,022 / 3,022
BraDD	SAR S1	$48^{2}$	0 2	6	0 2	official split
TimeMatch	optical pixel-set	16 px	10	30	0 9	cross-region

Appendix

Table 8: Benchmarks.

T

is the number of time steps after subsampling. FLAIR-2 is at 20 cm ground sampling distance, the rest at 10 m.

Encoder	Backbone	Input	Bands (opt./SAR)	Tapped blocks
Main EO encoders
CROMA	ViT-L	$120$	12 / 2	3, 5, 7, 11
DOFA	ViT-B	$224$	any †	3, 5, 7, 11
Prithvi-2	ViT-B	$224$	0 6 / –	3, 5, 7, 11
TerraMind	ViT-B	$224$	12 / 2	3, 5, 7, 11
SSL4EO-MAE	ViT-S/16	$224$	13 / 2	3, 5, 7, 11

Appendix

Table 9: Encoder configurations. Bands is the number of input channels expected (optical / SAR encoder). Natural-image backbones use their standard classification or segmentation head. † DOFA is wavelength-conditioned and accepts the dataset’s bands.

Encoder	Encoder params	Adapted matrices	SVF trainable	LoRA trainable
CROMA	303.0 M	96	98.3 K (0.03%)	3.15 M (1.04%)
DOFA	111.3 M	50	37.1 K (0.03%)	1.21 M (1.09%)
Prithvi-2	0 86.2 M	48	36.9 K (0.04%)	1.18 M (1.37%)
TerraMind	0 87.3 M	61	46.8 K (0.05%)	1.28 M (1.47%)
SSL4EO-MAE	0 22.6 M	48	18.4 K (0.08%)	0.59 M (2.60%)

Appendix

Table 10: Trainable parameters, excluding the decoder. Percentages are relative to the encoder.

Setting	Learning rate	Applies to
Default	$10^{-4}$	everything not listed below
Full fine-tuning	$2\times 10^{-5}$	CIFAR-10 (all backbones), VOC (CLIP, DINOv2)
LoRA	$5\times 10^{-4}$	VOC (DINOv3)
SVF	$5\times 10^{-3}$	VOC (DINOv3 and CLIP), Flowers-102 (CLIP)

Appendix

Table 11: Learning rates for the natural-image experiments. Appendix B.3 examines this sensitivity.

Explore similar work

SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation
Mar 20, 2026Víctor Barreiro, Johannes Jakubik, Francisco Argüello +1Geospatial Foundation Models Vision Foundation Model Adaptation
Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Spain · Departmento de Electrónica e Computación, Universidade de Santiago de Compostela, Spain · IBM Research Europe, Zurich, Switzerland
Scalable and Trustworthy Earth Observation Foundation Models
Jul 8, 2026Syed Usama Imtiaz, Mitra Nasr Azadani, Nasrin AlamdariGeospatial Foundation Models Vision Foundation Model Adaptation
Department of Civil and Environmental Engineering, Florida State University
SPECTRA: Band-Routed Embedding and Stage-Wise LoRA for Cross-Sensor Fine-Tuning of Geospatial Foundation Models
Date pendingXingyan Li, Jordan A. Caraballo-Vega, Jie Gong +2Remote Sensing Image Segmentation Geospatial Foundation Models
University of Maryland, Baltimore County Baltimore, Maryland, USA · NASA Goddard Space Flight Center Greenbelt, Maryland, USA

Mar 20, 2026, cs.CVJ/K move · Enter open · S save

SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation

Víctor Barreiro, Johannes Jakubik, Francisco Argüello, Dora B. Heras

Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Spain · Departmento de Electrónica e Computación, Universidade de Santiago de Compostela, Spain · IBM Research Europe, Zurich, Switzerland

Geospatial Foundation Models Vision Foundation Model Adaptation Remote Sensing Neural Network Pruning

Open paper AskPDF arXiv

Fine-tuning foundation models for Earth Observation is computationally expensive, with high training time and memory demands for both training and deployment. Parameter-efficient methods reduce training cost but retain full inference complexity, while post-hoc compression optimizes inference only after costly full fine-tuning. We introduce SIMPLER, a pre-fine-tuning architecture selection method that reduces inference and deployment costs by identifying an effective model depth before adaptation. SIMPLER exploits stabilization of representations in deeper layers of pre-trained vision transformers: it computes layer-wise representation similarity on unlabeled task data and applies an automated scoring function to select redundant layers, with no gradients, magnitude heuristics, or hyperparameter tuning required. On Prithvi-EO-2, SIMPLER prunes up to 79% of parameters while retaining 94% of baseline performance, yielding a 2.1x training speedup and 2.6x inference speedup. The method generalizes to TerraMind (a multimodal EO foundation model) and ImageNet-pretrained ViT-MAE, demonstrating applicability across tasks, architectures, and spectral modalities. Code is available at https://gitlab.citius.gal/hpc4rs/simpler.

Abstract

Explore similar work

SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observation

Scalable and Trustworthy Earth Observation Foundation Models

SPECTRA: Band-Routed Embedding and Stage-Wise LoRA for Cross-Sensor Fine-Tuning of Geospatial Foundation Models