Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

Arthur Ouaknine; David Rolnick

cs.CVOct 20, 2026

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

Authors: Arthur Ouaknine, David Rolnick

Organizations: McGill University · Mila – Quebec AI Institute

Abstract

Geospatial foundation models (GFMs) differ widely in pretraining objective, modality and resolution, and none leads across benchmarks. We ask whether their representations are complementary, and whether a single encoder can inherit that complementarity. We introduce Pangaea-Latents, an extension of the Pangaea benchmark providing frozen embeddings of $5$ GFMs over its $11$ datasets, along with reproduced baselines, and find GFM embeddings to be neither redundant nor disjoint. We then propose GeoMixture, a preliminary framework to align a ViT to several GFMs per dataset through a multi-representation alignment (mREPA) objective. Our encoder ties the best GFM in average rank across nine Pangaea datasets, trading peak performance for cross-task consistency.

Figures & tables

Panel of Figure 1 — Figure 1: Pangaea-Latents (a) (Sec. 3.1 ) is an extension of Pangaea with frozen embeddings of GFMs. GeoMixture (Sec. 3.3 ) is a framework aligning a ViT to GFMs through a multi-representation alignment (mREPA) objective in a single latent (b), or in a Mixture of Experts (c).

Panel of Figure 2 — Figure 2: Pairwise similarity (Eqs. 4 , 5 , 1 ) and complementarity (Eq. 6 ) between the $5$ GFM embeddings, averaged over the $11$ datasets of Pangaea-Latents ; per-dataset scores in App. F .

model	AI4Farms	CropMap	DynEarthNet	HLSBurns	MADOS	PASTIS	Sen1Floods11	SN7	xView2	Avg mIoU	Avg rank
UNet (baseline)	$\mathbf{45.41{\scriptstyle\,\pm\,0.71}}$	$45.95{\scriptstyle\,\pm\,2.11}$	$33.32{\scriptstyle\,\pm\,1.96}$	$\mathbf{85.64{\scriptstyle\,\pm\,0.61}}$	$57.66{\scriptstyle\,\pm\,3.00}$	$28.56{\scriptstyle\,\pm\,0.37}$	$\mathbf{90.88{\scriptstyle\,\pm\,0.87}}$	$\underline{62.43{\scriptstyle\,\pm\,0.16}}$	$42.21{\scriptstyle\,\pm\,2.32}$	54.67	4.33
CROMA	$\underline{27.59{\scriptstyle\,\pm\,0.27}}$	$\mathbf{58.41{\scriptstyle\,\pm\,4.58}}$	$30.80{\scriptstyle\,\pm\,1.07}$	$82.59{\scriptstyle\,\pm\,0.24}$	$\underline{65.63{\scriptstyle\,\pm\,1.58}}$	$38.49{\scriptstyle\,\pm\,0.69}$	$90.06{\scriptstyle\,\pm\,0.24}$	$59.22{\scriptstyle\,\pm\,0.28}$	$40.52{\scriptstyle\,\pm\,0.20}$	54.81	5.22
DOFA	$27.11{\scriptstyle\,\pm\,0.37}$	$47.53{\scriptstyle\,\pm\,5.02}$	$\underline{38.99{\scriptstyle\,\pm\,1.11}}$	$80.86{\scriptstyle\,\pm\,0.45}$	$60.56{\scriptstyle\,\pm\,2.64}$	$29.53{\scriptstyle\,\pm\,0.60}$	$88.22{\scriptstyle\,\pm\,0.20}$	$61.01{\scriptstyle\,\pm\,0.13}$	$59.15{\scriptstyle\,\pm\,0.10}$	54.77	5.33
Prithvi	$26.87{\scriptstyle\,\pm\,0.50}$	$44.91{\scriptstyle\,\pm\,2.43}$	$21.64{\scriptstyle\,\pm\,0.82}$	$83.08{\scriptstyle\,\pm\,0.36}$	$55.02{\scriptstyle\,\pm\,1.48}$	$33.93{\scriptstyle\,\pm\,0.45}$	$90.20{\scriptstyle\,\pm\,0.44}$	$56.56{\scriptstyle\,\pm\,0.28}$	$49.17{\scriptstyle\,\pm\,0.24}$	51.26	7.33
Scale-MAE	$22.52{\scriptstyle\,\pm\,1.87}$	$25.18{\scriptstyle\,\pm\,1.87}$	$35.97{\scriptstyle\,\pm\,1.75}$	$76.56{\scriptstyle\,\pm\,0.07}$	$55.60{\scriptstyle\,\pm\,2.25}$	$24.61{\scriptstyle\,\pm\,0.55}$	$74.85{\scriptstyle\,\pm\,0.09}$	$\mathbf{62.58{\scriptstyle\,\pm\,0.50}}$	$\mathbf{60.05{\scriptstyle\,\pm\,0.16}}$	48.66	6.67
TerraMind-L	$26.98{\scriptstyle\,\pm\,0.54}$	$\underline{54.67{\scriptstyle\,\pm\,2.82}}$	$35.22{\scriptstyle\,\pm\,1.42}$	$83.41{\scriptstyle\,\pm\,0.15}$	$\mathbf{70.99{\scriptstyle\,\pm\,2.31}}$	$\mathbf{42.54{\scriptstyle\,\pm\,0.44}}$	$\underline{90.78{\scriptstyle\,\pm\,0.13}}$	$60.22{\scriptstyle\,\pm\,0.70}$	$48.67{\scriptstyle\,\pm\,0.37}$	57.05	3.89

Table 1: PANGAEA frozen-encoder evaluation : mIoU as mean

\pm

std over

3

seeds, on nine Pangaea datasets. Our methods are trained with

\ell_{\mathrm{mse}}+\ell_{\cos}

of Eq. 3 . Best per dataset in bold, second best is underlined. FiveBillionPixels and Biomassters are omitted, the latter also being scored by RMSE. The full training and evaluation setups are detailed in App. D and B respectively.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Sensor(s)	Domain / task	Mod.	GSD (m) 1 1 1 GSD is that of the source product, taken from each dataset’s own publication; it is not declared in the benchmark’s configuration files.	Size	$T$	$C$	Splits
MADOS	Sentinel-2	Marine pollution, seg.	O	10	$240^{2}$	1	15	1433/642/728
HLSBurnScars	HLS (L8 + S2)	Wildfire, seg.	O	30	$512^{2}$	1	2	486/54/264
Sen1Floods11	Sentinel-1/2	Flood, seg.	O+S	10	$512^{2}$	1	2	252/89/90
AI4SmallFarms	Sentinel-2	Field boundaries, seg.	O	10	$496^{2}$	1	1	43/9/10
CropTypeMapping	Sentinel-1/2	Crop type, seg.	O+S	10	$64^{2}$	6	4	554/66/67
PASTIS	Sentinel-1/2	Crop type, seg.	O+S	10	$128^{2}$	6	20	1455/482/496

Appendix

Table 2: The eleven Pangaea datasets. Mod. is the modality set returned to the encoder (O = optical, S = SAR);

T

the number of time steps the loader returns;

C

the number of label classes as declared by the benchmark (

C=1

denotes a single-logit binary mask, or, for BioMassters, a continuous target). Size is the scene size the loader yields. Splits are train/val/test sample counts.

Task	Temporality	Decoder (frozen GFM)	Loss
Segmentation	$T=1$	UPerNet	Cross-entropy ( $C>1$ ) / Dice ( $C=1$ )
Segmentation	$T>1$	UPerNet + L-TAE temporal fusion	Cross-entropy
Change detection	$T=2$	Siamese-concatenation UPerNet	Cross-entropy ( $C>1$ ) / Dice ( $C=1$ )
Regression	$T=1$	Regression UPerNet	MSE
Regression	$T>1$	Regression UPerNet + L-TAE	MSE
UNet baseline (trained from scratch): UNet / Siamese UNet / regression UNet; time steps folded into channels.

Appendix

Table 3: Which evaluation components are used for which setup.

T

is the number of time steps returned by the loader. The from-scratch UNet baseline is the only model trained end-to-end.

Model	$d$	Input	Patch	Grid
CROMA (optical)	1024	120	8	$15\times 15$
DOFA	768	224	16	$14\times 14$
Prithvi	768	224	16	$14\times 14$
Scale-MAE	1024	224	16	$14\times 14$
TerraMind-L	1024	224	16	$14\times 14$

Appendix

Table 4: The five GFMs and the grid each produces per tile. Input is the model own window, to which every shared tile is resized; the token grid is

(\text{input}/\text{patch})^{2}

Dataset	Scene	$T$	Policy	Tiles
MADOS	$240^{2}$	1	centre	2803
HLSBurnScars	$512^{2}$	1	$3\times 3$ grid	7236
Sen1Floods11	$512^{2}$	1	$3\times 3$ grid	3879
AI4SmallFarms	$496^{2}$	1	$3\times 3$ grid	558
CropTypeMapping	$64^{2}$	6	whole scene	687
PASTIS	$128^{2}$	6	whole scene	2433

Appendix

Table 5: Tiling policy, subsampling cap and stored tiles per dataset. Caps are on samples per split, drawn with a fixed seed; the four capped datasets would otherwise dominate the dataset. Tiles are summed over train/val/test and are identical for all five GFMs.

	Learning rate (LR)
Dataset	Sum , MoE	Naive	Epochs	Steps	Batch	$n_{\text{train}}$
AI4SmallFarms	$10^{-4}$	$5{\times}10^{-4}$	500	6047	32	387
MADOS	$10^{-4}$	$10^{-4}$	300	6718	64	1433
SpaceNet7	$10^{-4}$	$5{\times}10^{-4}$	300	4500	64	960
CropTypeMapping	$10^{-4}$	$10^{-4}$	300	15582	64	3324
Sen1Floods11	$10^{-4}$	$10^{-3}$	200	3544	128	2268

Appendix

Table 6: Per-dataset alignment hyperparameters for the

\ell_{\mathrm{mse}}+\ell_{\cos}

runs, ordered by epoch budget. GeoMixture-Sum and GeoMixture-MoE use

10^{-4}

LR on every dataset; only GeoMixture-Naive departs from it, on three. Epochs, steps, batch size and

n_{\text{train}}

are shared by all three variants.

variant	$\ell$	AI4Farms	HLSBurns	MADOS	Sen1Floods11	SN7	Avg mIoU	Avg rank
GeoMixture-Naive	$\ell_{\mathrm{mse}}$	$\mathbf{27.49{\scriptstyle\,\pm\,0.41}}$	$83.31{\scriptstyle\,\pm\,0.09}$	$52.16{\scriptstyle\,\pm\,1.71}$	$\mathbf{90.59{\scriptstyle\,\pm\,0.01}}$	$61.32{\scriptstyle\,\pm\,0.36}$	62.98	4.00
	$\ell_{\cos}$	$\underline{27.31{\scriptstyle\,\pm\,0.39}}$	$82.31{\scriptstyle\,\pm\,0.88}$	$\underline{58.31{\scriptstyle\,\pm\,2.32}}$	$\underline{90.48{\scriptstyle\,\pm\,0.15}}$	$61.38{\scriptstyle\,\pm\,0.35}$	63.96	4.00
	$\ell_{\mathrm{mse}}+\ell_{\cos}$	$27.05{\scriptstyle\,\pm\,0.36}$	$83.21{\scriptstyle\,\pm\,0.14}$	$56.55{\scriptstyle\,\pm\,4.79}$	$90.27{\scriptstyle\,\pm\,0.21}$	$61.62{\scriptstyle\,\pm\,0.37}$	63.74	3.80
GeoMixture-Sum	$\ell_{\mathrm{mse}}$	$26.67{\scriptstyle\,\pm\,0.21}$	$82.45{\scriptstyle\,\pm\,0.60}$	$53.00{\scriptstyle\,\pm\,2.20}$	$90.16{\scriptstyle\,\pm\,0.18}$	$61.47{\scriptstyle\,\pm\,0.19}$	62.75	6.40
	$\ell_{\cos}$	$27.04{\scriptstyle\,\pm\,0.22}$	$82.36{\scriptstyle\,\pm\,1.04}$	$58.09{\scriptstyle\,\pm\,2.20}$	$89.78{\scriptstyle\,\pm\,0.06}$	$61.04{\scriptstyle\,\pm\,0.54}$	63.66	6.60
	$\ell_{\mathrm{mse}}+\ell_{\cos}$	$27.05{\scriptstyle\,\pm\,0.15}$	$83.20{\scriptstyle\,\pm\,0.66}$	$56.49{\scriptstyle\,\pm\,0.91}$	$90.25{\scriptstyle\,\pm\,0.28}$	$\underline{61.72{\scriptstyle\,\pm\,0.19}}$	63.74	4.00

Appendix

Table 7: Alignment loss ablation ( Pangaea , UPerNet decoder): mIoU as mean

\pm

std over

3

seeds, per GeoMixture variant

\times

alignment loss

\ell

of Eq. 3 . Best per dataset in bold, second best is underlined. Avg rank is a variant’s mean rank across the five datasets, where rank

1

is the best of the nine variants on that dataset (lower is better). Ranks are computed on unrounded values. The ablation covers the five datasets run with every loss; the remaining Pangaea datasets appear in Tab. 1 . Datasets acronyms AI4Farms, HLSBurns, and SN7 stands for AI4SmallFarms, HLSBurnScars, SpaceNet7 respectively.

Panel of Figure 3 — Figure 3: AI4SmallFarms similarity and complementarity scores.

Panel of Figure 4 — Figure 4: CropTypeMapping similarity and complementarity scores.

Panel of Figure 5 — Figure 5: DynamicEarthNet similarity and complementarity scores.

Panel of Figure 6 — Figure 6: FiveBillionPixels similarity and complementarity scores.

Panel of Figure 7 — Figure 7: HLSBurnScars similarity and complementarity scores.

Panel of Figure 8 — Figure 8: MADOS similarity and complementarity scores.

Panel of Figure 9 — Figure 9: PASTIS similarity and complementarity scores.

Panel of Figure 10 — Figure 10: Sen1Floods11 similarity and complementarity scores.

Panel of Figure 11 — Figure 11: SpaceNet7 similarity and complementarity scores.

Panel of Figure 12 — Figure 12: xView2 similarity and complementarity scores.

Panel of Figure 13 — Figure 13: BioMassters similarity and complementarity scores.

Geospatial Foundation Models

Geospatial foundation models turn satellite, map, and Earth-system data into reusable representations for spatial analysis and prediction.

See all 120 papers

Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts
Aug 17, 2026Nils Lehmann, Jakob Gawlikowski, Burak Ekim +2Geospatial Foundation Models Uncertainty Calibration
Chair of Data Science in Earth Observation, Technical University of Munich (TUM), Germany · Remote Sensing Technology Institute, German Aerospace Center (DLR), Germany · Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology (MIT), USA +1
From Pixels to Patches: Pooling Strategies for Earth Embeddings
Mar 2, 2026Isaac Corley, Caleb Robinson, Inbal Becker-Reshef +1Spatial Generalization Geospatial Foundation Models
Wherobots · Microsoft AI for Good Research Lab
Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models
Jun 3, 2026Shuwen Yu, Zhanxuan Hu, Yi Zhao +2Cross-Modal Representation Learning Vision Foundation Model Adaptation
Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China

Aug 17, 2026J/K move · Enter open · S save

Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts

Nils Lehmann, Jakob Gawlikowski, Burak Ekim, Isaac Corley+1

Chair of Data Science in Earth Observation, Technical University of Munich (TUM), Germany · Remote Sensing Technology Institute, German Aerospace Center (DLR), Germany · Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology (MIT), USA+1

Geospatial Foundation Models Uncertainty Calibration Distribution Shift Robustness Geospatial Representation Learning Remote Sensing Benchmarks

Open paper AskPDF arXiv

Geospatial Foundation Models (GeoFMs) are most commonly ranked and selected by accuracy on standard benchmark conditions via averaged ranks. We show that this protocol is too narrow: the promised deployment in critical EO tasks requires further angles of analysis, mainly calibration, the agreement between a model's confidence and its correctness. Across 16 frozen encoders, four classification and five segmentation datasets, and two orthogonal stress axes, every encoder degrades as corruption intensifies, and the ranking changes as well. Across the four classification benchmarks, EO-pretrained and ImageNet-pretrained encoders are indistinguishable on clean accuracy and clean calibration, and EO pretraining provides no more stability under shift than ImageNet pretraining. Under shift the GeoFMs drift further into overconfidence than the ImageNet-pretrained encoders, at every grade and in every corruption family. A centered kernel alignment (CKA) analysis ties this to representational rigidity: EO-pretrained embeddings move less under corruption while losing just as much task information and remaining overconfident. We apply three commonly explored uncertainty quantification methods and find that temperature scaling and deep ensembles cannot counteract the degradation, while a Gaussian-process probe roughly halves ECE under severe cloud only by tripling it on clean data. In selective prediction experiments, we find that confidence-based abstention cannot defer around confidently wrong predictions, and advocate that benchmark rankings and evaluations should therefore operate across a multitude of conditions and metrics to more holistically evaluate model development progress and close the gap to real world deployment scenarios.

Abstract

Geospatial Foundation Models

Explore similar work

Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts

From Pixels to Patches: Pooling Strategies for Earth Embeddings

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models