cs.CVOct 20, 2026

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

Authors: Arthur Ouaknine, David Rolnick

Organizations: McGill University · Mila – Quebec AI Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Geospatial foundation models (GFMs) differ widely in pretraining objective, modality and resolution, and none leads across benchmarks. We ask whether their representations are complementary, and whether a single encoder can inherit that complementarity. We introduce Pangaea-Latents, an extension of the Pangaea benchmark providing frozen embeddings of 55 GFMs over its 1111 datasets, along with reproduced baselines, and find GFM embeddings to be neither redundant nor disjoint. We then propose GeoMixture, a preliminary framework to align a ViT to several GFMs per dataset through a multi-representation alignment (mREPA) objective. Our encoder ties the best GFM in average rank across nine Pangaea datasets, trading peak performance for cross-task consistency.

Figures & tables

Figure 1: Pangaea-Latents (a) (Sec. 3.1 ) is an extension of Pangaea with frozen embeddings of GFMs. GeoMixture (Sec. 3.3 ) is a framework aligning a ViT to GFMs through a multi-representation alignment (mREPA) objective in a single latent (b), or in a Mixture of Experts (c).
Figure 2: Pairwise similarity (Eqs. 4 , 5 , 1 ) and complementarity (Eq. 6 ) between the 55 GFM embeddings, averaged over the 1111 datasets of Pangaea-Latents ; per-dataset scores in App. F .
modelAI4FarmsCropMapDynEarthNetHLSBurnsMADOSPASTISSen1Floods11SN7xView2Avg mIoUAvg rank
UNet (baseline)45.41±0.71\mathbf{45.41{\scriptstyle\,\pm\,0.71}}45.95±2.1145.95{\scriptstyle\,\pm\,2.11}33.32±1.9633.32{\scriptstyle\,\pm\,1.96}85.64±0.61\mathbf{85.64{\scriptstyle\,\pm\,0.61}}57.66±3.0057.66{\scriptstyle\,\pm\,3.00}28.56±0.3728.56{\scriptstyle\,\pm\,0.37}90.88±0.87\mathbf{90.88{\scriptstyle\,\pm\,0.87}}62.43±0.16‾\underline{62.43{\scriptstyle\,\pm\,0.16}}42.21±2.3242.21{\scriptstyle\,\pm\,2.32}54.674.33
CROMA27.59±0.27‾\underline{27.59{\scriptstyle\,\pm\,0.27}}58.41±4.58\mathbf{58.41{\scriptstyle\,\pm\,4.58}}30.80±1.0730.80{\scriptstyle\,\pm\,1.07}82.59±0.2482.59{\scriptstyle\,\pm\,0.24}65.63±1.58‾\underline{65.63{\scriptstyle\,\pm\,1.58}}38.49±0.6938.49{\scriptstyle\,\pm\,0.69}90.06±0.2490.06{\scriptstyle\,\pm\,0.24}59.22±0.2859.22{\scriptstyle\,\pm\,0.28}40.52±0.2040.52{\scriptstyle\,\pm\,0.20}54.815.22
DOFA27.11±0.3727.11{\scriptstyle\,\pm\,0.37}47.53±5.0247.53{\scriptstyle\,\pm\,5.02}38.99±1.11‾\underline{38.99{\scriptstyle\,\pm\,1.11}}80.86±0.4580.86{\scriptstyle\,\pm\,0.45}60.56±2.6460.56{\scriptstyle\,\pm\,2.64}29.53±0.6029.53{\scriptstyle\,\pm\,0.60}88.22±0.2088.22{\scriptstyle\,\pm\,0.20}61.01±0.1361.01{\scriptstyle\,\pm\,0.13}59.15±0.1059.15{\scriptstyle\,\pm\,0.10}54.775.33
Prithvi26.87±0.5026.87{\scriptstyle\,\pm\,0.50}44.91±2.4344.91{\scriptstyle\,\pm\,2.43}21.64±0.8221.64{\scriptstyle\,\pm\,0.82}83.08±0.3683.08{\scriptstyle\,\pm\,0.36}55.02±1.4855.02{\scriptstyle\,\pm\,1.48}33.93±0.4533.93{\scriptstyle\,\pm\,0.45}90.20±0.4490.20{\scriptstyle\,\pm\,0.44}56.56±0.2856.56{\scriptstyle\,\pm\,0.28}49.17±0.2449.17{\scriptstyle\,\pm\,0.24}51.267.33
Scale-MAE22.52±1.8722.52{\scriptstyle\,\pm\,1.87}25.18±1.8725.18{\scriptstyle\,\pm\,1.87}35.97±1.7535.97{\scriptstyle\,\pm\,1.75}76.56±0.0776.56{\scriptstyle\,\pm\,0.07}55.60±2.2555.60{\scriptstyle\,\pm\,2.25}24.61±0.5524.61{\scriptstyle\,\pm\,0.55}74.85±0.0974.85{\scriptstyle\,\pm\,0.09}62.58±0.50\mathbf{62.58{\scriptstyle\,\pm\,0.50}}60.05±0.16\mathbf{60.05{\scriptstyle\,\pm\,0.16}}48.666.67
TerraMind-L26.98±0.5426.98{\scriptstyle\,\pm\,0.54}54.67±2.82‾\underline{54.67{\scriptstyle\,\pm\,2.82}}35.22±1.4235.22{\scriptstyle\,\pm\,1.42}83.41±0.1583.41{\scriptstyle\,\pm\,0.15}70.99±2.31\mathbf{70.99{\scriptstyle\,\pm\,2.31}}42.54±0.44\mathbf{42.54{\scriptstyle\,\pm\,0.44}}90.78±0.13‾\underline{90.78{\scriptstyle\,\pm\,0.13}}60.22±0.7060.22{\scriptstyle\,\pm\,0.70}48.67±0.3748.67{\scriptstyle\,\pm\,0.37}57.053.89
Table 1: PANGAEA frozen-encoder evaluation : mIoU as mean ±\pm std over 33 seeds, on nine Pangaea datasets. Our methods are trained with ℓmse+ℓcos⁡\ell_{\mathrm{mse}}+\ell_{\cos} of Eq. 3 . Best per dataset in bold, second best is underlined. FiveBillionPixels and Biomassters are omitted, the latter also being scored by RMSE. The full training and evaluation setups are detailed in App. D and B respectively.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetSensor(s)Domain / taskMod.GSD (m) 1 1 1 GSD is that of the source product, taken from each dataset’s own publication; it is not declared in the benchmark’s configuration files.SizeTTCCSplits
MADOSSentinel-2Marine pollution, seg.O102402240^{2}1151433/642/728
HLSBurnScarsHLS (L8 + S2)Wildfire, seg.O305122512^{2}12486/54/264
Sen1Floods11Sentinel-1/2Flood, seg.O+S105122512^{2}12252/89/90
AI4SmallFarmsSentinel-2Field boundaries, seg.O104962496^{2}1143/9/10
CropTypeMappingSentinel-1/2Crop type, seg.O+S1064264^{2}64554/66/67
PASTISSentinel-1/2Crop type, seg.O+S101282128^{2}6201455/482/496
Appendix
Table 2: The eleven Pangaea datasets. Mod. is the modality set returned to the encoder (O = optical, S = SAR); TT the number of time steps the loader returns; CC the number of label classes as declared by the benchmark ( C=1C=1 denotes a single-logit binary mask, or, for BioMassters, a continuous target). Size is the scene size the loader yields. Splits are train/val/test sample counts.
TaskTemporalityDecoder (frozen GFM)Loss
SegmentationT=1T=1UPerNetCross-entropy ( C>1C>1 ) / Dice ( C=1C=1 )
SegmentationT>1T>1UPerNet + L-TAE temporal fusionCross-entropy
Change detectionT=2T=2Siamese-concatenation UPerNetCross-entropy ( C>1C>1 ) / Dice ( C=1C=1 )
RegressionT=1T=1Regression UPerNetMSE
RegressionT>1T>1Regression UPerNet + L-TAEMSE
UNet baseline (trained from scratch): UNet / Siamese UNet / regression UNet; time steps folded into channels.
Appendix
Table 3: Which evaluation components are used for which setup. TT is the number of time steps returned by the loader. The from-scratch UNet baseline is the only model trained end-to-end.
ModelddInputPatchGrid
CROMA (optical)1024120815×1515\times 15
DOFA7682241614×1414\times 14
Prithvi7682241614×1414\times 14
Scale-MAE10242241614×1414\times 14
TerraMind-L10242241614×1414\times 14
Appendix
Table 4: The five GFMs and the grid each produces per tile. Input is the model own window, to which every shared tile is resized; the token grid is (input/patch)2(\text{input}/\text{patch})^{2} .
DatasetSceneTTPolicyTiles
MADOS2402240^{2}1centre2803
HLSBurnScars5122512^{2}13×33\times 3 grid7236
Sen1Floods115122512^{2}13×33\times 3 grid3879
AI4SmallFarms4962496^{2}13×33\times 3 grid558
CropTypeMapping64264^{2}6whole scene687
PASTIS1282128^{2}6whole scene2433
Appendix
Table 5: Tiling policy, subsampling cap and stored tiles per dataset. Caps are on samples per split, drawn with a fixed seed; the four capped datasets would otherwise dominate the dataset. Tiles are summed over train/val/test and are identical for all five GFMs.
 Learning rate (LR)    
DatasetSum , MoENaiveEpochsStepsBatchntrainn_{\text{train}}
AI4SmallFarms10−410^{-4}5×10−45{\times}10^{-4}500604732387
MADOS10−410^{-4}10−410^{-4}3006718641433
SpaceNet710−410^{-4}5×10−45{\times}10^{-4}300450064960
CropTypeMapping10−410^{-4}10−410^{-4}30015582643324
Sen1Floods1110−410^{-4}10−310^{-3}20035441282268
Appendix
Table 6: Per-dataset alignment hyperparameters for the ℓmse+ℓcos⁡\ell_{\mathrm{mse}}+\ell_{\cos} runs, ordered by epoch budget. GeoMixture-Sum and GeoMixture-MoE use 10−410^{-4} LR on every dataset; only GeoMixture-Naive departs from it, on three. Epochs, steps, batch size and ntrainn_{\text{train}} are shared by all three variants.
variantℓ\ellAI4FarmsHLSBurnsMADOSSen1Floods11SN7Avg mIoUAvg rank
GeoMixture-Naiveℓmse\ell_{\mathrm{mse}}27.49±0.41\mathbf{27.49{\scriptstyle\,\pm\,0.41}}83.31±0.0983.31{\scriptstyle\,\pm\,0.09}52.16±1.7152.16{\scriptstyle\,\pm\,1.71}90.59±0.01\mathbf{90.59{\scriptstyle\,\pm\,0.01}}61.32±0.3661.32{\scriptstyle\,\pm\,0.36}62.984.00
ℓcos⁡\ell_{\cos}27.31±0.39‾\underline{27.31{\scriptstyle\,\pm\,0.39}}82.31±0.8882.31{\scriptstyle\,\pm\,0.88}58.31±2.32‾\underline{58.31{\scriptstyle\,\pm\,2.32}}90.48±0.15‾\underline{90.48{\scriptstyle\,\pm\,0.15}}61.38±0.3561.38{\scriptstyle\,\pm\,0.35}63.964.00
ℓmse+ℓcos⁡\ell_{\mathrm{mse}}+\ell_{\cos}27.05±0.3627.05{\scriptstyle\,\pm\,0.36}83.21±0.1483.21{\scriptstyle\,\pm\,0.14}56.55±4.7956.55{\scriptstyle\,\pm\,4.79}90.27±0.2190.27{\scriptstyle\,\pm\,0.21}61.62±0.3761.62{\scriptstyle\,\pm\,0.37}63.743.80
GeoMixture-Sumℓmse\ell_{\mathrm{mse}}26.67±0.2126.67{\scriptstyle\,\pm\,0.21}82.45±0.6082.45{\scriptstyle\,\pm\,0.60}53.00±2.2053.00{\scriptstyle\,\pm\,2.20}90.16±0.1890.16{\scriptstyle\,\pm\,0.18}61.47±0.1961.47{\scriptstyle\,\pm\,0.19}62.756.40
ℓcos⁡\ell_{\cos}27.04±0.2227.04{\scriptstyle\,\pm\,0.22}82.36±1.0482.36{\scriptstyle\,\pm\,1.04}58.09±2.2058.09{\scriptstyle\,\pm\,2.20}89.78±0.0689.78{\scriptstyle\,\pm\,0.06}61.04±0.5461.04{\scriptstyle\,\pm\,0.54}63.666.60
ℓmse+ℓcos⁡\ell_{\mathrm{mse}}+\ell_{\cos}27.05±0.1527.05{\scriptstyle\,\pm\,0.15}83.20±0.6683.20{\scriptstyle\,\pm\,0.66}56.49±0.9156.49{\scriptstyle\,\pm\,0.91}90.25±0.2890.25{\scriptstyle\,\pm\,0.28}61.72±0.19‾\underline{61.72{\scriptstyle\,\pm\,0.19}}63.744.00
Appendix
Table 7: Alignment loss ablation ( Pangaea , UPerNet decoder): mIoU as mean ±\pm std over 33 seeds, per GeoMixture variant ×\times alignment loss ℓ\ell of Eq. 3 . Best per dataset in bold, second best is underlined. Avg rank is a variant’s mean rank across the five datasets, where rank 11 is the best of the nine variants on that dataset (lower is better). Ranks are computed on unrounded values. The ablation covers the five datasets run with every loss; the remaining Pangaea datasets appear in Tab. 1 . Datasets acronyms AI4Farms, HLSBurns, and SN7 stands for AI4SmallFarms, HLSBurnScars, SpaceNet7 respectively.
Figure 3: AI4SmallFarms similarity and complementarity scores.
Figure 4: CropTypeMapping similarity and complementarity scores.
Figure 5: DynamicEarthNet similarity and complementarity scores.
Figure 6: FiveBillionPixels similarity and complementarity scores.
Figure 7: HLSBurnScars similarity and complementarity scores.
Figure 8: MADOS similarity and complementarity scores.
Figure 9: PASTIS similarity and complementarity scores.
Figure 10: Sen1Floods11 similarity and complementarity scores.
Figure 11: SpaceNet7 similarity and complementarity scores.
Figure 12: xView2 similarity and complementarity scores.
Figure 13: BioMassters similarity and complementarity scores.
More in this topic

Geospatial Foundation Models

Geospatial foundation models turn satellite, map, and Earth-system data into reusable representations for spatial analysis and prediction.

See all 120 papers

Explore similar work

CardsList
Aug 17, 2026

Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts

Geospatial Foundation Models (GeoFMs) are most commonly ranked and selected by accuracy on standard benchmark conditions via averaged ranks. We show that this protocol is too narrow: the promised deployment in critical EO tasks requires further angles of analysis, mainly calibration, the agreement between a model's confidence and its correctness. Across 16 frozen encoders, four classification and five segmentation datasets, and two orthogonal stress axes, every encoder degrades as corruption intensifies, and the ranking changes as well. Across the four classification benchmarks, EO-pretrained and ImageNet-pretrained encoders are indistinguishable on clean accuracy and clean calibration, and EO pretraining provides no more stability under shift than ImageNet pretraining. Under shift the GeoFMs drift further into overconfidence than the ImageNet-pretrained encoders, at every grade and in every corruption family. A centered kernel alignment (CKA) analysis ties this to representational rigidity: EO-pretrained embeddings move less under corruption while losing just as much task information and remaining overconfident. We apply three commonly explored uncertainty quantification methods and find that temperature scaling and deep ensembles cannot counteract the degradation, while a Gaussian-process probe roughly halves ECE under severe cloud only by tripling it on clean data. In selective prediction experiments, we find that confidence-based abstention cannot defer around confidently wrong predictions, and advocate that benchmark rankings and evaluations should therefore operate across a multitude of conditions and metrics to more holistically evaluate model development progress and close the gap to real world deployment scenarios.

Nils Lehmann, Jakob Gawlikowski, Burak Ekim +2

Chair of Data Science in Earth Observation, Technical University of Munich (TUM), Germany · Remote Sensing Technology Institute, German Aerospace Center (DLR), Germany · Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology (MIT), USA +1

Geospatial Foundation ModelsUncertainty CalibrationDistribution Shift RobustnessGeospatial Representation LearningRemote Sensing Benchmarks
AskPDF
Mar 2, 2026

From Pixels to Patches: Pooling Strategies for Earth Embeddings

Geospatial foundation models increasingly expose pixel-level embedding products that can be downloaded and reused without access to the underlying encoder. In this setting, downstream tasks with patch- or region-level labels require a post-hoc aggregation step that maps dense pixel embeddings to a single representation. The default choice, mean pooling, discards within-patch variability and can underperform under spatial distribution shift. To study this setting, we introduce EuroSAT-Embed: 81,000 embedding GeoTIFFs derived from three foundation models: AlphaEarth, OlmoEarth, and Tessera. Using these fixed embedding products, we benchmark 11 training-free pooling methods and 2 train-set-fitted baselines under both random and geographically disjoint test splits. Richer pooling schemes reduce the geographic generalization gap by over 50% relative to mean pooling and improve accuracy by up to 6% on spatial splits. We recommend a three-tier strategy: (1) mean as a baseline, (2) stats pooling (min/max/mean/std) as the default at 4x the embedding dimension, and (3) covariance pooling for peak accuracy. Across all three embedding products, simple distributional statistics improve spatial-split performance over mean pooling.

Isaac Corley, Caleb Robinson, Inbal Becker-Reshef +1

Wherobots · Microsoft AI for Good Research Lab

Spatial GeneralizationGeospatial Foundation ModelsRepresentation LearningRemote Sensing Benchmarks
AskPDF
Jun 3, 2026

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA

Shuwen Yu, Zhanxuan Hu, Yi Zhao +2

Yunnan Normal University, Kunming, China · Kunming University of Science and Technology, Kunming, China

Cross-Modal Representation LearningVision Foundation Model AdaptationCross-Modal Feature MatchingMultimodal Foundation ModelsVision-Language Alignment
AskPDF