How to Embed Matters: Evaluation of EO Embedding Design Choices

Luis Gilch; Isabelle Wittmann; Maximilian Nitsche; Johannes Jakubik; Arne Ewald; Thomas Brunschwiler

cs.CVMar 11, 2026

How to Embed Matters: Evaluation of EO Embedding Design Choices

Authors: Luis Gilch, Isabelle Wittmann, Maximilian Nitsche, Johannes Jakubik, Arne Ewald, Thomas Brunschwiler

Organizations: IBM Germany · IBM Research – Europe · NORDAKADEMIE Germany

Abstract

Earth observation (EO) missions produce petabytes of multispectral imagery, increasingly analyzed using large Geospatial Foundation Models (GeoFMs). Alongside end-to-end adaptation, workflows make growing use of intermediate representations as task-agnostic embeddings, enabling models to compute representations once and reuse them across downstream tasks. Consequently, when GeoFMs act as feature extractors, decisions about how representations are obtained, aggregated, and combined affect downstream performance and pipeline scalability. Understanding these trade-offs is essential for scalable embedding-based EO workflows, where compact embeddings can replace raw data while remaining broadly useful. We present a systematic analysis of embedding design in GeoFM-based EO workflows. Leveraging NeuCo-Bench, we study how backbone architecture, pretraining strategy, representation depth, spatial aggregation, and representation combination influence EO task performance. We demonstrate the usability of GeoFM embeddings by aggregating them into fixed-size representations more than 500x smaller than the raw input data. Across models, we find consistent trends: transformer backbones with mean pooling provide strong default embeddings, intermediate ResNet layers can outperform final layers, self-supervised objectives exhibit task-specific strengths, and combining embeddings from different objectives often improves robustness.

Figures & tables

Panel of Figure 1 — Figure 1: Per-task embedding performance across design choices. Distribution of regression performance across GeoFM backbones, self-supervised pretraining strategies, spatial aggregation methods, intermediate layers, and representation combinations. Performance is measured using mean $R^{2}$ (left), reflecting predictive accuracy, and the NeuCo Quality Score (right), which accounts for variability to reflect robustness. While most methods achieve similar peak accuracy for more saturated tasks, robustness varies, leading to clearer differentiation in the Quality Score; concatenated representations often rank among the most robust configurations in these cases. Boxplots summarize the distribution over all evaluated embedding variants; whiskers denote $1.5\times$ IQR and outliers are omitted. Markers indicate the single best-performing embedding configuration among all evaluated variants for the respective task.

Figure 2: Per-task Q-score comparison of ResNet-50 (left) and ViT-Small (right) FMs. We use final-layer embeddings with mean pooling; negative scores are clipped to zero. ResNet models score high on semantic/land-cover tasks but show little performance elsewhere. ViT models are more consistent across tasks and achieve meaningful performance beyond land cover: TerraMind is the most consistent overall, DINO is strong on land cover but weaker on other tasks, and FGMAE excels on the cloud-cover and biomass tasks. Radial axis is centered at

0

, and the maximal radius is set globally with a fixed buffer. A corresponding task

R^{2}

plot is provided in the supplementary.

Figure 3: Per-task Q-score comparison of spatial aggregation method for ResNet-50 (left) and ViT-Small (right). We use final-layer embeddings with mean, min, or max pooling (or the CLS token for ViT) and average scores across models; negative scores are clipped to zero. For ResNet, mean pooling performs best across tasks, with max pooling outperforming min pooling. For ViT, mean pooling again performs best, with CLS comparable on most tasks, while min and max pooling are similar but weaker, especially on continuous biophysical targets. A corresponding task

R^{2}

plot is provided in the supplementary material.

Figure 4: Per-task and overall $\Delta R^{2}$ (left) and $\Delta$ Q-score (right) for embedding concatenation. Top: Intra-method concatenation (Mean + CLS within the same ViT-Small SSL4EO model). Bottom: Inter-method concatenation (Mean + Mean across different SSL objectives). For each task, the baseline is the stronger individual embedding, and we report $\Delta=\text{score}_{\text{concat}}-\text{score}_{\text{baseline}}$ (zero indicates no change). We additionally report the overall gain relative to the stronger overall baseline. Intra-method (Mean+CLS) concatenation yields only modest improvements (typically $<0.04$ in $R^{2}$ and $<1$ Q-score point), indicating substantial redundancy between token aggregation strategies. In contrast, inter-method (Mean+Mean) concatenation produces larger overall gains and consistent per-task improvements, reflecting complementary strengths across SSL objectives. While per-task deltas remain moderate, overall gains demonstrate that diversity in pretraining objectives contributes more to complementarity than alternative token aggregation within a single model.

Figure 5: Layer-wise task-averaged performance (

R^{2}

, left; Q-score, right). Top: ViT-Small; bottom: ResNet-50 (SSL4EO). Representations are extracted from each layer (12 transformer blocks; 5 ResNet stages with output dimensions 64, 256, 512, 1024, 2048); negative task values are clipped before averaging. ViT performance increases in early layers and then saturates, whereas ResNet shows an inverted-U pattern, peaking at intermediate stages and degrading at the final layer. A resized last-layer reference is included for ResNet.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Biomass Mean	Biomass Std	Crops	Clouds	LC Agri	LC Forest	HI Mean	HI Std	Avg.
ResNet SoftCon (mean)	-0.282	-0.184	0.725	-0.022	0.825	0.806	0.070	-0.561	0.172
ResNet DeCur (mean)	-0.205	-0.127	0.807	-0.042	0.856	0.845	0.198	-0.427	0.238
ResNet MoCo (mean)	-0.139	-0.125	0.798	0.013	0.851	0.838	0.296	-0.332	0.275
ResNet DINO (mean)	0.053	0.005	0.835	-0.203	0.870	0.863	0.264	-0.282	0.301
ViT DINO (mean)	0.282	0.217	0.843	0.334	0.866	0.863	0.304	-0.129	0.447
ViT MoCo (mean)	0.375	0.293	0.762	0.338	0.827	0.824	0.471	0.158	0.506

Appendix

Table 1: Full per-task

R^{2}

scores for tested embedding methods (Mean pooling). Methods are sorted in ascending order by Avg. For each task, the best-performing method is highlighted in bold , and the second-best is underlined .

Method	Biomass Mean	Biomass Std	Crops	Clouds	LC Agri	LC Forest	HI Mean	HI Std	Avg.
ResNet SoftCon (max)	-1.360	-1.008	0.520	-0.546	0.719	0.703	-1.049	-1.984	-0.501
ResNet MoCo (max)	-0.977	-0.778	0.637	-0.328	0.756	0.754	-0.633	-1.474	-0.255
ResNet DeCur (max)	-0.781	-0.674	0.677	-0.386	0.776	0.770	-0.641	-1.637	-0.237
ResNet DINO (max)	-0.723	-0.589	0.683	-0.498	0.770	0.758	-0.519	-1.336	-0.182
ViT MAE (max)	0.171	0.132	0.433	0.148	0.705	0.696	-0.060	-0.499	0.216
ViT FGMAE (max)	0.155	0.110	0.451	0.101	0.709	0.711	-0.070	-0.390	0.222

Appendix

Table 2: Full per-task

R^{2}

scores for tested embedding methods (Max pooling). Methods are sorted in ascending order by Avg. For each task, the best-performing method is highlighted in bold , and the second-best is underlined .

Method	Biomass Mean	Biomass Std	Crops	Clouds	LC Agri	LC Forest	HI Mean	HI Std	Avg.
ResNet MoCo (min)	-0.680	-0.647	0.411	-3.071	0.258	0.242	-0.584	-0.801	-0.609
ResNet DeCur (min)	-1.017	-0.873	0.670	-2.105	0.713	0.703	-0.827	-1.531	-0.533
ResNet DINO (min)	-1.016	-0.779	0.718	-1.244	0.779	0.758	-0.833	-1.606	-0.403
ResNet SoftCon (min)	0.043	0.026	0.304	-0.253	0.317	0.238	-0.020	-0.171	0.060
ViT MAE (min)	0.131	0.107	0.460	0.121	0.694	0.673	-0.018	-0.456	0.214
ViT FGMAE (min)	0.175	0.136	0.427	0.109	0.709	0.708	0.021	-0.292	0.249

Appendix

Table 3: Full per-task

R^{2}

scores for tested embedding methods (Min pooling). Methods are sorted in ascending order by Avg. For each task, the best-performing method is highlighted in bold , and the second-best is underlined .

Method	Biomass Mean	Biomass Std	Crops	Clouds	LC Agri	LC Forest	HI Mean	HI Std	Avg.
ViT DINO (CLS)	0.324	0.236	0.860	0.134	0.878	0.873	0.328	-0.129	0.438
ViT SoftCon (CLS)	0.369	0.286	0.769	0.404	0.855	0.850	0.504	0.105	0.518
ViT MAE (CLS)	0.403	0.316	0.584	0.619	0.778	0.777	0.516	0.165	0.520
ViT FGMAE (CLS)	0.413	0.331	0.611	0.610	0.790	0.795	0.527	0.176	0.532
ViT MoCo (CLS)	0.386	0.311	0.798	0.431	0.854	0.852	0.537	0.131	0.537

Appendix

Table 4: Full per-task

R^{2}

scores for tested embedding methods (CLS token). Methods are sorted in ascending order by Avg. For each task, the best-performing method is highlighted in bold , and the second-best is underlined .

Method	Biomass Mean	Biomass Std	Crops	Clouds	LC Agri	LC Forest	HI Mean	HI Std	Avg.
TerraMind Small (min)	0.281	0.216	0.730	0.360	0.832	0.829	0.332	-0.081	0.437
TerraMind Small (max)	0.306	0.235	0.738	0.335	0.837	0.829	0.327	-0.099	0.438
TerraMind Small (mean)	0.511	0.384	0.852	0.671	0.900	0.896	0.637	0.239	0.636

Appendix

Table 5: Full per-task

R^{2}

scores for TerraMind ViT-Small (pooling variants). Results are reported for min, max, and mean pooling and sorted by Avg.

Figure 6: Per-task

R^{2}

comparison of ResNet-50 (left) and ViT-Small (right) FMs. Final-layer embeddings with mean pooling are used. In contrast to the main paper’s Q-score visualization, this plot reports raw predictive performance (

R^{2}

) per task. The overall ranking trends remain consistent: ResNet models perform strongly on semantic/land-cover tasks but show limited transfer beyond them, while ViT models are more balanced across tasks. TerraMind remains the most consistent ViT backbone, DINO is particularly strong on land-cover targets, and FGMAE performs well on cloud-cover and biomass tasks. The radial axis is centered at

0

, and the maximum radius is fixed globally with a constant buffer for comparability.

Figure 7: Per-task

R^{2}

comparison of spatial aggregation methods for ResNet-50 (left) and ViT-Small (right). Final-layer embeddings are evaluated using mean, min, or max pooling (and the CLS token for ViT), with scores averaged across models. This

R^{2}

view confirms the Q-score trends reported in the main paper: mean pooling consistently yields the strongest performance across tasks and backbones. For ResNet, max pooling generally outperforms min pooling but both degrade performance relative to mean pooling. For ViT, mean pooling again performs best, with CLS comparable on several semantic tasks, while min and max pooling are similar but systematically weaker—especially on non-land-cover targets. The radial axis is centered at

0

, with a fixed global maximum radius.

Figure 8: Per-task

R^{2}

radar plots for embedding concatenation experiments. We report results for all tested combinations, comparing the two individual baselines with their concatenated representation. The plots illustrate that concatenation typically preserves the stronger baseline and yields modest, task-dependent improvements. In particular, combinations such as SoftCon+DINO most consistently show positive deviations over the individual embeddings, suggesting measurable complementarity between pretraining objectives.

Figure 9: Layer-wise per-task downstream performance (

R^{2}

) for ViT-Small models pretrained on SSL4EO. Results are shown separately per task across layer depth. Semantic and land-cover targets exhibit increasing and saturating trends toward deeper layers, consistent with the averaged analysis in the main paper. Other tasks show early saturation or slight degradation at greater depth.

Figure 10: Layer-wise per-task downstream performance (Q-score) for ViT-Small models pretrained on SSL4EO. The robustness trends largely mirror the

R^{2}

behavior, confirming that depth-dependent effects are consistent across predictive accuracy and stability metrics.

Figure 11: Layer-wise per-task downstream performance (

R^{2}

) for ResNet-50 models pretrained on SSL4EO. Semantic and land-cover tasks show increasing and saturating trends similar to ViT models. In contrast, several other tasks exhibit a pronounced drop at the final-layer, intermediate layers frequently remain competitive with ViT final-layer embeddings.

Figure 12: Layer-wise per-task downstream performance (Q-score) for ResNet-50 models pretrained on SSL4EO. The robustness metric reinforces the

R^{2}

trends, highlighting stronger depth sensitivity in ResNet compared to ViT backbones.

Geospatial Representation Learning

Geospatial representation learning builds reusable embeddings of places, imagery, maps, mobility, and Earth data for spatial prediction and reasoning.

See all 128 papers

Better Together: Evaluating the Complementarity of Earth Embedding Models
May 18, 2026Thijs L van der Plas, Jacob JW Bakermans, Vishal Nedungadi +3Geospatial Foundation Models Geospatial Representation Learning
Wageningen University, NL · University College London, UK · University of Bonn, Germany
Earth Embeddings
Aug 4, 2026Adam J. Stewart, Heng Fang, Isaac A. Corley +1Geospatial Representation Learning Land Cover Classification
Chair of Data Science in Earth Observation, Technical University of Munich, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · KTH Royal Institute of Technology, Stockholm, Sweden +1
Recoverable Geographic Location Information in Earth-Observation Embeddings
Sep 24, 2026Peiwen Zhang, Kristie Hu, Jovana Knezevic +2Geospatial Foundation Models Geospatial Representation Learning
University of Waterloo · University of Cambridge · Aalto University

May 18, 2026J/K move · Enter open · S save

Better Together: Evaluating the Complementarity of Earth Embedding Models

Thijs L van der Plas, Jacob JW Bakermans, Vishal Nedungadi, Gabrielė Tijūnaitytė+2

Wageningen University, NL · University College London, UK · University of Bonn, Germany

Geospatial Foundation Models Geospatial Representation Learning Remote Sensing Feature Fusion

Open paper AskPDF arXiv

Earth embedding models transform Earth observation data into embeddings uniquely tied to locations on the Earth's surface. These models are typically evaluated in isolation, comparing the downstream task performance across different Earth embeddings. However, spatially aligned embeddings can naturally be fused, providing richer information per location, a capability that isolated evaluations fail to capture. We therefore propose assessing Earth embeddings by their complementarity: the performance gain of fused embeddings over the best single-model baseline. To operationalise this, we introduce an embedding complementarity index applicable to any embedding and task, and evaluate four Earth embedding models (AlphaEarth, Tessera, GeoCLIP, SatCLIP) in isolation, in all pairs, and jointly across six downstream tasks. Fused embeddings outperform the best single model in four out of six tasks, confirming that single-embedding evaluations often underestimate Earth embedding capabilities. Complementarity proves both task- and location-dependent. Further, for a land cover regression task, we find that complementarity is partially determined by the spatial scale of land cover classes. Complementarity reframes Earth embeddings: the greatest future gains may come not from any single Earth embedding model, but from combinations that are better together.

Abstract

Geospatial Representation Learning

Explore similar work

Better Together: Evaluating the Complementarity of Earth Embedding Models

Earth Embeddings

Recoverable Geographic Location Information in Earth-Observation Embeddings