Reuse, Don't Retrain: Efficient, Modality-Versatile Geospatial Foundation Models through Feature Distillation

Mario Hamon; Pierre Adorni; Minh-Tan Pham

cs.CVOct 23, 2026

Reuse, Don't Retrain: Efficient, Modality-Versatile Geospatial Foundation Models through Feature Distillation

Authors: Mario Hamon, Pierre Adorni, Minh-Tan Pham

Organizations: IRISA Universit´e Bretagne Sud, UMR 6074

Abstract

Geospatial foundation models (GeoFMs) are costly to pretrain, yet model families often repeat this process across backbone scales and sensing modalities. We investigate feature distillation as a more efficient way to reuse existing representations. First, we distill frozen CROMA and TerraMind teachers into smaller Vision Transformers through patch-level feature matching. Across nine PANGAEA tasks, distilled TerraMind Tiny, Small, and Base outperform their scale-matched released counterparts by 2.76, 3.02, and 1.87 mean IoU points, respectively; each also matches or exceeds the next-larger released model. Distilled CROMA-Base gains 2.05 points. TerraMind-Base required only 43 A6000 GPU-hours, approximately 0.5% of the reported cost of independent pretraining. Second, we introduce TEDI, an unpaired multi-expert approach combining TerraMind for multispectral imagery and DINOv3 for high-resolution RGB. Separate patch embeddings route unpaired inputs into a shared Transformer trunk, removing the need for co-registered data. TEDI-Large achieves the highest single-model mean in our PANGAEA-9 comparison (62.51 mIoU), outperforming both teachers overall while retaining their modality-specific strengths. TEDI-Base reaches 61.75 mIoU with 72% fewer parameters. These results show that distillation improves the GeoFM accuracy-compute frontier while consolidating complementary experts into a single versatile backbone.

Figures & tables

Figure 1: Accuracy–compute trade-off. Released TerraMind models, their single-teacher distilled counterparts (TerraMind-KD), and multi-expert TEDI students are evaluated on the same nine PANGAEA tasks. Marker area indicates parameter count. Distillation shifts the entire model family toward a better trade-off; adding a complementary HR expert further improves average performance. TEDI-B 1024 reallocates Base capacity toward a wider token embedding. Final comparisons use test scores; error bars show 3-run standard deviation.

Figure 2: TEDI training. Independent Sentinel-2 and HR samples are routed through modality-specific patch embeddings and a shared student trunk. Each sample is matched only to its corresponding frozen pretrained expert through a training-only projection head. Unlike a paired cross-modal objective, TEDI requires neither spatial correspondence nor a shared input representation across experts.

Model	Params.	Mean	Source
U-Net [ 10 ] †	${\sim}8$ M	58.82	reported
ViT-B [ 10 ] †	87M	55.61	reported
DOFA [ 23 ]	112M	54.51	reported
EoS-FM [ 1 ]	76M	60.18	reported
DEO [ 22 ]	87M	58.07 [-1pt] $\pm\,0.22$	reproduced
CROMA-Large	303M	57.57 [-1pt] $\pm\,0.09$	reproduced

Table 1: Context against PANGAEA baselines. Mean test mIoU on PANGAEA-9. DOFA, EoS-FM and DEO are modality-versatile published baselines. Reported means are recomputed from published per-task values after applying our fixed task subset and have no uncertainty estimates. Our entries show one sample standard deviation across three runs. † means fine-tuned, otherwise backbones are frozen.

Family	Backbone	Released	KD	Gain
TerraMind	Large	60.22 [-1pt] $\pm\,0.41$	–	–
	Base	58.82 [-1pt] $\pm\,0.32$	60.69 [-1pt] $\pm\,0.13$	$+1.87$
	Small	56.23 [-1pt] $\pm\,0.54$	59.25 [-1pt] $\pm\,0.49$	$+3.02$
	Tiny	54.22 [-1pt] $\pm\,0.13$	56.98 [-1pt] $\pm\,0.67$	$+2.76$
CROMA	Large	57.57 [-1pt] $\pm\,0.09$	–	–
	Base	55.83 [-1pt] $\pm\,0.66$	57.88 [-1pt] $\pm\,0.32$	$+2.05$

Table 2: Single-teacher model-family derivation. Mean test mIoU on PANGAEA-9 across three downstream runs, with one sample standard deviation shown below the mean. Gains compare matched backbone sizes. CROMA results use its Sentinel-2 encoder.

Model

Multispectral

High resolution

All

HLS

MAD

PAS

S1F

AI4

xV2

FBP

DEN

SN7

TerraMind-L

82.41 [-1pt]

\pm\,1.12

70.88 [-1pt]

\pm\,1.83

42.68 [-1pt]

\pm\,0.02

90.62 [-1pt]

\pm\,0.22

39.59 [-1pt]

\pm\,0.43

56.54 [-1pt]

\pm\,0.48

62.34 [-1pt]

\pm\,0.19

36.74 [-1pt]

\pm\,0.90

60.20 [-1pt]

\pm\,0.26

65.24 [-1pt]

\pm\,0.66

53.95 [-1pt]

\pm\,0.09

60.22 [-1pt]

\pm\,0.41

DINOv3-L

78.57 [-1pt]

\pm\,0.42

66.57 [-1pt]

\pm\,3.54

25.83 [-1pt]

\pm\,0.39

75.58 [-1pt]

\pm\,0.66

45.13 [-1pt]

\pm\,0.18

61.38 [-1pt]

\pm\,0.71

74.71 [-1pt]

\pm\,0.05

37.58 [-1pt]

\pm\,0.82

63.05 [-1pt]

\pm\,0.58

58.34 [-1pt]

\pm\,0.82

59.18 [-1pt]

\pm\,0.22

58.71 [-1pt]

\pm\,0.36

TEDI-L

83.64 [-1pt]

\pm\,0.40

69.36 [-1pt]

\pm\,7.47

41.60 [-1pt]

\pm\,0.41

90.97 [-1pt]

\pm\,0.34

40.86 [-1pt]

\pm\,0.58

61.32 [-1pt]

\pm\,1.26

72.57 [-1pt]

\pm\,0.58

38.94 [-1pt]

\pm\,0.93

63.34 [-1pt]

\pm\,0.12

65.29 [-1pt]

\pm\,1.47

59.04 [-1pt]

\pm\,0.06

62.51 [-1pt]

\pm\,0.83

TEDI-B

83.12 [-1pt]

\pm\,0.36

66.96 [-1pt]

\pm\,3.47

40.04 [-1pt]

\pm\,0.65

90.82 [-1pt]

\pm\,0.30

39.66 [-1pt]

\pm\,0.52

60.90 [-1pt]

\pm\,0.63

71.42 [-1pt]

\pm\,0.33

39.49 [-1pt]

\pm\,0.58

63.34 [-1pt]

\pm\,0.31

64.12 [-1pt]

\pm\,0.79

58.79 [-1pt]

\pm\,0.27

61.75 [-1pt]

\pm\,0.45

Expert oracle

82.41

70.88

42.68

90.62

45.13

61.38

74.71

37.58

63.05

66.34

59.18

63.16

Table 3: Multi-expert aggregation. Mean test mIoU on the nine PANGAEA-9 tasks. MS and HR average their respective five and four tasks, All over all nine. The expert oracle selects the better teacher independently for each task and therefore requires two deployed models. Results are averaged across three downstream runs, with one sample standard deviation shown below the mean. HLS=HLSBurnScars, MAD=Mados, PAS=PASTIS, S1F=Sen1Floods11, AI4=AI4SmallFarms, xV2=xView2, FBP=FiveBillionPixels, DEN=DynamicEarthNet, SN7=SpaceNet7.

Model	Params.	GFLOPs	FP32	Images/s	Latency p50/p95 (ms)	Peak MB	Distill h / GPU
TEDI-T/16	5.7M	2.0	23MB	165.02	6.07 / 6.43	34.92	56.29 / V100
TEDI-S/16	22M	6.7	88MB	145.96	7.33 / 7.60	101.06	91.07 / Titan RTX
TEDI-B/16	86M	24.6	344MB	86.62	12.88 / 13.24	353.12	63.75 / A6000
TEDI-B 1024 /16	90M	35.6	360MB	73.44	13.36 / 16.08	369.02	69.40 / A6000
TEDI-L/16	303M	84.3	1.21GB	30.86	38.80 / 39.76	1187.27	114.77 / A6000

Table 4: TEDI model and distillation resources. Parameter and storage counts include the shared trunk and both retained input embeddings; training-only heads are excluded. GFLOPs use ptflops with a standardized 10-channel

224\times 224

input. FP32 batch-1 inference uses an RTX 2080 Ti after 20 warm-up and 100 synchronized timed runs; peak memory includes resident weights. Distillation times are elapsed hours for the corresponding TEDI single-GPU runs and are not normalized across hardware.

Figure 3: Validation analyses. TerraMind-KD-Base val performance vs. distillation extent (left) and fraction of unique SSL4EO-S12 locations (right). Data-fraction runs hold update count fixed. Error bars show one sample standard deviation across three downstream runs.

Student	MS	HR	All	Params.
TerraMind-KD-B	65.41 [-1pt] $\pm\,0.14$	54.80 [-1pt] $\pm\,0.13$	60.69 [-1pt] $\pm\,0.13$	86M
DINOv3-KD-B	57.21 [-1pt] $\pm\,0.49$	57.98 [-1pt] $\pm\,0.89$	57.55 [-1pt] $\pm\,0.54$	86M
TEDI-B	64.12 [-1pt] $\pm\,0.79$	58.79 [-1pt] $\pm\,0.27$	61.75 [-1pt] $\pm\,0.45$	86M
Two students	65.41 [-1pt] $\pm\,0.14$	57.98 [-1pt] $\pm\,0.89$	62.11 [-1pt] $\pm\,0.37$	172M

Table 5: Teacher-multiplicity ablation. Avg. test mIoU across three downstream runs, with standard deviation. Students use the same architecture and optimization budget. Two-student deployment: MS tasks to TerraMind-KD and HR tasks to DINOv3-KD.

Student	Dim.	Depth	MLP	Params.	All
TEDI-B	768	12	$4.0\times$	86M	61.75 [-1pt] $\pm\,0.45$
TEDI-B 1024	1024	12	$1.5\times$	90M	62.32 [-1pt] $\pm\,0.13$
TEDI-L	1024	24	$4.0\times$	303M	62.51 [-1pt] $\pm\,0.83$
TerraMind-KD-B	768	12	$4.0\times$	86M	60.69 [-1pt] $\pm\,0.13$
TerraMind-KD-B 1024	1024	12	$1.5\times$	90M	60.60 [-1pt] $\pm\,0.18$
DINOv3-KD-B	768	12	$4.0\times$	86M	57.55 [-1pt] $\pm\,0.54$

Table 6: Width-allocation ablation. Avg. PANGAEA-9 test mIoU across three downstream runs, with one sample standard deviation. The B 1024 variants widen the token embedding while limiting parameter growth.

Geospatial Foundation Models

Geospatial foundation models turn satellite, map, and Earth-system data into reusable representations for spatial analysis and prediction.

See all 122 papers

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models
Oct 20, 2026Arthur Ouaknine, David RolnickGeospatial Foundation Models Geospatial Representation Learning
McGill University · Mila – Quebec AI Institute
Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction
Jul 2, 2026Clémentine Grethen, Florient Chouteau, Géraldine Morin +13D Foundation Models 3D Reconstruction
1 1 IRIT, University of Toulouse, France · 2 2 Airbus Defence and Space, Toulouse, France
GeoSANE: Learning Geospatial Representations from Models, Not Data
Mar 24, 2026Joelle Hanna, Damian Falk, Stella X. Yu +1Weight-Space Learning Geospatial Representation Learning
University of St.Gallen · University of Michigan and UC Berkeley · ESA Φ-Lab

Oct 20, 2026J/K move · Enter open · S save

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

Arthur Ouaknine, David Rolnick

McGill University · Mila – Quebec AI Institute

Geospatial Foundation Models Geospatial Representation Learning Representation Learning Latent Space Alignment

Open paper AskPDF arXiv

Geospatial foundation models (GFMs) differ widely in pretraining objective, modality and resolution, and none leads across benchmarks. We ask whether their representations are complementary, and whether a single encoder can inherit that complementarity. We introduce Pangaea-Latents, an extension of the Pangaea benchmark providing frozen embeddings of

5

GFMs over its

11

datasets, along with reproduced baselines, and find GFM embeddings to be neither redundant nor disjoint. We then propose GeoMixture, a preliminary framework to align a ViT to several GFMs per dataset through a multi-representation alignment (mREPA) objective. Our encoder ties the best GFM in average rank across nine Pangaea datasets, trading peak performance for cross-task consistency.

Abstract

Geospatial Foundation Models

Explore similar work

Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction

GeoSANE: Learning Geospatial Representations from Models, Not Data