TerraFlow: Multimodal, Multitemporal Representation Learning for Earth Observation

Nazar Puriy; Johannes Jakubik; Benedikt Blumenstiel; Konrad Schindler

cs.CVMar 13, 2026

TerraFlow: Multimodal, Multitemporal Representation Learning for Earth Observation

Authors: Nazar Puriy, Johannes Jakubik, Benedikt Blumenstiel, Konrad Schindler

Organizations: IBM Research Europe, Zurich, Switzerland · ETH Zurich, Zurich, Switzerland

Abstract

We propose TerraFlow, a novel approach to multimodal, multitemporal learning for Earth observation. TerraFlow builds on temporal training objectives that enable sequence-aware learning across space, time, and modality, while remaining robust to the variable-length inputs commonly encountered in real-world Earth observation data. Our experiments demonstrate superiority of TerraFlow over state-of-the-art foundation models for Earth observation across all temporal tasks of the GEO-Bench-2 benchmark. We additionally demonstrate that TerraFlow is able to make initial steps towards deep-learning based risk map prediction for natural disasters -- a task on which other state-of-the-art foundation models frequently collapse. TerraFlow outperforms state-of-the-art foundation models by up to 50% in F1 score and 24% in Brier score.

Figures & tables

Figure 1: Image encoders can be applied to temporal tasks using a late fusion approach (left) while TerraFlow uses temporal attention for early fusion (middle). We benchmark TerraFlow on four temporal GEO-Bench-2 datasets and challenging disaster risk maps from ImpactMesh and Kuro Siwo (right).

Figure 2: TerraFlow pretraining. Input and target patches are sampled from multiple modalities and timestamps, represented by either raw pixel values or tokens. TerraFlow uses a standard transformer encoder-decoder architecture, trained with cross entropy loss. The attention blocks apply RoPE to encode the relative temporal offset between timestamps in the queries and keys.

		Kuro Siwo	BioMassters	PASTIS	DEN
Model	Params	mIoU [%] ↑	custom ↑	mIoU [%] ↑	mIoU [%] ↑
Clay-V1 ViT-B [ 7 ]	100M	68.6 $\pm$ 0.3	95.8 $\pm$ 0.0	–	–
TerraMind 1.0 Base [ 16 ]	100M	67.9 $\pm$ 0.3	95.5 $\pm$ 0.0	–	–
DOFA ViT [ 34 ]	300M	69.7 $\pm$ 0.2	95.6 $\pm$ 0.0	–	–
DINOv3-ViT-L-SAT [ 24 ]	300M	69.0 $\pm$ 0.2	95.5 $\pm$ 0.0	–	–
TerraMind 1.0 Large [ 16 ]	300M	68.6 $\pm$ 0.2	96.0 $\pm$ 0.0	–	–

Table 1: Model comparison on GEO-Bench-2 temporal datasets. The top section reports publicly available GEO-Bench-2 results [ 25 ] that could be reproduced. PASTIS and DynamicEarthNet (DEN) values are omitted for baselines due to discrepancies during reproducibility checks; see supplementary material 0.D . BioMassters reports

(1-\mathrm{std})\cdot\mathrm{RMSE}

[%] as proposed in GEO-Bench-2.

Model	PT	TA	Loss	F1 Flood [%] ↑	Brier [%] ↓
ResNet18	✗	✗	CE	23.6 ± 0.3	14.8 ± 0.8
ResNet50	✗	✗	CE	23.0 ± 0.9	15.5 ± 0.8
ResNet101	✗	✗	CE	22.6 ± 1.5	15.3 ± 1.7
ViT-Tiny	✗	✓	CE	18.9 ± 0.0	11.6 ± 0.3
TerraMind 1.0 Tiny	✓	✗	CE	23.2 ± 0.1	12.1 ± 0.9
TerraMind 1.0 Tiny	✓	✓	CE	30.0 ± 0.7	11.0 ± 0.1

Table 2: Comparison of Kuro Siwo flood risk prediction performance across models. The top section reports models provided by Kuro Siwo [ 5 ] followed by Tiny and Base variants of TerraMind and TerraFlow.

Figure 3: Qualitative comparison on the Kuro Siwo flood risk prediction task. The first columns display two pre-event Sentinel-1 images with VV-VH-VV/VH pseudo coloring, along with the DEM. TerraFlow predicts logical risk maps and understands permanent water bodies while image-level models like TerraMind completely fail to accurately predict risk.

				ImpactMesh-Flood		ImpactMesh-Fire
Model	PT	TA	Loss	F1 Fl. [%] ↑	Brier [%] ↓	F1 Fi. [%] ↑	Brier [%] ↓
ViT-Tiny	✗	✓	CE	21.5 ± 2.9	5.8 ± 0.6	55.9 ± 0.8	21.1 ± 1.5
TerraMind 1.0 Tiny	✓	✗	CE	32.9 ± 0.8	6.2 ± 0.2	54.6 ± 1.0	20.9 ± 1.5
TerraMind 1.0 Tiny	✓	✓	CE	33.2 ± 1.9	5.2 ± 0.2	56.0 ± 0.5	20.3 ± 0.4
TerraFlow-Tiny-RS	✓	✓	CE	36.0 ± 1.1	5.0 ± 0.1	56.0 ± 1.7	19.4 ± 0.3
TerraFlow-Tiny-TDS	✓	✓	CE	38.1 ± 0.6	5.0 ± 0.1	56.3 ± 0.9	19.2 ± 0.8

Table 3: Performance comparison on ImpactMesh for flood and fire risk prediction, highlighting the impact of pretrained weights (PT), temporal attention (TA), and the training loss.

Figure 4: Qualitative comparison on the ImpactMesh-Fire risk prediction task. The inputs include two pre-event Sentinel-2 images, two Sentinel-1 images with VV-VH-VV/VH pseudo coloring, and DEM. No model, including TerraFlow, is able to learn meaningful patterns for fire prediction which may stem from the unpredictable human influence on many fire events.

Panel of Figure 5 — Figure 5: Downstream task performance on test sets as a function of the number of timesteps per sample, uniformly sampled from the available set.

Panel of Figure 6 — Figure 6: Effect of token subsampling: Random image subsampling reduces training time at the cost of accuracy, patch-level subsampling across timesteps and modalities achieves better results with fewer tokens.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Strategies used during the pretraining phase. Random Sampling (RS), where tokens are randomly masked as inputs and outputs. Temporal Disjoint Sampling (TDS), where input and output tokens do not share timesteps. TDS-Future, where input tokens correspond to earlier timesteps than the target tokens. Consistent Sampling, where the same tokens are sampled across modalities.

Figure 8: Qualitative comparison on the ImpactMesh-Flood risk prediction task. The inputs include two pre-event Sentinel-2 images, two Sentinel-1 images with VV-VH-VV/VH pseudo coloring, and DEM. We observe that the TerraFlow models are able to learn meaningful patterns for flood prediction while image-level models like TerraMind completely fail to accurately predict risk.

S1	S2	DEM	Rel.	ImpactMesh-Flood		ImpactMesh-Fire
S1	S2	DEM	DEM	TF-Base-RS	TF-Base-TDS	TF-Base-RS	TF-Base-TDS
✓	✗	✗	✗	20.2 $\pm$ 0.3	21.3 $\pm$ 1.9	51.7 $\pm$ 2.3	52.5 $\pm$ 1.4
✗	✓	✗	✗	38.2 $\pm$ 1.6	37.8 $\pm$ 2.9	54.7 $\pm$ 1.2	54.6 $\pm$ 3.0
✗	✗	✓	✗	5.7 $\pm$ 1.3	5.3 $\pm$ 2.5	44.5 $\pm$ 3.4	46.6 $\pm$ 3.0
✗	✗	✗	✓	17.8 $\pm$ 1.2	16.7 $\pm$ 0.9	49.0 $\pm$ 0.0	48.9 $\pm$ 0.1
✓	✓	✗	✗	37.8 $\pm$ 1.2	39.7 $\pm$ 0.8	54.5 $\pm$ 3.4	55.4 $\pm$ 1.5

Appendix

Table 4: TerraFlow-Base results on ImpactMesh flood and fire tasks show that multimodal inputs (S1+S2+DEM) perform best overall; Sentinel-2 is the strongest single modality, while DEM’s impact is task- and modality-dependent, benefiting floods mainly when combined with S2. We report F1 {Flood,Fire} [%]

\uparrow

as mean

\pm

std across three runs.

	PASTIS	BioMassters
Input	mIoU [%] ↑	RMSE ↓
Five timesteps	56.5 $\pm$ 0.1	0.138 $\pm$ 0.000
$\frac{1}{3}$ Patch Subsampling	56.6 $\pm$ 0.3	0.137 $\pm$ 0.000
Time-shuffled images	55.7 $\pm$ 0.1	0.138 $\pm$ 0.000

Appendix

Table 5: Comparison with TerraFlow-Base-TDS between the full, subsampled, and timestamp-shuffled variants using five timesteps on PASTIS and BioMassters. Performance is comparable overall, with subsampling yielding similar or slightly better results, while timestamp shuffling highlights differing sensitivity to temporal order across tasks.

Model	TA	mIoU $\uparrow$
UNet	✗	42.6
Prithvi-EO-1.0-100M	✓	42.7
Prithvi-EO-2.0-300M	✓	48.6
Prithvi-EO-2.0-600M	✓	50.7
TerraMind 1.0 Base	✗	50.3 $\pm$ 0.04
TerraMind 1.0 Base	✓	50.4 $\pm$ 0.08

Appendix

Table 6: Early comparison of segmentation performance on the MultiCrop dataset. Results for UNet, Prithvi, TerraMind 1.0 Base (with and without temporal attention), and TerraFlow-Base-RS. The results show that introducing early temporal attention (TA) combined with pretraining improves performance, with TerraFlow-Base-RS achieving performance comparable to Prithvi-EO-2.0-600M.

Figure 9: Preliminary MultiCrop results comparing TerraMind 1.0 Base, TerraMind with temporal attention, and TerraFlow-Base-RS, showing slower early convergence with temporal attention but improved final performance, and a larger gain when initializing from pretrained TerraFlow-Base weights.

Input	Floods mIoU ↑	Fire mIoU ↑
One pre-event timestep	34.9 ± 0.6	53.4 ± 0.7
Two pre-event timesteps	39.7 ± 0.4	56.6 ± 0.3

Appendix

Table 7: Model comparison for TerraFlow-Base-TDS using 1 or 2 pre-event timestamps for flood and fire disaster prediction tasks.

Model	PT	TA	F1 NW [%] ↑	F1 PW [%] ↑	F1 Flood [%] ↑	mIoU [%] ↑
UNet-ResNet18	✓	✗	98.7	76.0	79.9	75.1
UNet-ResNet50	✓	✗	98.7	78.2	80.1	76.2
UNet-ResNet101	✓	✗	98.7	79.3	78.9	76.1
DeepLab-ResNet18	✓	✗	98.7	77.4	78.5	75.1
ViT-Base	✗	✓	98.6 ± 0.1	74.9 ± 0.5	76.6 ± 0.9	73.1 ± 0.5
TerraMind 1.0 Base	✓	✗	98.6 ± 0.1	75.7 ± 1.2	77.4 ± 1.1	73.8 ± 0.9

Appendix

Table 8: Comparison of the Kuro Siwo segmentation task against the best model results reported in the original paper [ 5 ] , together with TerraFlow and TerraMind variants. Best values are shown in bold , second-best values are underlined .

Model	PASTIS	DEN
TerraMind 1.0 Base (GEO-Bench-2)	$42.2\pm 0.3$	$23.3\pm 2.2$
TerraMind 1.0 Large (GEO-Bench-2)	$43.1\pm 0.1$	$34.5\pm 0.6$
TerraMind 1.0 Base	$54.3\pm 0.5$	$35.8\pm 1.1$
TerraMind 1.0 Base-Rerun	$54.5\pm 0.3$	$32.0\pm 1.4$

Appendix

Table 9: Comparison between our originally reported TerraMind results in mIoU and a second configuration called Rerun, together with the reported results on GEO-Bench-2, where TerraMind 1.0 Large is the best-performing model inside the GEO-Bench-2 benchmark.

Model	RMSE $\downarrow$
TerraMind 1.0 Tiny	$16.7\pm 0.0$
TerraFlow-Tiny-RS	$16.2\pm 0.0$
TerraFlow-Tiny-TDS	$16.1\pm 0.0$
TerraMind 1.0 Base	$16.3\pm 0.0$
TerraFlow-Base-RS	$15.8\pm 0.0$
TerraFlow-Base-TDS	$15.8\pm 0.0$

Appendix

Table 10: Raw BioMassters results reporting RMSE (Mean

\pm

Std)

Figure 10: Mean runtime per training epoch in inference mode as a function of the number of timesteps

T

. Results are averaged over 50 runs with batch size 8. TerraMind exhibits approximately linear scaling since attention is not applied along the temporal dimension, whereas TerraFlow shows quadratic runtime growth due to temporal attention.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 760 papers

MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation
Oct 13, 2026Carmen Oliver Huidobro, Mikolaj Czerkawski, Kevin Monsalvez-Pozo +3Remote Sensing Geospatial Foundation Models
Image and Signal Processing (ISP) Group, Universitat de València, Valencia, Spain · Asterisk Labs, London, United Kingdom
Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding
Mar 7, 2026Lance Legel, Qin Huang, Brandon Voelker +9World Model Learning Geospatial Representation Learning
Ecological Intelligence Lab Ecodash.ai · School of Complex Adaptive Systems Arizona State University · Geosensing Systems Engineering & Sciences Lab University of Houston +6
SPEAR NeXT Causal Latent Forecasting Across Multiple Horizons for Spectral Temporal Earth Representation Learning
Sep 15, 2026Rajiv Ranjan, Udaiveer Singh, Shashank Tamaskar +1Multivariate Time Series Forecasting Geospatial Foundation Models
Plaksha University Mohali, Punjab, India · Purdue University West Lafayette, Indiana, USA

Oct 13, 2026J/K move · Enter open · S save

MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

Carmen Oliver Huidobro, Mikolaj Czerkawski, Kevin Monsalvez-Pozo, Cesar Aybar+2

Image and Signal Processing (ISP) Group, Universitat de València, Valencia, Spain · Asterisk Labs, London, United Kingdom

Remote Sensing Geospatial Foundation Models Multimodal Pretraining

Open paper AskPDF arXiv

Pretraining generalist Earth observation foundation models requires data that combine several sensors, varied temporal sampling and, increasingly, text, yet open datasets rarely provide all three in one consistent, easily accessible form. We introduce MajorTOM:Elliot Pretrain, a collection of 279,166 tiles of 10 km drawn from the Major-TOM grid by clustering AlphaEarth embeddings. Each tile stacks Sentinel-2 L1C, Landsat-8/9, Sentinel-1 RTC, Copernicus DEM and ESA WorldCover on a shared 1056x1056-pixel, 10 m grid, complemented by cloud masks, OpenStreetMap entities and ERA5 weather. The tiles are partitioned into three temporal regimes, Monotemporal (250,000 tiles, one acquisition), Monthly (12,500 tiles, twelve acquisitions spread over the annual cycle) and Burst (16,666 tiles, six acquisitions within 30 days), and are released as 6.3 TB of Cloud Optimized GeoTIFFs indexed by TACO v3 Parquet metadata, a standard that allows cloud streaming, efficient data access, and filtering before any pixel is read. A companion library derives sensor-tagged fact sheets, captions and 16 families of vision-language tasks from these layers at load time. We test our design choices in short pretraining runs of OlmoEarth-Base. First, a 25% subset of Major-TOM Core-S2L2A curated with AlphaEarth embeddings matches or exceeds the full corpus on 8 of 9 benchmarks. Second, on Elliot-Pretrain, adding Landsat-8/9 improves or matches Sentinel-2 only pretraining on every benchmark. Third, using DEM and WorldCover as prediction targets during pretraining further improves 8 of 9 benchmarks, while giving the same maps to the encoder as inputs degrades 5 of 9. Fourth, Sentinel-1 helps on 7 of 9 benchmarks, particularly on benchmarks with radar inputs. Fifth, rule-based captions match a small language model at no cost. The dataset is openly available at https://source.coop/major-tom/elliot-pretrain.

Abstract

Representation Learning

Explore similar work

MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding

SPEAR NeXT Causal Latent Forecasting Across Multiple Horizons for Spectral Temporal Earth Representation Learning