MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

Carmen Oliver Huidobro; Mikolaj Czerkawski; Kevin Monsalvez-Pozo; Cesar Aybar; Gustau Camps-Valls; Oscar J. Pellicer-Valero

cs.CVOct 13, 2026

MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

Authors: Carmen Oliver Huidobro, Mikolaj Czerkawski, Kevin Monsalvez-Pozo, Cesar Aybar, Gustau Camps-Valls, Oscar J. Pellicer-Valero

Organizations: Image and Signal Processing (ISP) Group, Universitat de València, Valencia, Spain · Asterisk Labs, London, United Kingdom

Abstract

Pretraining generalist Earth observation foundation models requires data that combine several sensors, varied temporal sampling and, increasingly, text, yet open datasets rarely provide all three in one consistent, easily accessible form. We introduce MajorTOM:Elliot Pretrain, a collection of 279,166 tiles of 10 km drawn from the Major-TOM grid by clustering AlphaEarth embeddings. Each tile stacks Sentinel-2 L1C, Landsat-8/9, Sentinel-1 RTC, Copernicus DEM and ESA WorldCover on a shared 1056x1056-pixel, 10 m grid, complemented by cloud masks, OpenStreetMap entities and ERA5 weather. The tiles are partitioned into three temporal regimes, Monotemporal (250,000 tiles, one acquisition), Monthly (12,500 tiles, twelve acquisitions spread over the annual cycle) and Burst (16,666 tiles, six acquisitions within 30 days), and are released as 6.3 TB of Cloud Optimized GeoTIFFs indexed by TACO v3 Parquet metadata, a standard that allows cloud streaming, efficient data access, and filtering before any pixel is read. A companion library derives sensor-tagged fact sheets, captions and 16 families of vision-language tasks from these layers at load time. We test our design choices in short pretraining runs of OlmoEarth-Base. First, a 25% subset of Major-TOM Core-S2L2A curated with AlphaEarth embeddings matches or exceeds the full corpus on 8 of 9 benchmarks. Second, on Elliot-Pretrain, adding Landsat-8/9 improves or matches Sentinel-2 only pretraining on every benchmark. Third, using DEM and WorldCover as prediction targets during pretraining further improves 8 of 9 benchmarks, while giving the same maps to the encoder as inputs degrades 5 of 9. Fourth, Sentinel-1 helps on 7 of 9 benchmarks, particularly on benchmarks with radar inputs. Fifth, rule-based captions match a small language model at no cost. The dataset is openly available at https://source.coop/major-tom/elliot-pretrain.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of MajorTOM:Elliot-Pretrain. The 279,166 tiles are selected from the land grid of the Major-TOM 10 km grid by spherical $k$ -means clustering of AlphaEarth embeddings (map: Monotemporal tiles). Each tile holds co-registered rasters on a $1056\times 1056$ -pixel, 10 m grid, its OpenStreetMap entities and administrative units, and tables with ERA5 weather at each acquisition, acquisition geometry and environmental and socio-economic variables. The example is a savanna tile in Simanjiro, Tanzania, shown with its cloud mask and OSM geometry. Each tile belongs to one of three temporal regimes. Captions and the 16 task families are generated from the tile’s layers at load time, and a task is generated only if its layers are given to the model. The caption and answers are abridged; text colours give the layer each fact comes from.

	Major-TOM	SSL4EO-S12	Satlas Pretrain	Terra Mesh	Elliot-Pretrain (ours)
Modalities
S2 L2A	✓	✓	✗	✓	✗
S2 L1C	✓	✓	✓	✓	✓
S1 GRD	✗	✓	✓	✓	✗
S1 RTC	✓ ⋆	✗	✗	✓	✓
Landsat (L8/9)	✗	✗	✓	✗	✓

Table 1: Comparison of large-scale EO pretraining datasets. ✓ present; ✗ absent; ✓ ⋆ partial coverage; † described for TerraMind Jakubik and others (2025) but not part of the public TerraMesh release.

Regime	Tiles	Share	Acquisitions per tile	Selection of dates
Monotemporal	250,000	89.6%	1	least cloudy available
Monthly	12,500	4.5%	12	greedy, maximal circular day-of-year spread
Burst	16,666	6.0%	6	greedy spread within the best 30-day window
Total	279,166	100%

Table 2: The three subdatasets, one per temporal regime. Each of the 279,166 selected tiles belongs to exactly one regime. Acquisitions are per sensor.

Figure 2: Example tiles across seven landscape types (one per row), with all co-registered layers and the per-tile context. Sentinel-2 L1C: RGB, true colour (bands B4, B3, B2); NIR, false-colour near-infrared (B8, B4, B3), in which vegetation appears red; SWIR, false-colour shortwave infrared (B12, B8, B4); cloud mask, OmniCloudMask classes. Landsat-8: RGB, true colour (B4, B3, B2); TIR, thermal infrared (band 10), a brightness temperature; cloud mask as above. Sentinel-1 RTC: radiometrically terrain-corrected backscatter in dB, with VV (red), VH (green) and VV $-$ VH (blue) as radar polarisations. Other layers: WorldCover, ESA land-cover map; DEM, Copernicus GLO-30 digital elevation model; NDVI, normalised difference vegetation index of Sentinel-2, $(\mathrm{B8}-\mathrm{B4})/(\mathrm{B8}+\mathrm{B4})$ ; vector map, OpenStreetMap features of the tile. Thermal, radar and elevation colours are stretched per tile, and class colours are given in the legends. Tile context: tile-centre latitude and longitude, Sentinel-2 acquisition date and place; ERA5 (ECMWF reanalysis) 2 m air temperature, relative humidity (RH) and 10 m wind speed at that date; Global Human Modification index (Human mod., 0 to 1); Critical Infrastructure Spatial Index (CISI); and the share of each WorldCover class in the tile (coloured bar).

Path	Monotemporal	Monthly / Burst	Content
<regime>/
COLLECTION.json	TACO collection description, column descriptions, band and radiometry contract
METADATA/
sample.parquet	One row per tile: grid codes, tile-level context, geometry, OSM counts, WorldCover and DEM statistics, cloud and Sentinel-1 summaries
children.parquet	One row per top-level entry of a tile (layer folders, osm.parquet , admin.parquet )
children__<layer>.parquet	One row per file (s2, l8, s1, cloud_s2, cloud_l8, dem, lc): dates, product, scores, geometry, ERA5, mask class fractions, Sentinel-1 pass

Table 3: Directory layout of the collection. The three sub-datasets monotemporal/ , monthly/ and burst/ share it; only the file names inside a tile’s layer folders differ. Monotemporal stores one acquisition per layer, Monthly and Burst one file per acquisition, with NN running over 00–11 (Monthly) or 00–05 (Burst). Static layers (DEM, land cover) are stored once.

Figure 3: Tile locations spread across the earth for the three temporal regimes

	Monotemporal	Monthly	Burst
Tiles	250,000	12,500	16,666
Images per sensor	250,000	150,000	99,996
Sentinel-2 acquisition years	2015–2026	2015–2026	2017–2025
Cloud fraction (masks), median	0.00	0.19	0.07
Sentinel-2 cloudiness bins (%)	100 / 0 / 0 / 0 / 0	74 / 6 / 8 / 3 / 10	85 / 5 / 5 / 1 / 3
Landsat-8 cloudiness bins (%)	94 / 3 / 3 / 0 / 0	69 / 8 / 10 / 4 / 9	75 / 11 / 11 / 3 / 0

Table 4: Statistics of the released regimes. Cloudiness bins are the share of images with cloudiness (1

-

clear score) in

<

0.1 / 0.1–0.3 / 0.3–0.7 / 0.7–0.9 /

>

0.9.

Figure 4: OlmoEarth Base on Core-S2L2A: downstream scores at matched optimizer steps for the full corpus (100%), a random 25% subset, a curated 25% subset and 512 random tiles, all trained for ten epochs (42,660 steps) in two parts; the dashed line marks step 21,330, where training was resumed with a restarted learning-rate schedule. The vertical axes of the two RMSE panels (Open-Canopy, BioMassters) are inverted so that up is better in every panel.

Figure 5: Change in downstream score at step 488 (OlmoEarth Base, Elliot-Pretrain, frozen encoder) relative to (a) the S2-only model and (b) the monotemporal S2+L8+DEM+WC model. Positive is better; RMSE tasks are sign-flipped. Note that dec = decoder-only target: the encoder never sees it and the decoder predicts it from the visible Sentinel-2 and Landsat patches; enc = encoder input: DEM and WorldCover are given to the encoder like extra sensor bands. Under each column: number of time steps T given to the model. Diamond = the model has a Sentinel-1 pathway and this cell is scored with Sentinel-1 + Sentinel-2 input (PASTIS, TreeSatAI, BioMassters); all other cells use optical input. The reference is the S2-only model in (a) and the monotemporal S2+L8+DEM(dec)+WC(dec) model in (b), each scored on its own optical input.

	Rule-based template (no LLM)	Small LLM (GPT-6-luna) fact sheet	Large LLM (GPT-6.1-sol) fact sheet	Large LLM (GPT-6.1-sol) fact sheet + image
Accuracy	8.74 [8.51, 8.95]	8.73 [8.54, 8.92]	9.39 [9.27, 9.51]	9.32 [9.19, 9.45]
Coverage	8.11 [7.91, 8.29]	6.81 [6.65, 6.96]	8.73 [8.60, 8.86]	8.45 [8.31, 8.59]
Salience	6.66 [6.51, 6.80]	6.92 [6.77, 7.06]	7.84 [7.71, 7.97]	7.96 [7.84, 8.07]
Spatial	6.92 [6.54, 7.30]	6.45 [6.09, 6.78]	7.42 [7.03, 7.81]	7.45 [7.09, 7.82]
Fluency	5.86 [5.72, 6.01]	7.38 [7.24, 7.51]	7.87 [7.80, 7.94]	8.02 [7.95, 8.09]
Mean of 5	7.26 [7.10, 7.41]	7.26 [7.13, 7.39]	8.25 [8.12, 8.38]	8.24 [8.12, 8.35]

Table 5: Caption quality on 100 stratified MajorTOM:Elliot Pretrain tiles. All captioners start from the same per-tile fact sheet. A blind judge (Claude Opus 5.5) saw the tile’s image, the fact sheet and the four captions in random order. It scored each caption on five axes (0–10), listed its factual errors, and ranked the four. Values are means over tiles, with bootstrap 95% confidence intervals in brackets. Costs are the measured API cost of these 100 captions (OpenRouter), scaled linearly. Best value per row in bold.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

	Major-TOM	MajorTOM:Elliot Pretrain
Storage format	GeoTIFFs embedded in Parquet columns	Cloud-Optimised GeoTIFF via TACO
Row groups	One row per row group	Up to 10,000 rows per row group
Band handling	One column per band, mixed resolutions	All bands at 10 m in a single COG
Pixel dimensions	Variable, not aligned to 32	1056 $\times$ 1056
Georeferencing	Per-band, mixed grids	Single MGRS-aligned CRS + affine
Compression	LZW (CPU only)	ZSTD + PREDICTOR=2 (3 for float layers) + INTERLEAVE=BAND

Appendix

Table 6: Summary of changes between Major-TOM and MajorTOM:Elliot Pretrain.

Sensor	Bands	Dtype	Predictor	DISCARD_LSB
Sentinel-2 L1C	13	UInt16	2	3 (all bands)
Landsat-8/9 OLI-TIRS	11	UInt32	2	3 $\times$ 9 (OLI), 10 $\times$ 2 (TIRS)
Copernicus DEM GLO-30	1	Int16 / Float32	2 / 3	2 / none
ESA WorldCover v200	1	UInt8	1	none (categorical)
Sentinel-1 RTC	2	Float32	3	mantissa truncated to 12 bits

Appendix

Table 7: Per-sensor compression parameters. All COGs use ZSTD and

352\times 352

-px internal tiles. For the elevation layer the first value is Monotemporal and the second Monthly and Burst; the Burst Landsat-8/9 rasters carry no bit trim.

Level	Temporal tolerance	Score requirement
N1	$\pm 8$ days	$>0.8$
N2	$\pm 16$ days	$>0.6$
N3	$\pm 30$ days	$>0.4$
N4	any date	none

Appendix

Table 8: Landsat-8/9 matching cascade. The first level with a candidate wins.

Family	Answer shape	Requires
caption	prose	–
dominant_cover	word	WorldCover
which_present	choice	–
locate_point , locate_bbox	point, box	–
features_all	several points	–
ndvi_grid	grid	S2

Appendix

Table 9: The 16 task families of majortom-elliot-tasks . “Requires” lists the layers that must be available to the model; series means a stack of more than one acquisition (Monthly, Burst); – means the family is built from the metadata and OpenStreetMap alone.

Table	Columns	Content
tile ( sample )	majortom:code_10km , code_100km , code_1000km	hierarchical grid codes of the tile
	terrain:elevation	mean elevation (m), Copernicus GLO-30
	climate:precipitation , climate:temperature	mean annual precipitation (GPM, m/year), mean annual land surface temperature (MODIS, K)
	soil:clay , sand , carbon , bulk_density , ph	surface soil properties as encoded by OpenLandMap: carbon in units of 5 g/kg, bulk density in units of 10 kg/m 3 , pH $\times 10$ ; null where no soil data exist (sea, Antarctica, Greenland)
	socio:gdp , population , human_modification , cisi	GDP (PPP, 2022; not per capita, aggregation undocumented), people per cell (Meta HRSL), Global Human Modification index, Critical Infrastructure Spatial Index
	admin:country , state , district	administrative names

Appendix

Table 10: Metadata columns of the release.

Check	Scope	Result
Sentinel-1 pairing date equals Sentinel-2 date	499,996 images	100%
Same CRS, geotransform and shape for all five rasters	300 tiles	100%
Stored geotransform equals file header	300 tiles	100%
Metadata nodata equals measured (Sentinel-2, Sentinel-1)	300 tiles	max. diff. 0
Valid WorldCover codes (Monotemporal; 0 = no data in Monthly, Burst)	300 tiles	100%
Sentinel-2/Landsat-8/9 shift within 1 / 3 px	300 tiles	81% / 89%

Appendix

Table 11: Validation summary. Metadata checks cover every tile, pixel checks 300 random Monotemporal tiles.

	Experiment 1 (data volume, curation)	Experiment 2 (modalities, partitions)
Pretraining data	Core-S2L2A: all 2,184,259 tiles; random 25% (546,064 tiles, seed 0, no replacement); curated 25% (the first 546,064 tiles of the AlphaEarth ordering, see below); 512 random tiles (seed 0)	Elliot-Pretrain: 250,000 Monotemporal tiles (modality ladder); monthly+burst (29,166 tiles), mono+monthly (262,500), mono+monthly+burst (279,166)
Encoder input	Sentinel-2 L2A, 12 bands	Sentinel-2 L1C, 12 bands, plus Landsat-8/9 and Sentinel-1 as stated per run (Table 18 )
Normalization	z-score, Core-S2L2A band statistics	z-score, Elliot-Pretrain band statistics (Sentinel-1: dB statistics)
Steps	42,660 (10 epochs of 4,266), in two parts of 21,330; the 25% subsets are repeated within an epoch	488 (Monotemporal and monthly+burst), 512 (mono+monthly), 545 (mono+monthly+burst)
Learning-rate schedule	Cosine, warmup 1.2% of the run; part two resumes from step 21,330 with the schedule recomputed over 42,660 steps (the rate jumps back to the cosine midpoint)	Cosine, warmup 1.2% of the run
Hardware, microbatch	one A100-80GB, 64, 16 dataloader workers (2 for the 512-tile run)	four GPUs (A30-24GB, or RTX PRO 6000 Blackwell for the Sentinel-1 runs), 32 per GPU, 8 workers per GPU

Appendix

Table 12: Configuration of the two pretraining experiments.

Model
Architecture	Latent masked image modelling, OlmoEarth Base encoder (88.8M parameters with Sentinel-2 only; 92.0M with Landsat-8, 95.1M with DEM or Sentinel-1 added, 98.3M with DEM and WorldCover, 101.4M with Sentinel-1, DEM and WorldCover) with a shallow predictor (decoder depth 4)
Patch size	1–8 (sampled per batch), stochastic depth 0.1
Position encoding	Mixed 3D RoPE (base 10,000, temporal coordinate scale $1/30$ , time in calendar days since 2000)
Band dropout	Random, maximum rate 0.2, on Sentinel-2 and Landsat
Tokenization	One band group for Sentinel-2 (12 bands, B10 dropped) and for Landsat (11 bands)
Optimization

Appendix

Table 13: Pretraining hyperparameters shared by both experiments (official v1.2 Base recipe).

Input	Role	Preparation	Missing data
Sentinel-2	encoder	Elliot-Pretrain: L1C top-of-atmosphere, Core-S2L2A: L2A; both go through the same Sentinel-2 slot, with B10 dropped (12 bands) and no conversion between levels; per-band z-score	none
Landsat-8/9	encoder	11 bands (9 reflectance, 2 brightness temperature), per-band z-score	an image with $\geq 90\%$ no data is treated as absent
Sentinel-1	encoder	VV and VH, linear power to dB ( $10\log_{10}$ , floor $10^{-6}$ ), per-band z-score with the dB statistics	no-data pixels are set to the band mean (zero after z-scoring); the 6.6% of tiles without a pass keep an image that is entirely mean-filled, no image is borrowed from another tile
DEM	decoder target (encoder input in the enc run)	Copernicus GLO-30 in the elevation slot, per-band z-score	none
WorldCover	decoder target (encoder input in the enc run)	ESA WorldCover v200 class codes, normalized as in OlmoEarth (categorical)	none
Time	all	timestamps $(\mathrm{day},\mathrm{month}-1,\mathrm{year})$ give a month embedding and the calendar-day coordinate of the 3D RoPE	padded steps masked

Appendix

Table 14: How each pretraining input is prepared. Statistics are those of the pretraining corpus (Appendix K ).

Benchmark (classes)	Input	Bands imputed	Time steps	Probe	Metric
So2Sat (17)	S2, 10 bands	B01 from B02, B09 from B8A, B10 from B11	1	KNN, $k=20$	accuracy
EuroSAT (10)	S2, 13 bands (L1C)	none (B10 dropped)	1	KNN, $k=20$	accuracy
BrickKiln (2)	S2, 13 bands (L1C)	none (B10 dropped)	1	KNN, $k=20$	accuracy
BigEarthNet (43)	S2, 12 bands (L2A)	none	1	KNN, $k=20$	micro-F1
TreeSatAI (15, multi-label)	S2 12 bands; S1 VV, VH	none; S1 ratio band dropped	1	KNN, $k=20$	macro-F1; 10000/2000/5044
PASTIS (19)	S2 (and S1) time series	as in the OlmoEarth loader	12	linear, lr 0.1	mIoU

Appendix

Table 15: Benchmarks. “Bands imputed” lists bands copied from a neighbouring band. Caps are the numbers of train/validation/test samples used when the split is subsampled (fixed seeded draw, identical for all models).

Benchmark	Random 25% $-$ 100%	512 tiles $-$ 100%	Curated 25% $-$ 100%	Curated $-$ random 25%
So2Sat (Acc.)	+0.49 [-0.02, +0.99] ( $p$ =0.056)	-1.30 [-1.77, -0.83]** ( $p$ =0.000)	+0.59 [+0.02, +1.17]* ( $p$ =0.043)	+0.11 [-0.58, +0.79] ( $p$ =0.729)
PASTIS (mIoU)	-1.17 [-1.75, -0.59]** ( $p$ =0.001)	+0.04 [-0.29, +0.38] ( $p$ =0.789)	-3.15 [-4.46, -1.83]** ( $p$ =0.000)	-1.98 [-3.46, -0.49]* ( $p$ =0.015)
EuroSAT (Acc.)	+0.24 [-0.16, +0.64] ( $p$ =0.208)	-1.38 [-1.97, -0.79]** ( $p$ =0.000)	+0.47 [-0.16, +1.10] ( $p$ =0.126)	+0.23 [-0.08, +0.54] ( $p$ =0.129)
MADOS (mIoU)	-0.77 [-1.19, -0.34]** ( $p$ =0.003)	-3.93 [-5.20, -2.67]** ( $p$ =0.000)	+2.32 [+1.78, +2.87]** ( $p$ =0.000)	+3.09 [+2.35, +3.83]** ( $p$ =0.000)
BrickKiln (Acc.)	-0.56 [-0.92, -0.20]** ( $p$ =0.006)	+0.89 [-0.04, +1.83] ( $p$ =0.059)	+0.06 [-0.40, +0.52] ( $p$ =0.773)	+0.62 [+0.28, +0.96]** ( $p$ =0.003)
BigEarthNet ( $\mu$ F1)	+0.32 [+0.02, +0.62]* ( $p$ =0.041)	-0.02 [-0.49, +0.46] ( $p$ =0.941)	+0.99 [+0.71, +1.28]** ( $p$ =0.000)	+0.68 [+0.41, +0.94]** ( $p$ =0.000)

Appendix

Table 16: Paired comparison of each data size with the full corpus (100%) over the ten checkpoints of the subset-size experiment (steps 4,266 to 42,660). Each entry is the mean difference to the 100% run at the same checkpoint (checkpoint scores averaged over three evaluation seeds), its 95% confidence interval (t distribution, 9 degrees of freedom) and the two-sided p-value of a paired t-test; *

p<0.05

, **

p<0.01

, without correction for the 36 comparisons. The last column compares the curated and the random 25% subsets directly (same size and steps). For the RMSE rows a negative difference is better. Checkpoints of one run are correlated and every run was pretrained once, so the test describes the training trajectory and does not include pretraining-seed variation.

Benchmark	100%	Random 25%	512 tiles	Curated 25%
So2Sat (Acc.)	37.73 $\pm$ 0.00 [37.73, 37.73]	38.74 $\pm$ 0.00 [38.74, 38.74]	37.53 $\pm$ 0.00 [37.53, 37.53]	38.74 $\pm$ 0.00 [38.74, 38.74]
PASTIS (mIoU)	31.22 $\pm$ 0.10 [30.98, 31.46]	29.89 $\pm$ 0.10 [29.65, 30.14]	31.97 $\pm$ 0.07 [31.79, 32.16]	30.21 $\pm$ 0.08 [30.02, 30.40]
EuroSAT (Acc.)	79.60 $\pm$ 0.00 [79.60, 79.60]	79.60 $\pm$ 0.00 [79.60, 79.60]	77.40 $\pm$ 0.00 [77.40, 77.40]	79.90 $\pm$ 0.00 [79.90, 79.90]
MADOS (mIoU)	51.81 $\pm$ 0.19 [51.33, 52.28]	50.55 $\pm$ 0.20 [50.07, 51.04]	45.23 $\pm$ 0.07 [45.05, 45.42]	53.74 $\pm$ 0.10 [53.49, 53.98]
BrickKiln (Acc.)	85.79 $\pm$ 0.00 [85.79, 85.79]	85.69 $\pm$ 0.00 [85.69, 85.69]	86.79 $\pm$ 0.00 [86.79, 86.79]	86.49 $\pm$ 0.00 [86.49, 86.49]
BigEarthNet ( $\mu$ F1)	49.22 $\pm$ 0.00 [49.21, 49.23]	49.57 $\pm$ 0.00 [49.57, 49.57]	49.15 $\pm$ 0.00 [49.15, 49.15]	50.48 $\pm$ 0.00 [50.48, 50.48]

Appendix

Table 17: Score at the last checkpoint (step 42,660): mean

\pm

standard deviation over three evaluation seeds (3622, 8956, 30991) and the 95% confidence interval of the mean (t distribution, 2 degrees of freedom) in brackets. The encoder is fixed, so this interval covers only the refitting of the probe or nearest-neighbour classifier, not pretraining-seed variation. Nearest-neighbour benchmarks give identical scores for all seeds.

Model	Other encoder inputs	Decoder targets	Pool	Steps
S2 only	—	—	Mono	488
S2+L8	L8	—	Mono	488
S2+L8+DEM (dec)	L8	DEM	Mono	488
S2+L8+DEM (dec)+WC (dec)	L8	DEM, WC	Mono	488
S2+L8+DEM (enc)+WC (enc)	L8, DEM, WC	—	Mono	488
S2+L8+S1	L8, S1	—	Mono	488

Appendix

Table 18: Pretrained models of Experiment 2. All use Sentinel-2 as an encoder input. dec = decoder-only target; enc = encoder input.

Pretraining run	Step	So2Sat	PASTIS	EuroSAT	MADOS	BrickKiln	BigEarthNet	TreeSatAI	Canopy	BioM
S2 only	488	30.4	26.0	69.0	37.6	81.6	41.1	17.1	6.22	55.13
S2+L8	488	31.2	28.5	70.7	39.5	86.5	42.5	17.1	6.23	54.22
S2+L8+DEM (dec)	488	29.3	25.2	69.4	36.6	83.9	41.0	17.1	6.31	55.20
S2+L8+DEM (dec)+WC (dec)	488	29.9	29.9	71.6	40.2	86.9	43.4	17.7	6.10	52.81
S2+L8+DEM (enc)+WC (enc)	488	26.1	23.0	69.6	33.6	81.2	40.6	17.3	6.43	54.95
S2+L8+S1	488	29.4	23.9	67.1	36.2	82.9	41.3	17.0	6.33	54.59

Appendix

Table 19: Scores behind Figure 5 with optical input, at step 488 of each run. Test split, evaluation seed 3622; 0–100 scale, RMSE for Open-Canopy (m) and BioMassters (Mg ha -1 ). dec = decoder-only target, enc = encoder input.

Pretraining run	Step	PASTIS	TreeSatAI	BioM
S2+L8+S1	488	26.1	18.7	52.02
S2+L8+S1+DEM (dec)	488	25.8	18.4	52.17
S2+L8+S1+DEM (dec)+WC (dec)	488	27.1	19.0	51.32

Appendix

Table 20: Scores of the Sentinel-1 models with Sentinel-1 input (Sentinel-1 and Sentinel-2 together), same checkpoints and protocol as Table 19 . These are the diamond cells of Figure 5 .

Explore similar work

Mar 13, 2026

TerraFlow: Multimodal, Multitemporal Representation Learning for Earth Observation

We propose TerraFlow, a novel approach to multimodal, multitemporal learning for Earth observation. TerraFlow builds on temporal training objectives that enable sequence-aware learning across space, time, and modality, while remaining robust to the variable-length inputs commonly encountered in real-world Earth observation data. Our experiments demonstrate superiority of TerraFlow over state-of-the-art foundation models for Earth observation across all temporal tasks of the GEO-Bench-2 benchmark. We additionally demonstrate that TerraFlow is able to make initial steps towards deep-learning based risk map prediction for natural disasters -- a task on which other state-of-the-art foundation models frequently collapse. TerraFlow outperforms state-of-the-art foundation models by up to 50% in F1 score and 24% in Brier score.

Nazar Puriy, Johannes Jakubik, Benedikt Blumenstiel +1

IBM Research Europe, Zurich, Switzerland · ETH Zurich, Zurich, Switzerland

Representation Learning Geospatial Foundation Models Multimodal Learning

Ask PDF

May 20, 2026

SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining

Earth observation (EO) foundation models (FMs) are increasingly trained on multisensor data, spanning multispectral imagery (MSI), synthetic aperture radar (SAR), and derived geospatial layers, but hyperspectral imagery (HSI) remains underrepresented. Conversely, existing hyperspectral FMs are trained on HSI alone, leaving joint pretraining and fusion of HSI with co-located EO sensors unexplored. We introduce SpectralEarth-FM, a hierarchical transformer for multisensor EO input with heterogeneous spectral dimensionality. The architecture combines spectral tokenization for hyperspectral inputs, sensor-specific encoders, a cross-sensor fusion module, and a shared hierarchical encoder, enabling joint processing of HSI and lower-channel observations. To pretrain SpectralEarth-FM, we curate SpectralEarth-MM, a dataset that co-locates HSI from three spaceborne sensors (EnMAP, EMIT, DESIS) with Sentinel-2, Landsat-8/9 optical imagery, Landsat land surface temperature (LST), and Sentinel-1 SAR, over common geographic footprints. It comprises approximately 2M globally distributed locations, 25M georeferenced patches, and over 40TB of data. Pretraining uses a Joint-Embedding Predictive Architecture (JEPA)-style objective that matches representations between global views and single-sensor local views from the same location. We evaluate SpectralEarth-FM on hyperspectral downstream tasks and standard EO benchmarks following the PANGAEA protocol, achieving state-of-the-art results across both evaluation settings.

Nassim Ait Ali Braham, Aaron Banze, Conrad M. Albrecht +3

Chair of Data Science in Earth Observation, Technical University of Munich, Germany · Remote Sensing Technology Institute, German Aerospace Center (DLR), Germany · Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany +3

Geospatial Foundation Models Multimodal Pretraining Hyperspectral Imaging Multimodal Sensor Fusion

Ask PDF

Mar 31, 2026

BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation

Vision-langugage models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text datasets with diverse textual annotations. Existing datasets predominantly include aerial Red-Green-Blue imagery, with short or weakly grounded captions, and provide limited diversity in annotation types. To address this limitation, we introduce BigEarthNet

.

txt, a large-scale, multi-sensor image-text dataset designed to advance instruction-driven image-text learning in Earth observation across multiple tasks. BigEarthNet

.

txt contains 464044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9.6M text annotations, including: i) geographically anchored captions describing land-use/land-cover (LULC) classes, their spatial relations, and environmental context; ii) visual question answering pairs relevant for different tasks; and iii) referring expression detection instructions for bounding box prediction. Through a comparative statistical analysis, we demonstrate that BigEarthNet

.

txt surpasses existing RS image-text datasets in textual richness and annotation type variety. We further establish a manually-verified benchmark split to evaluate VLMs in RS and CV. The results show the limitations of these models on tasks that involve complex LULC classes, whereas fine-tuning using BigEarthNet

.

txt results in consistent performance gains across all considered tasks.

Johann-Ludwig Herzog, Mathis Jürgen Adler, Leonard Hackel +5

BIFOLD and Technische Universität Berlin · University of Trento · National Technical University and National Observatory of Athens

Remote Sensing Remote Sensing Benchmarks Remote Sensing Visual Grounding Remote Sensing VQA

Ask PDF