GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing

Maram Hasan; Md Aminur Hossain; Savitra Roy; Souparna Bhowmik; Ayush V. Patel; Mainak Singha; Subhasis Chaudhuri; Muhammad Haris Khan

cs.CVApr 12, 2026

GeoMeld: Toward Semantically Grounded Foundation Models for Remote Sensing

Authors: Maram Hasan, Md Aminur Hossain, Savitra Roy, Souparna Bhowmik, Ayush V. Patel, Mainak Singha, Subhasis Chaudhuri, Muhammad Haris Khan, +1 more

Organizations: Indian Institute of Technology Bombay · Space Applications Centre, ISRO · University of Trento · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Effective foundation modeling in remote sensing requires spatially aligned heterogeneous modalities coupled with semantically grounded supervision, yet such resources remain limited at scale. We present GeoMeld, a large-scale multimodal dataset with approximately 2.5 million spatially aligned samples. The dataset spans diverse modalities and resolutions and is constructed under a unified alignment protocol for modality-aware representation learning. GeoMeld provides semantically grounded language supervision through an agentic captioning framework that synthesizes and verifies annotations from spectral signals, terrain statistics, and structured geographic metadata, encoding measurable cross-modality relationships within textual descriptions. To leverage this dataset, we introduce GeoMeld-FM, a pretraining framework that combines multi-pretext masked autoencoding over aligned modalities, JEPA representation learning, and caption-vision contrastive alignment. This joint objective enables the learned representation space to capture both reliable cross-sensor physical consistency and grounded semantics. Experiments demonstrate consistent gains in downstream transfer and cross-sensor robustness. Together, GeoMeld and GeoMeld-FM establish a scalable reference framework for semantically grounded multi-modal foundation modeling in remote sensing.

Figures & tables

Dataset	Modalities	Scale	Resolution	Tile Size	Meta-data	Foundation Model
MMEarth	S1, S2, Elevation, Slope, ETH-GCHM, Dynamic World, ESA WorldCover, Biome, Ecoregion, ERA5, Geo, Date	1.2M locations	10m	128×128	✓	MP-MAE
Skyscript	RGB + Text	2.6M	0.1m–30m	Variable	✓	SkyCLIP
Skysense	RGB, 10-band MS, SAR (VV/VH)	21.5M	0.3m, 10m	2048×2048 64×64	✓	SkySense
RS5M	RGB, S1, S2 (12-band), MS, Text	5M	0.5m–30m	Variable	✓	GeoRSCLIP
ChatEarthNet	S2, ESA WC 2020, Text	173k	10m	256×256	✓	–
Satlas Pretrain	S2, NAIP	856k	1m / 10m	8192×8192, 512×512	✓	SatlasNet

Table 1: Comparison of major multi-modal and vision-language Earth Observation datasets.

Panel of Figure 1 — Figure 1: (a) Global spatial distribution of datapoints presented in 1°x1°cell, representing the number of tiles in 10.68 × 10.68 $km^{2}$ . (b) Land-cover distribution in GeoMeld showing pixel-wise coverage, dominant (top-1) class, and top-3 classes per tile, highlighting broad biome diversity under landscape-scale sampling.

Figure 2: Agentic framework for generating semantically grounded captions. An Orchestrator aggregates modality-specific signals and metadata, a Captioner produces multiple candidate descriptions, an Evaluator ranks them via vision–text alignment, and a Verification agent enforces consistency with structured geospatial attributes. Comparision of generated captions against captions of other dataset (Right).

Figure 3: A sample from GeoMeld showing spatially aligned multi-modal inputs derived through a unified alignment protocol.

Figure 4: Temporal distribution of GeoMeld samples, showing the number of samples per month and overall seasonal coverage

Figure 5: GeoMeld-FM pretraining architecture. Sentinel-2 (12-band) tiles are patchified and randomly masked, and visible patches are encoded by a ConvNeXtV2 MAE encoder to produce latent tokens. The same masking pattern is used for the JEPA context encoder, while a separate non-overlapping mask forms the JEPA target view. Lightweight modality-specific decoders (MP-MAE) reconstruct or predict aligned modalities, including Sentinel-2, Sentinel-1, DEM, canopy height, and land-cover. In parallel, a JEPA branch forms context and target views of the same Sentinel-2 tile; a predictor maps the context to the target representation from an EMA teacher. For language grounding, captions are encoded and aligned with pooled vision embeddings using a symmetric contrastive (InfoNCE) objective. The model is trained jointly with three losses: MP-MAE reconstruction, S2-JEPA predictive loss, and caption-vision contrastive alignment.

Pretrain data	BigEarthNet20k (F1 $\uparrow$ ) FT/LP	So2Sat20k (Acc. $\uparrow$ ) FT/LP	Cashew1k (IoU $\uparrow$ ) FT	SAcrop3k (IoU $\uparrow$ ) FT
ImageNet	55.7 / 25.9	36.6 / 24.0	77.1	26.7
MMEarth (MP-MAE)	67.1 / 43.3	54.6 / 43.8	79.8	38.2
GeoMeld-S2	66.8 / 38.3	50.8 / 35.6	80.8	37.5
GeoMeld-FM (Full)	71.8 / 49.6	59.8 / 50.2	83.2	42.7

Table 2: Downstream evaluation results on GeoBench dataset. FT = full fine-tuning, LP = linear probing.

Variant	MP	JEPA	ITC	BigEarth20k	So2Sat20k	Cashew1k	SAcrop3k	I $\rightarrow$ T	T $\rightarrow$ I
				F1 $\uparrow$ (LP)	Acc. $\uparrow$ (LP)	IoU $\uparrow$ (FT)	IoU $\uparrow$ (FT)	R@5 $\uparrow$	R@5 $\uparrow$
S2-MAE baseline	$\times$	$\times$	$\times$	38.3	35.6	80.8	37.5	–	–
+ MP-MAE	$\checkmark$	$\times$	$\times$	44.8	45.2	81.3	39.6	–	–
+ MP-MAE + JEPA	$\checkmark$	$\checkmark$	$\times$	46.1	46.9	82.4	40.8	–	–
+ MP-MAE + ITC	$\checkmark$	$\times$	$\checkmark$	45.4	45.8	81.9	40.1	31.4	33.8
GeoMeld-FM (Full)	$\checkmark$	$\checkmark$	$\checkmark$	49.6	50.2	83.2	42.7	37.8	39.6

Table 3: Ablation study of GeoMeld-FM. MP = cross-modal multi-pretext reconstruction, LP = linear probing. Retrieval is reported as Recall@5 (R@5) for both image-to-text (I

\rightarrow

T) and text-to-image (T

\rightarrow

I) retrieval.

Explore similar work

FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales
May 27, 2026Jorge L. Rodriguez, Victor Angulo Morales, Areej Alwahas +6Remote Sensing Image Segmentation Geospatial Foundation Models
Biological and Environmental Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks
Jun 9, 2026Miaoxin Cai, Guanqun Wang, Wei Zhang +6Remote Sensing Image Understanding Multimodal Grounding
National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (SBIIP), Beijing Institute of Technology, Beijing 100081, China · Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China · School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China +5
Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing
Sep 8, 2026Paolo Fraccaro, Gabby Nyirjesy, Daniela Szwarcman +19Multimodal Foundation Models Remote Sensing
IBM Research, Yorktown Heights, NY, USA · The University of Alabama in Huntsville, Huntsville, AL, USA · University Space Research Association (USRA), Science and Technology Institute, Huntsville, AL, USA +7

May 27, 2026J/K move · Enter open · S save

FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales

Jorge L. Rodriguez, Victor Angulo Morales, Areej Alwahas, Mariana Elias Lara+5

Biological and Environmental Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia · Computer, Electrical and Mathematical Science and Engineering Division, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia

Remote Sensing Image Segmentation Geospatial Foundation Models Multimodal Pretraining Remote Sensing

Open paper AskPDF arXiv

Foundation models offer a promising route to transferable remote sensing representations, but many current approaches depend on very large pretraining datasets and fixed sensor configurations, limiting their suitability for ecological and environmental applications, where observations often vary across platforms, spatial and spectral resolutions, and available modalities. We introduce FLORO, a multimodal geospatial foundation model designed to learn transferable representations from a small but highly diverse remote sensing corpus. FLORO is pretrained using masked autoencoding on a heterogeneous combination of Sentinel-1, Sentinel-2, SkySAT imagery, elevation, and UAV-derived data. To accommodate sensor variability, FLORO incorporates availability-aware inputs that indicate which spectral bands and auxiliary modalities are present in each sample, enabling a unified input space across heterogeneous sensor configurations. We evaluated FLORO on the PANGAEA benchmark under a frozen-encoder protocol across scene classification, segmentation, and regression tasks. Despite being pretrained on a smaller corpus than competing foundation models, FLORO achieved strong and stable transfer across optical, optical-SAR, and optical-elevation benchmarks spanning medium-resolution satellite, airborne, and ultra-high-resolution UAV imagery. FLORO obtained the second-best average segmentation performance across six PANGAEA benchmarks, trailing only a recently introduced foundation model pretrained on over two orders of magnitude more images, remained competitive on scene classification, and was robust in regression tasks, while qualitative results showed improved preservation of spatial structure in flood, urban, biomass, and canopy-height prediction settings. In a separate controlled experiment on EuroSAT-MS, geo-positional encoding further improved classification relative to absolute positional encoding.

Abstract

Explore similar work

FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales

Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Tasks

Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing