EarthAlign: Learning Language-Aligned Representations of Earth System Data

Fadillah Adamsyah Maani; Numan Saeed; Jiaqi Ma; Dat Tien Nguyen; Youssef Wehbe; Duncan Watson-Parris; Salman Khan

cs.CVOct 10, 2026

EarthAlign: Learning Language-Aligned Representations of Earth System Data

Authors: Fadillah Adamsyah Maani, Numan Saeed, Jiaqi Ma, Dat Tien Nguyen, Youssef Wehbe, Duncan Watson-Parris, Salman Khan

Organizations: Mohamed bin Zayed University of Artificial Intelligence · ADIA Lab · University of California San Diego

Abstract

Interpreting Earth system conditions is central to understanding environmental risks and informing decisions across weather- and climate-sensitive sectors. Existing multimodal weather systems primarily target task-specific prediction, generation, or question answering, while general-purpose alignment between Earth system data and language remains underexplored due to the limited availability of diverse language supervision grounded in meteorological conditions. To fill this gap, we introduce EarthAlign, a large-scale dataset with broad geographic coverage spanning 2000-2024 that combines 10 expert-curated natural event and disaster data sources with climatology-derived normal-weather samples. Each sample includes temporal and geospatial information and four grounded caption styles aligned with 63-variable ERA5 meteorological data. Building on EarthAlign, we introduce MeteoLIP, a dual-encoder framework for learning aligned meteorology-language representations. MeteoLIP incorporates MetTok, which uses a geolocation- and time-conditioned query to aggregate variable-specific patch embeddings into a compact sequence of meteorology tokens. To comprehensively assess Earth system multimodal representations, we introduce three downstream tasks: cross-modal retrieval, weather-condition classification, and physical variable estimation. Our experiments show that MeteoLIP improves upon the corresponding CLIP and SigLIP baselines across these tasks. Further analyses highlight the importance of meteorological input, temporal and geospatial conditioning, and multi-style supervision for meteorology-language alignment. Thus, EarthAlign and MeteoLIP provide a foundation for learning and evaluating language-aligned representations of Earth system data. Our code, dataset, and weights are publicly available at https://github.com/mbzuai-oryx/EarthAlign.

Figures & tables

Figure 1: EarthAlign Overview. EarthAlign is a vision-language alignment dataset for the Earth system, comprising 553,738 samples that span 2000–2024, with a broad taxonomy of natural phenomena, broad geographic coverage, and diverse caption content to enable multimodal alignment.

Figure 2: Data curation pipeline. EarthAlign combines expert-curated natural-event records with climatology-based normal-weather samples. The resulting records contain standardized spatial and temporal information, labels, and four grounded caption styles for alignment with Earth system data.

Figure 3: MeteoLIP Overview. MeteoLIP consists of transformer-based meteorological and text encoders. MetTok converts multivariate meteorological input into a compact token sequence. During training, we randomly sample a caption style for each meteorological input and optimize a contrastive objective to align meteorological and textual representations in a shared embedding space.

Table 1: Comparison of CLIP and SigLIP baselines with their MeteoLIP counterparts. (a) Cross-modal retrieval. (b) Classification under text-prototype inference and linear probing . (c) Physical variable estimation of cyclone maximum sustained wind speed and minimum central pressure under the same two protocols. Bold and underline indicate the best and second-best results.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Years	Geo. Scope	Evidence Source	# NP Types	Size	Tasks
WeatherQA	2014–2020	CONUS	Expert forecast analyses	7	8,511	Severe weather MC-VQA and mesoscale case study generation
ClimateIQA	2023	Global	Meteorological heatmaps	3	26.3k img. (762.1k QA)	Meteorological anomaly VQA and description generation
Aquilon	1979–2022	Global	Expert-curated records & ERA5	N/R	444,244	Weather analysis, forecasting, and extreme event VQA
CLLMate	2015–2023	Global	Expert-verified news	13	7,747	Weather and climate phenomena and consequences MC-VQA
MeteorPred	2023–2024	China	Agency warnings	7	421,363	Severe weather VQA, MC-VQA, and structured generation tasks
EarthAlign	2000–2024	Global	Expert-curated records & ERA5	93	553,738	VL alignment: retrieval, classification, and physical variable estimation

Appendix

Table 8: Comparison of recent existing multimodal weather and climate datasets by temporal and geographic coverage, evidence source, natural-phenomenon types, dataset size, and evaluation tasks.

Figure 4: Effect of training data scale. We train Meteo-CLIP with 10%, 25%, 50%, and 100% of the training data and evaluate model performance across retrieval, classification, and physical variable estimation. Error bars denote standard deviation over three training runs.

Figure 5: Examples from EarthAlign. The global map illustrates geographically diverse natural-event records, while the bottom panels provide detailed examples of a tornado and hail event, and a landslide with their spatial and temporal metadata along with four grounded caption styles.

Panel of Figure 6 — Figure 6: Qualitative results using the MeteoLIP meteorology and text encoders. (a) M2T retrieval retrieves captions describing the same or closely related event across different caption styles. (b) The cold-wave example assigns the highest probability to cold wave, followed by winter weather and frost. (c) The similarity profiles over candidate physical values show peaks around the predicted minimum central pressure and maximum sustained wind speed.

Figure 7: Qualitative comparison of M2T retrieval using the full MetTok configuration and geolocation+time only, showing the top-5 retrieved texts for each ground-truth event, with green indicating meteorologically related events and red indicating unrelated phenomena. Full MetTok retrieves more meteorologically relevant events, while geolocation and time alone can retrieve unrelated phenomena.

Panel of Figure 8 — Figure 8: Examples of spatial grid-cell sampling for climatology-based normal-weather identification on California and Abu Dhabi.

Figure 9: Representative metadata records and their generated captions across record sources.

Figure 10: Distribution of generated caption lengths measured by using the CLIP text tokenizer. We report the overall distribution, distributions across the four caption styles, and distributions across record data sources.

Caption Style	Audited	Passed	Pass Rate
Natural Event Focused	110	110	100.00%
Physical Framing	110	108	98.18%
Impact Focused	110	98	89.09%
Atmospheric Pattern	110	105	95.45%
Overall	440	421	95.68%

Appendix

Table 11: Manual audit results for generated captions. A trained graduate annotator verified 10 samples from each record source across all caption styles, totaling 440 captions.

Figure 11: Regional distribution of EarthAlign records. Records spanning multiple regions contribute to more than one regional count.

Figure 12: Class distribution for the weather condition classification task in the training and test sets.

Figure 13: Confusion matrices under text-prototype inference and linear probing . Cell values report mean counts, while color intensity is normalized by true category to account for class imbalance.

	Classification		Sustained Wind		Central Pressure
Initialization	F1 $\uparrow$	Top-1 Acc. $\uparrow$	$R^{2}$ $\uparrow$	MAE $\downarrow$	$R^{2}$ $\uparrow$	MAE $\downarrow$
Scratch	77.02 $\pm$ 0.76	91.56 $\pm$ 0.16	72.55 $\pm$ 1.04	8.23 $\pm$ 0.18	78.89 $\pm$ 0.74	5.65 $\pm$ 0.13
WebLI	78.38 $\pm$ 0.64	91.88 $\pm$ 0.20	74.31 $\pm$ 0.28	7.89 $\pm$ 0.08	80.65 $\pm$ 0.37	5.36 $\pm$ 0.05
MeteoLIP	81.31 $\pm$ 0.47	92.97 $\pm$ 0.15	76.15 $\pm$ 0.35	7.56 $\pm$ 0.06	81.24 $\pm$ 0.38	5.24 $\pm$ 0.03

Appendix

Table 16: Effect of initialization on downstream fine-tuning using the same Meteo-SigLIP architecture. Bold and underline indicate the best and second-best results, respectively.

Panel of Figure 14 — Figure 14: Training dynamics of Meteo-CLIP. (a) CLIP test loss and retrieval, classification, and physical variable estimation performance across epochs. (b) Linear-probing classification performance using frozen meteorology encoder checkpoints at epochs 6, 12, 18, 24, and 30.

Figure 15: We compare ViT-B and ViT-L encoder variants using pretrained initialization, and additionally compare Meteo-CLIP models trained from scratch. We report M2T and T2M R@1 along with training and test loss to examine how model capacity interacts with initialization and meteorological-language inputs. The reported values are averaged over three training runs.

Cross-Modal Representation Learning

Cross-modal representation learning connects information across data types such as images, text, audio, biosignals, and sensor measurements.

See all 205 papers

Domain-specific language alignment of multimodal embeddings towards explainable predictions
Oct 16, 2026Gabrielė Tijūnaitytė, Thijs L. van der Plas, Mohamed-Bachir Belaid +3Multimodal Embedding Concept-Based Explanations
Artificial Intelligence Group, Wageningen University & Research, Wageningen, the Netherlands · Machine Learning in Earth Observation Lab, University of Bonn, Germany · NILU, Climate and environmental research institute, Kjeller, Norway +1
MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation
Oct 13, 2026Carmen Oliver Huidobro, Mikolaj Czerkawski, Kevin Monsalvez-Pozo +3Remote Sensing Geospatial Foundation Models
Image and Signal Processing (ISP) Group, Universitat de València, Valencia, Spain · Asterisk Labs, London, United Kingdom
AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models
Aug 16, 2026Md Aminur Hossain, Omkumar Vaghasiya, Rajeev Ranjan Dwivedi +2Remote Sensing VLMs Vision-Language Alignment
Space Applications Centre, ISRO, Ahmedabad, India · Indian Institute of Science Education and Research (IISER) Bhopal, India · CSRE, Indian Institute of Technology Bombay, India

Oct 16, 2026J/K move · Enter open · S save

Domain-specific language alignment of multimodal embeddings towards explainable predictions

Gabrielė Tijūnaitytė, Thijs L. van der Plas, Mohamed-Bachir Belaid, Rob Knapen+2

Artificial Intelligence Group, Wageningen University & Research, Wageningen, the Netherlands · Machine Learning in Earth Observation Lab, University of Bonn, Germany · NILU, Climate and environmental research institute, Kjeller, Norway+1

Multimodal Embedding Concept-Based Explanations Vision-Language Alignment Remote Sensing

Open paper AskPDF arXiv

Multimodal data are commonly fused into latent representations that improve downstream task performance but are not interpretable, limiting their application in domains that require explainable model decisions. Vision-language alignment has made image embeddings interpretable through language, but requires vast image-caption training data. However, captioned data are typically not available for multimodal embeddings, as relevant features cannot directly be observed and annotated from embeddings alone. Here, we propose a framework to language-align any embedding space that can be linked to interpretable auxiliary data. Captioning functions convert auxiliary data into partial captions; a soft contrastive loss weights pairs by the similarity of their properties that each caption describes; and a rank-based concept retrieval task evaluates alignment against the auxiliary data rather than the captions. We apply this framework to multimodal Earth embeddings to facilitate explainable predictions for two species distribution modelling tasks: S2BMS and SatBird. AlphaEarth embeddings beat previous state-of-the-art models on 6/7 metrics, while our framework language-aligns domain-specific concepts of the embedding space, outperforming previous Earth observation vision-language models on average concept retrieval for both datasets. Finally, aligned concepts recover species-habitat associations (

ρ

=0.44) and relate habitat descriptions to species predictions, showing that abstract embeddings can be interrogated in the language of domain experts.

Abstract

Cross-Modal Representation Learning

Explore similar work

Domain-specific language alignment of multimodal embeddings towards explainable predictions

MajorTOM:Elliot Pretrain, A Multi-Sensor, Multi-Temporal Pretraining Dataset for Earth Observation

AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models