Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding
Authors: Lance Legel, Qin Huang, Brandon Voelker, Daniel Neamati, Patrick Alan Johnson, Favyen Bastani, Jeff Rose, James Ryan Hennessy, Robert Guralnick, Douglas Soltis, Pamela Soltis, Shaowen Wang
Organizations: Ecological Intelligence Lab Ecodash.ai · School of Complex Adaptive Systems Arizona State University · Geosensing Systems Engineering & Sciences Lab University of Houston · Navigation & Autonomous Vehicles Lab Stanford University · Earth System Lab Allen Institute for Artificial Intelligence · Spatial Intelligence Lab SpatialLogic.com · Department of Computer Science Georgia Institute of Technology · Florida Museum of Natural History University of Florida · NSF Institute for Geospatial Understanding University of Illinois Urbana-Champaign
We present DeepEarth, a self-supervised multi-modal world model with Earth4D, a novel planetary-scale 4D space-time positional encoder. Earth4D extends 3D multi-resolution hash encoding to include time, efficiently scaling across the planet over centuries with sub-meter, sub-second precision. Multi-modal encoders (e.g. vision-language models) are fused with Earth4D embeddings and trained via masked reconstruction. We demonstrate Earth4D's expressive power by achieving state-of-the-art performance on an ecological forecasting benchmark. Earth4D with learnable hash probing surpasses a multi-modal foundation model pre-trained on substantially more data. Access open source code and download models at: https://github.com/legel/deepearth
Figures & tables
Figure 1: DeepEarth Overview. Masked multi-modal data ( e.g. images, text) sampled around an event ( e.g. pollination) are encoded and fused with Earth4D space-time embeddings. These universal tokens are jointly encoded, and then masked data is inductively decoded and simulated.
Figure 2: Earth4D Space-Time Positional Encoding. A planetary-scale 4D encoder with fully decomposable spatio-temporal representation. Four grids ( xyz , xyt , yzt , xzt ) are each learned in 3D space and computed in parallel. Each grid has multiple resolution levels (Appendix A ), enabling deep learning of complex joint distributions in multi-modal data across space-time scales.
Figure 3: Earth4D LFMC Prediction Performance. (Top) Distribution of absolute errors in percentage point predictions across 13,297 test samples, showing median error of 7.1pp. (A) Geographic error distribution across CONUS shows low error in well-sampled regions. (B) Temporal predictions closely track ground truth LFMC measurements across seasons (2017–2023).
Model
Data Inputs
MAE (pp)
RMSE (pp)
R²
Galileo (Pre-Trained)
( x,y,z,t ) + Species Type + Remote Sensing
12.6
18.9
0.72
Earth4D (Learned Hashing)
( x,y,z,t ) + Species Name
11.7
18.7
0.783
Table 1: State-of-the-Art Ecological Forecasting Benchmark. Earth4D surpasses the pre-trained Galileo foundation model without satellite imagery, weather data, or topography.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Earth4D Space-Time Scales. Default 24 × 24 × 24 levels for each xyz , xyt , yzt , xzt grid. Each level stores up to 222 entries, with each entry storing a 2D feature. Requires 724M trainable parameters ( ∼ 11 GB GPU memory during training). Parallelizable across levels and spatio-temporal boundaries. Outputs 192D per (x,y,z,t) coordinate from 4 grids × 24 levels × 2D feature per level. Hashing saves memory vs. naive requirement, e.g., (228)3=1025 at level 24.
Figure 5: Earth4D Hash Collision Analysis. (Table) 10 (x,y,z,t) point distribution scenarios that were simulated to analyze hash collisions in Earth4D memory. (Graph) Shows results for 1M point simulations across all 24 levels.
We propose TerraFlow, a novel approach to multimodal, multitemporal learning for Earth observation. TerraFlow builds on temporal training objectives that enable sequence-aware learning across space, time, and modality, while remaining robust to the variable-length inputs commonly encountered in real-world Earth observation data. Our experiments demonstrate superiority of TerraFlow over state-of-the-art foundation models for Earth observation across all temporal tasks of the GEO-Bench-2 benchmark. We additionally demonstrate that TerraFlow is able to make initial steps towards deep-learning based risk map prediction for natural disasters -- a task on which other state-of-the-art foundation models frequently collapse. TerraFlow outperforms state-of-the-art foundation models by up to 50% in F1 score and 24% in Brier score.
Nazar Puriy, Johannes Jakubik, Benedikt Blumenstiel +1
IBM Research Europe, Zurich, Switzerland · ETH Zurich, Zurich, Switzerland
Recent research in geospatial machine learning demonstrates that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks with limited labeled data. However, most benchmark datasets have few data modalities and poor global representation, limiting the ability to evaluate multimodal pretrained models at global scales. In order to fill this gap, we introduce MMEarth-Bench, a collection of five new environmental tasks with 12 modalities, globally distributed data, and both random and geographic test splits. We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor. Moreover, a simple randomly initialized multimodal model is competitive given enough labeled data. Although data is abundant, models can currently only make use of the modalities on which they were pretrained. To solve this problem, we propose using all the modalities available at test time as auxiliary tasks for test-time adaptation. Our model-agnostic method for test-time training with multimodal reconstruction (TTT-MMR) can improve performance across all models and tasks on both test splits. Furthermore, geographic batching leads to a good trade-off between regularization and specialization during TTT, which is especially beneficial for long-tail distributions. Our dataset, code, and visualization tool are linked on the project page: lgordon99.github.io/mmearth-bench.
Lucia Gordon, Serge Belongie, Christian Igel +1
Harvard University, USA · University of Copenhagen, Denmark
Earth observation is inherently dynamic, yet temporal information in many foundation models is learned through reconstruction, invariance, or retrospective sequence summarization. SPEAR NeXT is introduced as a compact pixel-wise multimodal spectral temporal foundation model in which temporal self supervision is formulated as past only, multi horizon latent Earth state prediction. Instantaneous states are first encoded by the pretrained SPEAR model from optical, radar, and environmental observations into compact 32 dimensional embeddings. Their temporal evolution is then modeled by a causally masked Trans former that predicts multiple future latent states from pre ceding observations. Relative temporal order is represented using Rotary Position Embeddings, while month and year embeddings encode seasonal phase and interannual con text.