Self-Supervised Multi-Modal World Model with 4D Space-Time Embedding
Authors: Lance Legel, Qin Huang, Brandon Voelker, Daniel Neamati, Patrick Alan Johnson, Favyen Bastani, Jeff Rose, James Ryan Hennessy, Robert Guralnick, Douglas Soltis, Pamela Soltis, Shaowen Wang
Organizations: Ecological Intelligence Lab Ecodash.ai · School of Complex Adaptive Systems Arizona State University · Geosensing Systems Engineering & Sciences Lab University of Houston · Navigation & Autonomous Vehicles Lab Stanford University · Earth System Lab Allen Institute for Artificial Intelligence · Spatial Intelligence Lab SpatialLogic.com · Department of Computer Science Georgia Institute of Technology · Florida Museum of Natural History University of Florida · NSF Institute for Geospatial Understanding University of Illinois Urbana-Champaign
We present DeepEarth, a self-supervised multi-modal world model with Earth4D, a novel planetary-scale 4D space-time positional encoder. Earth4D extends 3D multi-resolution hash encoding to include time, efficiently scaling across the planet over centuries with sub-meter, sub-second precision. Multi-modal encoders (e.g. vision-language models) are fused with Earth4D embeddings and trained via masked reconstruction. We demonstrate Earth4D's expressive power by achieving state-of-the-art performance on an ecological forecasting benchmark. Earth4D with learnable hash probing surpasses a multi-modal foundation model pre-trained on substantially more data. Access open source code and download models at: https://github.com/legel/deepearth
Figures & tables
Figure 1: DeepEarth Overview. Masked multi-modal data ( e.g. images, text) sampled around an event ( e.g. pollination) are encoded and fused with Earth4D space-time embeddings. These universal tokens are jointly encoded, and then masked data is inductively decoded and simulated.
Figure 2: Earth4D Space-Time Positional Encoding. A planetary-scale 4D encoder with fully decomposable spatio-temporal representation. Four grids ( xyz , xyt , yzt , xzt ) are each learned in 3D space and computed in parallel. Each grid has multiple resolution levels (Appendix A ), enabling deep learning of complex joint distributions in multi-modal data across space-time scales.
Figure 3: Earth4D LFMC Prediction Performance. (Top) Distribution of absolute errors in percentage point predictions across 13,297 test samples, showing median error of 7.1pp. (A) Geographic error distribution across CONUS shows low error in well-sampled regions. (B) Temporal predictions closely track ground truth LFMC measurements across seasons (2017–2023).
Model
Data Inputs
MAE (pp)
RMSE (pp)
R²
Galileo (Pre-Trained)
( x,y,z,t ) + Species Type + Remote Sensing
12.6
18.9
0.72
Earth4D (Learned Hashing)
( x,y,z,t ) + Species Name
11.7
18.7
0.783
Table 1: State-of-the-Art Ecological Forecasting Benchmark. Earth4D surpasses the pre-trained Galileo foundation model without satellite imagery, weather data, or topography.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Earth4D Space-Time Scales. Default 24 × 24 × 24 levels for each xyz , xyt , yzt , xzt grid. Each level stores up to 222 entries, with each entry storing a 2D feature. Requires 724M trainable parameters ( ∼ 11 GB GPU memory during training). Parallelizable across levels and spatio-temporal boundaries. Outputs 192D per (x,y,z,t) coordinate from 4 grids × 24 levels × 2D feature per level. Hashing saves memory vs. naive requirement, e.g., (228)3=1025 at level 24.
Figure 5: Earth4D Hash Collision Analysis. (Table) 10 (x,y,z,t) point distribution scenarios that were simulated to analyze hash collisions in Earth4D memory. (Graph) Shows results for 1M point simulations across all 24 levels.