Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing
Authors: Paolo Fraccaro, Gabby Nyirjesy, Daniela Szwarcman, Himanshu Patil, Vishal Gaur, Rohit Lal, Rachel A. Slank, Geoffrey Dawson, +14 more
Organizations: IBM Research, Yorktown Heights, NY, USA · The University of Alabama in Huntsville, Huntsville, AL, USA · University Space Research Association (USRA), Science and Technology Institute, Huntsville, AL, USA · NASA Goddard Space Flight Center, Greenbelt, MD, USA · SETI Institute, Mountain View, CA, USA · NASA Ames Research Center, Moffett Field, CA, USA · University of Maryland, Baltimore County (UMBC), Baltimore, MD, USA · Howard University, Washington, DC, USA · NASA Marshall Space Flight Center, Huntsville, AL, USA · NASA Headquarters, Washington, DC, USA
We present a multimodal foundation model for lunar remote sensing, pretrained from scratch on SomBench, a geographically partitioned corpus of nearly two million co-registered tile bundles spanning 11 modalities at two spatial scales (1 m/pixel and 100 m/pixel). The model adapts the TerraMind masked-token architecture with two lunar-specific extensions: acquisition geometry is provided as explicit context, and meter- and hundred-meter-scale tiles are trained jointly so that a single set of weights covers both resolutions. FlexiViT patch embeddings allow adaptation to different patch sizes without retraining, while modality-wise inputs enable flexible multimodal fine-tuning. Qualitative generation experiments suggest the model learns meaningful cross-modal correspondences, including terrain derivatives from elevation and illumination-consistent reflectance from geometry. We evaluate on four benchmarks: crater detection at WAC and NAC scales, irregular mare patch (IMP) segmentation, and polar ice prospectivity regression. Across tasks, the pretrained model matches or outperforms ImageNet-pretrained baselines and an architecturally identical random-init control. On multimodal ice prospectivity regression, pretrained variants achieve the best results, while the random-init model outperforms most baselines, suggesting gains arise from both the architecture and pretraining. Label efficiency is notable for WAC crater detection, where the pretrained model trained on 50% of the data exceeds the strongest ImageNet baseline trained on the full dataset. Among adaptation strategies, LoRA matches or surpasses full fine-tuning on crater detection and IMP segmentation while using far fewer trainable parameters, whereas full fine-tuning performs best for ice prospectivity regression. We release the pretrained checkpoint, benchmark datasets, and fine-tuning code to support reproducible lunar AI research.
Figures & tables
Figure 1: Dataset construction. Each optical observation defines the sample extent; auxiliary products are co-registered onto the anchor grid, extracted as multimodal tile bundles, and partitioned geographically.
Modality
Key
Native res.
Ch.
Source
Pretrain
WAC family (low-resolution track, 51.2 km tiles)
Visible reflectance
vis
100 m
5
LROC WAC
Dense
Ultraviolet reflectance
uv
500 m
2
LROC WAC
Dense
Topography
dtm
60 m
1
SLDEM2015
Dense
Slope
slope
60 m
1
SLDEM2015
Dense
Aspect
aspect
60 m
2
SLDEM2015
Dense
Table 1: Corpus composition. Channels marked “multi” are multi-band stacks (e.g., 24 Diviner Tbol phases plus the acquisition-matched phase). The Pretrain column gives the representation each layer receives during pretraining: dense layers are patch-embedded and tokenized as full images, forming the nine image-like modalities; scalar layers are averaged over the tile footprint and contribute a single discrete token each, together forming the static-map context modality. With the per-tile optical metadata, this gives the 11 pretraining modalities.
Name
Science Theme
Image Size (px)
Resolution (m/px)
Target
Train
Val
Test
Object Detection
Robbins craters
Impact processes
512 × 512
100
1 class
800
100
100
NAC hand labeled
Impact processes
256 × 256
1
1 class
645
59
62
Segmentation
Irregular Mare Patches
Volcanic History
256 × 256
1
1 class
100
20
10
Regression
Table 2: Overview of SomBench benchmark datasets used for the validation of our model.
Figure 2: Model overview. Left: input modalities include low- and high-resolution images (LROC WAC and NAC), DTM-derived topographic variables (DTM, slope, aspect), per-tile optical metadata, and static context maps. Center-left: modality-wise tokenization in parallel branches. Image modalities are processed by patch-embedding layers ( Pixel-level input ) and VQ-VAE tokenizers ( Token-level input ); metadata and static maps are encoded with a text tokenizer ( Sequence input ). Center-right: correlation learning follow the TerraMind training procedure. A subset of the modality tokens is sampled as encoder inputs, while a separate subset is sampled as prediction targets. The encoder processes the input tokens, and the decoder predicts the target tokens under a cross-entropy objective. Right: The pretrained model can be used for multimodal generation and multimodal fine-tuning with native support for multiple input modalities. The added FlexiViT support enables fine-tuning at patch sizes different from those used in pretraining.
- Emission, incidence, and phase angles describing viewing geometry - sub-solar azimuth, latitude, and longitude anchoring the solar reference frame - tile center coordinates
Photometric properties
WAC_NORM_REF_[321,360,415,566,604,643,689] ALBEDO
Photometrically normalized WAC reflectance products and broadband albedo.
Composition and maturity proxies from Kaguya-derived spectral mineralogy, space-weathering iron partitioning, and titanium.
Volatile-related
HYDROGEN
Hydrogen abundance proxy.
Physical and geophysical
GRAVITY ROCK_ABUNDANCE ROUGHNESS
Physical surface characteristics and geophysical products.
Table 3: Context variables derived from static maps and optical metadata. For each tile, the corresponding map is averaged over the tile footprint and quantized into a single token.
Figure 3: Example of generation from DTM to slope and aspect for the WAC family.
Figure 4: Example of generation from DTM to slope and aspect for the NAC family.
Figure 5: Example of generation from DTM and metadata to WAC for the same location.
Figure 6: First example of generation from WAC to all other modalities.
Figure 7: Second example of generation from WAC to all other modalities.
Model
mAP
AP@50
AP@75
50% training data
ResNet-50 (ImageNet)
0.1993 ± 0.0007
0.4960 ± 0.0036
0.1572 ± 0.0038
ResNet-50 (random)
0.1873 ± 0.0011
0.4663 ± 0.0029
0.1472 ± 0.0041
ViT-B MAE (ImageNet)
0.2167 ± 0.0022
0.5656 ± 0.0092
0.1670 ± 0.0042
ConvNeXt-B (IN22k)
0.2266 ± 0.0030
0.5684 ± 0.0030
0.1840 ± 0.0040
ConvNeXtV2-B (IN22k)
0.2303 ± 0.0034
0.5845 ± 0.0036
0.1829 ± 0.0087
Table 4: Crater detection on LRO WAC (context-scale) as a function of training-data fraction. Test-set bounding-box mean average precision. Values are mean ± standard deviation over five random seeds. Bold indicates the best value per column; underline the second best; differences smaller than the reported seed spread should not be read as a ranking.
Model
mAP
AP@50
AP@75
ResNet-50 (ImageNet)
0.1411 ± 0.0036
0.4374 ± 0.0196
0.0895 ± 0.0055
ResNet-50 (random)
0.1296 ± 0.0034
0.4159 ± 0.0171
0.0787 ± 0.0057
ViT-B MAE (ImageNet)
0.1236 ± 0.0119
0.3798 ± 0.0460
0.0703 ± 0.0109
ConvNeXt-B (IN22k)
0.1464 ± 0.0060
0.4400 ± 0.0249
0.0933 ± 0.0039
ConvNeXtV2-B (IN22k)
0.1345 ± 0.0085
0.3949 ± 0.0545
0.0783 ± 0.0107
SwinV2-B (ImageNet)
0.1552 ± 0.0086
0.4586 ± 0.0308
0.1090 ± 0.0146
Table 5: Crater detection on LRO NAC (meter-scale) at 100% training data. Test-set bounding-box mean average precision. Values are mean ± standard deviation over five random seeds. Bold indicates the best value per column; underline the second best; differences smaller than the reported seed spread should not be read as a ranking.
Figure 8: Example of predictions for the WAC Robbins Crater Catalog (first row) and the NAC Hand Labeled Dataset (second row). The first column ( a and c ) shows predictions over ground truth for NASA-IBM LFM, with the second column ( b and d ) showing the same for SwinV2-B. Green: ground truth; blue: predictions at confidence ≥ 0.5.
Model
IoU 1
F1 1
ResNet-50 (ImageNet)
0.4731 ± 0.0114
0.6423 ± 0.0105
ResNet-50 (random)
0.3496 ± 0.0323
0.5175 ± 0.0355
ViT-B MAE (ImageNet)
0.5541 ± 0.0369
0.7125 ± 0.0298
ConvNeXt-B (IN22k)
0.5374 ± 0.0197
0.6989 ± 0.0168
ConvNeXtV2-B (IN22k)
0.5687 ± 0.0181
0.7249 ± 0.0146
SwinV2-B (ImageNet)
0.5555 ± 0.0159
0.7141 ± 0.0132
Table 6: Irregular Mare Patch (IMP) segmentation. IMP delineation is a binary dense-prediction task, so we report test-set metrics on the positive (IMP) class only: intersection-over-union (IoU 1 ) and F1 score (F1 1 ). Values are mean ± standard deviation over five random seeds. Bold indicates the best mean per column and underline the second best; differences smaller than the reported seed spread should not be read as a ranking.
Figure 9: Qualitative IMP segmentation predictions. Our model and the baselines produce broadly comparable outputs; in the second example, the pretrained NASA-IBM LFM identifies IMP extents that none of the baselines detect. Green overlays mark ground-truth IMP extents; colored overlays show model predictions. The colors used in the maps in the top row are for visualization purposes only and do not convey any specific meaning.
Model
RMSE ↓
MAE ↓
R2↑
ResNet-50 (ImageNet)
0.0788 ± 0.0004
0.0568 ± 0.0004
0.9162 ± 0.0009
ResNet-50 (random)
0.0695 ± 0.0007
0.0495 ± 0.0005
0.9347 ± 0.0013
ViT-B MAE (ImageNet)
0.0917 ± 0.0009
0.0674 ± 0.0007
0.8865 ± 0.0021
ConvNeXt-B (IN22k)
0.0442 ± 0.0008
0.0301 ± 0.0005
0.9737 ± 0.0009
ConvNeXtV2-B (IN22k)
0.0439 ± 0.0008
0.0298 ± 0.0006
0.9739 ± 0.0010
SwinV2-B (ImageNet)
0.0377 ± 0.0004
0.0256 ± 0.0003
0.9808 ± 0.0004
Table 7: Polar ice prospectivity regression on the full eight-modality polar stack (aspect, slope, DICE, TMAX, LPSR, LPSR_DEN, LPSR_DIS, CUR). Test-set metrics against the prospectivity map in [ 44 ] : RMSE and MAE lower-is-better, R2 higher-is-better. Values are mean ± standard deviation over five random seeds. Bold indicates the best value per column; underline the second best.
Number of input modalities m
Model
2
3
4
5
6
7
8
(as, sl)
+DICE
+TMAX
+LPSR
+DEN
+DIS
+CUR
ConvNeXt-B (IN22k)
0.2083 ( ± 0.0086)
0.0614 ( ± 0.0007)
0.0522 ( ± 0.0007)
0.0513 ( ± 0.0004)
0.0509 ( ± 0.0005)
0.0517 ( ± 0.0007)
0.0437 ( ± 0.0002)
NASA-IBM LFM (ps8)
0.1981 ( ± 0.0063)
0.0434 ( ± 0.0006)
0.0412 ( ± 0.0009)
0.0393 ( ± 0.0002)
0.0403 ( ± 0.0010)
0.0398 ( ± 0.0004)
0.0288 ( ± 0.0011)
Table 8: Polar ice prospectivity regression, modality-count ablation. Each column corresponds to results from models trained on a fixed set of polar input variables, starting from the two pretrained modalities (aspect, slope) at m=2 and incrementally adding polar-stack modalities not seen during pretraining, in the order DICE, TMAX, LPSR, LPSR_DEN, LPSR_DIS, and finally CUR at m=8 . This isolates the contribution of each additional benchmark-only modality on top of the pretrained backbone. Values are test-set RMSE (lower is better). Values are mean ± standard deviation over three random seeds. Bold indicates the best value per column.
Figure 10: Qualitative ice prospectivity predictions on test patches. NASA-IBM LFM recovers PSR-boundary sharpness and fine spatial detail that the ImageNet-pretrained baseline smooths out. The last row shows our model prediction using only slope, aspect, and DICE, illustrating competitive performance even with a drastically reduced input set.
Tokenizer
Modality
GPUs
Global Batch
Epochs
Warmup Epochs
Peak LR
NAC
nac_aspect_3m
Aspect
16
512
100
2
3.06e-6
nac_dtm_3m
DTM
16
512
100
2
3.06e-6
nac
Panchromatic
16
512
100
2
3.06e-6
nac_slope_3m
Slope
16
512
60
2
3.06e-6
WAC
Table 9: Per-run training configuration for the nine domain-specific lunar tokenizers. All runs share a per-GPU batch size of 32.
Figure 11: Additional generation examples from WAC to all other modalities.
Decades of orbital missions have produced multi-modal remote sensing data for the Moon, spanning optical imagery, spectroscopy, thermal emission, radar, gravity, and elemental composition. Yet these datasets remain fragmented across archives, and no benchmark exists for evaluating machine learning on lunar data. We introduce Moonstone, the first multi-modal foundation model benchmark for lunar remote sensing. Our contributions are: (1) a 28-channel, 128 pixels-per-degree (~237 m) global lunar pretraining dataset from seven instrument families across five missions, (2) MG-MAE, a modality-grouped masked autoencoder with per-group convolutional tokenizers, a shared Vision Transformer encoder, attention masking for missing modalities, coverage-adaptive masking for heterogeneous spatial coverage, and spectral continuity regularization for physically plausible reconstructions, and (3) a benchmark of six downstream tasks covering classification, regression, and segmentation. MG-MAE pretrained features outperform scratch baselines on all tasks and surpass both ImageNet-pretrained and vanilla MAE baselines by large margins. Data and code are available at https://huggingface.co/datasets/ayushprd/Moonstone and https://github.com/ayushprd/Moonstone .
Ayush Prasad, Swarnalee Mazumder
Space and Earth Observation Centre, Finnish Meteorological Institute, Finland · Data Management, German Climate Computing Centre, Germany
The renewed global focus on lunar exploration, driven by the prospect of in-situ resource utilization and a sustained human presence on the Moon, has created growing demand for accurate, large-scale characterization of the lunar surface. Although vast quantities of orbital remote-sensing data have been collected, scientific analysis and resource mapping remain fragmented by heterogeneous multiinstrument observations, sparse labels, and bespoke task-specific modelling workflows. Here we introduce LunarFM, a multimodal foundation model that learns a general representation of the lunar surface from diverse orbital measurements. LunarFM assimilates observations from six instruments across three lunar missions, mapping 18 input channels to a shared embedding space. We demonstrate that this embedding space supports a diverse range of downstream applications, including similarity search, few-shot resource mapping, mineral abundance regression, and geological unit classification, enabling efficient scientific investigation and resource-oriented analysis. We provide a machine-learning-ready dataset of co-registered multimodal observations spanning latitudes from 70°S to 70°N, a pretrained multimodal masked autoencoder, and a companion embedding dataset providing a joint 768-dimensional representation of lunar surface properties. All code and data are available at https://lunarfm.trillium.tech/
Marc Girona-Mata, Jakob Gawlikowski, Sumit Goski +6
German Aerospace Center (DLR) · European Space Resources Innovation Center (ESRIC) · Universidad de Antioquia
Effective foundation modeling in remote sensing requires spatially aligned heterogeneous modalities coupled with semantically grounded supervision, yet such resources remain limited at scale. We present GeoMeld, a large-scale multimodal dataset with approximately 2.5 million spatially aligned samples. The dataset spans diverse modalities and resolutions and is constructed under a unified alignment protocol for modality-aware representation learning. GeoMeld provides semantically grounded language supervision through an agentic captioning framework that synthesizes and verifies annotations from spectral signals, terrain statistics, and structured geographic metadata, encoding measurable cross-modality relationships within textual descriptions. To leverage this dataset, we introduce GeoMeld-FM, a pretraining framework that combines multi-pretext masked autoencoding over aligned modalities, JEPA representation learning, and caption-vision contrastive alignment. This joint objective enables the learned representation space to capture both reliable cross-sensor physical consistency and grounded semantics. Experiments demonstrate consistent gains in downstream transfer and cross-sensor robustness. Together, GeoMeld and GeoMeld-FM establish a scalable reference framework for semantically grounded multi-modal foundation modeling in remote sensing.
Maram Hasan, Md Aminur Hossain, Savitra Roy +6
Indian Institute of Technology Bombay · Space Applications Centre, ISRO · University of Trento +1