Multimodal-Multiresolution Foundation Model for Lunar Remote Sensing
Authors: Paolo Fraccaro, Gabby Nyirjesy, Daniela Szwarcman, Himanshu Patil, Vishal Gaur, Rohit Lal, Rachel A. Slank, Geoffrey Dawson, +14 more
Organizations: IBM Research, Yorktown Heights, NY, USA · The University of Alabama in Huntsville, Huntsville, AL, USA · University Space Research Association (USRA), Science and Technology Institute, Huntsville, AL, USA · NASA Goddard Space Flight Center, Greenbelt, MD, USA · SETI Institute, Mountain View, CA, USA · NASA Ames Research Center, Moffett Field, CA, USA · University of Maryland, Baltimore County (UMBC), Baltimore, MD, USA · Howard University, Washington, DC, USA · NASA Marshall Space Flight Center, Huntsville, AL, USA · NASA Headquarters, Washington, DC, USA
We present a multimodal foundation model for lunar remote sensing, pretrained from scratch on SomBench, a geographically partitioned corpus of nearly two million co-registered tile bundles spanning 11 modalities at two spatial scales (1 m/pixel and 100 m/pixel). The model adapts the TerraMind masked-token architecture with two lunar-specific extensions: acquisition geometry is provided as explicit context, and meter- and hundred-meter-scale tiles are trained jointly so that a single set of weights covers both resolutions. FlexiViT patch embeddings allow adaptation to different patch sizes without retraining, while modality-wise inputs enable flexible multimodal fine-tuning. Qualitative generation experiments suggest the model learns meaningful cross-modal correspondences, including terrain derivatives from elevation and illumination-consistent reflectance from geometry. We evaluate on four benchmarks: crater detection at WAC and NAC scales, irregular mare patch (IMP) segmentation, and polar ice prospectivity regression. Across tasks, the pretrained model matches or outperforms ImageNet-pretrained baselines and an architecturally identical random-init control. On multimodal ice prospectivity regression, pretrained variants achieve the best results, while the random-init model outperforms most baselines, suggesting gains arise from both the architecture and pretraining. Label efficiency is notable for WAC crater detection, where the pretrained model trained on 50% of the data exceeds the strongest ImageNet baseline trained on the full dataset. Among adaptation strategies, LoRA matches or surpasses full fine-tuning on crater detection and IMP segmentation while using far fewer trainable parameters, whereas full fine-tuning performs best for ice prospectivity regression. We release the pretrained checkpoint, benchmark datasets, and fine-tuning code to support reproducible lunar AI research.
Figures & tables
Figure 1: Dataset construction. Each optical observation defines the sample extent; auxiliary products are co-registered onto the anchor grid, extracted as multimodal tile bundles, and partitioned geographically.
Modality
Key
Native res.
Ch.
Source
Pretrain
WAC family (low-resolution track, 51.2 km tiles)
Visible reflectance
vis
100 m
5
LROC WAC
Dense
Ultraviolet reflectance
uv
500 m
2
LROC WAC
Dense
Topography
dtm
60 m
1
SLDEM2015
Dense
Slope
slope
60 m
1
SLDEM2015
Dense
Aspect
aspect
60 m
2
SLDEM2015
Dense
Table 1: Corpus composition. Channels marked “multi” are multi-band stacks (e.g., 24 Diviner Tbol phases plus the acquisition-matched phase). The Pretrain column gives the representation each layer receives during pretraining: dense layers are patch-embedded and tokenized as full images, forming the nine image-like modalities; scalar layers are averaged over the tile footprint and contribute a single discrete token each, together forming the static-map context modality. With the per-tile optical metadata, this gives the 11 pretraining modalities.
Name
Science Theme
Image Size (px)
Resolution (m/px)
Target
Train
Val
Test
Object Detection
Robbins craters
Impact processes
512 × 512
100
1 class
800
100
100
NAC hand labeled
Impact processes
256 × 256
1
1 class
645
59
62
Segmentation
Irregular Mare Patches
Volcanic History
256 × 256
1
1 class
100
20
10
Regression
Table 2: Overview of SomBench benchmark datasets used for the validation of our model.
Figure 2: Model overview. Left: input modalities include low- and high-resolution images (LROC WAC and NAC), DTM-derived topographic variables (DTM, slope, aspect), per-tile optical metadata, and static context maps. Center-left: modality-wise tokenization in parallel branches. Image modalities are processed by patch-embedding layers ( Pixel-level input ) and VQ-VAE tokenizers ( Token-level input ); metadata and static maps are encoded with a text tokenizer ( Sequence input ). Center-right: correlation learning follow the TerraMind training procedure. A subset of the modality tokens is sampled as encoder inputs, while a separate subset is sampled as prediction targets. The encoder processes the input tokens, and the decoder predicts the target tokens under a cross-entropy objective. Right: The pretrained model can be used for multimodal generation and multimodal fine-tuning with native support for multiple input modalities. The added FlexiViT support enables fine-tuning at patch sizes different from those used in pretraining.
- Emission, incidence, and phase angles describing viewing geometry - sub-solar azimuth, latitude, and longitude anchoring the solar reference frame - tile center coordinates
Photometric properties
WAC_NORM_REF_[321,360,415,566,604,643,689] ALBEDO
Photometrically normalized WAC reflectance products and broadband albedo.
Composition and maturity proxies from Kaguya-derived spectral mineralogy, space-weathering iron partitioning, and titanium.
Volatile-related
HYDROGEN
Hydrogen abundance proxy.
Physical and geophysical
GRAVITY ROCK_ABUNDANCE ROUGHNESS
Physical surface characteristics and geophysical products.
Table 3: Context variables derived from static maps and optical metadata. For each tile, the corresponding map is averaged over the tile footprint and quantized into a single token.
Figure 3: Example of generation from DTM to slope and aspect for the WAC family.
Figure 4: Example of generation from DTM to slope and aspect for the NAC family.
Figure 5: Example of generation from DTM and metadata to WAC for the same location.
Figure 6: First example of generation from WAC to all other modalities.
Figure 7: Second example of generation from WAC to all other modalities.
Model
mAP
AP@50
AP@75
50% training data
ResNet-50 (ImageNet)
0.1993 ± 0.0007
0.4960 ± 0.0036
0.1572 ± 0.0038
ResNet-50 (random)
0.1873 ± 0.0011
0.4663 ± 0.0029
0.1472 ± 0.0041
ViT-B MAE (ImageNet)
0.2167 ± 0.0022
0.5656 ± 0.0092
0.1670 ± 0.0042
ConvNeXt-B (IN22k)
0.2266 ± 0.0030
0.5684 ± 0.0030
0.1840 ± 0.0040
ConvNeXtV2-B (IN22k)
0.2303 ± 0.0034
0.5845 ± 0.0036
0.1829 ± 0.0087
Table 4: Crater detection on LRO WAC (context-scale) as a function of training-data fraction. Test-set bounding-box mean average precision. Values are mean ± standard deviation over five random seeds. Bold indicates the best value per column; underline the second best; differences smaller than the reported seed spread should not be read as a ranking.
Model
mAP
AP@50
AP@75
ResNet-50 (ImageNet)
0.1411 ± 0.0036
0.4374 ± 0.0196
0.0895 ± 0.0055
ResNet-50 (random)
0.1296 ± 0.0034
0.4159 ± 0.0171
0.0787 ± 0.0057
ViT-B MAE (ImageNet)
0.1236 ± 0.0119
0.3798 ± 0.0460
0.0703 ± 0.0109
ConvNeXt-B (IN22k)
0.1464 ± 0.0060
0.4400 ± 0.0249
0.0933 ± 0.0039
ConvNeXtV2-B (IN22k)
0.1345 ± 0.0085
0.3949 ± 0.0545
0.0783 ± 0.0107
SwinV2-B (ImageNet)
0.1552 ± 0.0086
0.4586 ± 0.0308
0.1090 ± 0.0146
Table 5: Crater detection on LRO NAC (meter-scale) at 100% training data. Test-set bounding-box mean average precision. Values are mean ± standard deviation over five random seeds. Bold indicates the best value per column; underline the second best; differences smaller than the reported seed spread should not be read as a ranking.
Figure 8: Example of predictions for the WAC Robbins Crater Catalog (first row) and the NAC Hand Labeled Dataset (second row). The first column ( a and c ) shows predictions over ground truth for NASA-IBM LFM, with the second column ( b and d ) showing the same for SwinV2-B. Green: ground truth; blue: predictions at confidence ≥ 0.5.
Model
IoU 1
F1 1
ResNet-50 (ImageNet)
0.4731 ± 0.0114
0.6423 ± 0.0105
ResNet-50 (random)
0.3496 ± 0.0323
0.5175 ± 0.0355
ViT-B MAE (ImageNet)
0.5541 ± 0.0369
0.7125 ± 0.0298
ConvNeXt-B (IN22k)
0.5374 ± 0.0197
0.6989 ± 0.0168
ConvNeXtV2-B (IN22k)
0.5687 ± 0.0181
0.7249 ± 0.0146
SwinV2-B (ImageNet)
0.5555 ± 0.0159
0.7141 ± 0.0132
Table 6: Irregular Mare Patch (IMP) segmentation. IMP delineation is a binary dense-prediction task, so we report test-set metrics on the positive (IMP) class only: intersection-over-union (IoU 1 ) and F1 score (F1 1 ). Values are mean ± standard deviation over five random seeds. Bold indicates the best mean per column and underline the second best; differences smaller than the reported seed spread should not be read as a ranking.
Figure 9: Qualitative IMP segmentation predictions. Our model and the baselines produce broadly comparable outputs; in the second example, the pretrained NASA-IBM LFM identifies IMP extents that none of the baselines detect. Green overlays mark ground-truth IMP extents; colored overlays show model predictions. The colors used in the maps in the top row are for visualization purposes only and do not convey any specific meaning.
Model
RMSE ↓
MAE ↓
R2↑
ResNet-50 (ImageNet)
0.0788 ± 0.0004
0.0568 ± 0.0004
0.9162 ± 0.0009
ResNet-50 (random)
0.0695 ± 0.0007
0.0495 ± 0.0005
0.9347 ± 0.0013
ViT-B MAE (ImageNet)
0.0917 ± 0.0009
0.0674 ± 0.0007
0.8865 ± 0.0021
ConvNeXt-B (IN22k)
0.0442 ± 0.0008
0.0301 ± 0.0005
0.9737 ± 0.0009
ConvNeXtV2-B (IN22k)
0.0439 ± 0.0008
0.0298 ± 0.0006
0.9739 ± 0.0010
SwinV2-B (ImageNet)
0.0377 ± 0.0004
0.0256 ± 0.0003
0.9808 ± 0.0004
Table 7: Polar ice prospectivity regression on the full eight-modality polar stack (aspect, slope, DICE, TMAX, LPSR, LPSR_DEN, LPSR_DIS, CUR). Test-set metrics against the prospectivity map in [ 44 ] : RMSE and MAE lower-is-better, R2 higher-is-better. Values are mean ± standard deviation over five random seeds. Bold indicates the best value per column; underline the second best.
Number of input modalities m
Model
2
3
4
5
6
7
8
(as, sl)
+DICE
+TMAX
+LPSR
+DEN
+DIS
+CUR
ConvNeXt-B (IN22k)
0.2083 ( ± 0.0086)
0.0614 ( ± 0.0007)
0.0522 ( ± 0.0007)
0.0513 ( ± 0.0004)
0.0509 ( ± 0.0005)
0.0517 ( ± 0.0007)
0.0437 ( ± 0.0002)
NASA-IBM LFM (ps8)
0.1981 ( ± 0.0063)
0.0434 ( ± 0.0006)
0.0412 ( ± 0.0009)
0.0393 ( ± 0.0002)
0.0403 ( ± 0.0010)
0.0398 ( ± 0.0004)
0.0288 ( ± 0.0011)
Table 8: Polar ice prospectivity regression, modality-count ablation. Each column corresponds to results from models trained on a fixed set of polar input variables, starting from the two pretrained modalities (aspect, slope) at m=2 and incrementally adding polar-stack modalities not seen during pretraining, in the order DICE, TMAX, LPSR, LPSR_DEN, LPSR_DIS, and finally CUR at m=8 . This isolates the contribution of each additional benchmark-only modality on top of the pretrained backbone. Values are test-set RMSE (lower is better). Values are mean ± standard deviation over three random seeds. Bold indicates the best value per column.
Figure 10: Qualitative ice prospectivity predictions on test patches. NASA-IBM LFM recovers PSR-boundary sharpness and fine spatial detail that the ImageNet-pretrained baseline smooths out. The last row shows our model prediction using only slope, aspect, and DICE, illustrating competitive performance even with a drastically reduced input set.
Tokenizer
Modality
GPUs
Global Batch
Epochs
Warmup Epochs
Peak LR
NAC
nac_aspect_3m
Aspect
16
512
100
2
3.06e-6
nac_dtm_3m
DTM
16
512
100
2
3.06e-6
nac
Panchromatic
16
512
100
2
3.06e-6
nac_slope_3m
Slope
16
512
60
2
3.06e-6
WAC
Table 9: Per-run training configuration for the nine domain-specific lunar tokenizers. All runs share a per-GPU batch size of 32.
Figure 11: Additional generation examples from WAC to all other modalities.