cs.CVSep 17, 2026
SaveMarsRecon: Self-Supervised and Multimodal Surface Representations for Mars
Organizations: Department of Electrical and Computer Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA · Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA
Abstract
High-resolution orbital imagery offers a rich record of the Martian surface, but sparse geological labels limit supervised representation learning. We present MarsRecon, a geospatially aware pipeline for learning visual and multimodal representations from HiRISE observations of Olympus Mons. The pipeline calibrates NASA Planetary Data System products, extracts valid georeferenced patches, and trains a masked autoencoder on unlabeled imagery. Increasing input resolution and filtering invalid tokens reduced held-out reconstruction loss from 0.1751 to 0.1342 in the principal Stage A model series. We then freeze the visual encoder and align its features with observation text, coordinates, and local--global image context. The strongest current local-primary model achieves image-to-text recall@10 of 0.3787, text-to-image recall@10 of 0.9161, and local-to-global recall@10 of 0.4350 on the held-out test split. These results establish a working Mars-specific pretraining and retrieval pipeline; further crop-overlap controls and downstream geological evaluations are needed to assess the broader utility of its embeddings.
Figures & tables
| Run | Image | Patch px | Epochs | Best val loss | Interpretation |
| 128/p8 e20 | 128 | 8 | 20 | 0.1751 | Baseline p8 configuration |
| 128/p8 e80 | 128 | 8 | 80 | 0.1510 | Longer training helped but remained above 256/p8 |
| 256/p8 validmask v1 | 256 | 8 | 20 | 0.1479 | Higher resolution plus valid-mask-aware training |
| 256/p8 strict80 | 256 | 8 | 20 | 0.1342 | Best p8 metrics-of-record |
| 256/p4 strict80 | 256 | 4 | 20 | 0.0964 | Lower loss but caveated by easier tiny-token reconstruction |
Table 1: Stage A masked-autoencoder ablations. Within the main line, higher resolution and stricter valid-token filtering improve held-out reconstruction loss. The row is shown as a caveated ablation because small patch tokens can make reconstruction easier without proving better downstream representation quality.
| Run | Align | I2T@10 | T2I@10 | L2G@10 | Interpretation |
| B1a geo+ | 0.4918 | 0.3671 | 0.9068 | n/a | Image/text/coordinate alignment baseline |
| Center proxy | 0.5090 | 0.3867 | 0.9228 | 0.6189 | Strong but easier nested-crop control |
| True expanded 0.015 | 0.4451 | 0.3129 | 0.9057 | 0.2402 | Roadmap-aligned but difficult |
| Local-primary 0.015 lg0.25 | 0.4995 | 0.3744 | 0.8939 | 0.2121 | Recovers image/text quality but weak local/global retrieval |
| Local-primary 0.010 lg0.25 final | 0.4981 | 0.3787 | 0.9161 | 0.4350 | Best practical candidate so far |
Table 2: Stage B held-out retrieval summary. Center-crop retrieval is strong but too easy. True expanded global context is harder. Local-primary training with smaller global context gives the most promising current local/global retrieval.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Run | N | Align | I2T@10 | T2I@10 | GlobalText | L2G@10 | G2L@10 | Overlap@10 |
| B1a geo latlon | 12285 | 0.4736 | 0.3410 | 0.9289 | n/a | n/a | n/a | n/a |
| B1a geo+ coords rff-siren | 12285 | 0.4918 | 0.3671 | 0.9068 | n/a | n/a | n/a | n/a |
| center-crop proxy | 12285 | 0.5090 | 0.3867 | 0.9228 | n/a | 0.6189 | 0.6170 | n/a |
| true expanded 0.015 | 12285 | 0.4451 | 0.3129 | 0.9057 | n/a | 0.2402 | 0.2470 | n/a |
| train-only augment | 12285 | 0.0820 | 0.0609 | 0.1310 | n/a | 0.2401 | 0.2447 | n/a |
| dual text expanded | 12285 | 0.4463 | 0.2595 | 0.9036 | n/a | 0.2214 | 0.2109 | n/a |
Table 3: Stage B full metrics ledger. This table preserves the broader run history used to interpret the compact Stage B result table in the main text. Center-crop is a high-capacity/easy control, train-only augmentation is a negative result, and local-primary runs are the strongest current practical candidates.
Figure 3: MarsRecon pipeline architecture. Stage 0 ingests NASA PDS HiRISE products, performs reprojection and radiometric calibration, and emits valid patch-level training samples. Stage A trains a masked autoencoder on valid Mars surface tokens. Stage B extends the frozen visual backbone into a multimodal embedding space using text, coordinate, and local/global context.
Figure 4: Dataset loader and sampler architecture. The data system connects PDS product discovery, Mars CRS/projection handling, radiometric calibration, valid-footprint estimation, and patch sampling into one trainable HiRISE dataset interface.
Figure 5: Authentic Stage B retrieval examples from the held-out test set. Each row shows a real local query patch, the model’s top retrieved global-context crop, and the top five retrieved rationale strings from the final local-primary / Stage B run. Green text bars mark the true rationale when it appears in the top five; orange bars show confounding alternatives. Thumbnails are grayscale contrast-stretched for display, but the ranks and similarity scores are computed directly from exported model embeddings.
| Band | Wavelength range | Primary use |
| Blue–Green | 400–600 nm | Frost, ice, atmospheric scattering, and color context |
| RED | 550–850 nm | High-resolution geomorphology, structural texture, and topographic interpretation |
| Near-Infrared | 800–1000 nm | Mineralogical differentiation and spectral contrast where available |
Table 4: HiRISE spectral bands used by the MarsRecon data pipeline. The RED channel provides the highest-resolution structural texture signal, while blue-green and near-infrared channels provide additional spectral context where color products are available [ 1 , 25 ] .
| Channel | Mean | Std | Min | Max | P2 | P98 | CV / DRCR | |
| Near-infrared | 0.1965 | 0.0569 | 0.0097 | 0.7700 | 0.0928 | 0.3125 | 0.290 / 1.118 | |
| RED | 0.1668 | 0.0391 | 0.0097 | 0.3600 | 0.0879 | 0.2471 | 0.235 / 0.954 | |
| Blue-green | 0.0723 | 0.0211 | 0.0097 | 0.3184 | 0.0389 | 0.1270 | 0.292 / 1.218 |
Table 5: Radiometrically calibrated dataset statistics. Statistics are computed over the Olympus patch corpus after DN-to-I/F conversion. P2/P98 are the 2nd and 98th percentiles, CV is the coefficient of variation, and DRCR is .
Figure 6: Radiometrically calibrated pixel distributions. The histograms summarize calibrated I/F values for the Olympus patch corpus and show why dataset-level normalization and percentile-aware visualization are useful for training and qualitative inspection.
| Parameter | Example value | Role in MarsRecon |
| OBSERVATION_ID | TRA_000873_2015 | Unique orbital capture identifier. |
| PRODUCT_ID | TRA_000873_COLOR | Product-specific identifier, including color/RED distinction. |
| OBSERVATION_START_TIME | 2006-10-03T12:51:33.015 | Timestamp for temporal and mission-phase context. |
| MAP_PROJECTION_TYPE | EQUIRECTANGULAR | Per-product map projection used before transformation into the common Mars geographic CRS. |
| MAP_SCALE | 0.25 m/pixel | Spatial resolution at the projected center latitude. |
| CENTER_LATITUDE / LONGITUDE | 20.000 ∘ , 180.000 ∘ | Planetocentric map center and coordinate context. |
Table 6: Representative PDS metadata fields used by the data pipeline. These fields make each patch geospatially and physically interpretable rather than a detached image crop.
Figure 7: HiRISE coverage and Olympus subset. Blue points show available HiRISE sample locations in the plotted range, while red points show the project subset selected around Olympus Mons. The base topographic map is from USGS Mars topography products [ 26 ] .
Figure 8: Valid-support estimation for a HiRISE strip. The gray rectangle is the metadata bounding box, while the valid image footprint is narrower and angled. Valid-mask-aware sampling prevents the model from learning artificial nodata patterns.
Figure 9: Example Olympus patches. Random examples from the selected subset illustrate the range of surface textures, illumination conditions, and strip geometries seen during training.
Figure 10: HiRISE targeting vocabulary. The word clouds summarize the most prominent terms in observation rationales after stopword filtering. They show that HiRISE text is strongly geologic rather than generic metadata, with repeated concepts such as craters, dunes, layers, polar processes, slopes, deposits, and channels.
Figure 11: Top rationale words and phrases. The script counts filtered unigrams and adjacent token bigrams, making the text distribution auditable rather than relying only on the word-cloud visualization.
Figure 12: From words to geological categories. Geographic-feature counts track named Mars feature types such as craters, chasmata, valles, fossae, and montes. Theme counts group rationales into science categories including impact craters, gullies/slopes, aeolian processes, polar/ice/frost, layers/stratigraphy, mineralogy, volcanic processes, tectonics, and landing-site/engineering observations.
Figure 13: Observation map colored by primary science theme. Each observation is assigned a primary theme from its matched keyword list and plotted by longitude/latitude. This connects the Stage B text signal back to physical Mars geography.
Figure 14: Theme-specific Mars maps. Small multiples reduce overplotting by drawing one hex-density map per science theme, revealing where different rationale classes cluster spatially.
Figure 15: Dominant theme per Mars region. Each spatial bin is colored by the locally dominant primary theme, with opacity scaled by log observation count. This preserves a single-map view while reducing the overplotting problem of point-wise thematic maps.
Figure 16: Theme imbalance and latitude dependence. The treemap shows the global class imbalance in rationale-derived science themes, while the stacked latitude-band plot shows that theme frequencies vary by equatorial, mid-latitude, and polar regions. This helps explain why Stage B should combine text with location rather than treating rationales as globally exchangeable labels.
Figure 17: Valid-center region from overlap constraints. The green polygon shows where patch centers can be placed while maintaining the minimum required overlap with the valid strip footprint. This converts a nodata-prone bounding-box problem into a constrained sampling problem.
Figure 18: Simple grid versus independent strip packing. The strip-aware method adapts each row to the valid center region, improving valid patch yield over rigid column-locked grids for angled orbital strips.
Figure 19: Sampler robustness across strip geometries. Across synthetic angled strip footprints, the boundary-aware method consistently finds more valid patch centers than a simple grid.
Figure 20: Boundary-aware patch sampling. The sampler increases valid patch density by adapting candidate centers to the valid strip footprint rather than relying on a rigid grid over the metadata bounding box.
Figure 21: Sampling sensitivity. The sweeps show how patch yield and coverage respond to stricter valid-overlap requirements and different patch-overlap strides. These curves justify treating patch sampling as an explicit design choice rather than a hidden preprocessing detail.
Figure 22: Preprocessing workflow. JPEG2000 HiRISE products are transformed into tiled, metadata-preserving assets for efficient random patch access during training.
More in this topic
Geospatial Representation Learning
Geospatial representation learning builds reusable embeddings of places, imagery, maps, mobility, and Earth data for spatial prediction and reasoning.
See all 133 papers