HiPredict: Predicting the missing HiRISE RED4 spectral channel with machine learning
Authors: Ardan Suphi, Ben Moseley, Gareth S Collins, Valentin T Bickel, Vidhya G Rangarajan
Organizations: Department of Earth Science and Engineering, Imperial College London, London, UK · Center for Space and Habitability, University of Bern, Bern, Switzerland · Institute of Space Research, German Aerospace Center (DLR), Berlin, Germany
The High Resolution Imaging Science Experiment (HiRISE) onboard the Mars Reconnaissance Orbiter (MRO) acquires imagery of Mars at 25--30 cm/pixel, supporting studies of martian geology, climate history, and landing-site characterization. Its 14-detector focal plane assembly includes several overlapping detectors that enable the acquisition of three-channel colour-infrared images, which are important for identifying compositional variations and studying active surface processes. In 2023, a detector in the center of the focal plane assembly (RED4) began failing intermittently, creating a gap in acquired imagery and reducing the instrument's colour-imaging capability by half. The HiRISE processing pipeline provides a synthetic red (SYN4) channel obtained by linearly scaling the co-located infrared (IR10) channel. However, this approach cannot capture observed non-linear relationships between the colour channels, limiting its accuracy and applicability. Here, we present HiPredict, a machine-learning framework that reconstructs RED4 from co-located IR10 and blue-green (BG12) imagery, neighbouring red-channel observations, and acquisition metadata. The model employs a U-Net architecture to learn non-linear spatiotemporal-spectral relationships between these inputs and is trained on 81,717 training samples, representing 13,625 observations. Evaluated on 23,348 test samples, representing 3,892 pre-failure observations, HiPredict reconstructs RED4 with a Normalised Mean Absolute Error of 1.1%, substantially outperforming SYN4 (3.1%). The reconstructed products improve colour image recovery and more closely represent features of interest such as icy craters, stratigraphy exposed along ridge crests and chloride deposits. We release HiPredict as an open software tool. This work demonstrates the potential of machine learning to mitigate data loss as planetary missions age and naturally degrade.
Figures & tables
Figure 1: Three recently-acquired HiRISE images (RED bandpass-only) showing the central gap that results from the RED4 failure. The gap induced by the RED9 failure is not directly visible.
Figure 2: A comparison showing the impact of the two CCD failures. As the BG12 and IR10 detectors overlap with RED4, this band can be predicted but as there is nothing which images in the RED9 spatial area, this data cannot be recovered. Observation ESP_023753_1660 ( −13.8236∘,309.643∘E ).
Figure 3: Overview of HiPredict - (A) Overview of the HiPredict architecture showing the inputs and two model heads. The inputs, top to bottom, are the two band images from the target location, the overlap images from neighbouring CCDs, and metadata. (B) The spatial head, a U-Net based architecture. (C) The brightness head, a lightweight convolutional operator.
TDI (%)
Binning (%)
128
64
Other
1
2
Other
Train
75.2
24.8
0.0
34.0
66.0
0.0
True
70.2
24.4
5.3
30.6
65.8
3.6
Table 1: TDI and binning distributions of the training and full HiRISE datasets.
NMAE (%) ↓
SSIM ↑
PWT 1% ↑
Name
mean
median
mean
median
mean
median
Linear
4.012
3.508
0.542
0.596
4.926
2.632
Linear with meta
4.234
3.594
0.544
0.588
4.851
2.634
Basic UNet
2.064
1.339
0.687
0.747
13.841
12.750
Customised UNet
6.453
2.990
0.632
0.690
6.729
2.924
SYN4
3.118
1.545
0.663
0.722
11.069
9.148
Table 2: Performance metrics for unseen data. Arrows indicate if higher or lower values are better.
Figure 4: Top row: True HiRISE images using the original RED4. Bottom row: HiPredict predictions for the same areas. Images are plotted in ‘IRB’ format, meaning the infrared/red/blue-green bands are mapped to the red/green/blue channels to create the colour image, which helps to highlight subtle surface composition differences. Left column: ESP_055232_1595 ( −20.1668∘,226.817∘E ). Middle column: ESP_056705_1440 ( −35.4419∘,342.328∘E ). Right column: ESP_012940_1655 ( −14.0974∘,318.282∘E ).
Figure 5: Using high SYN4 error as a proxy for colourful and complex images, we can evaluate HiPredict performance on this subset.
Figure 6: Log-transformed SHAP analysis of six most consequential metadata values provided to the model, excluding the top 10 outliers. Points to the left of center have a positive impact on model performance, points to the right a negative impact. Red points indicate a larger value for that metadata item and blue a lower value. Verticality indicates density of data points. The complete SHAP plot can be seen in Supplemental Information S6
Figure 7: A deeper comparison of the models using icy impacts as examples.
Image reconstruction recovers images, volumes, or scenes from incomplete, noisy, indirect, or compressed measurements using physical models and learned priors.
High-resolution orbital imagery offers a rich record of the Martian surface, but sparse geological labels limit supervised representation learning. We present MarsRecon, a geospatially aware pipeline for learning visual and multimodal representations from HiRISE observations of Olympus Mons. The pipeline calibrates NASA Planetary Data System products, extracts valid georeferenced patches, and trains a masked autoencoder on unlabeled imagery. Increasing input resolution and filtering invalid tokens reduced held-out reconstruction loss from 0.1751 to 0.1342 in the principal Stage A model series. We then freeze the visual encoder and align its features with observation text, coordinates, and local--global image context. The strongest current local-primary model achieves image-to-text recall@10 of 0.3787, text-to-image recall@10 of 0.9161, and local-to-global recall@10 of 0.4350 on the held-out test split. These results establish a working Mars-specific pretraining and retrieval pipeline; further crop-overlap controls and downstream geological evaluations are needed to assess the broader utility of its embeddings.
Akshay Naik, Marius F. R. Juston, Jay Mahajan
Department of Electrical and Computer Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA · Department of Computer Science University of Illinois Urbana-Champaign Urbana, IL 61801-3080, USA
Aerial navigation on Mars requires vision-based pipelines that are robust to the diverse illumination conditions and terrain morphology of the Martian surface. A key bottleneck for training and evaluating such methods is the scarcity of large-scale, annotated aerial datasets. We present MARTIAN, an open-source Blender-based rendering framework that leverages real HiRISE orbital map products to synthesize realistic aerial views of the Martian terrain under controllable lighting conditions and at varying altitudes. MARTIAN generates observations with accurate pose annotations, directly addressing the scarcity of training data for vision-based navigation on Mars. The framework has been validated through its deployment in concurrent work on map-based localization systems for Ingenuity and future Mars rotorcraft, where synthetically trained deep image matchers were successfully evaluated on real Mars imagery. MARTIAN is publicly available at: https://github.com/nasa-jpl/martian.
Dario Pisanti, Georgios Georgakis
Space Robotics Research Group, SnT, University of Luxembourg. · Jet Propulsion Laboratory, California Institute of Technology.
We present MarsFM, an image-conditioned latent flow-matching model for local Martian relief estimation from single-band HiRISE RED orthoimagery. The method combines a pretrained generative prior with stereo-derived geometric supervision and a differentiable Lunar--Lambert shading objective. Relief, normal, gradient, curvature, and ordinal terms constrain complementary aspects of terrain structure, while a positive-affine-invariant image comparison constrains rendered appearance. An evaluation comprising 2024 gathered patch records per integration-step count yields mean affine-aligned RMSE between 0.0935 and 0.0957 in normalized signed-log relief space for one to twenty Euler steps. These scores measure agreement with VAE-reconstructed references on positive-reference support. Their narrow range supports low-step inference under this protocol. Spatial, differential, and spectral diagnostics show broad terrain correspondence alongside smoothing, amplitude compression, and boundary mismatch. MarsFM provides a framework for combining learned terrain priors with image-based constraints; establishing improved physical terrain resolution requires matched baselines and independent high-resolution reference data. Data: https://huggingface.co/datasets/SuperComputer/mars_hirise_dtm_processed-6aa9b66ba461e07f; code: https://github.com/Marius-Juston/MarsRecon.
Marius F. R. Juston
Department of Industrial and Enterprise Systems Engineering The Grainger College of Engineering University of Illinois Urbana-Champaign, Urbana IL 61801-3080, USA