SpecTM: Spectral Targeted Masking for Trustworthy Foundation Models

Syed Usama Imtiaz; Mitra Nasr Azadani; Nasrin Alamdari

cs.AIMar 23, 2026

SpecTM: Spectral Targeted Masking for Trustworthy Foundation Models

Authors: Syed Usama Imtiaz, Mitra Nasr Azadani, Nasrin Alamdari

Organizations: Department of Civil and Environmental Engineering Florida State University, Tallahassee, FL, USA

Abstract

Foundation models are now increasingly being developed for Earth observation (EO), yet they often rely on stochastic masking that do not explicitly enforce physics constraints; a critical trustworthiness limitation, in particular for predictive models that guide public health decisions. In this work, we propose SpecTM (Spectral Targeted Masking), a physics-informed masking design that encourages the reconstruction of targeted bands from cross-spectral context during pretraining. To achieve this, we developed an adaptable multi-task (band reconstruction, bio-optical index inference, and 8-day-ahead temporal prediction) self-supervised learning (SSL) framework that encodes spectrally intrinsic representations via joint optimization, and evaluated it on a downstream microcystin concentration regression model using NASA PACE hyperspectral imagery over Lake Erie. SpecTM achieves R^2 = 0.695 (current week) and R^2 = 0.620 (8-day-ahead) predictions surpassing all baseline models by (+34% (0.51 Ridge) and +99% (SVR 0.31)) respectively. Our ablation experiments show targeted masking improves predictions by +0.037 R^2 over random masking. Furthermore, it outperforms strong baselines with 2.2x superior label efficiency under extreme scarcity. SpecTM enables physics-informed representation learning across EO domains and improves the interpretability of foundation models.

Figures & tables

Fig. 1: (A) PACE OCI Level-3 hyperspectral imagery (122 bands, 2 km, 8-day composite) is combined with meteorological predictors (GridMET; 52 features). Spectral tokens are constructed and augmented with positional encoding, while key diagnostic bands (e.g., PC absorption, chlorophyll-a red, and red-edge regions) are masked to enforce physics-guided learning. A multi-head self-attention transformer processes the inputs and is jointly trained on three tasks. (B) The pretrained encoder is frozen and coupled with a trainable MLP head to predict microcystin concentrations 8 days ahead, evaluated using R², RMSE, and correlation $\rho$ and lists Bio-optical indices and meteorological variables used.

Fig. 2: Experimental results. (a) SpecTM outperforms all baselines for both current-week (+34%) and 8-day-ahead (+99%) microcystin prediction. (b) Ablation analysis: targeted masking improves over random by +0.037 $R^{2}$ ; SSL pretraining provides +0.18 $R^{2}$ gain over random initialization.

Fig. 3: SSL pretraining validation. (a) Single-sample spectral reconstruction showing visible context bands (blue), true masked values (red), and predicted values (green) across diagnostic wavelengths (615–720 nm). (b) Aggregate reconstruction accuracy across 4,096 samples ( $r{=}1.000$ , RMSE ${=}0.018$ ). Shaded region: 95% CI.

Fig. 4: Label efficiency under data scarcity. SpecTM achieves $1.8\times$ (current) and $2.2\times$ (8-day-ahead) improvement over baseline at 5% labeled data. Shaded regions: $\pm$ 1 SD across five stratified subsamples.

Explore similar work

HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing
Sep 29, 2026Li Pang, Xinqiao Wu, Jing Yao +5Synthetic Data Augmentation Vision Foundation Model Adaptation
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China +4
Scalable and Trustworthy Earth Observation Foundation Models
Jul 8, 2026Syed Usama Imtiaz, Mitra Nasr Azadani, Nasrin AlamdariGeospatial Foundation Models Vision Foundation Model Adaptation
Department of Civil and Environmental Engineering, Florida State University
Reuse or Relearn? A Spectral View of Earth Observation Foundation Models
Sep 26, 2026Mehmet Ozgur Turkoglu, Valerio Marsocci, Dominik J. Mühlematter +3Fine-Tuning Geospatial Foundation Models
Agroscope, EO of Agroecosystems · ESA, Φ-lab · ETH Zurich

Sep 29, 2026J/K move · Enter open · S save

HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing

Li Pang, Xinqiao Wu, Jing Yao, Pedram Ghamisi+4

School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China+4

Synthetic Data Augmentation Vision Foundation Model Adaptation Hyperspectral Imaging

Open paper AskPDF arXiv

Hyperspectral remote sensing provides dense spectral measurements that are indispensable for material-level Earth observation, yet the construction of a general-purpose hyperspectral foundation model remains difficult. Two bottlenecks are especially limiting. First, large hyperspectral corpora rarely provide high spatial resolution together with reliable dense annotations. Second, many hyperspectral models are still trained almost from scratch, so the geometric and interactive priors learned by modern vision foundation models are not fully reused. To alleviate these issues, we \highlight{present} HyperSAM, a promptable hyperspectral foundation model that couples a data-centric hyperspectral synthesis pipeline with a spectral adaptation architecture based on Segment Anything Model 3 (SAM3). On the data side, HyperSAM synthesizes full-spectrum hyperspectral cubes from high-resolution SpaceNet multispectral imagery through a physics-informed abundance-transfer generator, while SAM3-derived pseudo-masks provide object-centric supervision. On the model side, the latest implementation uses a frozen SAM3 RGB image branch, a trainable hyperspectral side encoder initialized from the RGB vision transformer (ViT), ControlNet-style zero-initialized feature injection, and a lightweight mixture-of-experts mask refiner. To enhance training robustness against noisy pseudo-labels, Cross-modal Sample Selection (CromSS)-style confidence selection is incorporated for noisy-label weighting. Extensive experiments show that HyperSAM obtains strong generalization on diverse hyperspectral tasks (e.g., classification, anomaly detection, change detection, target detection, and airborne oil-spill mapping) and that high-quality synthetic hyperspectral data can be more effective than simply scaling noisy hyperspectral supervision.

Abstract

Explore similar work

HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing

Scalable and Trustworthy Earth Observation Foundation Models

Reuse or Relearn? A Spectral View of Earth Observation Foundation Models