SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing

Xiaokang Zhang; Bo Li; Chufeng Zhou; Weikang Yu; Lefei Zhang

cs.CVMar 8, 2026

SIGMAE: A Spectral-Index-Guided Foundation Model for Multispectral Remote Sensing

Authors: Xiaokang Zhang, Bo Li, Chufeng Zhou, Weikang Yu, Lefei Zhang

Organizations: School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence and Automation, Wuhan University of Science and Technology, Wuhan, 430081, China · School of Electronic Information, Wuhan University of Science and Technology, Wuhan, 430081, China · Helmholtz-Zentrum Dresden-Rossendorf, Freiberg, 09599, Germany ·

Abstract

Pretraining and fine-tuning have emerged as a new paradigm in remote sensing image interpretation. Among them, Masked Autoencoder (MAE)-based pretraining stands out for its strong capability to learn general feature representations via reconstructing masked image regions. However, applying MAE to multispectral remote sensing images remains challenging due to complex backgrounds, indistinct targets, and the lack of semantic guidance during masking, which hinders the learning of underlying structures and meaningful spatial-spectral features. To address this, we propose a simple yet effective approach, Spectral Index-Guided MAE (SIGMAE), for multispectral image pretraining. The core idea is to incorporate domain-specific spectral indices as prior knowledge to guide dynamic token masking toward informative regions. SIGMAE introduces Semantic Saliency-Guided Dynamic Token Masking (SSDTM), a curriculum-style strategy that quantifies each patch's semantic richness and internal heterogeneity to adaptively select the most informative tokens during training. By prioritizing semantically salient regions and progressively increasing sample difficulty, SSDTM enhances spectrally rich and structurally aware representation learning, mitigates overfitting, and reduces redundant computation compared with random masking. Extensive experiments on five widely used datasets covering various downstream tasks, including scene classification, semantic segmentation, object extraction and change detection, demonstrate that SIGMAE outperforms other pretrained geospatial foundation models. Moreover, it exhibits strong spatial-spectral reconstruction capability, even with a 90% mask ratio, and improves complex target recognition under limited labeled data. The source codes and model weights will be released at https://github.com/zxk688/SIGMAE.

Figures & tables

Figure 1: Performance comparison of remote sensing foundation models across five diverse datasets, where our SIGMAE achieved superior generalization capability.

Figure 2: Overview of the proposed SIGMAE framework: an asymmetric encoder–decoder architecture equipped with Semantic Saliency-guided Dynamic Token Masking (SSDTM) that uses spectral-domain priors to adaptively select informative regions and enhance feature discriminability during reconstruction.

Figure 3: Details of network structures of the (a) encoder, (b) reconstruction decoder and (c) for downstream tasks.

Dataset	Task	Image Size	Fine-tuning	Testing
FOD	Seg.	$224\times 224$	6,087	3,704
Wildfire Detection	Seg.	$224\times 224$	2,263	4,053
EuroSAT	Cls.	$64\times 64$	21,600	5,400
SegMunich	Seg.	$128\times 128$	39,402	9,846
OSCD	CD	$224\times 224$	7,160	3600

Table 1: Summary of datasets used in the downstream tasks

Model	Arch.	Pretrained EO Data	Learning Strategy	Params (M)	Patch Volume
CROMA (NIPS’24)	ViT	SSL4EO-S12 ( Wang et al., 2023 )	Contrastive	396.13	3M
SatlasNet (ICCV’23)	Swin-T	SatlasPretrain ( Bastani et al., 2023 )	Supervised	128.57	856K
S12-MAE (GRSM’23)	ViT	SSL4EO-S12 ( Wang et al., 2023 )	MIM	61.99	3M
ScaleMAE (ICCV’23)	ViT	fMoW-RGB ( Christie et al., 2018 )	MIM	396.21	363.6K
SpectralGPT (TPAMI’24)	ViT	fMoW-S2, BigEarthNet ( Sumbul et al., 2019 )	MIM	614.75	1.47M
SoftCon (TGRS’24)	ViT	SSL4EO-S12 ( Wang et al., 2023 )	Contrastive	242.19	3M

Table 2: Foundation models used in benchmarking, with architecture, pretraining data, strategy, size, patch volume, and release year.

Hyperparameters	Pretraining	Fine-tuning
Optimizer	AdamW	AdamW
Base learning rate	1e-4	1e-4
Weight decay	0.05	0.001
Adam $\beta$	(0.9, 0.95)	(0.9, 0.999)
Layer-wise lr decay	–	0.75
Batch size	900	20

Table 3: Parameter Settings in the Pretraining and Fine-tuning.

Model	FOD				Wildfires Detection				OSCD				EuroSAT
	mIoU	F1	Recall	Precision	mIoU	F1	Recall	Precision	mIoU	F1	Recall	Precision	mIoU	OA	mF1
CROMA	57.10	62.87	72.68	59.05	89.52	89.52	95.59	84.18	58.92	66.98	76.53	62.80	95.13	97.61	97.49
S12-MAE	56.59	62.09	60.58	64.15	89.31	89.28	84.59	94.53	57.54	65.09	76.72	60.90	95.51	97.85	97.73
SatlasNet	57.00	62.79	62.62	62.98	90.70	90.78	94.96	86.96	55.22	61.58	87.67	57.39	95.09	97.59	97.44
ScaleMAE	56.16	61.49	67.24	58.66	85.79	85.25	82.15	88.60	52.26	57.09	72.60	54.70	94.55	97.25	97.14
SpectralGPT	55.69	60.68	74.28	56.93	89.97	90.00	85.44	95.07	61.66	70.50	79.49	63.30	95.27	97.94	97.72

Table 4: Quantitative Results on FOD, Wildfires, OSCD, and EuroSAT Datasets for Segmentation, Change Detection, and Classification Tasks (%)

Method	Background	Arable land	Perm. Crops	Pastures	Forests	Surface water	Shrub	Open spaces	Wetlands	Mine, dump	Artificial veg.	Urban fabric	Buildings	mF1
CROMA	90.64	82.77	21.42	64.07	89.12	70.51	20.62	33.32	44.72	40.57	27.79	78.33	67.25	56.24
S12-MAE	89.48	81.89	22.25	62.24	88.51	71.25	16.73	34.80	44.08	40.94	25.51	76.32	64.25	55.25
SatlasNet	90.81	81.69	17.98	61.32	88.43	81.30	16.45	43.81	50.44	47.08	22.44	75.93	63.69	57.03
ScaleMAE	85.12	81.14	20.04	60.82	87.62	70.49	15.77	29.45	44.29	39.47	25.52	77.32	65.57	54.05
SpectralGPT	87.80	83.34	31.24	65.53	89.11	85.84	21.31	46.05	46.41	52.59	23.98	78.97	65.18	59.80
DOFA	89.91	81.97	21.51	62.41	88.65	71.09	18.71	33.04	43.11	39.78	27.99	77.57	66.21	55.86

Table 5: Quantitative Results on SegMunich Dataset (%)

Figure 4: Statistical characterization of spectral attributes and patch-level saliency indicators in BigEarthNet-S2 dataset. (a) Probability density functions of raw spectral indices (NDVI, NDWI, and NDBI) within the BigEarthNet-S2 dataset. (b) Distribution of the sum of absolute values derived from the indices, representing semantic richness. (c) Probability density of the resulting SSM.

Model Variants	FOD		Wildfire		EuroSAT		SegMunich		OSCD
	mIoU	F1	mIoU	F1	mIoU	mF1	mIoU	mF1	mIoU	F1
SIGMAE w/o SSDTM	59.19	65.96	89.92	89.96	94.54	97.15	46.16	59.98	65.04	74.51
MAE w/ Index Fusion	59.70	67.03	89.06	89.04	95.16	97.50	46.31	59.96	64.96	74.44
SIGMAE w/o $\gamma(e)$	61.23	68.78	90.67	90.73	94.58	97.18	46.44	60.16	66.07	75.70
SIGMAE (Full)	61.21	68.87	91.12	91.20	95.53	97.70	47.45	60.90	65.45	75.03

Table 6: Ablation Studies (%)

Figure 5: Spectral reconstruction performance comparison.

Figure 6: Image reconstruction performance comparison with varied masking ratios of $50\%$ , $75\%$ and $90\%$ , respectively.

Figure 7: Convergence performance comparison with different models and different mask ratios.

Figure 8: Segmentation performance on the Floating object detection and wildfire detection datasets.

Figure 9: Semantic segmentation performance on the SegMunich dataset.

Figure 10: Change detection performance on the OSCD dataset.

Remote Sensing Image Understanding

Remote sensing image understanding uses AI to interpret satellite and aerial imagery for objects, land cover, change, hazards, and spatial relationships.

See all 207 papers

Apr 30, 2026

A generalised pre-training strategy for deep learning networks in semantic segmentation of remotely sensed images

In the segmentation of remotely sensed images, deep learning models are typically pre-trained using large image databases like ImageNet before fine-tuned on domain-specific datasets. However, the performance of these fine-tuned models is often hindered by the large domain gaps (i.e., differences in scenes and modalities) between ImageNet's images and remotely sensed images being processed. Therefore, many researchers have undertaken efforts to establish large-scale domain-specific image datasets for pre-training, aiming to enhance model performance. However, establishing such datasets is often challenging, requiring significant effort, and these datasets often exhibit limited generaliza-bility to other application scenarios. To address these issues, this study introduces a novel yet simple pre-training strategy designed to guide a model away from learning domain-specific features in a pre-training dataset during pre-training, thereby improving the generalisation ability of the pre-trained model. To evaluate the strategy's effectiveness, deep learning models are pre-trained on ImageNet and subsequently fine-tuned on four semantic segmentation datasets with diverse scenes and modalities, including iSAID, MFNet, PST900 and Potsdam. Experimental results show that the proposed pre-training strategy led to state-of-the-art accuracies on all four datasets, namely 67.4% mIoU for iSAID, 56.9% mIoU for MFNet, 84.22% mIoU for PST900, 91.88% mF1 for Potsdam. This research lays the groundwork for developing a unified foundation model applicable to both computer vision and remote sensing applications.

Yuan Fang, Yuanzhi Cai, Jagannath Aryal +4

Department of Civil Engineering, Xi’an Jiaotong-Liverpool University, Suzhou 215000, China · CSIRO Mineral Resources, Kensington, WA 6151, Australia · Department of Infrastructure Engineering, The University of Melbourne, Melbourne VIC 3010, Australia

Remote Sensing Image Segmentation Domain Generalization Remote Sensing

Ask PDF

Apr 22, 2026

Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

Multi-spectral imagery is a valuable input signal for Remote Sensing applications, such as land-use and land-cover classification and environmental monitoring. However, generalist Large Multi-modal Models (LMMs) are typically trained on RGB images, limiting their applicability to the RGB domain. At the same time, training multi-spectral multi-modal models is expensive and produces uniquely specialized models. To address this, we propose a novel training-free approach that introduces multi-spectral data within the inference pipeline of standard RGB-only LMMs, allowing large gains in performance. Our approach leverages the LMMs' understanding of the visual space by adapting non-RGB inputs to that space and injecting domain-specific information and Chain-of-Thought reasoning as instructions. We demonstrate this with the Gemini 2.5 model and observe strong Zero-Shot performance gains on popular Remote Sensing benchmarks. These results highlight the potential for geospatial professionals to leverage powerful generalist models for specialized sensor inputs, benefiting from rich reasoning capabilities grounded in specialized data.

Dahun Kim, Ganesh Satish Mallya, Anelia Angelova

Google DeepMind

Remote Sensing Image Understanding Multimodal Large Language Models Multimodal Reasoning

Ask PDF

Mar 8, 2026

Local Precise Refinement: A Dual-Gated Mixture-of-Experts for Enhancing Foundation Model Generalization against Spectral Shifts

Domain Generalization Semantic Segmentation (DGSS) in spectral remote sensing is severely challenged by spectral shifts across diverse acquisition conditions, which cause significant performance degradation for models deployed in unseen domains. While fine-tuning foundation models is a promising direction, existing methods employ global, homogeneous adjustments. This "one-size-fits-all" tuning struggles with the spatial heterogeneity of land cover, causing semantic confusion. We argue that the key to robust DGSS lies not in a single global adaptation, but in performing fine-grained, spatially-adaptive refinement of a foundation model's features. To achieve this, we propose SpectralMoE, a novel fine-tuning framework for DGSS. It operationalizes this principle by utilizing a Mixture-of-Experts (MoE) architecture to perform local precise refinement on the foundation model's features, incorporating depth features estimated from selected RGB bands of the spectral remote sensing imagery to guide the fine-tuning process. Specifically, SpectralMoE employs a dual-gated MoE architecture that independently routes visual and depth features to top-k selected experts for specialized refinement, enabling modality-specific adjustments. A subsequent cross-attention mechanism then judiciously fuses the refined structural cues into the visual stream, mitigating semantic ambiguities caused by spectral variations. Extensive experiments show that SpectralMoE sets a new state-of-the-art on multiple DGSS benchmarks across hyperspectral, multispectral, and RGB remote sensing imagery.

Xi Chen, Maojun Zhang, Yu Liu +1

National University of Defense Technology

Domain Generalization for Semantic Segmentation Remote Sensing Image Segmentation Domain Generalization Semantic Segmentation Vision Foundation Model Adaptation Mixture of Experts

Ask PDF

Abstract

Remote Sensing Image Understanding

Explore similar work

A generalised pre-training strategy for deep learning networks in semantic segmentation of remotely sensed images

Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

Local Precise Refinement: A Dual-Gated Mixture-of-Experts for Enhancing Foundation Model Generalization against Spectral Shifts