HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing

Li Pang; Xinqiao Wu; Jing Yao; Pedram Ghamisi; Jun Zhou; Zhengchao Chen; Deyu Meng; Xiangyong Cao

cs.CVSep 29, 2026

HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing

Authors: Li Pang, Xinqiao Wu, Jing Yao, Pedram Ghamisi, Jun Zhou, Zhengchao Chen, Deyu Meng, Xiangyong Cao

Organizations: School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China · State Key Laboratory of Remote Sensing and Digital Earth, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China · Helmholtz-Zentrum Dresden-Rossendorf, 09599 Freiberg, Germany · Faculty of Electrical and Computer Engineering, University of Iceland, 101 Reykjavik, Iceland · School of Information and Communication Technology, Griffith University, Nathan, QLD 4111, Australia · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China

Abstract

Hyperspectral remote sensing provides dense spectral measurements that are indispensable for material-level Earth observation, yet the construction of a general-purpose hyperspectral foundation model remains difficult. Two bottlenecks are especially limiting. First, large hyperspectral corpora rarely provide high spatial resolution together with reliable dense annotations. Second, many hyperspectral models are still trained almost from scratch, so the geometric and interactive priors learned by modern vision foundation models are not fully reused. To alleviate these issues, we \highlight{present} \textbf{HyperSAM}, a promptable hyperspectral foundation model that couples a data-centric hyperspectral synthesis pipeline with a spectral adaptation architecture based on Segment Anything Model 3 (SAM3). On the data side, HyperSAM synthesizes full-spectrum hyperspectral cubes from high-resolution SpaceNet multispectral imagery through a physics-informed abundance-transfer generator, while SAM3-derived pseudo-masks provide object-centric supervision. On the model side, the latest implementation uses a frozen SAM3 RGB image branch, a trainable hyperspectral side encoder initialized from the RGB vision transformer (ViT), ControlNet-style zero-initialized feature injection, and a lightweight mixture-of-experts mask refiner. To enhance training robustness against noisy pseudo-labels, Cross-modal Sample Selection (CromSS)-style confidence selection is incorporated for noisy-label weighting. Extensive experiments show that HyperSAM obtains strong generalization on diverse hyperspectral tasks (e.g., classification, anomaly detection, change detection, target detection, and airborne oil-spill mapping) and that high-quality synthetic hyperspectral data can be more effective than simply scaling noisy hyperspectral supervision.

Figures & tables

Fig. 1: Cross-task performance comparison between HyperSAM and representative baselines. Results are normalized from hyperspectral classification (HC), hyperspectral anomaly detection (HAD), hyperspectral change detection (HCD), and hyperspectral target detection (HTD) metrics.

Fig. 2: Comparison of data sources and training paradigms among representative hyperspectral foundation models. HyperSAM builds an object-centric hyperspectral training corpus by synthesizing full-spectrum HSI from high-resolution multispectral SpaceNet imagery and pairing it with SAM3-generated pseudo-masks.

Fig. 3: Visual comparison of training data. HyperFree samples are often limited by coarse spatial details or weaker object boundaries, whereas the HyperSAM synthetic training data contain sharper outlines, richer object categories, and pseudo-masks suitable for promptable foundation-model training.

Fig. 4: Physics-informed hyperspectral reconstruction pipeline. Multispectral SpaceNet imagery is decomposed into abundance and illumination maps, which are transferred through a hyperspectral endmember library to synthesize full-spectrum cubes aligned with object-level pseudo-masks.

Fig. 5: Shared-PCA visualization of 800 randomly sampled synthetic spectra and 800 real AVIRIS spectra from Hyper-Seg. The core overlap indicates physically plausible synthetic coverage.

Fig. 6: Spatially grounded comparison between synthesized and real hyperspectral data. Left: representative synthetic and real AVIRIS image examples, with red dots marking sampled locations. Right: 25th–75th percentile spectral envelopes formed after independently normalizing each spectrum. The overlapping shapes provide qualitative evidence of material-dependent consistency while retaining visible distributional differences.

Fig. 7: HyperSAM architecture. A frozen SAM3 RGB branch preserves objectness, boundary, and prompt-response priors, while a trainable hyperspectral side encoder injects full-spectrum residual features through zero-initialized adapters. The mask decoder is augmented with a lightweight multi-scale MoE refinement path for adapting mask logits and quality scores to hyperspectral imagery.

Fig. 8: Representative analysis of pseudo-mask uncertainty and confidence-aware weighting. (a) RGB image used to provide the SAM3 spatial prior. (b) Estimated uncertainty map, in which unreliable regions concentrate around ambiguous boundaries, fragmented objects, and complex background structures. (c) Confidence-weight map, which assigns larger weights to stable regions and smaller weights to uncertain pseudo-label pixels.

Task	Supervision or prompt	Mask generation	Feature or score construction	Decision rule
HC	One randomly sampled support pixel per class from a seeded pool	32 points/side; IoU 0.3; stability 0.4; NMS 0.7	Select the smallest covering mask and average its features; use the single-pixel embedding if no mask covers the support	Assign candidate masks to the nearest prototype by cosine similarity
HAD	No labels, prompts, or target spectra	96 points/side; IoU 0.4; stability 0.4; NMS 0.7	Suppress large background masks and retain compact proposals; default area-ratio threshold is 0.0009	Convert retained mask proposals into an anomaly response map
HCD	No changed-pixel labels	32 points/side for each date; IoU 0.3; stability 0.4; NMS 0.7	Average dense features inside masks and compute cosine-distance change scores for both temporal directions	Merge the two directional maps by maximum response and threshold at quantile 0.74
HTD	One target spectrum or one support prompt	32 points/side; IoU 0.3; stability 0.4; NMS 0.7	Locate a target-like support region and use its mean feature as the target prototype	Score candidate masks by cosine similarity to the target prototype and apply light spatial post-processing

TABLE I: Task-specific inference protocols for HyperSAM. The same model parameters are used for all tasks.

HC (Indian Pines / PaviaU)
Metric	SSFTT [ 43 ]	TGRS-ViT [ 44 ]	HyperSIGMA-LP [ 6 ]	DOFA-LP [ 12 ]	HyperFree [ 7 ]	SpectralEarth [ 16 ]	SAM3 [ 15 ]	Ours	SSFTT [ 43 ]	TGRS-ViT [ 44 ]	HyperSIGMA-LP [ 6 ]	DOFA-LP [ 12 ]	HyperFree [ 7 ]	SpectralEarth [ 16 ]	SAM3 [ 15 ]	Ours
OA (%) $\uparrow$	58.43	54.14	41.74	60.01	51.35	57.22	62.49	69.37	69.34	63.99	65.27	64.22	75.23	65.48	84.61	89.12
AA (%) $\uparrow$	70.07	65.28	46.99	73.64	62.07	69.90	75.80	77.46	71.59	75.85	72.34	63.09	62.49	56.02	85.04	87.38
Kappa (%) $\uparrow$	53.41	48.24	33.50	56.09	47.40	52.28	58.60	65.11	60.65	56.01	54.67	54.36	64.53	54.57	80.45	85.81
HAD (Beach-1 / Beach-2)
Metric	RXD [ 4 ]	Auto-AD [ 45 ]	HyperSIGMA [ 6 ]	DOFA [ 12 ]	TDD [ 46 ]	HyperFree [ 7 ]	SpectralEarth [ 16 ]	Ours	RXD [ 4 ]	Auto-AD [ 45 ]	HyperSIGMA [ 6 ]	DOFA [ 12 ]	ADLR [ 47 ]	HyperFree [ 7 ]	SpectralEarth [ 16 ]	Ours

TABLE II: Condensed comparison on representative datasets.

Fig. 9: Qualitative comparison of hyperspectral classification results. Compared to the baselines, HyperSAM produces more spatially coherent classification maps with sharper land-cover boundaries and significantly less salt-and-pepper noise, demonstrating the benefit of region-level promptable priors.

Fig. 10: Qualitative comparison on the Beach-1 anomaly detection scene from the Airport-Beach-Urban benchmark [ 38 ] . HyperSAM localizes the target with compact mask-level responses while suppressing most background structures.

Fig. 11: Qualitative comparison on the Beach-2 anomaly detection scene from the Airport-Beach-Urban benchmark [ 38 ] . HyperSAM reduces false alarms from shoreline and water-background regions and highlights coherent anomalous targets.

Fig. 12: Qualitative comparison on the Hermiston change detection scene [ 40 ] . HyperSAM produces more complete changed regions and fewer fragmented artifacts than the compared foundation-model baselines.

Fig. 13: Qualitative comparison on the Nuance Cri target detection scene [ 41 ] . HyperSAM remains competitive with strong spectral detectors while producing region-consistent target responses.

Fig. 14: Qualitative comparison on the Airport target detection scene from the Airport-Beach-Urban benchmark [ 38 ] . HyperSAM benefits from object-level mask features and achieves stronger clutter suppression than several spectral or foundation-model baselines. The false-color composite uses the closest available bands to 700, 546.1, and 438.8 nm as red, green, and blue, respectively.

Configuration	Component							Downstream performance
	Training data			Network structure			Training loss	PaviaU OA $\uparrow$	Beach-2 ODP $\uparrow$	Hermiston IoU $\uparrow$	Airport ODP $\uparrow$
	HyperFree	Raw-MS	Syn-HSI	RGB	HSI	MoE	Conf.	PaviaU OA $\uparrow$	Beach-2 ODP $\uparrow$	Hermiston IoU $\uparrow$	Airport ODP $\uparrow$
HyperFree training data	✓	–	–	✓	✓	✓	✓	79.33	1.3903	68.40	1.2500
Raw multispectral input	–	✓	–	✓	✓	✓	✓	85.84	1.3576	66.73	1.3088
Synthetic hyperspectral input	–	–	✓	✓	✓	–	✓	84.92	1.4452	66.61	1.3544
Without frozen RGB branch	–	–	✓	–	✓	✓	✓	84.89	1.3126	66.45	1.3208

TABLE III: Component-wise ablation of HyperSAM. Components are grouped into training data, network structure, and confidence-aware training.

Initialization	PaviaU OA $\uparrow$	Beach-2 ODP $\uparrow$	Hermiston IoU $\uparrow$	Airport ODP $\uparrow$
Random	83.59	1.3967	68.43	1.3126
Zero	89.12	1.4641	70.31	1.3297

TABLE IV: Ablation of the feature-injection initialization strategy. All other settings are unchanged.

Configuration	PaviaU OA $\uparrow$	Beach-2 ODP $\uparrow$	Hermiston IoU $\uparrow$	Airport ODP $\uparrow$
w/o $L_{1}$ reconstruction	81.64	1.3846	63.88	1.2715
w/o cosine consistency $L_{\cos}$	80.26	1.2681	61.15	1.2194
w/o projection consistency $L_{\mathrm{proj}}$	87.91	1.4218	68.74	1.3312
w/o abundance sparsity $L_{\mathrm{1/2}}$	83.52	1.3467	65.21	1.2916
w/o total variation $L_{\mathrm{tv}}$	88.73	1.4479	69.42	1.3421
Full generator objective	89.12	1.4641	70.31	1.3297

TABLE V: Leave-one-term-out ablation of the hyperspectral data-generation objective. All variants use the same training data, initialization, and optimization schedule.

Fig. 15: Top-1 routing frequency of the three MoE mask-refinement experts across mask-size ranges, showing scale-dependent specialization without expert collapse.

Configuration	PaviaU OA $\uparrow$	Beach-2 ODP $\uparrow$	Hermiston IoU $\uparrow$	Airport ODP $\uparrow$
w/o Dice	85.06	1.4159	67.16	1.3226
w/o BCE	86.24	1.4273	67.81	1.3396
w/o confidence weighting	87.48	1.4524	68.96	1.3587
w/o $\mathcal{L}_{\mathrm{bal}}$	87.17	1.4592	68.27	1.3472
Full training objective	89.12	1.4641	70.31	1.3297

TABLE VI: Ablation of the HyperSAM training-loss components. All variants use the same architecture, corpus, initialization, optimization schedule, and downstream protocol.

Fig. 16: Ablation of CromSS-style confidence-aware robust learning [ 22 ] . The comparison reports representative HC, HAD, HCD, and HTD metrics with and without noisy-label weighting and cross-view consistency.

Fig. 17: Oil-spill mapping results on the HOSD Gulf of Mexico GM13 and GM17 scenes. From left to right, the figure shows the input false-color composite, ground truth, and HyperSAM prediction for each scene. The composites use the closest available bands to 700, 546.1, and 438.8 nm as red, green, and blue, respectively. Red and blue in the label maps denote oil spill and background (clean seawater or non-oil sea surface), respectively.

Method	OA (%)	AA (%)	Kappa (%)	Time (s)	Energy (Wh)	CO 2 e (g)
HyperSAM	66.91	74.09	62.02	21.11 $\pm$ 1.98	0.624 $\pm$ 0.223	0.2964 $\pm$ 0.1059
HyperFree	50.36	58.06	43.62	23.40 $\pm$ 18.94	0.244 $\pm$ 0.111	0.1159 $\pm$ 0.0527
SSFTT	54.85	65.07	49.51	8.30 $\pm$ 0.06	0.087 $\pm$ 0.001	0.0413 $\pm$ 0.0005
TGRS-ViT	49.73	63.49	44.92	8.16 $\pm$ 0.13	0.120 $\pm$ 0.002	0.0570 $\pm$ 0.0010
HyperSIGMA-LP	40.69	46.99	33.53	27.01 $\pm$ 37.63	0.274 $\pm$ 0.401	0.1302 $\pm$ 0.1905
DOFA-LP	59.32	64.48	55.34	13.69 $\pm$ 9.88	0.181 $\pm$ 0.168	0.0860 $\pm$ 0.0798

TABLE VII: Accuracy, energy, and operational carbon cost on Indian Pines. Costs are the mean and standard deviation over three cold-start replays of that same configuration and include downstream training when required followed by full-scene inference.

Fig. 18: Accuracy–energy trade-off on Indian Pines. The horizontal axis reports measured GPU board energy for downstream training, adaptation, and/or inference.

Explore similar work

Hyperspectral Image Models: Technical Report
Sep 30, 2026Tanishq Rachamalla, Aryan Das, Srishti Kaushik +1Hyperspectral Image Deep Learning
Department of Information Technology Siddhartha Academy of Higher Education Vijayawada, Andhra Pradesh 521108, India · Department of Computer Science and Engineering Vellore Institute of Technology Bhopal, Madhya Pradesh 466114, India · Department of Computer and Information Sciences Indira Gandhi National Open University New Delhi 110068, India +1
HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone
May 17, 2026Guanyiman Fu, Jingtao Li, Zihang Cheng +8Hyperspectral Image
Griffith University, Australia · Wuhan University, China · Nanjing University of Science and Technology, China +2
Benchmarking Hyperspectral Foundation Models for Hyperspectral Unmixing
Sep 23, 2026Edgard Dabier, Christophe Kervazo, Pietro Gori +1Hyperspectral Image Wireless Foundation Models
LTCI, T´el´ecom Paris, Institut Polytechnique de Paris, Palaiseau, France

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

Hyperspectral Image Models: Technical Report

Tanishq Rachamalla, Aryan Das, Srishti Kaushik, Swalpa Kumar Roy

Department of Information Technology Siddhartha Academy of Higher Education Vijayawada, Andhra Pradesh 521108, India · Department of Computer Science and Engineering Vellore Institute of Technology Bhopal, Madhya Pradesh 466114, India · Department of Computer and Information Sciences Indira Gandhi National Open University New Delhi 110068, India+1

Hyperspectral Image Deep Learning Technical Report

Open paper AskPDF arXiv

Hyperspectral remote sensing has advanced across diverse deep learning paradigms, including spectral spatial CNNs, Vision Transformers, Mamba, graph neural networks, Kolmogorov Arnold networks, and self supervised masked autoencoding. Yet progress remains hindered by fragmented repositories, incompatible tensor conventions, and non standardized evaluation. Hyperspectral Image Models addresses these challenges through a modular framework unifying 55 representative models across six paradigms with a common registry, automatic 4D/5D tensor adaptation, and standardized constructors. It integrates 24 benchmark scenes from Airborne, Spaceborne, UAV, and Mars CRISM sensors, with caching, label remapping, PCA, explicit band selection or raw spectra, optional spatial max pooling, and arbitrary PxP patch extraction. To prevent inflated accuracy from overlapping windows, it supports class balanced random partitioning and spatially disjoint regional blocking with Chebyshev guard bands that eliminate train test pixel overlap. Experiments use a single config.yaml with deterministic seeds and complete provenance, generating LaTeX benchmark tables and classification maps. Across 1,320 model scene evaluations and 6,600 seeded runs, scene difficulty dominates architecture, with mean accuracy ranging from 96.40% on Botswana to 56.70% on Houston 2018, versus a 15 point spread across paradigm means. No paradigm universally dominates, while sub 1 M parameter models can match architectures two orders of magnitude larger. Code is publicly available at https://github.com/Tanishq251/Hyperspectral-Image-Models.

Abstract

Explore similar work

Hyperspectral Image Models: Technical Report

HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone

Benchmarking Hyperspectral Foundation Models for Hyperspectral Unmixing