Unmixing-Guided Spatial-Spectral Mamba with Clustering Tokens for Hyperspectral Image Classification

Yimin Zhu; Lincoln Linlin Xu

cs.CVApr 10, 2026

Unmixing-Guided Spatial-Spectral Mamba with Clustering Tokens for Hyperspectral Image Classification

Authors: Yimin Zhu, Lincoln Linlin Xu

Organizations: Department of Geomatics Engineering, University of Calgary, Canada

Abstract

Although hyperspectral image (HSI) classification is critical for supporting various environmental applications, it is a challenging task due to the spectral-mixture effect, the spatial-spectral heterogeneity and the difficulty to preserve class boundaries and details. This letter presents a novel unmixing-guided spatial-spectral Mamba with clustering tokens for improved HSI classification, with the following contributions. First, to disentangle the spectral mixture effect in HSI for improved pattern discovery, we design a novel spectral unmixing network that not only automatically learns endmembers and abundance maps from HSI but also accounts for endmember variabilities. Second, to generate Mamba token sequences, based on the clusters defined by abundance maps, we design an efficient Top-K token selection strategy to adaptively sequence the tokens for improved representational capability. Third, to improve spatial-spectral feature learning and detail preservation, based on the Top-K token sequences, we design a novel unmixing-guided spatial-spectral Mamba module that greatly improves traditional Mamba models in terms of token learning and sequencing. Fourth, to learn simultaneously the endmember-abundance patterns and classification labels, a multi-task scheme is designed for model supervision, leading to a new unmixing-classification framework that outputs not only accurate classification maps but also a comprehensive spectral-library and abundance maps. Comparative experiments on four HSI datasets demonstrate that our model can greatly outperform the other state-of-the-art approaches. Code is available at https://github.com/GSIL-UCalgary/Unmixing_guided_Mamba.git

Figures & tables

Figure 1: Traditional Mamba token sequencing treats an entire image or uses a pre-defined way—despite its highly heterogeneous spatial patterns—as a single long sequence, which suffers from being inefficient at modeling fine-scale structures. In contrast, our Unmixing-guided per-abundance token sequencing decomposes the hyperspectral image into physically meaningful abundance maps, which better capture subtle and weak patterns and edges, and constructs a dedicated token sequence from each abundance, leading to non-local, sparse, dynamic, and adaptive Mamba token sequence.

Figure 2: The proposed model contains (1) a semi-blind spectral unmixing branch for abundance learning and endmember variability modeling subsection II-B , (2) adaptive Top- K tokens selection from abundance subsection II-C , (3) the unmixing-guided spatial-spectral Mamba block for learning selected sparse tokens subsection II-D , and (4) a multi-task learning part for balancing different tasks subsection II-E . Model outputs classification map, abundance map, and endmember with variability.

QUH-Tangdaowan
Class No.	Class Name	Train Num	Methods									Ablation Study
Class No.	Class Name	Train Num	RF	SSRN	SS-ConvNeXt	DSNet	S2VNet	MambaHSI	SFMamba	SDMamba	Ours	W/O $\text{Pos}_{\text{um}}$	W/O $\text{Pos}_{\text{cls}}$	W/O Top- K	W/O Variability
1	Rubber track	258	97.70	99.38	99.29	99.27	99.85	99.94	99.71	99.64	99.72	99.72	99.55	99.56	99.81
2	Flagging	555	92.94	97.02	97.85	97.54	97.72	99.59	98.78	97.62	99.18	98.45	98.60	77.63	99.14
3	Sandy	340	87.41	94.04	93.60	90.71	94.33	98.09	95.81	94.39	98.21	98.23	96.06	93.03	98.04
4	Asphalt	606	96.03	99.05	99.42	99.20	99.40	99.80	99.33	99.35	99.18	99.49	98.93	91.96	99.49

Table I: Performance on the four datasets.

Panel of Figure 3 — Figure 3: The classification map of (a) RF (b) SSRN (c) SS-ConvNeXt (d) DSNet (e) S 2 VNet (f) MambaHSI (g) SFMamba (h) SDMamba (i) Ours (j) PCA image (k) Ground Truth. QUH-Tangdaowan, Pavia University, Houston, Fanglu Tea Farm, from top to bottom. Yellow boxes highlight the detailed preservation.

Panel of Figure 4 — Figure 4: The estimated abundance map from (a) DSNet, (b) S 2 VNet, (c) Ours, and the estimated (d) endmember variability, (e) Initialized endmember.

	QUH-Tangdaowan					Pavia University
#Num $P$	18	23	28	33	#Num $P$	9	14	19	24
OA	98.00	98.17	98.08	98.05	OA	98.15	98.59	97.93	97.98
AA	95.63	96.12	95.14	96.59	AA	97.47	97.92	97.02	97.40
Kappa	97.73	97.92	97.81	97.78	Kappa	97.55	98.14	97.26	97.33
	Houston					Fanglu Tea Farm
#Num $P$	15	20	25	30	#Num $P$	10	15	20	25

Table II: Impact of endmember number

P

Explore similar work

Token Clustering and Semantic Sequence Mamba for Hyperspectral Image Classification
Sep 23, 2026Yimin Zhu, Mahmood Elahi, Lincoln Linlin XuVisual SSMs Hyperspectral Image Classification
Department of Geomatics Engineering, University of Calgary, Canada · Department of Electrical and Software Engineering, University of Calgary, Canada
mHC-HSI: Clustering-Guided Hyper-Connection Mamba for Hyperspectral Image Classification
Mar 3, 2026Yimin Zhu, Zack Dewis, Quinn Ledingham +6Hyperspectral Image Classification Hyperspectral Imaging
Department of Geomatics Engineering, University of Calgary, Canada · Department of Geography, Faculty of Arts, The University of Jordan, Amman, Jordan
AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification
Sep 6, 2026Nafisa Anjum, Satavisa Dey Borno, Ananna Saha +4Hyperspectral Image Classification Hyperspectral Imaging
North South University, Dhaka, Bangladesh · Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh

Sep 23, 2026J/K move · Enter open · S save

Token Clustering and Semantic Sequence Mamba for Hyperspectral Image Classification

Yimin Zhu, Mahmood Elahi, Lincoln Linlin Xu

Department of Geomatics Engineering, University of Calgary, Canada · Department of Electrical and Software Engineering, University of Calgary, Canada

Visual SSMs Hyperspectral Image Classification

Open paper AskPDF arXiv

Although hyperspectral images (HSIs) provide rich spectral-spatial information, accurate pixel-level classification remains challenging because of spectral-spatial heterogeneity and complex spatial structures. Existing vision state space models (Mamba) typically construct sequences according to predefined spatial neighborhoods, without explicitly accounting for semantic similarity or spatial non-stationarity. To address this limitation, we propose Token Clustering and Semantic Sequence Mamba (STMamba), which organizes sparse tokens into semantically coherent sequences for hyperspectral image classification with the following features. First, at the macro level, a hierarchical encoder decoder progressively selects semantic tokens with the Token Clustering Module (TCM) and restores dense features using a parameter-free Cross-scale Neighborhood Attention (CNA) Upsampler. Second, at the micro level, TCM first identifies representative cluster centers through density-aware clustering and estimates soft memberships based on feature similarity. A quadtree-based dynamic selection strategy then retains sparse and spatially distributed tokens from each semantic cluster, forming coherent semantic-token sequences while reducing redundant pixel-wise representations. Third, parallel Spatial and Spectral Semantic-wise Sequencing Mamba (SWSM) modules capture complementary long-range spatial and spectral dependencies within homogeneous semantic token sequences while suppressing irrelevant interactions across heterogeneous regions. Experimental results on three large-scale benchmark datasets demonstrate that STMamba outperforms the SOTA methods with respect to quantitative and qualitative results.

Abstract

Explore similar work

Token Clustering and Semantic Sequence Mamba for Hyperspectral Image Classification

mHC-HSI: Clustering-Guided Hyper-Connection Mamba for Hyperspectral Image Classification

AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification