AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification

Nafisa Anjum; Satavisa Dey Borno; Ananna Saha; Mir Faiyaz Hossain; Sifat Momen; Nabeel Mohammed; Shafin Rahman

cs.CVSep 6, 2026

AGSA-Net: Abundance-Guided Self-Attention Network for Spectral Unmixing-Aware Hyperspectral Remote Sensing Image Classification

Authors: Nafisa Anjum, Satavisa Dey Borno, Ananna Saha, Mir Faiyaz Hossain, Sifat Momen, Nabeel Mohammed, Shafin Rahman

Organizations: North South University, Dhaka, Bangladesh · Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh

Abstract

Hyperspectral image (HSI) classification plays a vital role in remote sensing applications, including agriculture, environmental monitoring, and urban analysis. However, its performance remains challenged by high spectral redundancy, noise sensitivity, and the difficulty of jointly modeling local material composition and long-range spectral dependencies. To address this, we propose AGSA-Net, an abundance-guided self-attention network that explicitly integrates spectral unmixing priors into the classification process. AGSA Net first estimates physically meaningful subpixel abundance maps subject to non-negativity and sum-to-one constraints, regularized by hybrid linear-nonlinear reconstruction decoder. The learned abundances are then used to construct an abundance affinity prior that guides a spectral transformer to emphasize class-discriminative interactions, and the resulting transformer features are fused with compact abundance descriptors for final prediction; in contrast to existing approaches that use abundance as auxiliary or concatenated features. Experiments on Indian Pines, Augsburg, and Berlin demonstrate the benefit of incorporating abundance- guided contextual modeling, particularly in heterogeneous urban scenes. The source code and trained models are available at: https://github.com/nnuvi/AGSA-Net

Figures & tables

Panel of Figure 1 — Fig. 1: Overview of the proposed AGSA-Net for HSI classification. The framework comprises an unmixing branch and a transformer branch. (a) In contrast to prior approaches that treat abundance maps only as auxiliary inputs or concatenated features, (b) AGSA-Net converts the estimated abundances into a pixel-wise material affinity matrix and injects it directly into the multi-head self-attention as an additive bias, thereby explicitly governing attention formation. This design enforces contextual interactions driven by sub-pixel material composition rather than spectral similarity alone. Final predictions are obtained by fusing abundance-aware transformer features with compact abundance representations.

Fig. 2: Proposed AGSA-Net architecture. Given an input patch $\mathbf{P}_{i,j}\in\mathbb{R}^{p\times p\times C}$ , the encoder in the first branch estimates pixel-wise abundances $\mathbf{v}_{i,j}\in\mathbb{R}^{K\times p\times p}$ , which are flattened and normalized to compute the abundance affinity matrix. The decoder then reconstructs the patch using both linear and nonlinear mixing modules. In parallel, the spectral transformer tokenizes the same patch, projects the tokens into a latent space, and applies multi-head self-attention guided by the padded affinity matrix $\mathbf{AF}$ through an additive bias term. The classification token from the final transformer layer is fused with the abundance-derived features to produce the final logits.

Dataset	Indian Pines	Augsburg	Berlin
Sensor	AVIRIS	HySpex	EnMap
Wavelength (nm)	400–2500	400–2500	400–2500
Spatial resolution	20 m	30 m	30 m
Spatial size	145 $\times$ 145	332 $\times$ 485	1723 $\times$ 476
Spectral bands	200	180	244
Classes	16	7	8

TABLE I: Summary of the HSI datasets used in experiments.

Class Name	Train	Test	2-D CNN [ 50 ]	3-D CNN [ 9 ]	GRU [ 26 ]	ViT [ 29 ]	MorphConv [ 33 ]	SSFTT [ 31 ]	WFCG [ 28 ]	DSNet [ 21 ]	CACFTNet [ 37 ]	GraphGST [ 34 ]	MambaHSI+ [ 42 ]	S2Mamba [ 41 ]	AGSA-Net (Ours)
Forest	146	13361	85.76	93.02	81.10	82.44	93.21	95.52	91.65	93.19	98.23	87.89	95.42	87.07	94.66
Residential Area	264	30065	87.44	92.68	84.51	88.77	95.83	96.39	98.96	95.85	96.60	92.89	96.86	93.68	96.95
Industrial Area	21	3830	69.56	21.46	84.41	79.27	32.69	25.93	50.97	62.85	66.40	42.64	64.88	63.55	73.68
Low Plants	248	26609	81.39	78.29	82.25	80.51	88.75	87.02	87.33	91.28	92.86	82.89	88.76	93.47	93.99
Allotment	52	523	41.68	58.70	32.10	33.35	63.67	45.70	52.35	66.92	68.64	53.35	55.64	43.98	69.41
Commercial Area	7	1638	14.10	9.46	5.56	12.39	15.81	11.78	1.05	11.05	16.12	10.32	10.93	12.45	4.58

TABLE II: Class-wise classification accuracy (%) on the Augsburg dataset

Class	Train	Test	2-D CNN [ 50 ]	3-D CNN [ 9 ]	GRU [ 26 ]	ViT [ 29 ]	MorphConv [ 33 ]	SSFTT [ 31 ]	WFCG [ 28 ]	DSNet [ 21 ]	CACFTNet [ 37 ]	GraphGST [ 34 ]	MambaHSI+ [ 42 ]	S2Mamba [ 41 ]	AGSA-Net (Ours)
Forest	443	54511	77.16	73.10	75.55	66.74	55.24	47.84	76.46	75.44	73.55	84.93	81.42	70.24	76.51
Residential	423	268219	66.33	55.27	57.60	57.86	72.47	73.58	68.80	74.69	84.41	72.97	81.79	74.05	79.31
Industrial	499	19067	21.37	43.92	73.13	55.85	26.25	30.59	42.06	54.91	38.76	53.75	32.35	41.51	49.26
LowPlants	376	58906	69.11	68.76	71.70	76.25	74.27	84.73	67.58	81.79	62.63	86.15	73.94	75.77	89.26
Soil	331	17095	68.15	84.05	82.95	80.04	84.88	92.20	97.91	74.19	69.01	83.18	71.36	76.02	80.51
Allotment	280	13025	58.52	32.48	22.30	64.17	61.21	49.21	72.80	63.71	29.25	77.47	33.65	64.19	57.83

TABLE III: Class-wise classification accuracy (%) on the Berlin dataset

Class Name	Train	Test	2-D CNN [ 50 ]	3-D CNN [ 9 ]	GRU [ 26 ]	ViT [ 29 ]	MorphConv [ 33 ]	SSFTT [ 31 ]	WFCG [ 28 ]	DSNet [ 21 ]	CACFTNet [ 37 ]	GraphGST [ 34 ]	MambaHSI+ [ 42 ]	S2Mamba [ 41 ]	AGSA-Net (Ours)
Corn Notill	50	1384	77.24	65.25	72.33	66.55	71.53	85.62	94.87	83.96	95.88	95.81	74.57	94.44	91.04
Corn Mintill	50	784	84.57	59.57	83.16	88.52	84.57	90.43	90.69	94.90	99.74	98.85	87.63	100.00	96.56
Corn	50	184	92.39	94.57	73.37	88.59	97.83	97.28	100.00	99.46	100.00	100.00	95.65	100.00	99.46
Grass Pasture	50	447	94.63	89.71	87.02	97.09	93.74	90.60	94.41	98.21	97.76	97.09	87.92	98.43	97.32
Grass Trees	50	697	78.62	95.12	81.35	85.65	92.25	95.95	99.43	97.27	100.00	98.57	90.24	100.00	100.00
Hay Windrowed	50	439	92.26	98.63	94.76	98.86	98.86	98.63	100.00	99.54	100.00	99.54	95.67	100.00	98.41

TABLE IV: Class-wise classification accuracy (%) on the Indian Pines dataset

Panel of Figure 7 — Fig. 3: False-color image, ground truth and classification maps obtained using the Transformer, DSNet, and AGSA-Net models on the Indian Pines dataset.

Panel of Figure 8 — Fig. 4: False-color image, ground truth, and classification maps obtained using the Transformer, DSNet, and AGSA-Net models on the Augsburg dataset.

Panel of Figure 9 — Fig. 5: False-color image, ground truth and classification maps obtained using the Transformer, DSNet, and AGSA-Net models on the Berlin dataset.

Panel of Figure 10 — Fig. 6: Confusion matrices obtained using AGSA-Net on the Berlin and Augsburg datasets.

Fig. 7: Effect of training sample ratio on overall accuracy (OA) across the evaluated datasets and different models, showing performance variations under different ratios of training data.

Model	# of Params	FLOPs	Inference Time
Model	(Millions) $\downarrow$	(Millions) $\downarrow$	(ms) $\downarrow$
DSNet [ 21 ]	0.42	11.00	0.021
SSFTT [ 31 ]	0.15	1.43	0.022
GraphGST [ 34 ]	2.14	2.15	0.094
CACFTNet [ 37 ]	1.86	118.52	0.285
ViT [ 29 ]	5.37	267.31	0.265

TABLE V: Computational complexity comparison on the Indian Pines dataset (patch size

7\times 7

\downarrow

denotes that lower is better. Bold indicates the best result, and underlined indicates the second-best result.

Dataset	Sim. Type	Same-Class	Different-Class	Gap
Indian Pines	Spectral	0.9939	0.9233	0.0706
	Abundance	0.9562	0.8403	0.1159
Augsburg	Spectral	0.9769	0.9425	0.0344
	Abundance	0.9671	0.8958	0.0713
Berlin	Spectral	0.9811	0.9535	0.0276
	Abundance	0.8844	0.7857	0.0987

TABLE VI: Similarity Comparison Between Abundance and Spectral Representations

Description	Equation	Indian Pines			Augsburg			Berlin
Description	Equation	OA	AA	$\kappa$	OA	AA	$\kappa$	OA	AA	$\kappa$
Shuffled Guidance	$QK^{\top}+\lambda PA_{abu}P^{\top}$	80.72	88.40	78.01	84.32	62.47	77.33	68.92	62.97	55.47
Spectral Guidance	$QK^{\top}+\lambda A_{spec}$	79.73	88.32	77.03	84.68	63.27	77.87	70.65	64.44	57.30
No Guidance	$QK^{\top}$	80.92	89.55	78.35	84.96	60.66	78.24	72.00	60.54	57.69
Proposed Abundance Guidance	$QK^{\top}+\lambda A_{abu}$	81.99	89.12	79.40	85.57	64.00	79.25	72.29	64.63	59.09

TABLE VII: Controlled intervention analysis of different attention guidance strategies.

Fig. 8: Visualization of abundance-guided self-attention behavior for the Augsburg (top) and Indian Pines (bottom) datasets. For each dataset, the panels show: (1) query token location, (2) attention map without guidance, (3) attention map with abundance guidance, (4) abundance similarity map, and (5) attention difference ( $\Delta$ Attention) between guided and unguided settings. The alignment score above each row denotes the correlation between the attention distribution and abundance similarity before and after introducing abundance guidance.

Modules				Remarks	Indian Pines			Augsburg			Berlin
UA	AGM	Fusion	Trans	Remarks	OA	AA	$\kappa$	OA	AA	$\kappa$	OA	AA	$\kappa$
$\times$	$\times$	$\times$	✓	Transformer only	81.30	89.14	78.68	82.22	56.97	74.12	68.00	62.92	50.65
✓	✓	$\times$	✓	No abundance fusion	81.99	89.12	79.40	85.57	64.00	79.25	72.29	64.63	59.09
✓	$\times$	✓	$\times$	CNN backbone (DSNet)	93.08	96.31	92.08	89.30	67.20	84.62	72.09	65.34	59.27
✓	$\times$	✓	✓	No abu-guided self-attn	94.72	97.36	93.96	90.48	67.41	86.26	76.04	64.77	63.50
✓	✓	✓	✓	AGSA-Net	96.04	97.61	95.45	91.26	68.47	87.46	75.31	65.04	63.04

TABLE VIII: Core component analysis of the proposed AGSA-Net on Indian Pines, Augsburg, and Berlin datasets.

Fig. 9: Training dynamics of the abundance guidance weight

\lambda

and validation accuracy on the Indian Pines, Augsburg, and Berlin datasets. The circle indicates the value of

\lambda

at the epoch where the best validation accuracy is achieved, while the star denotes the highest validation accuracy during training.

Dataset	Metric	0.5U	0.75U	U	1.25U	1.5U	2U
Pines	Best OA	0.9343	0.9413	0.9604	0.9431	0.9478	0.9455
	Best AA	0.9645	0.9700	0.9761	0.9671	0.9694	0.9746
	Kappa	0.9248	0.9328	0.9546	0.9348	0.9401	0.9376
Berlin	Best OA	0.7211	0.7187	0.7531	0.7465	0.7214	0.7503
	Best AA	0.6501	0.6368	0.6504	0.6173	0.6693	0.5911
	Kappa	0.5912	0.5847	0.6304	0.6170	0.5945	0.6131

TABLE IX: Sensitivity analysis of the number of endmembers (

K

). Best OA, AA and Kappa values for each dataset are highlighted in bold.

Panel of Figure 19 — Fig. 10: t-SNE visualization of feature representations from the Indian Pines dataset using (a) raw spectral features, (b) Transformer, (c) DSNet, and (d) AGSA-Net. AGSA-Net produces the most compact and well-separated clusters, indicating more discriminative feature representations.

Explore similar work

Unmixing-Guided Spatial-Spectral Mamba with Clustering Tokens for Hyperspectral Image Classification
Apr 10, 2026Yimin Zhu, Lincoln Linlin XuHyperspectral Image Classification Hyperspectral Imaging
Department of Geomatics Engineering, University of Calgary, Canada
MixerCA: An Efficient and Accurate Model for High-Performance Hyperspectral Image Classification
Apr 28, 2026Mohammed Q. Alkhatib, Ali JamaliHyperspectral Imaging Efficient Attention
aCollege of Engineering and IT, University of Dubai, Dubai, 14143, UAE · bDepartment of Geography, Simon Fraser University, 8888 University Dr, Burnaby, BC V5A 1S6, Canada
Representative Spectral Correlation Network for Multi-source Remote Sensing Image Classification
Apr 30, 2026Chuanzheng Gong, Feng Gao, Junyan Lin +2Remote Sensing Image Understanding Feature Selection
State Key Laboratory of Physical Oceanography, Ocean University of China, Qingdao 266100, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China · Department of Electrical and Computer Engineering, Mississippi State University, Starkville, MS 39762 USA

Apr 10, 2026J/K move · Enter open · S save

Unmixing-Guided Spatial-Spectral Mamba with Clustering Tokens for Hyperspectral Image Classification

Yimin Zhu, Lincoln Linlin Xu

Department of Geomatics Engineering, University of Calgary, Canada

Hyperspectral Image Classification Hyperspectral Imaging Mamba

Open paper AskPDF arXiv

Although hyperspectral image (HSI) classification is critical for supporting various environmental applications, it is a challenging task due to the spectral-mixture effect, the spatial-spectral heterogeneity and the difficulty to preserve class boundaries and details. This letter presents a novel unmixing-guided spatial-spectral Mamba with clustering tokens for improved HSI classification, with the following contributions. First, to disentangle the spectral mixture effect in HSI for improved pattern discovery, we design a novel spectral unmixing network that not only automatically learns endmembers and abundance maps from HSI but also accounts for endmember variabilities. Second, to generate Mamba token sequences, based on the clusters defined by abundance maps, we design an efficient Top-K token selection strategy to adaptively sequence the tokens for improved representational capability. Third, to improve spatial-spectral feature learning and detail preservation, based on the Top-K token sequences, we design a novel unmixing-guided spatial-spectral Mamba module that greatly improves traditional Mamba models in terms of token learning and sequencing. Fourth, to learn simultaneously the endmember-abundance patterns and classification labels, a multi-task scheme is designed for model supervision, leading to a new unmixing-classification framework that outputs not only accurate classification maps but also a comprehensive spectral-library and abundance maps. Comparative experiments on four HSI datasets demonstrate that our model can greatly outperform the other state-of-the-art approaches. Code is available at https://github.com/GSIL-UCalgary/Unmixing_guided_Mamba.git

Abstract

Explore similar work

Unmixing-Guided Spatial-Spectral Mamba with Clustering Tokens for Hyperspectral Image Classification

MixerCA: An Efficient and Accurate Model for High-Performance Hyperspectral Image Classification

Representative Spectral Correlation Network for Multi-source Remote Sensing Image Classification