Multi-task Disentangling Fusion Mamba-CNN for MODIS Spectral-Temporal Unmixing with Endmember Variability Modeling
Authors: Motasem Alkayid, Yimin Zhu, Naser El-Sheimy, Lincoln Linlin Xu
Organizations: Department of Geomatics Engineering, University of Calgary, Canada · Department of Geography, Faculty of Arts, The University of Jordan, Amman, Jordan
Spectral-temporal unmixing of MODIS time series is essential for applications such as abundance mapping and spectral library construction, yet it remains challenging due to (i) spectral-temporal information coupling in MODIS data, (ii) intra-class endmember spectral variability and inter-class signature ambiguity, and (iii) the instability of naively fusing multiple expert representations. This paper presents the Multi- task Disentangling Fusion Mamba (MDF-Mamba) to address these limitations. To resolve spectral-temporal coupling, we design a disentangling architecture that pairs per-channel spectral experts with a dedicated bidirectional state-space (Mamba) temporal branch. Convolutional modules each process a single spectral channel, while the Mamba branch encodes the cross-channel seasonal trajectory, forming a multi-branch expert ensemble with soft inter-expert reweighting based on squeeze-and-excitation. For the integration of these spectral and temporal expert outputs, we introduce mHCFusion, adapting the manifold-constrained hyper-connection framework to merge the outputs in a stable and interpretable manner. Endmember variability is addressed by extending the linear mixing model with pixel-wise learnable spectral variability weights that capture intra-class endmember variation. Finally, we design a multi-task training objective that combines supervised abundance estimation with self-supervised spectral reconstruction and endmember regularization terms on the same training data, yielding a highly efficient framework for spectral-temporal unmixing. Experiments on MODIS time series covering the Alberta and Saskatchewan provinces show that MDF-Mamba achieves the best performance on 12 of 14 evaluation metrics across both datasets (7 of 7 on Alberta; 5 of 7 on Saskatchewan), ranking second on the remaining two.
Figures & tables
Type
Model
Abundance
Reconstruction
Classification
MAE ↓
RMSE ↓
MSE ↓
SAD (rad) ↓
OA ↑
AA ↑
κ↑
Traditional
FCLSU [ 3 ]
0.0633
0.1530
0.0106
0.2445
0.5256
0.3290
0.4256
SGSNMF [ 6 ]
0.0703
0.1668
0.0118
0.2512
0.4734
0.3258
0.3750
SUnSAL [ 5 ]
0.0666
0.1547
0.0104
0.2469
0.5124
0.2984
0.4035
CNN/AE
PGMSU [ 9 ]
0.0329
0.0842
0.0116
0.2344
0.6847
0.4152
0.6159
SUnCNN [ 10 ]
0.0309
0.0823
0.0141
0.2678
0.6985
0.4308
0.6328
TABLE I: Quantitative performance comparison on the Alberta MODIS dataset. Bold indicates the best result; underline indicates the second best. ↓ indicates lower is better; ↑ indicates higher is better.
Type
Model
Abundance
Reconstruction
Classification
MAE ↓
RMSE ↓
MSE ↓
SAD (rad) ↓
OA ↑
AA ↑
κ↑
Traditional
FCLSU [ 3 ]
0.0729
0.2217
0.0126
0.2560
0.5572
0.2729
0.4495
SGSNMF [ 6 ]
0.0691
0.2481
0.0312
0.3498
0.5229
0.1857
0.3822
SUnSAL [ 5 ]
0.0807
0.2359
0.0132
0.2533
0.4607
0.2417
0.3476
CNN/AE
PGMSU [ 9 ]
0.0546
0.1748
0.0143
0.2617
0.6986
0.3832
0.6286
SUnCNN [ 10 ]
0.0509
0.1685
0.0162
0.2836
0.7176
0.3874
0.6493
TABLE II: Quantitative performance comparison on the Saskatchewan MODIS dataset. Bold indicates the best result; underline indicates the second best. ↓ indicates lower is better; ↑ indicates higher is better.
Type
Model
Abundance
Reconstruction
Classification
MAE ↓
RMSE ↓
MSE ↓
SAD (rad) ↓
OA ↑
AA ↑
κ↑
Traditional
FCLSU [ 3 ]
0.7291
0.7526
0.0231
0.3375
0.5874
0.2166
0.4315
SGSNMF [ 6 ]
0.6591
0.7217
0.0317
0.3540
0.6466
0.2887
0.5368
SUnSAL [ 5 ]
0.7180
0.7464
0.0242
0.3364
0.5901
0.2304
0.4367
CNN/AE
PGMSU [ 9 ]
0.3726
0.4360
0.0145
0.2562
0.8984
0.5979
0.8724
SUnCNN [ 10 ]
0.3001
0.3793
0.0183
0.2963
0.9134
0.6457
0.8913
TABLE III: Quantitative performance comparison on the Alberta filtered dataset (80% purity). Bold indicates the best result; underline indicates the second best. ↓ indicates lower is better; ↑ indicates higher is better.
Type
Model
Abundance
Reconstruction
Classification
MAE ↓
RMSE ↓
MSE ↓
SAD (rad) ↓
OA ↑
AA ↑
κ↑
Traditional
FCLSU [ 3 ]
0.0552
0.2006
0.0270
0.3438
0.6503
0.2004
0.4774
SGSNMF [ 6 ]
0.0567
0.2113
0.0342
0.3576
0.6303
0.2353
0.4819
SUnSAL [ 5 ]
0.0548
0.2052
0.0273
0.3429
0.6493
0.2008
0.4754
CNN/AE
PGMSU [ 9 ]
0.0312
0.1176
0.0191
0.2789
0.8539
0.5477
0.8080
SUnCNN [ 10 ]
0.0296
0.1135
0.0227
0.3066
0.8615
0.5874
0.8175
TABLE IV: Quantitative performance comparison on the Saskatchewan filtered dataset (80% purity). Bold indicates the best result; underline indicates the second best. ↓ indicates lower is better; ↑ indicates higher is better.
Fig. 5: Spectral reconstruction evaluation on the Alberta test set. (a) Patch-mean original versus reconstructed spectra for dominant classes across the 138-band spectral-temporal sequence. (b) Per-feature reconstruction RMSE over the 23-step temporal index. (c) Pixel-level original versus reconstructed reflectance scatter, with overall MSE, RMSE, SAD, and R2 annotated. (d) Per-class reconstruction SAD.
Fig. 6: Unified classification comparison on the Alberta dataset. Top two rows: Complete classification maps for all 13 models with a standardized 14-class colormap consistently maintained across the manuscript; white squares on the Ground Truth map mark twelve 60×60 sub-regions (a-l) centered on distinct land cover classes. Bottom grid: Zoomed-in 13×12 comparison of all 13 models within each highlighted region. Regions were selected to represent a diverse set of median-performing and challenging cases.
Fig. 10: Learned temporal endmember profiles and concordance analysis for the P=12 active land cover classes, ordered from top to bottom by lowest RMSE (best prediction) to highest RMSE (worst prediction); the two sub-polar lichen-moss classes are omitted (Section IV-A). Columns 1 - 6: Per-branch temporal profiles (NDVI, EVI, Red, NIR, Blue, MIR) showing R=10 variant spread (shaded), learned mean endmember (solid colored line), GT reference (solid black), and geographic initialization (dashed). Column 7 (Corr.): Per-class scatter of GT vs. learned mean endmember across all L=138 spectral-temporal bands, colored by spectral branch, with the 1:1 identity line (dashed) and annotated Lin’s concordance correlation coefficient (CCC) [ 52 ] , RMSE, and SAD. CCC measures agreement with the 1:1 line and penalizes both offset and scale differences. Classes whose GT reference is nearly flat (Shrubland, Snow/Ice, Urban, Broadleaf Forest, Sub-polar Taiga) produce the highest RMSE and lowest CCC, consistent with poor endmember convergence being data-imposed rather than architectural. Y-axis auto-scaled per row; X-axis spans the full 2010 calendar year (23 composites).
Configuration
MAE ↓
OA ↑
κ↑
Recon-MSE ↓
Recon-SAD ↓
Full MDF-Mamba
0.0255
0.7122
0.6575
0.0067
0.1843
− Temporal/phenology branch (C1)
0.0305
0.7036
0.6415
0.0067
0.186
− Learned fusion weighting (C2)
0.0372
0.6759
0.6009
0.0072
0.190
− Endmember variability (C3)
0.0255
0.7122
0.6575
0.0474
0.481
TABLE V: Part I: component lesion study. One module of the single trained MDF-Mamba is disabled at inference (shared weights, zero retraining variance). Bold : full model.
Configuration
MAE ↓
OA ↑
κ↑
Recon-MSE ↓
Endm-mSAD ↓
Endm-mRMSE ↓
Full MDF-Mamba (proposed)
0.0255
0.7122
0.6575
0.0067
0.00017
0.0458
(C3) Endmember variant count R and initialization
R=1 (no variant bank, standard LMM)
0.0290
0.7012
0.6358
0.0568
0.0260
0.1111
R=10 , random init (no GT purity)
0.0294
0.7016
0.6372
0.0798
0.0150
0.6537
(C4) Multi-task reconstruction objective
No reconstruction loss ( − Maha, − SAD)
0.0264
0.7101
0.6515
0.0281
0.00231
0.1429
TABLE VI: Part II: design ablations on the Alberta test set. Each row retrains the full MDF-Mamba with one design decision changed; all other settings remain fixed. Bold : proposed configuration (not necessarily the best value in each column).
Seed
MAE ↓
RMSE ↓
OA ↑
κ↑
Recon-MSE ↓
Recon-SAD ↓
42
0.0255
0.0654
0.7122
0.6575
0.0067
0.1843
123
0.0268
0.0681
0.7113
0.6492
0.0064
0.1845
7
0.0267
0.0679
0.7119
0.6499
0.0068
0.1874
Mean ± Std
0.0263±0.0006
0.0671±0.0012
0.7118±0.0004
0.6522±0.0038
0.0066±0.0002
0.1854±0.0014
TABLE VII: Multi-seed reproducibility on the Alberta test set. All three seeds are evaluated on the full test set under identical conditions. Mean and standard deviation computed over three independent training runs.
Model
Params (M)
GFLOPs
FCLSU
0.00
1.62
SGSNMF
0.00
3.16
SUnSAL
0.00
1.85
DeepTrans-HSU
26.82
1.04
PGMSU
2.15
8.82
SUnCNN
1.58
7.37
TABLE VIII: Computational complexity profile of all 11 baselines compared to MDF-Mamba. Measured on input shape (1,138,64,64) . Traditional non-parametric models (FCLSU, SGSNMF, SUnSAL) contain no trainable weights (Params = 0.00) and their GFLOPs are analytically estimated based on standard convergence iteration limits (e.g., ADMM, multiplicative updates). UNMamba GFLOPs are similarly estimated analytically.
Component
Params (M)
GFLOPs
Mem (MB)
patches/s
MDF-Mamba (full)
3.33
14.93
180.9
48.8
CNN Branches ( × 6)
2.761
11.32
-
-
Temporal Mamba
0.126
2.46
-
-
mHC Fusion
0.072
0.29
-
-
ELMM + Recon Head
0.122
0.00
-
-
Miscellaneous (stem, SE, FiLM)
0.249
0.86
-
-
TABLE IX: Internal computational profile of MDF-Mamba components. Throughput measured as patches per second (batch size 1).
Class
F1
Transfer quality
Cropland
0.934
✓ Excellent - stable agricultural phenology
Water
0.816
✓ Excellent - spectrally invariant
Needleleaf
0.776
✓ Good - consistent boreal canopy
Barren
0.758
✓ Good - low-reflectance signature stable
Grassland
0.688
Moderate
Wetland
0.533
Moderate
TABLE X: Class-level transfer metrics for the Alberta → Saskatchewan cross-provincial evaluation (OA: 0.7485, κ : 0.6881, MAE: 0.0492). Classes marked ✓ (F1 >0.70 ) transfer consistently; classes marked ✗ (F1 <0.30 ) are limited by domain-specific spectral signatures rare or absent in the Alberta training distribution.
Method
OA
AA
κ
MAE
RMSE
SAD (rad)
DFMamba [ 58 ]
0.6658
0.4122
0.5923
0.0383
0.0939
1.5708
MambaLG [ 59 ]
0.6837
0.4324
0.6142
0.0366
0.0914
1.5708
MambaHSI [ 60 ]
0.6951
0.4442
0.6297
0.0342
0.0869
1.5708
MHSSMamba [ 61 ]
0.5253
0.2340
0.4057
0.0620
0.1260
1.5708
Mamba-in-Mamba [ 62 ]
0.5403
0.2471
0.4283
0.0593
0.1231
1.5708
MDF-Mamba (Ours)
0.7122
0.4721
0.6575
0.0255
0.0654
0.1843
TABLE XI: Quantitative comparison against recent generic Mamba models on the Alberta test set. Without dedicated physics constraints (ELMM), generic state-space architectures fail to achieve stable spectral reconstruction or accurate fractional abundance estimation.
Subpixel mapping (SPM) of crop types from \SI{250}{m} Moderate Resolution Imaging Spectroradiometer (MODIS) time series to field-scale Agriculture and Agri-Food Canada (AAFC) labels supports acreage estimation and agricultural monitoring. Three difficulties remain: (i)spectral-temporal coupling and mixed pixels that break direct coarse-to-fine classifiers; (ii)intra-class endmember variability that a single spectrum per crop cannot capture; and (iii)uniform refinement on the high-resolution (HR) grid, which treats every subpixel the same, so most of the effort goes to easy pixels while errors remain in the minority class. UnMixMamba addresses them as follows. For(i), a temporal-spectral-spatial decoupled (TSSD) factorization estimates simplex fractions and drives AAFC classification from the same mixture field, instead of mapping low-resolution (LR) observations directly to HR logits. For(ii), ×8 abundance super-resolution and a learnable per-crop variant endmember bank reconstruct the MODIS cube at LR and HR under a linear mixing model. For(iii), the predicted crop fractions decide where a fixed Mamba token budget is spent, so HR refinement focuses on the minority class instead of the full grid. On Alberta 2015 under a fixed spatial-block protocol, UnMixMamba reaches OA 0.8397, mIoU 0.6640, AA 0.7837, and Kappa 0.7715, ahead of ST-DRes, ESPCN, FineCrop, AgriFM, and classical SPM baselines. The task is closed-set four-crop mapping inside a known agricultural mask.
Motasem Alkayid, Yimin Zhu, Naser El-Sheimy +1
Department of Geomatics Engineering, University of Calgary, Canada · Department of Geography, Faculty of Arts, The University of Jordan, Amman, Jordan
Although hyperspectral image (HSI) classification is critical for supporting various environmental applications, it is a challenging task due to the spectral-mixture effect, the spatial-spectral heterogeneity and the difficulty to preserve class boundaries and details. This letter presents a novel unmixing-guided spatial-spectral Mamba with clustering tokens for improved HSI classification, with the following contributions. First, to disentangle the spectral mixture effect in HSI for improved pattern discovery, we design a novel spectral unmixing network that not only automatically learns endmembers and abundance maps from HSI but also accounts for endmember variabilities. Second, to generate Mamba token sequences, based on the clusters defined by abundance maps, we design an efficient Top-K token selection strategy to adaptively sequence the tokens for improved representational capability. Third, to improve spatial-spectral feature learning and detail preservation, based on the Top-K token sequences, we design a novel unmixing-guided spatial-spectral Mamba module that greatly improves traditional Mamba models in terms of token learning and sequencing. Fourth, to learn simultaneously the endmember-abundance patterns and classification labels, a multi-task scheme is designed for model supervision, leading to a new unmixing-classification framework that outputs not only accurate classification maps but also a comprehensive spectral-library and abundance maps. Comparative experiments on four HSI datasets demonstrate that our model can greatly outperform the other state-of-the-art approaches. Code is available at https://github.com/GSIL-UCalgary/Unmixing_guided_Mamba.git
Yimin Zhu, Lincoln Linlin Xu
Department of Geomatics Engineering, University of Calgary, Canada
Tree species classification from Moderate Resolution Imaging Spectroradiometer (MODIS) time series is critical for environmental monitoring, yet challenging due to subtle inter-species signatures, strong spatial-spectral-temporal coupling, and the difficulty of modeling large-scale geographical correlations. This letter presents a Graph-regulated Disentangled Sparse Mamba model (Geo-GDS-Mamba) with three contributions. First, a mini-batch graph-regulated stream models geographic correlations among samples via a Radial-Basis-Function kernel over their physical coordinates, independent of spectral content, and fuses this context through a graph convolutional network without full-batch memory costs. Second, a disentangling Mamba architecture separates the spatial, spectral, and temporal information coupled within each sample into three independent processing streams. Third, sparse token modules adaptively retain the top 30% most informative tokens per branch via learned attention or similarity scores, mitigating Mamba's correlation-decay limitation while reducing computational cost. Experiments on annual MOD13Q1 data across two Canadian provinces show Geo-GDS-Mamba achieves 95.35% OA (94.73% Kappa) in Alberta, outperforming eleven state-of-the-art baselines, and generalizes without fine-tuning to unseen Saskatchewan data at 87.63% OA, using only 1.74M parameters and 0.18 GFLOPs. Ablation studies confirm each component's contribution, led by a 4.15-point OA gain from the geographic graph stream alone.
Motasem Alkayid, Zhengsen Xu, Naser El-Sheimy +1
Department of Geomatics Engineering, University of Calgary, Canada · Department of Geography, Faculty of Arts, The University of Jordan, Amman, Jordan