Organizations: Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · Faculty of Geographical Science, Beijing Normal University, Beijing, 100875, China · School of Electronic and Communication Engineering, Guangzhou University, Guangzhou University, Guangzhou, Guangzhou, 510006, China
Deep learning methods, including Convolutional Neural Networks, Transformers and Mamba, have achieved remarkable success in hyperspectral image (HSI) classification. Nevertheless, existing methods exhibit inflexible integration of local-global representations, inadequate handling of spectral-spatial scale disparities across heterogeneous bands, and susceptibility to the Hughes phenomenon under high-dimensional sample heterogeneity. To address these challenges, we propose Local-Global Expert Spatial-Spectral Transformer (LGEST), a novel framework that synergistically combines three key innovations. The LGEST first employs a Deep Spatial-Spectral Autoencoder (DSAE) to generate compact yet discriminative embeddings through hierarchical nonlinear compression, preserving 3D neighborhood coherence while mitigating information loss in high-dimensional spaces. Secondly, a Cross-Interactive Mixed Expert Feature Pyramid (CIEM-FPN) leverages cross-attention mechanisms and residual mixture-of-experts layers to dynamically fuse multi-scale features, adaptively weighting spectral discriminability and spatial saliency through learnable gating functions. Finally, a Local-Global Expert System (LGES) processes decomposed features via sparsely activated expert pairs: convolutional sub-experts capture fine-grained textures, while transformer sub-experts model long-range contextual dependencies, with a routing controller dynamically selecting experts based on real-time feature saliency. Extensive experiments on four benchmark datasets demonstrate that LGEST consistently outperforms state-of-the-art methods.
Figures & tables
Figure 1: The architecture of proposed LGEST. The network processes input data with a deep spatial-spectral auto-encoder (DSAE), which serves as a local feature extractor. It then uses a cross-interactive mixed expert feature pyramid to fuse features across different scales via parallel and downsampling branches, enhancing interactions through the Cross-Interactive mixed Expert Feature Pyramid (CIEM-FPN). This is followed by the local-global expert system (LGES), divided into two expert groups handling local and global information. The router selects the activated expert from sub-experts within each group, which are linear layers.
Figure 2: The structure of the cross-interactive mixed expert feature pyramid (CIEM-FPN). (a) shows the overall framework of CIEM-FPN, which incorporates a novel cross-interactive mixed expert module (CIEM) that enhances model comprehension and generation by creating associations between different input sequences.(b) presents the detailed structure of the CIEM module. To enable cross-interactive functionality, these inputs can originate from both the parallel structure and the upper output. (c) illustrates the RMoE layer, in which the experts are learnable matrices. (d) illustrates the cross-attention mechanism employed in CIEM.
Class
SVM
2D CNN
3D CNN
HybridSN
SyCNN
ViT
Cross ViT
Deep ViT
CaiT
HiT
Morphformer
SS-TMNet
GSC-ViT
LGEST (ours)
1
0.00 ± 0.00
15.56 ± 21.20
0.00 ± 0.00
0.00 ± 0.00
73.79 ± 8.25
0.00 ± 0.00
0.00 ± 0.00
7.95 ± 16.48
2.73 ± 5.45
14.19 ± 16.39
42.62 ± 23.14
68.73 ± 11.55
63.30 ± 30.15
86.46 ± 9.72
2
36.47 ± 2.36
88.73 ± 1.52
64.04 ± 3.58
60.05 ± 21.33
84.13 ± 1.33
48.68 ± 5.89
60.89 ± 5.29
78.18 ± 4.71
75.02 ± 2.33
86.47 ± 5.69
90.83 ± 1.71
87.35 ± 4.47
92.47 ± 1.39
92.51 ± 0.87
3
0.00 ± 0.00
87.70 ± 1.61
66.25 ± 6.79
45.77 ± 28.93
85.64 ± 3.64
10.76 ± 5.95
26.65 ± 20.21
71.36 ± 9.57
70.62 ± 3.68
87.36 ± 2.90
87.99 ± 1.82
87.70 ± 2.20
88.78 ± 0.38
91.25 ± 1.44
4
0.00 ± 0.00
91.35 ± 5.33
14.33 ± 16.14
24.53 ± 33.52
91.30 ± 2.80
36.88 ± 16.35
61.66 ± 13.63
87.78 ± 5.11
86.50 ± 2.86
86.87 ± 10.28
91.78 ± 5.64
89.03 ± 5.24
91.98 ± 4.12
94.24 ± 2.88
5
0.23 ± 0.23
92.15 ± 1.98
83.09 ± 10.96
38.52 ± 31.57
88.75 ± 2.27
13.84 ± 11.51
56.83 ± 6.31
57.34 ± 13.77
49.31 ± 15.23
82.94 ± 11.04
91.53 ± 2.18
87.31 ± 1.96
94.13 ± 0.70
93.43 ± 1.98
6
66.95 ± 0.55
93.22 ± 1.32
85.85 ± 4.15
62.89 ± 16.27
90.52 ± 1.96
58.41 ± 5.82
73.54 ± 8.45
82.43 ± 2.75
73.46 ± 1.89
90.63 ± 2.20
92.38 ± 2.04
92.49 ± 1.26
94.14 ± 0.49
93.76 ± 1.22
Table 1: Classification results for the IndianPines dataset (10% training samples).
Class
2D CNN
3D CNN
HybridSN
SyCNN
ViT
Cross ViT
Deep ViT
CaiT
HiT
SSFTT
Morphformer
SS-TMNet
GSC-ViT
LGEST (ours)
1
86.06 ± 1.52
91.22 ± 2.83
92.73 ± 0.94
91.75 ± 0.98
55.30 ± 4.30
76.22 ± 2.71
78.72 ± 0.85
70.51 ± 5.77
93.72 ± 1.25
90.53 ± 6.93
90.61 ± 4.34
89.99 ± 7.37
95.00 ± 0.62
91.72 ± 1.61
2
57.62 ± 9.53
65.44 ± 11.37
49.31 ± 16.05
63.21 ± 8.43
0.00 ± 0.00
0.82 ± 2.08
1.45 ± 2.57
0.80 ± 1.61
22.61 ± 14.75
73.37 ± 13.53
72.10 ± 11.72
73.62 ± 4.14
67.53 ± 13.93
77.77 ± 4.23
3
51.43 ± 6.90
67.50 ± 15.67
77.63 ± 10.51
81.42 ± 7.37
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
73.18 ± 10.19
67.84 ± 24.07
71.44 ± 24.15
75.99 ± 7.31
70.91 ± 9.22
75.47 ± 5.66
4
41.69 ± 9.14
50.13 ± 9.48
51.99 ± 20.42
63.04 ± 12.31
29.10 ± 21.65
38.95 ± 17.36
50.43 ± 3.34
25.23 ± 20.85
31.94 ± 18.55
77.06 ± 13.31
74.65 ± 13.11
66.28 ± 11.48
60.58 ± 3.08
77.25 ± 5.77
5
64.78 ± 12.26
60.13 ± 11.54
37.63 ± 16.43
53.64 ± 12.49
5.60 ± 8.74
44.29 ± 24.05
53.83 ± 23.55
14.35 ± 16.74
89.49 ± 5.52
72.93 ± 14.29
67.16 ± 23.72
82.07 ± 7.40
71.87 ± 8.73
95.79 ± 2.45
6
53.46 ± 10.36
81.64 ± 5.69
73.99 ± 6.46
75.47 ± 9.69
0.00 ± 0.00
0.00 ± 0.00
3.21 ± 7.54
0.00 ± 0.00
87.78 ± 6.72
72.03 ± 27.06
77.59 ± 15.60
70.48 ± 17.80
72.81 ± 8.24
90.03 ± 7.67
Table 2: Classification results for the KSC dataset (10% training samples).
Class
SVM
3D CNN
HybridSN
SyCNN
ViT
Cross ViT
Deep ViT
CaiT
RVT
HiT
Morphformer
SS-TMNet
GSC-ViT
LGEST (ours)
1
90.94 ± 0.21
86.75 ± 1.72
83.54 ± 5.73
89.17 ± 1.11
79.78 ± 1.34
80.08 ± 2.46
86.50 ± 2.91
83.41 ± 3.72
88.36 ± 3.04
89.26 ± 1.68
92.63 ± 1.51
78.82 ± 24.36
91.79 ± 0.78
93.69 ± 1.94
2
96.56 ± 0.41
84.53 ± 3.22
83.45 ± 6.53
88.74 ± 2.34
63.64 ± 5.01
67.29 ± 5.82
76.70 ± 4.00
66.97 ± 4.18
89.50 ± 3.17
85.94 ± 2.11
93.44 ± 1.22
87.15 ± 10.45
93.66 ± 0.74
94.77 ± 1.18
3
59.63 ± 32.19
97.33 ± 1.59
93.32 ± 5.38
96.47 ± 1.60
95.24 ± 0.94
97.08 ± 1.46
97.51 ± 1.87
97.05 ± 0.82
97.50 ± 0.73
98.10 ± 1.03
98.48 ± 0.68
97.34 ± 1.56
98.16 ± 0.91
98.34 ± 1.34
4
93.87 ± 0.32
84.62 ± 3.48
78.82 ± 5.64
85.06 ± 2.97
64.98 ± 6.30
65.14 ± 7.32
76.68 ± 6.51
66.24 ± 5.60
84.97 ± 3.55
79.25 ± 4.31
93.93 ± 1.45
88.98 ± 7.77
94.24 ± 0.91
91.71 ± 1.62
5
90.97 ± 0.22
92.38 ± 3.09
91.79 ± 6.27
97.17 ± 1.08
87.35 ± 2.10
97.19 ± 1.05
98.47 ± 1.03
95.78 ± 0.89
99.44 ± 0.48
98.67 ± 0.64
99.21 ± 0.50
94.14 ± 10.30
99.83 ± 0.19
99.84 ± 0.15
6
89.94 ± 0.21
85.74 ± 1.15
85.82 ± 4.24
84.19 ± 2.62
79.63 ± 4.89
85.25 ± 1.34
83.03 ± 4.31
81.36 ± 3.86
84.47 ± 2.96
85.50 ± 0.89
90.48 ± 3.77
89.13 ± 3.97
88.23 ± 1.97
93.59 ± 1.34
Table 3: Classification results for the Houston2013 dataset (10% training samples).
Class
SVM
2D CNN
3D CNN
HybridSN
SyCNN
ViT
Cross ViT
Deep ViT
SSFTT
HiT
Morphformer
SS-TMNet
GSC-ViT
LGEST (ours)
1
74.34 ± 1.28
99.13 ± 0.31
95.12 ± 1.88
87.72 ± 22.49
97.26 ± 0.95
85.94 ± 3.12
89.96 ± 3.00
92.45 ± 2.62
98.31 ± 0.78
97.34 ± 2.65
98.36 ± 0.93
99.10 ± 0.51
99.20 ± 0.41
99.30 ± 0.27
2
27.18 ± 14.88
64.75 ± 7.57
59.79 ± 13.99
50.97 ± 28.43
83.74 ± 5.91
58.59 ± 3.23
64.29 ± 6.50
67.76 ± 7.26
75.38 ± 27.09
75.16 ± 9.31
87.14 ± 4.97
81.33 ± 3.20
84.17 ± 7.31
86.52 ± 3.67
3
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
1.02 ± 2.82
3.11 ± 4.96
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
2.39 ± 5.47
9.61 ± 17.94
26.74 ± 31.27
20.60 ± 31.25
48.96 ± 40.50
74.51 ± 24.81
4
85.37 ± 0.56
91.14 ± 2.13
88.82 ± 0.66
90.01 ± 5.91
93.53 ± 0.69
90.76 ± 0.48
92.13 ± 0.84
92.51 ± 0.91
94.44 ± 1.01
94.04 ± 1.77
94.45 ± 1.46
94.74 ± 1.13
96.88 ± 1.21
96.68 ± 0.98
5
0.00 ± 0.00
19.05 ± 21.20
44.35 ± 19.88
19.09 ± 27.35
57.19 ± 15.82
0.00 ± 0.01
7.92 ± 9.82
10.18 ± 8.02
37.26 ± 30.46
38.35 ± 14.23
57.82 ± 14.87
62.91 ± 18.67
60.47 ± 28.11
63.38 ± 16.32
6
1.86 ± 3.11
96.45 ± 1.71
92.14 ± 3.78
69.91 ± 35.94
92.14 ± 3.81
49.70 ± 28.56
71.19 ± 21.21
75.20 ± 18.12
96.02 ± 1.84
95.44 ± 2.54
96.77 ± 1.38
96.05 ± 1.96
95.46 ± 3.72
95.40 ± 1.78
Table 4: Classification results for the WHU-Hi-LongKou dataset (0.1% training samples).
Figure 3: Classification map of different methods on the IndianPines dataset (with 10% training samples).
Figure 4: Classification map of different methods on the KSC dataset (with 10% training samples).
Figure 5: Classification map of different methods on the WHU-Hi-LongKou dataset (with 0.1% training samples).
Figure 6: Classification map of different methods on the Houston2013 dataset (with 10% training samples).
Figure 7: Visualization of t-SNE data analysis on the IndianPines dataset.
Methods
F(G)
P(M)
OA(%) ↑
RvT
21.11
8.94
90.14 ± 0.90
CaiT
16.79
163.66
75.76 ± 1.52
SyCNN
194.11
45.81
88.14 ± 1.19
LGEST
108.32
32.99
93.69 ± 0.44
Table 5: Complexity analysis on the Indian Pines dataset.
Methods
OA(%) ↑
AA(%) ↑
Kappa(%) ↑
DSAE
91.50
74.49
90.30
DSAE w/Local Expert Group
93.05
77.25
92.05
LGEST
93.67
79.54
92.76
w/o CIEM-FPN
92.82
72.54
91.78
w/o Local Expert Group
93.28
77.31
92.31
w/o Global Expert Group
93.33
78.60
92.37
Table 6: Ablation studies on key components of LGEST.
Figure 8: Visualization of t-SNE data analysis on different modules in LGEST. The area marked by the red box is the confusing part.
Datasets
Number
F(G)
P(M)
OA(%) ↑
AA(%) ↑
Kappa(%) ↑
Indian Pines
[2,2]
107.26
22.36
93.41 ± 0.61
80.48 ± 2.41
92.47 ± 0.70
[4,4]
108.32
32.99
93.69 ± 0.44
80.72 ± 3.00
92.80 ± 0.50
[8,8]
110.45
54.24
93.56 ± 1.11
80.78 ± 4.13
92.63 ± 0.45
[16,16]
114.70
96.75
93.45 ± 0.47
79.53 ± 3.49
92.51 ± 0.54
KSC
[2,2]
106.77
20.58
92.59 ± 1.44
90.32 ± 1.89
91.75 ± 1.60
[4,4]
107.66
29.43
91.44 ± 1.21
88.63 ± 1.66
90.46 ± 1.34
Table 7: Ablation study on the number of experts. (best results in bold)
Figure 9: Ablation study with training percentages on four datasets.
Datasets
Patch Size
OA(%) ↑
AA(%) ↑
Kappa(%) ↑
Indian Pines
9×9
95.07 ± 0.74
86.86 ± 0.77
94.37 ± 0.85
11×11
95.50 ± 0.32
83.07 ± 0.96
94.87 ± 0.36
13×13
95.01 ± 0.65
84.55 ± 2.80
94.30 ± 0.74
15×15
93.56 ± 1.11
80.78 ± 4.13
92.63 ± 0.45
17×17
91.69 ± 0.72
75.48 ± 1.44
90.51 ± 0.82
KSC
9×9
89.21 ± 1.21
83.28 ± 1.72
87.97 ± 1.35
Table 8: Ablation study on the patch sizes (best results in bold)
In the hyperspectral image (HSI) classification task, each pixel is categorized into a specific land-cover category or material. Convolutional neural networks (CNNs) and transformers have been widely used to extract local and non-local features in HSI classification. Recent works have utilized a multi-scale vision transformer (ViT) to enhance spectral feature capture and yield promising results. However, most existing methods still face challenges in the effective joint use of spatial-spectral information and in preserving information across layers during the propagation process. To address these issues, we propose a synergistic CNN-Transformer network with pooling attention fusion for HSI classification, which collaboratively utilizes CNNs and ViT to process spatial and spectral features separately. Specifically, we propose a Twin-Branch Feature Extraction (TBFE) module, which employs 3D and 2D convolution in parallel to comprehensively extract spectral and spatial features from HSI. A hybrid pooling attention (HPA) module is designed to aggregate spatial attention. Moreover, a cascade transformer encoder is employed for global spectral feature extraction, and a simple yet efficient cross-layer feature fusion (CFF) module is designed to reduce the loss of crucial information in the previous network layers. Extensive experiments are conducted on several representative datasets to demonstrate the superior performance of our proposed method compared to the state-of-the-art works. Code is available at https://github.com/chenpeng052/SCT-Net.git.
Peng Chen, Wenxuan He, Feng Qian +2
College of Engineering, Shantou University, Shantou, 515063, Guangdong, China · School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, China · Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun, 130033, Jilin, China +2
Hyperspectral image classification (HSIC) has been significantly advanced by deep learning methods that exploit rich spatial-spectral correlations. However, existing approaches still face fundamental limitations: transformer-based models suffer from poor scalability due to the quadratic complexity of self-attention, while recent Fourier transform-based methods typically rely on 2D spatial FFTs and largely ignore critical inter-band spectral dependencies inherent to hyperspectral data. To address these challenges, we propose Hybrid GFNet (HGFNet), a novel architecture that integrates localized 3D convolutional feature extraction with frequency-domain global filtering via GFNet-style blocks for efficient and robust spatial-spectral representation learning. HGFNet introduces three complementary frequency transforms tailored to hyperspectral imagery: Spectral Fourier Transform (a 1D FFT along the spectral axis), Spatial Fourier Transform (a 2D FFT over spatial dimensions), and Spatial-Spatial Fourier Transform (a 3D FFT jointly over spectral and spatial dimensions), enabling comprehensive and high-dimensional frequency modeling. The 3D convolutional layers capture fine-grained local spatial-spectral structures, while the Fourier-based global filtering modules efficiently model long-range dependencies and suppress noise. To further mitigate the severe class imbalance commonly observed in HSIC, HGFNet incorporates an Adaptive Focal Loss (AFL) that dynamically adjusts class-wise focusing and weighting, improving discrimination for underrepresented classes.
Muhammad Ahmad
SDAIA-KFUPM, Joint Research Center for Artificial Intelligence (JRCAI), King Fahd University of Petroleum and Minerals, Dhahran, 31261, Saudi Arabia.
Hyperspectral imaging enables fine-grained recognition of materials by capturing rich spectral signatures, but learning robust classifiers is challenging due to high dimensionality, spectral redundancy, limited labeled data, and strong domain shifts. Beyond earth observation, labeled HSI data is often scarce and imbalanced, motivating compact models for generic hyperspectral classification across diverse acquisition regimes. We propose the lightweight Spectral-Spatial Fusion Transformer (SSFT), which factorizes representation learning into spectral and spatial pathways and integrates them via cross-attention to capture complementary wavelength-dependent and structural information. We evaluate our SSFT on the challenging HSI-Benchmark, a heterogeneous multi-dataset benchmark covering earth observation, fruit condition assessment, and fine-grained material recognition. SSFT achieves state-of-the-art overall performance, ranking first while using less than 2% of the parameters of the previous leading method. We further evaluate transfer to the substantially larger SpectralEarth benchmark under the official protocol, where SSFT remains competitive despite its compact size. Ablation studies show that both spectral and spatial pathways are crucial, with spatial modeling contributing most, and that SSFT remains robust without data augmentation.
Alexander Musiat, Nikolas Ebert, Oliver Wasenmüller