LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification

Jiawen Wen; Suixuan Qiu; Zihang Luo; Xiaofei Yang; Haotian Shi

cs.CVMar 25, 2026

LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification

Authors: Jiawen Wen, Suixuan Qiu, Zihang Luo, Xiaofei Yang, Haotian Shi

Organizations: Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · Faculty of Geographical Science, Beijing Normal University, Beijing, 100875, China · School of Electronic and Communication Engineering, Guangzhou University, Guangzhou University, Guangzhou, Guangzhou, 510006, China

Abstract

Deep learning methods, including Convolutional Neural Networks, Transformers and Mamba, have achieved remarkable success in hyperspectral image (HSI) classification. Nevertheless, existing methods exhibit inflexible integration of local-global representations, inadequate handling of spectral-spatial scale disparities across heterogeneous bands, and susceptibility to the Hughes phenomenon under high-dimensional sample heterogeneity. To address these challenges, we propose Local-Global Expert Spatial-Spectral Transformer (LGEST), a novel framework that synergistically combines three key innovations. The LGEST first employs a Deep Spatial-Spectral Autoencoder (DSAE) to generate compact yet discriminative embeddings through hierarchical nonlinear compression, preserving 3D neighborhood coherence while mitigating information loss in high-dimensional spaces. Secondly, a Cross-Interactive Mixed Expert Feature Pyramid (CIEM-FPN) leverages cross-attention mechanisms and residual mixture-of-experts layers to dynamically fuse multi-scale features, adaptively weighting spectral discriminability and spatial saliency through learnable gating functions. Finally, a Local-Global Expert System (LGES) processes decomposed features via sparsely activated expert pairs: convolutional sub-experts capture fine-grained textures, while transformer sub-experts model long-range contextual dependencies, with a routing controller dynamically selecting experts based on real-time feature saliency. Extensive experiments on four benchmark datasets demonstrate that LGEST consistently outperforms state-of-the-art methods.

Figures & tables

Figure 1: The architecture of proposed LGEST. The network processes input data with a deep spatial-spectral auto-encoder (DSAE), which serves as a local feature extractor. It then uses a cross-interactive mixed expert feature pyramid to fuse features across different scales via parallel and downsampling branches, enhancing interactions through the Cross-Interactive mixed Expert Feature Pyramid (CIEM-FPN). This is followed by the local-global expert system (LGES), divided into two expert groups handling local and global information. The router selects the activated expert from sub-experts within each group, which are linear layers.

Figure 2: The structure of the cross-interactive mixed expert feature pyramid (CIEM-FPN). (a) shows the overall framework of CIEM-FPN, which incorporates a novel cross-interactive mixed expert module (CIEM) that enhances model comprehension and generation by creating associations between different input sequences.(b) presents the detailed structure of the CIEM module. To enable cross-interactive functionality, these inputs can originate from both the parallel structure and the upper output. (c) illustrates the RMoE layer, in which the experts are learnable matrices. (d) illustrates the cross-attention mechanism employed in CIEM.

Class	SVM	2D CNN	3D CNN	HybridSN	SyCNN	ViT	Cross ViT	Deep ViT	CaiT	HiT	Morphformer	SS-TMNet	GSC-ViT	LGEST (ours)
1	0.00 $\pm$ 0.00	15.56 $\pm$ 21.20	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	73.79 $\pm$ 8.25	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	7.95 $\pm$ 16.48	2.73 $\pm$ 5.45	14.19 $\pm$ 16.39	42.62 $\pm$ 23.14	68.73 $\pm$ 11.55	63.30 $\pm$ 30.15	86.46 $\pm$ 9.72
2	36.47 $\pm$ 2.36	88.73 $\pm$ 1.52	64.04 $\pm$ 3.58	60.05 $\pm$ 21.33	84.13 $\pm$ 1.33	48.68 $\pm$ 5.89	60.89 $\pm$ 5.29	78.18 $\pm$ 4.71	75.02 $\pm$ 2.33	86.47 $\pm$ 5.69	90.83 $\pm$ 1.71	87.35 $\pm$ 4.47	92.47 $\pm$ 1.39	92.51 $\pm$ 0.87
3	0.00 $\pm$ 0.00	87.70 $\pm$ 1.61	66.25 $\pm$ 6.79	45.77 $\pm$ 28.93	85.64 $\pm$ 3.64	10.76 $\pm$ 5.95	26.65 $\pm$ 20.21	71.36 $\pm$ 9.57	70.62 $\pm$ 3.68	87.36 $\pm$ 2.90	87.99 $\pm$ 1.82	87.70 $\pm$ 2.20	88.78 $\pm$ 0.38	91.25 $\pm$ 1.44
4	0.00 $\pm$ 0.00	91.35 $\pm$ 5.33	14.33 $\pm$ 16.14	24.53 $\pm$ 33.52	91.30 $\pm$ 2.80	36.88 $\pm$ 16.35	61.66 $\pm$ 13.63	87.78 $\pm$ 5.11	86.50 $\pm$ 2.86	86.87 $\pm$ 10.28	91.78 $\pm$ 5.64	89.03 $\pm$ 5.24	91.98 $\pm$ 4.12	94.24 $\pm$ 2.88
5	0.23 $\pm$ 0.23	92.15 $\pm$ 1.98	83.09 $\pm$ 10.96	38.52 $\pm$ 31.57	88.75 $\pm$ 2.27	13.84 $\pm$ 11.51	56.83 $\pm$ 6.31	57.34 $\pm$ 13.77	49.31 $\pm$ 15.23	82.94 $\pm$ 11.04	91.53 $\pm$ 2.18	87.31 $\pm$ 1.96	94.13 $\pm$ 0.70	93.43 $\pm$ 1.98
6	66.95 $\pm$ 0.55	93.22 $\pm$ 1.32	85.85 $\pm$ 4.15	62.89 $\pm$ 16.27	90.52 $\pm$ 1.96	58.41 $\pm$ 5.82	73.54 $\pm$ 8.45	82.43 $\pm$ 2.75	73.46 $\pm$ 1.89	90.63 $\pm$ 2.20	92.38 $\pm$ 2.04	92.49 $\pm$ 1.26	94.14 $\pm$ 0.49	93.76 $\pm$ 1.22

Table 1: Classification results for the IndianPines dataset (10% training samples).

Class	2D CNN	3D CNN	HybridSN	SyCNN	ViT	Cross ViT	Deep ViT	CaiT	HiT	SSFTT	Morphformer	SS-TMNet	GSC-ViT	LGEST (ours)
1	86.06 $\pm$ 1.52	91.22 $\pm$ 2.83	92.73 $\pm$ 0.94	91.75 $\pm$ 0.98	55.30 $\pm$ 4.30	76.22 $\pm$ 2.71	78.72 $\pm$ 0.85	70.51 $\pm$ 5.77	93.72 $\pm$ 1.25	90.53 $\pm$ 6.93	90.61 $\pm$ 4.34	89.99 $\pm$ 7.37	95.00 $\pm$ 0.62	91.72 $\pm$ 1.61
2	57.62 $\pm$ 9.53	65.44 $\pm$ 11.37	49.31 $\pm$ 16.05	63.21 $\pm$ 8.43	0.00 $\pm$ 0.00	0.82 $\pm$ 2.08	1.45 $\pm$ 2.57	0.80 $\pm$ 1.61	22.61 $\pm$ 14.75	73.37 $\pm$ 13.53	72.10 $\pm$ 11.72	73.62 $\pm$ 4.14	67.53 $\pm$ 13.93	77.77 $\pm$ 4.23
3	51.43 $\pm$ 6.90	67.50 $\pm$ 15.67	77.63 $\pm$ 10.51	81.42 $\pm$ 7.37	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	73.18 $\pm$ 10.19	67.84 $\pm$ 24.07	71.44 $\pm$ 24.15	75.99 $\pm$ 7.31	70.91 $\pm$ 9.22	75.47 $\pm$ 5.66
4	41.69 $\pm$ 9.14	50.13 $\pm$ 9.48	51.99 $\pm$ 20.42	63.04 $\pm$ 12.31	29.10 $\pm$ 21.65	38.95 $\pm$ 17.36	50.43 $\pm$ 3.34	25.23 $\pm$ 20.85	31.94 $\pm$ 18.55	77.06 $\pm$ 13.31	74.65 $\pm$ 13.11	66.28 $\pm$ 11.48	60.58 $\pm$ 3.08	77.25 $\pm$ 5.77
5	64.78 $\pm$ 12.26	60.13 $\pm$ 11.54	37.63 $\pm$ 16.43	53.64 $\pm$ 12.49	5.60 $\pm$ 8.74	44.29 $\pm$ 24.05	53.83 $\pm$ 23.55	14.35 $\pm$ 16.74	89.49 $\pm$ 5.52	72.93 $\pm$ 14.29	67.16 $\pm$ 23.72	82.07 $\pm$ 7.40	71.87 $\pm$ 8.73	95.79 $\pm$ 2.45
6	53.46 $\pm$ 10.36	81.64 $\pm$ 5.69	73.99 $\pm$ 6.46	75.47 $\pm$ 9.69	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	3.21 $\pm$ 7.54	0.00 $\pm$ 0.00	87.78 $\pm$ 6.72	72.03 $\pm$ 27.06	77.59 $\pm$ 15.60	70.48 $\pm$ 17.80	72.81 $\pm$ 8.24	90.03 $\pm$ 7.67

Table 2: Classification results for the KSC dataset (10% training samples).

Class	SVM	3D CNN	HybridSN	SyCNN	ViT	Cross ViT	Deep ViT	CaiT	RVT	HiT	Morphformer	SS-TMNet	GSC-ViT	LGEST (ours)
1	90.94 $\pm$ 0.21	86.75 $\pm$ 1.72	83.54 $\pm$ 5.73	89.17 $\pm$ 1.11	79.78 $\pm$ 1.34	80.08 $\pm$ 2.46	86.50 $\pm$ 2.91	83.41 $\pm$ 3.72	88.36 $\pm$ 3.04	89.26 $\pm$ 1.68	92.63 $\pm$ 1.51	78.82 $\pm$ 24.36	91.79 $\pm$ 0.78	93.69 $\pm$ 1.94
2	96.56 $\pm$ 0.41	84.53 $\pm$ 3.22	83.45 $\pm$ 6.53	88.74 $\pm$ 2.34	63.64 $\pm$ 5.01	67.29 $\pm$ 5.82	76.70 $\pm$ 4.00	66.97 $\pm$ 4.18	89.50 $\pm$ 3.17	85.94 $\pm$ 2.11	93.44 $\pm$ 1.22	87.15 $\pm$ 10.45	93.66 $\pm$ 0.74	94.77 $\pm$ 1.18
3	59.63 $\pm$ 32.19	97.33 $\pm$ 1.59	93.32 $\pm$ 5.38	96.47 $\pm$ 1.60	95.24 $\pm$ 0.94	97.08 $\pm$ 1.46	97.51 $\pm$ 1.87	97.05 $\pm$ 0.82	97.50 $\pm$ 0.73	98.10 $\pm$ 1.03	98.48 $\pm$ 0.68	97.34 $\pm$ 1.56	98.16 $\pm$ 0.91	98.34 $\pm$ 1.34
4	93.87 $\pm$ 0.32	84.62 $\pm$ 3.48	78.82 $\pm$ 5.64	85.06 $\pm$ 2.97	64.98 $\pm$ 6.30	65.14 $\pm$ 7.32	76.68 $\pm$ 6.51	66.24 $\pm$ 5.60	84.97 $\pm$ 3.55	79.25 $\pm$ 4.31	93.93 $\pm$ 1.45	88.98 $\pm$ 7.77	94.24 $\pm$ 0.91	91.71 $\pm$ 1.62
5	90.97 $\pm$ 0.22	92.38 $\pm$ 3.09	91.79 $\pm$ 6.27	97.17 $\pm$ 1.08	87.35 $\pm$ 2.10	97.19 $\pm$ 1.05	98.47 $\pm$ 1.03	95.78 $\pm$ 0.89	99.44 $\pm$ 0.48	98.67 $\pm$ 0.64	99.21 $\pm$ 0.50	94.14 $\pm$ 10.30	99.83 $\pm$ 0.19	99.84 $\pm$ 0.15
6	89.94 $\pm$ 0.21	85.74 $\pm$ 1.15	85.82 $\pm$ 4.24	84.19 $\pm$ 2.62	79.63 $\pm$ 4.89	85.25 $\pm$ 1.34	83.03 $\pm$ 4.31	81.36 $\pm$ 3.86	84.47 $\pm$ 2.96	85.50 $\pm$ 0.89	90.48 $\pm$ 3.77	89.13 $\pm$ 3.97	88.23 $\pm$ 1.97	93.59 $\pm$ 1.34

Table 3: Classification results for the Houston2013 dataset (10% training samples).

Class	SVM	2D CNN	3D CNN	HybridSN	SyCNN	ViT	Cross ViT	Deep ViT	SSFTT	HiT	Morphformer	SS-TMNet	GSC-ViT	LGEST (ours)
1	74.34 $\pm$ 1.28	99.13 $\pm$ 0.31	95.12 $\pm$ 1.88	87.72 $\pm$ 22.49	97.26 $\pm$ 0.95	85.94 $\pm$ 3.12	89.96 $\pm$ 3.00	92.45 $\pm$ 2.62	98.31 $\pm$ 0.78	97.34 $\pm$ 2.65	98.36 $\pm$ 0.93	99.10 $\pm$ 0.51	99.20 $\pm$ 0.41	99.30 $\pm$ 0.27
2	27.18 $\pm$ 14.88	64.75 $\pm$ 7.57	59.79 $\pm$ 13.99	50.97 $\pm$ 28.43	83.74 $\pm$ 5.91	58.59 $\pm$ 3.23	64.29 $\pm$ 6.50	67.76 $\pm$ 7.26	75.38 $\pm$ 27.09	75.16 $\pm$ 9.31	87.14 $\pm$ 4.97	81.33 $\pm$ 3.20	84.17 $\pm$ 7.31	86.52 $\pm$ 3.67
3	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	1.02 $\pm$ 2.82	3.11 $\pm$ 4.96	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	2.39 $\pm$ 5.47	9.61 $\pm$ 17.94	26.74 $\pm$ 31.27	20.60 $\pm$ 31.25	48.96 $\pm$ 40.50	74.51 $\pm$ 24.81
4	85.37 $\pm$ 0.56	91.14 $\pm$ 2.13	88.82 $\pm$ 0.66	90.01 $\pm$ 5.91	93.53 $\pm$ 0.69	90.76 $\pm$ 0.48	92.13 $\pm$ 0.84	92.51 $\pm$ 0.91	94.44 $\pm$ 1.01	94.04 $\pm$ 1.77	94.45 $\pm$ 1.46	94.74 $\pm$ 1.13	96.88 $\pm$ 1.21	96.68 $\pm$ 0.98
5	0.00 $\pm$ 0.00	19.05 $\pm$ 21.20	44.35 $\pm$ 19.88	19.09 $\pm$ 27.35	57.19 $\pm$ 15.82	0.00 $\pm$ 0.01	7.92 $\pm$ 9.82	10.18 $\pm$ 8.02	37.26 $\pm$ 30.46	38.35 $\pm$ 14.23	57.82 $\pm$ 14.87	62.91 $\pm$ 18.67	60.47 $\pm$ 28.11	63.38 $\pm$ 16.32
6	1.86 $\pm$ 3.11	96.45 $\pm$ 1.71	92.14 $\pm$ 3.78	69.91 $\pm$ 35.94	92.14 $\pm$ 3.81	49.70 $\pm$ 28.56	71.19 $\pm$ 21.21	75.20 $\pm$ 18.12	96.02 $\pm$ 1.84	95.44 $\pm$ 2.54	96.77 $\pm$ 1.38	96.05 $\pm$ 1.96	95.46 $\pm$ 3.72	95.40 $\pm$ 1.78

Table 4: Classification results for the WHU-Hi-LongKou dataset (0.1% training samples).

Figure 3: Classification map of different methods on the IndianPines dataset (with 10% training samples).

Figure 4: Classification map of different methods on the KSC dataset (with 10% training samples).

Figure 5: Classification map of different methods on the WHU-Hi-LongKou dataset (with 0.1% training samples).

Figure 6: Classification map of different methods on the Houston2013 dataset (with 10% training samples).

Figure 7: Visualization of t-SNE data analysis on the IndianPines dataset.

Methods	F(G)	P(M)	OA(%) $\uparrow$
RvT	21.11	8.94	90.14 $\pm$ 0.90
CaiT	16.79	163.66	75.76 $\pm$ 1.52
SyCNN	194.11	45.81	88.14 $\pm$ 1.19
LGEST	108.32	32.99	93.69 $\pm$ 0.44

Table 5: Complexity analysis on the Indian Pines dataset.

Methods	OA(%) $\uparrow$	AA(%) $\uparrow$	Kappa(%) $\uparrow$
DSAE	91.50	74.49	90.30
DSAE w/Local Expert Group	93.05	77.25	92.05
LGEST	93.67	79.54	92.76
w/o CIEM-FPN	92.82	72.54	91.78
w/o Local Expert Group	93.28	77.31	92.31
w/o Global Expert Group	93.33	78.60	92.37

Table 6: Ablation studies on key components of LGEST.

Figure 8: Visualization of t-SNE data analysis on different modules in LGEST. The area marked by the red box is the confusing part.

Datasets	Number	F(G)	P(M)	OA(%) $\uparrow$	AA(%) $\uparrow$	Kappa(%) $\uparrow$
Indian Pines	[2,2]	107.26	22.36	93.41 $\pm$ 0.61	80.48 $\pm$ 2.41	92.47 $\pm$ 0.70
	[4,4]	108.32	32.99	93.69 $\pm$ 0.44	80.72 $\pm$ 3.00	92.80 $\pm$ 0.50
	[8,8]	110.45	54.24	93.56 $\pm$ 1.11	80.78 $\pm$ 4.13	92.63 $\pm$ 0.45
	[16,16]	114.70	96.75	93.45 $\pm$ 0.47	79.53 $\pm$ 3.49	92.51 $\pm$ 0.54
KSC	[2,2]	106.77	20.58	92.59 $\pm$ 1.44	90.32 $\pm$ 1.89	91.75 $\pm$ 1.60
	[4,4]	107.66	29.43	91.44 $\pm$ 1.21	88.63 $\pm$ 1.66	90.46 $\pm$ 1.34

Table 7: Ablation study on the number of experts. (best results in bold)

Figure 9: Ablation study with training percentages on four datasets.

Datasets	Patch Size	OA(%) $\uparrow$	AA(%) $\uparrow$	Kappa(%) $\uparrow$
Indian Pines	9×9	95.07 $\pm$ 0.74	86.86 $\pm$ 0.77	94.37 $\pm$ 0.85
	11×11	95.50 $\pm$ 0.32	83.07 $\pm$ 0.96	94.87 $\pm$ 0.36
	13×13	95.01 $\pm$ 0.65	84.55 $\pm$ 2.80	94.30 $\pm$ 0.74
	15×15	93.56 $\pm$ 1.11	80.78 $\pm$ 4.13	92.63 $\pm$ 0.45
	17×17	91.69 $\pm$ 0.72	75.48 $\pm$ 1.44	90.51 $\pm$ 0.82
KSC	9×9	89.21 $\pm$ 1.21	83.28 $\pm$ 1.72	87.97 $\pm$ 1.35

Table 8: Ablation study on the patch sizes (best results in bold)

Explore similar work

A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification
Apr 26, 2026Peng Chen, Wenxuan He, Feng Qian +2Hybrid CNN-Transformer Architectures Transformer Attention
College of Engineering, Shantou University, Shantou, 515063, Guangdong, China · School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, China · Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun, 130033, Jilin, China +2
3D Fourier-based Global Feature Extraction for Hyperspectral Image Classification
Mar 17, 2026Muhammad AhmadHyperspectral Image Classification Frequency-Domain Feature Learning
SDAIA-KFUPM, Joint Research Center for Artificial Intelligence (JRCAI), King Fahd University of Petroleum and Minerals, Dhahran, 31261, Saudi Arabia.
SSFT: A Lightweight Spectral-Spatial Fusion Transformer for Generic Hyperspectral Classification
Apr 17, 2026Alexander Musiat, Nikolas Ebert, Oliver WasenmüllerRepresentation Learning Hyperspectral Imaging
Mannheim University of Applied Sciences, Germany

Apr 26, 2026J/K move · Enter open · S save

A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

Peng Chen, Wenxuan He, Feng Qian, Guangyao Shi+1

College of Engineering, Shantou University, Shantou, 515063, Guangdong, China · School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, 430074, Hubei, China · Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun, 130033, Jilin, China+2

Hybrid CNN-Transformer Architectures Transformer Attention Hyperspectral Image Classification

Open paper AskPDF arXiv

In the hyperspectral image (HSI) classification task, each pixel is categorized into a specific land-cover category or material. Convolutional neural networks (CNNs) and transformers have been widely used to extract local and non-local features in HSI classification. Recent works have utilized a multi-scale vision transformer (ViT) to enhance spectral feature capture and yield promising results. However, most existing methods still face challenges in the effective joint use of spatial-spectral information and in preserving information across layers during the propagation process. To address these issues, we propose a synergistic CNN-Transformer network with pooling attention fusion for HSI classification, which collaboratively utilizes CNNs and ViT to process spatial and spectral features separately. Specifically, we propose a Twin-Branch Feature Extraction (TBFE) module, which employs 3D and 2D convolution in parallel to comprehensively extract spectral and spatial features from HSI. A hybrid pooling attention (HPA) module is designed to aggregate spatial attention. Moreover, a cascade transformer encoder is employed for global spectral feature extraction, and a simple yet efficient cross-layer feature fusion (CFF) module is designed to reduce the loss of crucial information in the previous network layers. Extensive experiments are conducted on several representative datasets to demonstrate the superior performance of our proposed method compared to the state-of-the-art works. Code is available at https://github.com/chenpeng052/SCT-Net.git.

Abstract

Explore similar work

A Synergistic CNN-Transformer Network with Pooling Attention Fusion for Hyperspectral Image Classification

3D Fourier-based Global Feature Extraction for Hyperspectral Image Classification

SSFT: A Lightweight Spectral-Spatial Fusion Transformer for Generic Hyperspectral Classification