Spectral-spatial vision test-time training for hyperspectral image classification

Xiang Zhao

eess.IVNov 2, 2026

Spectral-spatial vision test-time training for hyperspectral image classification

Authors: Xiang Zhao

Organizations: College of Mathematical Sciences, Harbin Engineering University, 145 Nantong Street, Nangang District, Harbin 150001, China

Abstract

Hyperspectral image (HSI) classification requires the representation of contiguous spectral signatures and spatial context under limited labeled supervision. Vision test-time training (TTT) offers an alternative to fixed-weight token mixing by performing self-supervised inner-model updates on the input sequence, but its application to center-pixel HSI classification remains insufficiently characterized. We present ViTTT-HSI, a spectral-spatial adaptation of the vision test-time training (ViT3) architecture. A spectral-aware embedding stem extracts ordered-band information using spectral convolution and forms spatial tokens without reducing their spatial resolution. Two ViT3-based TTT encoder blocks perform patch-conditioned token mixing, followed by the fusion of center-pixel and neighborhood-average features for classification. We evaluate the framework on Indian Pines, Pavia University, Salinas, and Houston 2013 using ten random seeds and dataset-specific training fractions of 1% or 5%. The mean overall accuracies are 97.6390%, 98.8753%, 99.3292%, and 98.4283%, respectively. Ablations reveal a substantial contribution from the TTT inner update on Indian Pines and a smaller positive contribution on Pavia University, while the relative benefits of the spectral stem and feature fusion depend on the dataset. The results demonstrate the feasibility of using patch-conditioned vision TTT for spectral--spatial HSI classification and highlight the role of the inner adaptation mechanism.

Figures & tables

Figure 1: Overall architecture of ViTTT-HSI. (a) A centered hyperspectral patch is encoded without spatial downsampling, processed by two TTT encoder blocks, and classified using a center–global feature head. (b) Each encoder block uses residual convolutional positional encoding (CPE), a pre-normalized TTT mixer inherited from ViT 3 [ 3 ] , and a pre-normalized feed-forward network. (c) The center token and spatial average are concatenated for center-pixel prediction.

C

D

N

, and

K

denote spectral bands, token dimension, token count, and classes.

Figure 2: Implementation details of the spectral-aware stem and ViT 3 -based TTT mixer. (a) Spectral convolution, pooling, channel folding, and spatial depthwise convolution produce 64-dimensional spatial tokens. (b) Four 16-dimensional simplified swish-gated linear unit (SwiGLU) heads and a separate 16-channel depthwise convolution branch each perform a one-step, unlabeled key–value inner update and then evaluate query features using the patch-specific fast weights. Both branch designs and their analytical update mechanism follow ViT 3 [ 3 ] . Fast weights do not persist between patches.

Metric	SFMamba	HyperTransXNet	S 2 Mamba	DSFormer	SSFTT	ViTTT-HSI
Class 01	71.19 $\pm$ 25.21	67.86 $\pm$ 20.05	0.48 $\pm$ 1.00	79.05 $\pm$ 18.87	59.05 $\pm$ 28.14	96.43 $\pm$ 2.31
Class 02	91.73 $\pm$ 3.88	93.68 $\pm$ 2.91	93.17 $\pm$ 2.57	92.65 $\pm$ 1.91	91.76 $\pm$ 2.50	96.01 $\pm$ 2.21
Class 03	92.44 $\pm$ 3.46	93.14 $\pm$ 2.97	93.14 $\pm$ 3.55	96.88 $\pm$ 2.33	94.50 $\pm$ 2.23	96.74 $\pm$ 1.48
Class 04	93.00 $\pm$ 5.47	95.16 $\pm$ 4.01	91.74 $\pm$ 6.72	93.80 $\pm$ 4.07	93.94 $\pm$ 7.06	98.22 $\pm$ 1.71
Class 05	92.48 $\pm$ 4.78	93.52 $\pm$ 3.04	91.68 $\pm$ 3.52	93.82 $\pm$ 2.58	93.36 $\pm$ 3.01	93.93 $\pm$ 3.32
Class 06	98.34 $\pm$ 1.15	97.07 $\pm$ 2.22	99.59 $\pm$ 0.28	98.86 $\pm$ 1.05	98.63 $\pm$ 0.96	99.60 $\pm$ 0.28

Table 1: Class-wise recall and overall performance on Indian Pines (%, ten-seed mean

\pm

standard deviation). All metrics, including kappa, are expressed as percentages. Class IDs follow the reference labels; bold marks the highest mean in each row.

Metric	SFMamba	HyperTransXNet	S 2 Mamba	DSFormer	SSFTT	ViTTT-HSI
Class 01	98.70 $\pm$ 2.79	99.36 $\pm$ 0.82	99.90 $\pm$ 0.24	97.64 $\pm$ 5.07	99.84 $\pm$ 0.30	99.85 $\pm$ 0.41
Class 02	99.99 $\pm$ 0.03	99.96 $\pm$ 0.11	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00	99.91 $\pm$ 0.16	99.99 $\pm$ 0.02
Class 03	98.69 $\pm$ 3.80	99.16 $\pm$ 1.15	99.98 $\pm$ 0.04	99.78 $\pm$ 0.53	99.98 $\pm$ 0.07	99.53 $\pm$ 0.50
Class 04	98.95 $\pm$ 2.13	97.56 $\pm$ 2.60	98.66 $\pm$ 1.24	99.43 $\pm$ 0.33	99.02 $\pm$ 0.70	99.39 $\pm$ 1.34
Class 05	98.69 $\pm$ 1.92	98.55 $\pm$ 1.02	98.64 $\pm$ 2.20	98.91 $\pm$ 1.60	99.02 $\pm$ 1.23	98.01 $\pm$ 0.83
Class 06	99.89 $\pm$ 0.20	99.83 $\pm$ 0.36	99.83 $\pm$ 0.53	100.00 $\pm$ 0.00	99.89 $\pm$ 0.19	99.90 $\pm$ 0.23

Table 3: Class-wise recall and overall performance on Salinas (%, ten-seed mean

\pm

standard deviation). All metrics, including kappa, are expressed as percentages. Class IDs follow the reference labels; bold marks the highest mean in each row.

Panel of Figure 3 — Figure 3: Classification maps on Indian Pines: (a) ground truth; (b) SFMamba; (c) HyperTransXNet; (d) S 2 Mamba; (e) DSFormer; (f) SSFTT; (g) ViTTT-HSI. Each model’s prediction appears above its error map.

Panel of Figure 5 — Figure 5: Classification maps on Salinas. Panel order and the error-map convention follow Fig. 4 .

Figure 7: Training cross-entropy across seeds 3405–3414: (a) Indian Pines; (b) Pavia University; (c) Salinas; (d) Houston 2013. Lines and bands show the mean and

\pm

one sample standard deviation of the available runs at each epoch; stopped runs are not extended.

Figure 8: Validation cross-entropy across seeds 3405–3414. The line/band convention and panels (a)–(d) follow Fig. 7 ; stopped runs are not extended.

Dataset	Variant	OA	AA
Indian	Linear, no TTT	93.76 $\pm$ 0.88	89.45 $\pm$ 3.13
Indian	Spectral, no TTT	94.19 $\pm$ 0.95	90.91 $\pm$ 2.16
Indian	Linear, TTT	97.42 $\pm$ 0.46	94.85 $\pm$ 1.89
Indian	Spectral, TTT	97.64 $\pm$ 0.58	95.18 $\pm$ 1.83
Pavia	Linear, no TTT	98.42 $\pm$ 0.38	97.72 $\pm$ 0.64
Pavia	Spectral, no TTT	98.46 $\pm$ 0.35	97.78 $\pm$ 0.69

Table 5: Embedding and TTT ablation on Indian Pines and Pavia University (%, ten seeds).

Dataset	Mixer setting	OA	AA
Indian	No TTT	94.19 $\pm$ 0.95	90.91 $\pm$ 2.16
Indian	Frozen inner	94.08 $\pm$ 0.65	90.18 $\pm$ 2.06
Indian	Full	97.64 $\pm$ 0.58	95.18 $\pm$ 1.83
Pavia	No TTT	98.46 $\pm$ 0.35	97.78 $\pm$ 0.69
Pavia	Frozen inner	98.63 $\pm$ 0.31	98.11 $\pm$ 0.52
Pavia	Full	98.88 $\pm$ 0.49	98.51 $\pm$ 0.72

Table 6: Ablation of the TTT inner update (%, mean

\pm

standard deviation; ten seeds).

Dataset	Head	OA
Indian	Center	97.43 $\pm$ 0.40
Indian	Global	95.04 $\pm$ 0.69
Indian	Center + Global	97.64 $\pm$ 0.58
Pavia	Center	98.90 $\pm$ 0.41
Pavia	Global	94.71 $\pm$ 0.71
Pavia	Center + Global	98.88 $\pm$ 0.49

Table 7: Classification-head ablation (%, ten-seed OA mean

\pm

standard deviation).

Dataset	$P$	Full TTT	No TTT	Difference
Indian	11	97.34	95.30	+2.04
Indian	21	97.24	95.17	+2.07
Indian	29	97.69	94.80	+2.89
Pavia	11	97.46	97.36	+0.11
Pavia	21	98.74	98.21	+0.53
Pavia	29	98.72	98.50	+0.22

Table 8: Patch-size ablation on Indian Pines and Pavia University (%, mean OA over three seeds). The last column is the Full TTT minus No TTT difference in percentage points.

Variant	OA	AA
Full	90.16 $\pm$ 1.04	87.39 $\pm$ 2.65
No TTT	88.76 $\pm$ 0.85	85.56 $\pm$ 6.92

Table 9: Spatial-block evaluation on Pavia University using patch size 11 (%, mean

\pm

standard deviation across three seeds).

Dataset	Parameters (M)	GFLOPs	Train (s)	Test (s)
Indian	0.0923	0.167	139.1	4.04
Pavia	0.0914	0.165	42.2	11.15
Salinas	0.0923	0.145	93.3	20.44
Houston 2013	0.0922	0.166	219.5	4.33

Table 10: Model complexity and seed-3405 runtime of ViTTT-HSI. GFLOPs denote the estimated major forward arithmetic operations per input patch; M denotes millions of parameters, and runtime is reported in seconds.

Hyperspectral Image Classification

Hyperspectral image classification assigns land-cover, material, or biological labels using the spectral and spatial information in image pixels.

See all 58 papers

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift
May 18, 2026Haozhe Si, Yuxuan Wan, Yuqing Wang +2Representation Learning Masked Autoencoders
Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, IL, USA · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, IL, USA
VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification
Mar 1, 2026Abdellah Zakaria Sellam, Fadi Abdeladhim Zidi, Salah Eddine Bekhouche +4Hybrid Mamba-Transformer Architectures Hyperspectral Image Classification
Institute of Applied Sciences and Intelligent Systems (ISASI), CNR, 73100 Lecce, Italy · Dept. of Engineering for Innovation, University of Salento, 73100 Lecce, Italy · VSC Laboratory, Department of Electronics and Automation, University of Biskra, Biskra, Algeria +4
SSFT: A Lightweight Spectral-Spatial Fusion Transformer for Generic Hyperspectral Classification
Apr 17, 2026Alexander Musiat, Nikolas Ebert, Oliver WasenmüllerRepresentation Learning Hyperspectral Imaging
Mannheim University of Applied Sciences, Germany

May 18, 2026J/K move · Enter open · S save

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

Haozhe Si, Yuxuan Wan, Yuqing Wang, Minh N. Do+1

Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign, IL, USA · Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, IL, USA

Representation Learning Masked Autoencoders Hyperspectral Imaging Efficient ViTs

Open paper AskPDF arXiv

Modeling hyperspectral imagery (HSI) across different sensors presents a fundamental challenge due to variations in wavelength coverage, band sampling, and channel dimensionality. As a result, models trained under a fixed spectral configuration often fail to generalize to other sensors. Existing Vision Transformer (ViT) approaches either rely on implicit spectral modeling with fixed channel assumptions or adopt explicit spatial-spectral attention with prohibitive computational cost, leading to a fundamental trade-off between efficiency and expressiveness. In this work, we introduce Low-rank Efficient Spatial-Spectral ViT (LESSViT), a sensor-flexible architecture for cross-spectral generalization. LESSViT is built on LESS Attention, a structured low-rank factorization that models joint spatial-spectral interactions through separable spatial and spectral components, reducing the complexity of full spatial-spectral attention from

\mathcal{O}(N^2 C^2)

\mathcal{O}(rNC)

, where

N

is the number of spatial tokens,

C

is the number of spectral channels, and

r

is the rank of the low-rank approximation. We further incorporate channel-agnostic patch embedding and wavelength-aware positional encoding to support flexible spectral inputs. To enable efficient and robust pretraining, we introduce a hyperspectral masked autoencoder (HyperMAE) with decoupled spatial-spectral masking and hierarchical channel sampling. We evaluate LESSViT under a cross-spectral generalization setting that simulates cross-sensor variability. Experiments on the SpectralEarth benchmark demonstrate that LESSViT improves robustness under spectral shifts while remaining competitive in-distribution, and explicit and efficient spatial-spectral modeling is essential for scalable and generalizable hyperspectral representation learning. The code is available through the project page: https://uiuctml.github.io/LESSViT/

Abstract

Hyperspectral Image Classification

Explore similar work

LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift

VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification

SSFT: A Lightweight Spectral-Spatial Fusion Transformer for Generic Hyperspectral Classification