See the Future, Classify the Present: Stochastic Latent World Modeling for Hyperspectral Images

Muhammad Ahmad; Gemine Vivone; Ronny Hänsch; Ahmed Sohaib; Manuel Mazzara; Danfeng Hong

cs.CVOct 19, 2026

See the Future, Classify the Present: Stochastic Latent World Modeling for Hyperspectral Images

Authors: Muhammad Ahmad, Gemine Vivone, Ronny Hänsch, Ahmed Sohaib, Manuel Mazzara, Danfeng Hong

Organizations: Khanpur Institute of Technology, Khanpur 64100, Pakistan · National Research Council - Institute of Methodologies for Environmental Analysis (CNR-IMAA), 85050 Tito, Italy · Department of SAR Technology, German Aerospace Center (DLR), Germany · Institute of Software Development and Engineering, Innopolis University, Innopolis, 420500, Russia ·

Abstract

Hyperspectral image classification (HSIC) relies on static spatial-spectral representations learned from labeled samples, limiting its ability to model structured dependencies in the latent representation space. To address this limitation, this work proposes Stochastic Latent World Modeling (LWM), a trajectory-based representation-learning framework that enriches classification through stochastic latent-state prediction. Given a hyperspectral patch, LWM tokenizes and encodes spectral and spatial information separately using dedicated transformer branches, then fuses their outputs into a unified representation. This representation is further organized into an environment state that captures center, neighborhood, spectral, and global contextual information. A stochastic latent transition module then predicts multiple future latent states through Gaussian parameterization and reparameterization, with recurrent refinement used to model dependencies across successive states. The resulting latent trajectory is adaptively aggregated by a rollout planner to construct a planned representation, which is integrated with the base representation for final classification. By extending static spatial-spectral encoding with stochastic multi-step latent-state modeling, LWM enables the classifier to exploit structured dependencies in the representation space and provides richer features for HSIC. Across five benchmark datasets, LWM achieves overall accuracies ranging from 89.21% to 99.52%, demonstrating consistent classification performance under limited-label settings.

Figures & tables

Fig. 1 : Overview of LWM framework. (A) Input hyperspectral patches are preprocessed with spectral band and spatial pixel masking. (B) Each patch is decomposed into spectral tokens (per-band) and spatial tokens (per-pixel), which are encoded by dedicated transformer branches and fused through a joint transformer to produce a unified representation and a global scene embedding (CLS token) for base classification. (C) An environment state is constructed by aggregating center, neighborhood, spectral, and global features, and is propagated through a stochastic latent transition model to generate multi-step latent rollouts via Gaussian parameterization and recurrent refinement. A rollout planner adaptively aggregates latent states into a planned representation for trajectory-based classification, which is combined with the base prediction. (D) The model is trained using masked reconstruction, spectral and spatial latent dynamics, stochastic rollout regularization with KL constraint, trajectory supervision, and physically motivated regularization. (E) During inference, masking is disabled and prediction is obtained through a forward pass with latent rollout and trajectory aggregation.

Branch		OA	AA	$\kappa$	$\Delta$
DFC18	Base	0.9100	0.9198	0.8858	+0.0149
	Trajectory	0.9113	0.9273	0.8976	+0.0136
	Final	0.9249	0.9352	0.9021	–
Dioni	Base	0.8648	0.8835	0.8343	+0.0523
	Trajectory	0.9037	0.9208	0.8837	+0.0134
	Final	0.9171	0.9346	0.8979	–

TABLE I : Branch-wise performance of the Base, Trajectory, and Final prediction branches.

\Delta

denotes the OA improvement of the final fused prediction over the corresponding branch. Best results are in boldface.

Dataset	CR (%)	HR (%)	NG (%)	OA oracle (%)	AGR (%)	JSD
DFC18	1.76	1.25	0.51	93.67	96.31	0.0196
Dioni	6.41	1.51	4.89	92.89	90.86	0.0817
Loukia	6.37	1.96	4.41	91.08	90.74	0.0433
HH	0.25	0.17	0.08	99.28	99.54	0.0033
TD	0.11	0.18	-0.06	99.64	99.70	0.0017

TABLE II : Trajectory branch correction analysis. Higher CR, NG, and OA oracle indicate stronger complementary information provided by the latent rollout branch.

Dataset	Perturbation	Level	OA	AA	$\boldsymbol{\kappa}$
DFC18	Missing Bands	10%	0.9132	0.9341	0.8872
	Missing Bands	20%	0.8752	0.9130	0.8389
	Missing Bands	30%	0.7925	0.8514	0.7355
	Gaussian Noise	$\sigma=0.01$	0.9249	0.9352	0.9020
	Gaussian Noise	$\sigma=0.03$	0.9247	0.9351	0.9018
	Gaussian Noise	$\sigma=0.05$	0.9242	0.9344	0.9012

TABLE III : Robustness evaluation under spectral band removal and additive Gaussian noise.

Panel of Figure 5 — Fig. 2 : Attention-based interpretability analysis on the DFC18 dataset. (a) GT. (b) Spatial distribution of the learned attention responses. (c) Attention responses overlaid on GT, illustrating the regions that contribute most strongly to the classification decision. (d) Error map, where green pixels indicate correctly classified samples and red pixels indicate misclassified samples.

Panel of Figure 6 — Fig. 3 : Attention-based analysis on the Dioni dataset.

Panel of Figure 7 — Fig. 4 : Attention-based analysis on the Loukia dataset.

Panel of Figure 8 — Fig. 5 : Attention-based analysis on the HH dataset.

Panel of Figure 9 — Fig. 6 : Attention-based analysis on the TD dataset.

Dataset	Setting	Loss $\downarrow$	OA $\uparrow$	AA $\uparrow$	$\boldsymbol{\kappa}$ $\uparrow$	MCC $\uparrow$	Log Loss $\downarrow$	Top3 $\uparrow$	Macro-F1 $\uparrow$	Brier $\downarrow$	ECE $\downarrow$
DFC18	SST only	0.5794	0.8992	0.9083	0.8683	0.8689	0.3925	0.9830	0.8919	0.1447	0.0385
	SST+Mask Recon	0.6267	0.9076	0.9190	0.8796	0.8802	0.3390	0.9841	0.8994	0.1630	0.0493
	SST+World	0.5196	0.9175	0.9324	0.8925	0.8931	0.2832	0.9878	0.9030	0.1260	0.0280
	LWM w/o Physical	0.4601	0.9225	0.9325	0.8991	0.8996	0.2942	0.9877	0.9082	0.1234	0.0296
	Full LWM	0.4100	0.9249	0.9352	0.9021	0.9024	0.2783	0.9899	0.9226	0.1192	0.0283
	Relative Gain (%)	-29.23	+2.86	+2.96	+3.89	+3.86	-29.10	+0.70	+3.44	-17.62	-26.49

TABLE IV : Ablation study of the proposed LWM framework. Relative Gain (%) is computed with respect to the SST-only baseline. For OA, AA,

\kappa

, MCC, Top3, and Macro-F1, higher values are better, whereas for Loss, Log Loss, Brier Score, and ECE, lower values are preferred. Best values are highlighted in bold.

Dataset	Metric	Methods
Dataset	Metric	LWM	WF	DF	MCMTN	SMFAT	MSST	S2CIFT	S2CAT
Training Samples (%)		10	50	50	50	50	50	50	50
DFC18	OA	92.4997 $\pm$ 0.1096	91.4101 $\pm$ 0.5559	89.6109 $\pm$ 0.0707	90.4070 $\pm$ 0.1673	91.9506 $\pm$ 0.1864	87.0541 $\pm$ 0.3509	91.6866 $\pm$ 0.0159	91.5293 $\pm$ 0.2206
	AA	93.5275 $\pm$ 0.0126	88.9789 $\pm$ 0.1071	81.3057 $\pm$ 0.3993	88.0639 $\pm$ 0.1983	85.6948 $\pm$ 0.1386	78.4143 $\pm$ 0.7389	90.9763 $\pm$ 0.3494	88.9707 $\pm$ 0.9180
	$\kappa$	90.2191 $\pm$ 0.0231	89.9807 $\pm$ 0.7378	86.2516 $\pm$ 0.1730	87.2409 $\pm$ 0.1456	89.0046 $\pm$ 0.1064	82.7725 $\pm$ 0.4699	89.6388 $\pm$ 0.0193	89.1599 $\pm$ 0.3376
Loukia	OA	89.2107 $\pm$ 0.1094	88.3205 $\pm$ 0.2095	88.0258 $\pm$ 0.9530	80.4684 $\pm$ 0.2983	82.7826 $\pm$ 0.1956	76.2885 $\pm$ 1.2358	87.3655 $\pm$ 0.1885	87.2983 $\pm$ 0.3037

TABLE V : Comparison with recent state-of-the-art HSIC methods. Results are reported as mean

\pm

standard deviation over five runs. The best and second-best results are highlighted in bold and underlined italics, respectively. The training sample ratio used by each method is reported in the row “Training Samples (%)”.

Panel of Figure 12 — Fig. 7 : Qualitative comparison on the DFC18 dataset. The proposed LWM framework exhibits improved spatial consistency and reduced boundary confusion compared with existing transformer-based methods. Error maps reveal fewer fragmented predictions and more coherent class structures, indicating that latent trajectory modeling enhances representation robustness under spectrally similar categories.

Panel of Figure 13 — Fig. 8 : Qualitative comparison on the Dioni dataset.

Panel of Figure 14 — Fig. 9 : Qualitative comparison on the Loukia dataset.

Panel of Figure 15 — Fig. 10 : Qualitative comparison on the HH dataset.

Panel of Figure 16 — Fig. 11 : Qualitative comparison on the TD dataset.

Hyperspectral Image Classification

Hyperspectral image classification assigns land-cover, material, or biological labels using the spectral and spatial information in image pixels.

See all 53 papers

VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification
Mar 1, 2026Abdellah Zakaria Sellam, Fadi Abdeladhim Zidi, Salah Eddine Bekhouche +4Hybrid Mamba-Transformer Architectures Hyperspectral Image Classification
Institute of Applied Sciences and Intelligent Systems (ISASI), CNR, 73100 Lecce, Italy · Dept. of Engineering for Innovation, University of Salento, 73100 Lecce, Italy · VSC Laboratory, Department of Electronics and Automation, University of Biskra, Biskra, Algeria +4
Token Clustering and Semantic Sequence Mamba for Hyperspectral Image Classification
Sep 23, 2026Yimin Zhu, Mahmood Elahi, Lincoln Linlin XuVisual SSMs Hyperspectral Image Classification
Department of Geomatics Engineering, University of Calgary, Canada · Department of Electrical and Software Engineering, University of Calgary, Canada
LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification
Mar 25, 2026Jiawen Wen, Suixuan Qiu, Zihang Luo +2Hyperspectral Image Classification Hybrid CNN-Transformer Architectures
Information Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · Faculty of Geographical Science, Beijing Normal University, Beijing, 100875, China · School of Electronic and Communication Engineering, Guangzhou University, Guangzhou University, Guangzhou, Guangzhou, 510006, China

Mar 1, 2026J/K move · Enter open · S save

VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification

Abdellah Zakaria Sellam, Fadi Abdeladhim Zidi, Salah Eddine Bekhouche, Ihssen Houhou+3

Institute of Applied Sciences and Intelligent Systems (ISASI), CNR, 73100 Lecce, Italy · Dept. of Engineering for Innovation, University of Salento, 73100 Lecce, Italy · VSC Laboratory, Department of Electronics and Automation, University of Biskra, Biskra, Algeria+4

Hybrid Mamba-Transformer Architectures Hyperspectral Image Classification Hyperspectral Imaging

Open paper AskPDF arXiv

Accurate classification of hyperspectral imagery (HSI) is often frustrated by the tension between high-dimensional spectral data and the extreme scarcity of labeled training samples. While hierarchical models like LoLA-SpecViT have demonstrated the power of local windowed attention and parameter-efficient fine-tuning, the quadratic complexity of standard Transformers remains a barrier to scaling. We introduce VP-Hype, a framework that rethinks HSI classification by unifying the linear-time efficiency of State-Space Models (SSMs) with the relational modeling of Transformers in a novel hybrid architecture. Building on a robust 3D-CNN spectral front-end, VP-Hype replaces conventional attention blocks with a Hybrid Mamba-Transformer backbone to capture long-range dependencies with significantly reduced computational overhead. Furthermore, we address the label-scarcity problem by integrating dual-modal Visual and Textual Prompts that provide context-aware guidance for the feature extraction process. Our experimental evaluation demonstrates that VP-Hype establishes a new state of the art in low-data regimes. Specifically, with a training sample distribution of only 2%, the model achieves Overall Accuracy (OA) of 99.69% on the Salinas dataset and 99.45% on the Longkou dataset. These results suggest that the convergence of hybrid sequence modeling and multi-modal prompting provides a robust path forward for high-performance, sample-efficient remote sensing.

Abstract

Hyperspectral Image Classification

Explore similar work

VP-Hype: A Hybrid Mamba-Transformer Framework with Visual-Textual Prompting for Hyperspectral Image Classification

Token Clustering and Semantic Sequence Mamba for Hyperspectral Image Classification

LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification