Beyond RGB: Benchmarking and Enhancing MLLMs for Hyperspectral Image Understanding via Training-Free Reasoning Framework

Xinyu Zhang; Zurong Mai; Qingmei Li; Xiaoya Fan; Zjin Liao; Haoyuan Liang; Yibin Wen; Yuhang Chen

cs.CVApr 10, 2026

Beyond RGB: Benchmarking and Enhancing MLLMs for Hyperspectral Image Understanding via Training-Free Reasoning Framework

Authors: Xinyu Zhang, Zurong Mai, Qingmei Li, Xiaoya Fan, Zjin Liao, Haoyuan Liang, Yibin Wen, Yuhang Chen, +8 more

Organizations: Sun Yat-sen University · Tsinghua Shenzhen International Graduate School · Southwest University · National Supercomputing Center in Shenzhen · China Agricultural University · Southwest Jiaotong University

Abstract

Multimodal Large Language Models (MLLMs) have achieved strong performance on RGB image understanding, yet their ability to use spectral evidence beyond the visible range remains largely unexplored. Hyperspectral imagery (HSI) provides dense spectral measurements that reveal material and environmental cues unavailable in RGB, but current MLLMs cannot directly ingest high-dimensional HSI. To study this gap, we introduce HM-Bench, an evidence-grounded benchmark for hyperspectral image understanding with MLLMs. HM-Bench contains 19,337 question--answer pairs from 2,178 hyperspectral samples across 13 task categories, covering general perception, spectral reasoning, and spatial--spectral reasoning. To make HSI accessible to the native image--text interfaces of existing MLLMs, we further propose VSR$^{2}, a training-free Visual--Spectral--Report Reasoning framework. VSR^{2} represents each HSI sample with three aligned views: an RGB image for visual semantics and spatial context, a PCA-based image for dominant spectral variation, and a structured report for quantitative spectral--spatial evidence. Under a controlled RGB-only versus VSR^{2} evaluation protocol, experiments on 17 representative MLLMs show that HSI information improves average accuracy from 38.26% to 40.35%, with gains varying substantially across models and task categories. These results indicate that HSI information (beyond RGB) is useful for current MLLMs, while robust hyperspectral reasoning remains an open challenge.

Figures & tables

Figure 1: Limitations of current MLLMs and benchmarks in hyperspectral understanding. We tackle these challenges by proposing HM-Bench and exploring VSR 2 .

Benchmark	Input Modality	QA	Tasks	BR-Eval	TF-Enh
MMBench ( Liu et al. 2024b )	RGB	$\sim$ 3k	20	✗	✗
VRSBench ( Li, Ding, and Elhoseiny 2024 )	RGB	12.3k	10	✗	✗
AgroBench ( Shinoda et al. 2025 )	RGB	4.3k	7	✗	✗
AgroMind ( Li et al. 2025 )	RGB	28k	13	✗	✗
XLRS-Bench ( Wang et al. 2025c )	RGB	32k	16	✗	✗
RSHR-Bench ( Dang et al. 2025 )	RGB	8.2k	13	✗	✗

Table 1: Comparison between existing benchmarks and HM-Bench. “BR-Eval” indicates support for beyond-RGB evaluation, and “TF-Enh” indicates support for training-free input enhancement for MLLMs.

Figure 2: Statistical overview of HM-Bench. (a) Question-type distribution across six task scenarios. (b) Sample and QA-pair counts across six capability dimensions. (UAV: Unmanned Aerial Vehicle.)

Figure 3: HM-Bench construction and VSR 2 framework overview. (a) HM-Bench construction pipeline. Built from 23 public hyperspectral datasets through preprocessing, adaptive cropping, QA generation, and expert verification, yielding 19,337 evidence-grounded QA pairs. (b) Overview of VSR 2 . Each hyperspectral cube is converted into three aligned views, enabling controlled zero-shot evaluation of single-view and multi-view input settings.

Figure 4: Hierarchical task taxonomy of HM-Bench, showing 13 hyperspectral tasks under perception and reasoning, together with data statistics and representative VQA examples.

Model	Report	PCA	RGB	w/o PCA	w/o Report	VSR 2
Random	25.13	25.13	25.13	25.13	25.13	25.13
Closed-source Models
Claude-Sonnet-4-6 ( Anthropic, P.B.C. 2026 )	36.57	39.48	40.95	39.65	43.11	44.01
GPT-5.4-Mini ( OpenAI 2026 )	37.45	42.35	39.88	40.58	43.70	44.28
Grok-4 ( de Carvalho Souza and Weigang 2025 )	31.68	32.78	34.67	38.70	43.74	39.50
Open-source Models

Table 2: Overall performance under different VSR 2 input settings on HM-Bench. Average accuracy (%) over all tasks under a unified zero-shot protocol. Bold indicates the best result per column (input setting); underlined indicates the second-best per column. The Random row is excluded from ranking.

Figure 5: Main results, task-wise analysis, and ablations of VSR 2 on HM-Bench. The subfigures present task-wise single-view performance, model-level sensitivity to the number of PCA components, dataset-level PCA explained variance, and the effect of the fusion prompt.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Hierarchical task taxonomy of HM-Bench.

Dataset	Sensor	Resolution	Bands	Wavelength	QAs	Patch Size	Scene Type
IndianPines ( Baumgardner, Biehl, and Landgrebe 2015 )	AVIRIS	20m	200	0.4 – 2.5 $\mu$ m	450	40 $\times$ 40	Agriculture/Nature
Salinas ( Hou et al. 2021a )	AVIRIS	3.7m	204	0.4 – 2.5 $\mu$ m	1,080	64 $\times$ 64	Agriculture
Xiongan ( Yi et al. 2020 )	VNIR Imaging Spectrometer (SITP)	0.5m	250	0.4 – 1 $\mu$ m	3,003	256 $\times$ 256	Agriculture
Houston 2013 ( Debes et al. 2014 )	CASI-1500	2.5m	144	0.364 – 1.046 $\mu$ m	501	100 $\times$ 100	Urban/Nature
Houston 2018 ( Xu et al. 2019 )	CASI-1500	2.5m	144	0.374 – 1.047 $\mu$ m	501	100 $\times$ 100	Urban/Nature
WashingtonDC ( Purdue Research Foundation 1996 )	HYDICE	—	191	0.4 – 2.4 $\mu$ m	612	48 $\times$ 48	Urban/Nature

Appendix

Table 3: Detailed statistics of the HSI datasets included in HM-Bench.

Category	Definition	Rejected QA candidate	Reason for rejection
Hallucination	The MLLM generates surface-plausible content that cannot be supported by the available hyperspectral evidence.	Task: Vegetation Health/Stress Diagnosis Question: Based on the spectral characteristics of the vegetation in this block, what is the estimated chlorophyll content? Options: A. Low (SPAD $<$ 20) B. Moderate (SPAD 30–40) C. High (SPAD $>$ 45) D. Extremely high (SPAD $>$ 60) Answer: C	The question requests a quantitative chlorophyll estimate, but the available spectral-spatial evidence does not provide sufficient calibration information for biochemical retrieval. The MLLM incorrectly associates vegetation appearance with a specific physiological parameter and produces a seemingly plausible but unsupported quantitative claim.
Triviality	The QA pair is correctly formatted but provides limited evaluation value because the answer can be obtained from overly obvious visual cues without requiring meaningful spectral-spatial analysis.	Task: Presence Detection Question: Does vegetation exist in this image patch? Options: A. Yes B. No Answer: A	The image patch is almost entirely occupied by visually dominant vegetation, making the answer directly observable from RGB appearance or simple visual inspection. Although the QA pair is valid, it does not require hyperspectral evidence or non-visible spectral information, resulting in limited capability discrimination.
Format Issue	The generated content may be semantically meaningful but violates the predefined QA schema or evaluation protocol.	Task: Spectral Unmixing Question: Based on the spectral characteristics of pixel, which endmember combination is most likely? Options: A. Concrete and Asphalt … D. Healthy vegetation and Concrete Answer: The pixel contains mixed vegetation and impervious surface signals due to transitional spectral responses.	The benchmark requires the answer to be represented only by the option index (e.g., “B”). The generated response contains additional free-form explanation, which violates the predefined answer format and prevents automatic evaluation.

Appendix

Table 4: Representative rejection cases during MLLM-based QA generation.

Figure 7: Example of the pseudo-RGB view generated from a hyperspectral sample.

Figure 8: Examples of PCA-based spectral views constructed using the top 6, 12, and 24 principal components from the same hyperspectral sample.

Dataset	#Bands	EV@1	EV@3	EV@6	EV@12	EV@24	#PCs for 95% EV
BayArea_2013 ( Cooper et al. 2020 )	224	82.52	99.81	99.96	99.99	99.99	2
BayArea_2015 ( Cooper et al. 2020 )	224	54.10	99.44	99.81	99.93	99.96	2
Botswana ( Hou et al. 2021a )	145	94.62	99.40	99.67	99.78	99.87	2
HanChuan ( Zhong et al. 2020 )	274	92.21	99.45	99.67	99.74	99.81	2
Hermiston_2004 ( Hou et al. 2021b )	242	76.77	93.59	96.67	97.99	98.97	4
Hermiston_2007 ( Hou et al. 2021b )	242	73.11	90.17	95.29	97.19	98.56	6

Appendix

Table 5: PCA explained variance summary across datasets in HM-Bench. We report the cumulative explained variance (EV, %) captured by the top 1, 3, 6, 12, and 24 principal components, together with the number of principal components required to reach 95% explained variance.

Model	Input Mod.	# PCs
		6	12	24
Qwen3-VL-4B	PCA-only	39.57	40.96	40.34
	w/o Report	39.23	42.97	42.63
	VSR²	40.47	43.23	41.98
InternVL3.5	PCA-only	37.08	43.08	38.10
	w/o Report	42.91	43.80	41.72

Appendix

Table 6: Ablation on the number of principal components in the PCA view. Using 12 components yields the best or most stable performance across models and input configurations.

Figure 9: An example of structured report input generated from the same data block as the PCA composite image above.

Model	Report	PCA	RGB	w/o PCA	w/o Report	VSR 2
Random	25.13	25.13	25.13	25.13	25.13	25.13
Closed-source Models
Claude-Sonnet-4-6 ( Anthropic, P.B.C. 2026 )	36.57	39.48	40.95	39.65	43.11	44.01
GPT-5.4-Mini ( OpenAI 2026 )	37.45	42.35	39.88	40.58	43.70	44.28
Grok-4 ( de Carvalho Souza and Weigang 2025 )	31.68	32.78	34.67	38.70	43.74	39.50
Open-source Models

Appendix

Table 7: Overall performance under different VSR 2 input settings on HM-Bench. Average accuracy (%) over all tasks under a unified zero-shot protocol. Bold indicates the best result per column and underlined indicates the second-best. The Random row is excluded from ranking. The comparison illustrates the relative contribution of RGB, PCA, and Report views, as well as the effect of partial and full multi-view fusion.

Model	Input	FR		TQ	SL			CI		SE		CD
		SFR	LCC	PD	Count	OLR	RD	SAD	SU	VHS	EPS	BCI	CAL	CSA
Claude-Sonnet-4.6	Report-only	41.54	23.39	39.80	34.57	30.38	47.30	39.58	39.62	48.58	45.37	30.36	21.60	33.33
	PCA-only	58.20	42.17	50.07	26.73	29.02	43.77	32.05	42.65	49.26	56.48	38.08	14.80	30.00
	RGB-only	48.67	29.59	56.26	24.70	31.64	42.43	32.36	41.83	46.32	53.01	36.35	12.41	30.05
GPT-5.4-Mini	Report-only	39.60	19.77	32.96	39.05	27.24	43.77	60.27	37.32	47.85	44.52	31.96	20.41	29.17
	PCA-only	50.60	35.48	46.06	32.05	28.55	42.00	64.18	47.30	41.79	51.31	35.55	12.41	35.00

Appendix

Table 8: Task-wise results under three single-view settings. Accuracy (%) of all evaluated models on 13 HM-Bench tasks using only one input view at a time: Report-only, PCA-only, or RGB-only. This table highlights modality-specific strengths and reveals which task types benefit more from visual, spectral-contrast, or textual evidence.

Figure 10: A question case of the Spectral Feature Recognition

Figure 11: A question case of the Land Cover Classification

Figure 12: A question case of the Presence Detection

Figure 13: A question case of the Counting

Figure 14: A question case of the Object Location Relationship

Figure 15: A question case of the Spectral Unmixing

Figure 16: A question case of the Vegetation Health/Stress Diagnosis

Figure 17: A question case of the Region Delineation

Figure 18: A question case of the Environmental Pollution Severity Assessment

Figure 19: A question case of the Spectral Anomaly Detection

Figure 20: A question case of the Basic Change Identification

Figure 21: A question case of the Change Area Localization

Figure 22: A question case of the Change Statistical Analysis

Explore similar work

Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning
Apr 22, 2026Dahun Kim, Ganesh Satish Mallya, Anelia AngelovaRemote Sensing Image Understanding Multimodal Large Language Models
Google DeepMind
A PyTorch Library for Hyperspectral Image Models: Technical Report
Sep 30, 2026Tanishq Rachamalla, Aryan Das, Srishti Kaushik +1Remote Sensing Benchmarks Hyperspectral Image Classification
Department of Information Technology Siddhartha Academy of Higher Education Vijayawada, Andhra Pradesh 521108, India · Department of Computer Science and Engineering Vellore Institute of Technology Bhopal, Madhya Pradesh 466114, India · Department of Computer and Information Sciences Indira Gandhi National Open University New Delhi 110068, India +1
SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images
Apr 30, 2026Jialu Shen, Han Lyu, Suyang Zhong +5Visual Question Answering Multimodal Model Evaluation
DP Technology

Apr 22, 2026J/K move · Enter open · S save

Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

Dahun Kim, Ganesh Satish Mallya, Anelia Angelova

Google DeepMind

Remote Sensing Image Understanding Multimodal Large Language Models Multimodal Reasoning

Open paper AskPDF arXiv

Multi-spectral imagery is a valuable input signal for Remote Sensing applications, such as land-use and land-cover classification and environmental monitoring. However, generalist Large Multi-modal Models (LMMs) are typically trained on RGB images, limiting their applicability to the RGB domain. At the same time, training multi-spectral multi-modal models is expensive and produces uniquely specialized models. To address this, we propose a novel training-free approach that introduces multi-spectral data within the inference pipeline of standard RGB-only LMMs, allowing large gains in performance. Our approach leverages the LMMs' understanding of the visual space by adapting non-RGB inputs to that space and injecting domain-specific information and Chain-of-Thought reasoning as instructions. We demonstrate this with the Gemini 2.5 model and observe strong Zero-Shot performance gains on popular Remote Sensing benchmarks. These results highlight the potential for geospatial professionals to leverage powerful generalist models for specialized sensor inputs, benefiting from rich reasoning capabilities grounded in specialized data.

Abstract

Explore similar work

Unlocking Multi-Spectral Data for Multi-Modal Models with Guided Inputs and Chain-of-Thought Reasoning

A PyTorch Library for Hyperspectral Image Models: Technical Report

SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images