ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery

Ayush Shrivastava; Kirtan Gangani; Laksh Jain; Mayank Goel; Nipun Batra

cs.CVFeb 16, 2026

ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery

Authors: Ayush Shrivastava, Kirtan Gangani, Laksh Jain, Mayank Goel, Nipun Batra

Organizations: Indian Institute of Technology, Gandhinagar, India · Carnegie Mellon University, Pittsburgh, USA

Abstract

Vision language models (VLMs) achieve strong performance on RGB imagery, but they do not generalize to thermal images. Thermal sensing plays a critical role in settings where visible light fails, including nighttime surveillance, search and rescue, autonomous driving, and medical screening. Unlike RGB imagery, thermal images encode physical temperature rather than color or texture, requiring perceptual and reasoning capabilities that existing RGB-centric benchmarks do not evaluate. We introduce ThermEval-B, a structured benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal vision language understanding. ThermEval-B integrates public datasets with our newly collected ThermEval-D, the first dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments. Evaluating 25 open-source and closed-source VLMs, we find that models consistently fail at temperature-grounded reasoning, degrade under colormap transformations, and default to language priors or fixed responses, with only marginal gains from prompting or supervised fine-tuning. These results demonstrate that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions, positioning ThermEval as a benchmark to drive progress in thermal vision language modeling.

Figures & tables

Figure 1. Thermal imagery enables critical perception tasks in settings where RGB fails, but VLMs trained predominantly on RGB exhibit systematic errors when applied to thermal images, driven by modality mismatch and language priors.

Dataset	Temp	BBox	Seg	Rel	Subj	Objective
Charlotte ( Ashrafi et al., 2022 )	✓	✗	✗	✗	10	Facial thermography
M3FD ( Liu et al., 2022 )	✗	✓	✗	✗	many	Multi-modal object detection
LCAS Physio. ( Cosar et al., 2018 )	✓	✗	✗	✗	5	Physiological monitoring
LCAS RGB-D-T ( Cosar and Bellotto, 2019 )	✓	✓	✗	✗	15	Human re-identification
FLIR ADAS ( FLIR, 2024 )	✗	✓	✗	✗	many	ADAS object detection
LLVIP ( Jia et al., 2021 )	✗	✓	✗	✗	many	Low-light pedestrian detection

Table 1. Comparison of thermal datasets, summarizing temperature data (Temp), bounding boxes (BBoxes), segmentation masks (Seg), annotator reliability (Rel), subject counts (Subj), and primary research objectives.

Figure 2. ThermEval defines seven evaluation tasks covering modality identification (T1–T2), human counting (T3), colorbar interpretation (T4), thermal reasoning (T5), and temperature estimation (T6–T7), designed to probe complementary aspects of thermal vision language understanding.

Task	Subtask	InternVL (Simple)	InternVL (Contextual)	MiniCPM (Simple)	MiniCPM (Contextual)	Qwen2.5-VL (Simple)	Qwen2.5-VL (Contextual)	BLIP-2 (Simple)	BLIP-2 (Contextual)
T1	FLIR (ACC $\uparrow$ )	0.96	0.97 ( +0.01 )	0.95	0.96 ( +0.01 )	0.71	0.96 ( +0.25 )	0.46	0.34 ( -0.12 )
	LLVIP (ACC $\uparrow$ )	1.00	1.00 (+0.00)	0.98	0.99 ( +0.01 )	0.72	0.96 ( +0.24 )	0.22	0.43 ( +0.21 )
T2	FLIR (ACC $\uparrow$ )	0.86	0.99 ( +0.13 )	0.91	0.89 ( -0.02 )	0.61	0.98 ( +0.37 )	0.77	0.67 ( -0.10 )
	LLVIP (ACC $\uparrow$ )	0.97	1.00 ( +0.03 )	0.93	0.95 ( +0.02 )	0.80	0.99 ( +0.19 )	0.77	0.77 (+0.00)
T3	FLIR (MAE $\downarrow$ )	2.70	2.53 ( -0.17 )	3.70	2.90 ( -0.80 )	3.78	3.20 ( -0.58 )	4.69	4.65 ( -0.04 )
	LLVIP (AME $\downarrow$ )	0.73	0.60 ( -0.13 )	1.09	0.80 ( -0.29 )	1.09	0.88 ( -0.21 )	2.99	2.94 ( -0.05 )

Table 4. Performance of models before and after contextual prompt ablations.

\downarrow

indicates lower and

\uparrow

indicates higher is better. Colored deltas indicate improvement ( green ) or degradation ( red ) relative to the simple prompt.

Task	Subtask	Best Model (Zero-shot)	Human	Qwen 2.5 Zero-shot	Qwen 2.5 Finetuned	Qwen $\boldsymbol{\Delta}$
T1	FLIR (Acc $\uparrow$ )	1.00	0.97	0.98	1.00	+0.02
	LLVIP (Acc $\uparrow$ )	1.00	0.98	1.00	1.00	0.00
T2	FLIR (Acc $\uparrow$ )	1.00	0.98	0.99	1.00	+0.01
	LLVIP (Acc $\uparrow$ )	1.00	0.98	1.00	1.00	0.00
T3	FLIR (MAE $\downarrow$ )	2.72	1.73	3.55	1.85	$-1.70$
	LLVIP (MAE $\downarrow$ )	0.51	0.30	0.89	0.55	$-0.34$

Table 5. Comparison of finetuned Qwen-VL 2.5 with human performance and other models.

\downarrow

indicates lower is better and

\uparrow

indicates higher is better. “–” indicates not applicable. Green in Qwen

\boldsymbol{\Delta}

column indicates improvement.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3. Images from ThermEval-D dataset. The top row shows the images having a single person in the scene whereas the second row shows the images having more than one person in the scene. Colorbars were added programatically during task evaluation

Metric	Annotator Pairs			Mean of Pairs
Metric	1 & 2	1 & 3	2 & 3	Mean of Pairs
Bounding Box IoU	0.77	0.74	0.77	0.76
Segmentation IoU	0.72	0.71	0.73	0.72
Bounding Box Dice	0.87	0.85	0.87	0.86
Segmentation Dice	0.84	0.83	0.84	0.84

Appendix

Table 6. Inter-annotator agreement using IoU and Dice

Figure 4. Demonstrates some of the images from the FLIR-ADAS dataset, which is used for Tasks T-1, T-2, and T-3. Top row shows thermal images while the bottom shows RGB for different scenes. More information regarding the tasks could be obtained from section 3.1 .

Figure 5. Demonstrates some of the images from the LLVIP dataset, which is used for Tasks T-1, T-2, and T-3.Top row shows thermal images while the bottom shows RGB for different scenes. More information regarding the tasks could be obtained from section 3.1 .

ID	Task Name	Subtask / Source	# Samples
T1	Modality ID	FLIR, LLVIP	10,000
T2	Modality ID (Colormap)	FLIR, LLVIP	10,000
T3	Counting	FLIR, LLVIP	20,000
T4	Colorbar Inference	Detection	2,098
		Position	4,196
		Maximum	1,049

Appendix

Table 7. VQA sample counts and data sources per task in ThermEval.

Model	Params (B)	Gray		Magma		Spring		Summer		Viridis
Model	Params (B)	FLIR	LLVIP	FLIR	LLVIP	FLIR	LLVIP	FLIR	LLVIP	FLIR	LLVIP
SMOL-256M	0.3	0.002	0.003	0.634	0.553	0.268	0.575	0.250	0.241	0.519	0.456
Jina-VLM	2.0	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
PaliGemma-2	3.0	0.995	0.997	1.000	1.000	0.992	1.000	0.915	1.000	1.000	1.000
Phi-3	4.0	0.262	0.669	0.978	0.910	0.865	0.931	0.907	0.863	0.936	0.937
Phi-3.5	4.0	0.751	0.899	1.000	0.992	1.000	1.000	0.995	1.000	1.000	0.998

Appendix

Table 8. Accuracy of VLMS on Task-2: Modality Identification under colormap transformation with results shown separately for FLIR and LLVIP datasets. Higher numbers are better.

Model	Params (B)	FLIR				LLVIP
Model	Params (B)	MAE ↓	STD ↓	Bias *	RMSE ↓	MAE ↓	STD ↓	Bias *	RMSE ↓
SMOL-256M	0.3	4.31	5.56	-4.30	7.03	1.77	1.73	-1.63	2.38
Jina-VLM	2.0	3.82	5.22	-3.80	6.45	0.62	0.90	-0.45	1.01
PaliGemma-2	3.0	4.03	5.37	-4.01	6.70	1.02	1.30	-0.90	1.58
Phi-3	4.0	3.59	4.96	-3.52	6.08	1.22	1.23	-1.17	1.70
Phi-3.5	4.0	4.42	100.08	-2.14	100.10	1.07	1.16	-1.01	1.53

Appendix

Table 9. Regression metrics for Task-3 : Human Counting using FLIR and LLVIP datasets. ↓ indicates lower is better. *Bias closer to 0 is better.

Figure 6. Demonstrates various colormaps used for Task T-2. Colormaps used were ‘gray’, ‘magma’, ‘viridis’, ‘spring’ and ‘summer’

Model	Params (B)	Detection	Position	Extraction
		Accuracy	Accuracy	Acc Max	Acc Min	Acc	MAE Max	MAE Min	MAE
SMOL-256M	0.3	0.414	0.000	0.990	0.824	0.907	0.22	3.0	1.61
Jina-VLM	2.0	1.00	0.994	1.000	1.000	1.000	0.0	0.0	0.0
PaliGemma-2	3.0	0.961	0.785	1.000	0.955	0.978	0.0	0.12	0.06
Phi-3	4.0	1.00	0.958	1.000	0.995	0.998	0.0	0.0	0.0
Phi-3.5	4.0	1.00	1.00	0.999	1.000	1.000	0.0	0.0	0.0

Appendix

Table 10. Model evaluation for Task-4: colorbar interpretation task, assessing the ability to detect, position, and extract temperature values. Acc Max and Acc Min denotes the accuracy of correctly identifying maximum and minimum values of the colorbar. MAE Max and MAE Min denotes the MAE is estimating Max and Min temperature of the colorbar.

Model	Params (B)	Arrow				Coordinates				Region
		MAE $\downarrow$	RMSE $\downarrow$	BIAS *	STD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	BIAS *	STD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	BIAS *	STD $\downarrow$
ChartGemma	3.0	3.84	5.05	1.19	4.91	13.74	28.06	8.43	26.77	2.33	3.44	-0.61	3.39
TinyCharts	3.0	5.66	6.64	-2.92	5.96	6.68	16.62	1.32	16.57	5.03	5.92	-0.27	5.92
ChartInstruct	7.0	3.96	5.94	-1.47	5.75	9.37	33.94	1.09	33.93	4.38	6.30	-0.81	6.25
SMOL-256M	0.3	5.74	6.87	5.35	4.31	11.26	33.12	5.45	32.67	3.37	4.55	1.95	4.11
Jina-VLM	2.0	3.26	4.34	0.37	4.32	4.22	14.95	-1.12	14.90	1.95	2.83	-0.16	2.83

Appendix

Table 11. Regression metrics for Task-6: Temperature Estimation on ThermEval Dataset. ↓ indicates lower is better. *Bias closer to 0 is better.

Model	Params (B)	2ft				6ft				10ft
		MAE $\downarrow$	RMSE $\downarrow$	BIAS *	STD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	BIAS *	STD $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	BIAS *	STD $\downarrow$
ChartGemma	3.0	1.20	1.49	-0.66	1.33	1.02	1.28	-0.30	1.25	1.08	1.36	0.06	1.36
TinyCharts	3.0	3.51	4.12	1.12	3.96	3.40	3.79	2.04	3.19	3.53	3.79	2.70	2.66
ChartInstruct	7.0	2.96	3.18	-1.45	2.83	2.76	2.98	-1.62	2.50	2.37	2.66	-1.75	2.01
SMOL-256M	0.3	2.40	2.85	0.72	2.75	2.67	3.02	0.35	3.00	2.45	2.82	0.26	2.81
Jina-VLM	0.3	1.24	1.57	-0.97	1.23	1.05	1.31	-0.90	0.96	0.81	1.01	-0.56	0.83

Appendix

Table 12. Regression metrics for Task-7: Temperature Estimation at varying depth on ThermEval dataset. ↓ indicates lower is better. *Bias closer to 0 is better.

Figure 7. Illustration of how a thermal image gets augmented for co-ordinate and arrow/marker based subtask

Figure 8. MAE of counterfactual temperature estimation vs. perturbation across regions. Red and green curves represent the model before and after fine-tuning, respectively. "Overall" denotes the aggregate across forehead, chest, and nose.

Error	OVERALL		FOREHEAD		CHEST		NOSE
Error	No FT	After FT	No FT	After FT	No FT	After FT	No FT	After FT
1	2.38	1.28	2.41	1.27	2.54	1.05	2.15	1.55
2	2.19	1.68	2.11	1.58	2.50	1.38	1.93	2.14
3	1.86	2.28	1.65	2.16	2.13	1.87	1.79	2.92
-1	3.61	1.42	3.95	1.40	3.24	1.56	3.61	1.27
-2	4.36	2.15	4.98	2.17	3.70	2.30	4.39	1.94

Appendix

Table 13. Comparison of MAE Results: Without Fine-Tuning vs. After Fine-Tuning

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception
May 21, 2026Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape +1Thermal Imaging Visual Question Answering
Department of Electrical & Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA
T-CLIP: Enabling Thermal Perception for Contrastive Language-Image Pretraining
May 30, 2026Tayeba Qazi, Ayush Maheshwari, Prerana Mukherjee +1Thermal Imaging Image Captioning
Indian Institute of Technology Delhi, India · NVIDIA AI Technology Center, India · Jawaharlal Nehru University, India
Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI
Oct 21, 2026Han Hu, He ChenHuman Activity Recognition VLMs for Robotics
Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China.

May 21, 2026J/K move · Enter open · S save

Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception

Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape, Vishal M. Patel

Department of Electrical & Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA

Thermal Imaging Visual Question Answering Vision-Language Models VLM Adaptation Vision-Language Grounding

Open paper AskPDF arXiv

Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: https://thusharakart.github.io/Thermo-VL

Abstract

VLM Evaluation

Explore similar work

Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception

T-CLIP: Enabling Thermal Perception for Contrastive Language-Image Pretraining

Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI