cs.CVOct 21, 2026

Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI

Authors: Han Hu, He Chen

Organizations: Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Thermal cameras enable robots to perceive human activities in darkness while revealing less personally identifiable appearance information and fewer details of the surrounding environment than RGB cameras. As vision-language models (VLMs), originally developed for RGB imagery, are increasingly integrated into embodied systems for scene understanding and high-level planning, thermal sensing offers a promising way to extend activity perception to low-light and privacy-sensitive settings. However, the failure modes of VLMs in thermal human activity understanding for human-robot interaction (HRI) remain underexplored. We fill this gap by evaluating seven locally deployable VLMs, including models developed for physical AI and robotics, on two person-centric thermal datasets. Our diagnostic tasks examine fine-grained action recognition, activity timeline reconstruction, and the ability to withhold unsupported answers. Paired RGB recordings serve as a performance reference, while controlled variations in frame sampling, thermal input presentation, and reasoning are used to assess whether the observed errors can be mitigated. Our results show that models can recognize broad activity categories but struggle with fine-grained actions such as coughing, sneezing, and yawning. They also produce inaccurate event timelines and hallucinate actions involving objects. Simple thermal input preprocessing and additional reasoning offer only limited, model-dependent gains. Abstention behavior also varies across models even under explicit instructions, limiting their reliability for practical robotic use.

Figures & tables

Fig. 1: Task examples: (a) fine-grained action recognition and (b) action–object interaction temporal grounding.
Fig. 2: Action recognition on 865 thermal clips. (a) Exact and category-wise accuracy. (b) Per-category recall (clip counts in parentheses). (c) Pooled category confusion, row-normalized.
SettingQwen3.5-4BQwen3.6-27B
Exact-action accuracy (%), 865 clips
2 FPS37.846.9
5 FPS42.952.3
10 FPS43.752.7
Accuracy change (pp), 293 clips
JET−2.0-2.0+0.3+0.3
TABLE I: Frame sampling and input transformations.
Fig. 3: Interaction recognition and temporal grounding on 118 thermal recordings. (a) Joint pair and event F1. (b) Event F1 for the joint task and temporal grounding with given interactions.
Fig. 4: The sit–stool event in Fig. 1 b: Qwen3.6 recovers it, Cosmos3 predicts a fragment, and RoboBrain2.5 omits it.
Fig. 5: RGB–thermal comparison in the joint task. (a) Pair F1. (b) Thermal (open) to RGB (filled) operating points: pair-matched event counts and mean tIoU over matched events. Scores use elapsed seconds.
Fig. 6: Temporal-evidence interventions for tested models. (a) Target selection with target events vs. unrelated activities. (b) Target selection and answer changes after reversing or block-shuffling events.
ModelCoverageAnswered acc.Correct backoff
Qwen3.5-4B50.155.067.7
Qwen3.6-27B97.252.16.1
Qwen3.8-27B85.949.125.0
Gemma-4-31B-IT92.745.016.1
Qwen3-VL-8B93.546.215.3
Cosmos3-Nano82.244.731.2
TABLE II: Explicit-backoff results (%).
More in this topic

Human Activity Recognition

Human activity recognition classifies actions and behaviors from wearable sensors, video, radio signals, and other measurements.

See all 228 papers

Explore similar work

CardsList
Feb 16, 2026

ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery

Vision language models (VLMs) achieve strong performance on RGB imagery, but they do not generalize to thermal images. Thermal sensing plays a critical role in settings where visible light fails, including nighttime surveillance, search and rescue, autonomous driving, and medical screening. Unlike RGB imagery, thermal images encode physical temperature rather than color or texture, requiring perceptual and reasoning capabilities that existing RGB-centric benchmarks do not evaluate. We introduce ThermEval-B, a structured benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal vision language understanding. ThermEval-B integrates public datasets with our newly collected ThermEval-D, the first dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments. Evaluating 25 open-source and closed-source VLMs, we find that models consistently fail at temperature-grounded reasoning, degrade under colormap transformations, and default to language priors or fixed responses, with only marginal gains from prompting or supervised fine-tuning. These results demonstrate that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions, positioning ThermEval as a benchmark to drive progress in thermal vision language modeling.

Ayush Shrivastava, Kirtan Gangani, Laksh Jain +2

Indian Institute of Technology, Gandhinagar, India · Carnegie Mellon University, Pittsburgh, USA

VLM EvaluationThermal ImagingVisual Question AnsweringVLM Robustness
AskPDF
May 21, 2026

Thermo-VL: Extending Vision-Language Models to Thermal Infrared Perception

Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present Thermo-VL, a wavelength-aware VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show strong gains on challenging thermal-only and RGB+thermal reasoning tasks, highlighting the value of prompt-conditioned multispectral fusion. Our dataset and code are publicly available at: https://thusharakart.github.io/Thermo-VL

Rusiru Thushara, Yasiru Ranasinghe, Jay Paranjape +1

Department of Electrical & Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA

Thermal ImagingVisual Question AnsweringVision-Language ModelsVLM AdaptationVision-Language Grounding
AskPDF
Mar 5, 2026

Safe-Night VLA: Seeing the Unseen via Thermal-Perceptive Vision-Language-Action Models for Safety-Critical Manipulation

Current Vision-Language-Action (VLA) models rely primarily on RGB perception, preventing them from capturing modalities such as thermal signals that are imperceptible to conventional visual sensors. Moreover, end-to-end generative policies lack explicit safety constraints, making them fragile when encountering obstacles and novel scenarios outside the training distribution. To address these limitations, we propose Safe-Night VLA, a multimodal manipulation framework that enables robots to see the unseen while enforcing rigorous safety constraints for thermal-aware manipulation in unstructured environments. Specifically, Safe-Night VLA integrates long-wave infrared thermal perception into a pre-trained vision-language backbone, enabling semantic reasoning grounded in thermodynamic properties. To ensure safe execution under out-of-distribution conditions, we incorporate a safety filter via control barrier functions, which provide deterministic workspace constraint enforcement during policy execution. We validate our framework through real-world experiments on a Franka manipulator, introducing a novel evaluation paradigm featuring temperature-conditioned manipulation, subsurface target localization, and reflection disambiguation, while maintaining constrained execution at inference time. Results demonstrate that Safe-Night VLA outperforms RGB-only baselines and provide empirical evidence that foundation models can effectively leverage non-visible physical modalities for robust manipulation.

Dian Yu, Qingchuan Zhou, Bingkun Huang +2

Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), 80992 Munich, Germany.

Robotic ManipulationVision-Language-Action ModelsRobotic PerceptionSafe Robot Manipulation
AskPDF