In recent human-robot collaboration environments, there is a growing focus on integrating diverse sensor data beyond visual information to enable safer and more intelligent task execution. Although thermal data can be crucial for enhancing robot safety and operational efficiency, its integration has been relatively overlooked in prior research. This paper proposes a novel Vision-Language-Action (VLA) framework that incorporates thermal information for robot task execution. The proposed system leverages a Vision-Language Model (VLM) as a high-level planner to interpret complex natural language commands and decompose them into simpler sub-tasks. This approach facilitates efficient data collection and robust reasoning for complex operations. Unlike conventional methods that rely solely on visual data, our approach integrates thermal information, enabling the robot to perceive physical properties and proactively ensure environmental safety. Experimental results from real-world task scenarios validate the feasibility of our proposed framework, suggesting its potential to enhance task success rates and safety compared to existing vision-based systems.
Figures & tables
Fig. 1: We propose ThermoAct . (a) illustrates a VLM Planner that decomposes a high-level user instruction into specific sub-task descriptions . (b) depicts a VLA Executor that receives these descriptions as input prompts to predict low-level actions. By leveraging temperature cues from thermal imaging, ThermoAct is able to perform temperature-aware tasks beyond existing approaches.
Fig. 2: Hierarchical Collaboration between VLM Planner and VLA Executor. (a) The VLM Planner receives RGB-Thermal images and a structured guideline prompt containing role definitions and output examples. (b) Based on the thermal information, the VLM analyzes the environment context and decomposes the instruction into executable sub-tasks. (c) Sub-task Decomposition with VLM and Action Execution with VLA.
Fig. 3: The figure shows five main task environments (Tasks 1–5), with the actual thermal input images displayed above each task. Tasks 1–3 correspond to daily-life manipulation tasks, while Tasks 4–5 focus on safety-related scenarios.
Fig. 4: Process of Task 5, where the system turns off the heated hair straightener and successfully generalizes to unseen ones using the learned data.
Method
30 episode
50 episode
70 episode
RGB-RGB
60.0%
80.0%
90.0%
Ours(RGB-T)
43.3 ± 11.5%
83.3 ± 5.8%
86.7 ± 5.8%
TABLE I: Performance of Fine-Tuned Tasks by Episode
Task Scenario
Sub-task
Success Rate (%)
FlatVLA
RGB-RGB
Ours (RGB-T)
Task 1: Bring warm water and an apple
pick up [warm water] from [floor]
-
40
90
place [warm water] to the [right side] of [empty plate]
-
100
90
pick up [an apple] from [fruit plate]
-
70
70
place [an apple] on the [empty plate]
-
80
70
TABLE II: Comprehensive Success Rates Comparison. This table integrates sub-task performances and overall averages. All success rates are reported based on N=10 independent trials per sub-task.
Fig. 5: Performance on subtasks requiring thermal perception, including picking up warm water (Task 1), picking up a cold Coke (Task 2), placing an object into the appropriate cup (Task 3), picking up a heated battery (Task 4), and turning off a hair straightener (Task 5).
Current Vision-Language-Action (VLA) models rely primarily on RGB perception, preventing them from capturing modalities such as thermal signals that are imperceptible to conventional visual sensors. Moreover, end-to-end generative policies lack explicit safety constraints, making them fragile when encountering obstacles and novel scenarios outside the training distribution. To address these limitations, we propose Safe-Night VLA, a multimodal manipulation framework that enables robots to see the unseen while enforcing rigorous safety constraints for thermal-aware manipulation in unstructured environments. Specifically, Safe-Night VLA integrates long-wave infrared thermal perception into a pre-trained vision-language backbone, enabling semantic reasoning grounded in thermodynamic properties. To ensure safe execution under out-of-distribution conditions, we incorporate a safety filter via control barrier functions, which provide deterministic workspace constraint enforcement during policy execution. We validate our framework through real-world experiments on a Franka manipulator, introducing a novel evaluation paradigm featuring temperature-conditioned manipulation, subsurface target localization, and reflection disambiguation, while maintaining constrained execution at inference time. Results demonstrate that Safe-Night VLA outperforms RGB-only baselines and provide empirical evidence that foundation models can effectively leverage non-visible physical modalities for robust manipulation.
Dian Yu, Qingchuan Zhou, Bingkun Huang +2
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), 80992 Munich, Germany
In robotics, Vision-Language-Action (VLA) models that integrate diverse multimodal signals from multi-view inputs have emerged as an effective approach. However, most prior work adopts static fusion that processes all visual inputs uniformly, which incurs unnecessary computational overhead and allows task-irrelevant background information to act as noise. Inspired by the principles of human active perception, we propose a dynamic information fusion framework designed to maximize the efficiency and robustness of VLA models. Our approach introduces a lightweight adaptive routing architecture that analyzes the current text prompt and observations from a wrist-mounted camera in real-time to predict the task-relevance of multiple camera views. By conditionally attenuating computations for views with low informational utility and selectively providing only essential visual features to the policy network, Our framework achieves computation efficiency proportional to task relevance. Furthermore, to efficiently secure large-scale annotation data for router training, we established an automated labeling pipeline utilizing Vision-Language Models (VLMs) to minimize data collection and annotation costs. Experimental results in real-world robotic manipulation scenarios demonstrate that the proposed approach achieves significant improvements in both inference efficiency and control performance compared to existing VLA models, validating the effectiveness and practicality of dynamic information fusion in resource-constrained, real-time robot control environments.
Language-guided manipulation can depend on physical properties that visible appearance does not reveal. Temperature is one such property, but object datasets for robot learning rarely associate measured temperatures with object appearance and geometry. We present HEARTH, an object-centric RGB-thermal-3D dataset of 90 physical objects from 18 everyday categories, comprising 145 captured object states. Our pipeline maps apparent surface temperatures onto reconstructed meshes through camera calibration and pose transfer. The dataset includes raw temperature measurements, camera parameters, RGB-textured meshes, and thermal textures for simulation. We use these assets to construct three LIBERO-derived tasks and collect 1,200 demonstrations for fine-tuning a pretrained vision-language-action (VLA) model, π0.5. In an ablation study, adding thermal observations to the VLA increases success on temperature-dependent object-selection tasks from 35.0% for the RGB-only baseline to 75.0%. These results demonstrate the utility of HEARTH for training robot policies to follow temperature-related instructions.