cs.RONov 18, 2026

Assessing Vision-Language Models for Early Prediction of Unsafe Situations in Robotics

Authors: Muhammad Farooq, Aitor Arrieta, Paolo Arcaini, Shaukat Ali, Thomas Peyrucain

Organizations: OsloMet – Oslo Metropolitan University, Norway · Mondragon Unibertsitatea, Spain · Simula Research Laboratory, Norway · PAL Robotics, Spain

Abstract

Vision-Language Models (VLMs) process visual information and textual instructions and, by using their ability to understand and reason about context, are well-suited for predicting robot failures before they occur. However, evidence supporting this remains limited. To address this, we present an empirical evaluation of VLMs for predicting robot failures. In particular, we experiment with Qwen, InternVL, and LLaVA-OneVision across three robotic case studies (HuRoN, Pal Robotics, and CrowdBot), analyzing both video and odometry data for early failure warning (i.e., before an unsafe situation occurs). Specifically, we evaluate their performance in classifying situations as safe, potentially unsafe, and unsafe, together with uncertainty quantified using various metrics, and study the relationship between uncertainty and performance. Our results showed that Qwen provides the strongest overall classification and prediction-reliability performance, while InternVL shows stronger early-warning capability in several configurations. The results further show that the effect of robot-state information and temporal history depends on the model and evaluation objective. Overall, our findings indicate that general-purpose VLMs can support early robot-safety monitoring, but the most suitable configuration depends on whether the priority is accurate assessment of safety-state, reliable confidence and uncertainty, or timely warning before unsafe events.

Figures & tables

Explore similar work

CardsList
  1. Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

    Aug 12, 2026Vladyslava Rudas, Dmytro KuzmenkoVLM EvaluationRobot Safety

  2. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

    Jun 25, 2026Mingyang Lyu, Yinqian Sun, Yiyang Jia +5RoboticsRobot Safety

  3. FailBench: How Reliable are VLMs at Judging Robot Task Success?

    Sep 3, 2026Zaruhi Navasardyan, Tatul Danielyan, Hrant DavtyanVLM EvaluationRobot Failure Detection