Predictive Semantic Safety: From Visual Physical Reasoning to Safety-Critical Control
Authors: Taekyung Kim, Salem Fradi, Yanning Dai, Mateusz Ostaszewski, Jürgen Schmidhuber
Organizations: Department of Robotics, University of Michigan, Ann Arbor, MI 48109, USA · Center of Excellence in Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Switzerland. · Universit`a della Svizzera italiana (USI), Switzerland. · Scuola universitaria professionale della Svizzera italiana (SUPSI), Switzerland.
Physical interactions can create future hazards that are not apparent from the robot's current geometric surroundings. We present a framework termed Predictive Semantic Safety (PSS), which connects visual physical reasoning to backup-based safety filtering. A vision-language model (VLM) predicts physical events and their timing or directly predicts object displacements. An explicit motion model converts event hypotheses into object trajectories. Split conformal prediction calibrates position errors jointly across specified objects, observation times, and future times; geometric shape bounds convert the resulting position regions into predicted object occupancy. PSS evaluates a prescribed backup maneuver against this occupancy and derives input-affine constraints for minimally modifying the nominal input while preserving backup feasibility under the robot dynamics and input limits. MuJoCo experiments with a Unitree Go1 consider falling fixtures, impact-driven support loss, and contact propagation. PSS achieves a safe episode rate of 99.3%, compared with 43.3% for a Backup Control Barrier Function baseline that only uses current obstacle geometry.
Figures & tables
Fig. 1: Motivating example of predictive semantic safety. A partially detached ceiling fixture remains above an apparently clear route, but its anticipated fall may leave the robot with insufficient control authority to avoid collision.
Fig. 2: Overview of PSS, illustrated for the ceiling fixture. RGB-D observations and measured motion are used to predict a physical event and its timing with a VLM. The event hypothesis and an explicit motion model determine future object motion, which is combined with geometric shape bounds and calibrated uncertainty to obtain unsafe occupancy. A Backup CBF safety filter evaluates a prescribed backup maneuver against this occupancy and modifies the nominal command when necessary. The VLM panel shows an excerpt from a recorded model output. MuJoCo trials are shown in Fig. 4 .
Fig. 3: Backup feasibility with and without physical-event prediction. (a) Semantic-Off evaluates the backup using current obstacle geometry, so the resulting motion may continue toward the object’s future occupancy. (b) Semantic-On evaluates the same backup against predicted future occupancy and modifies the nominal motion while the backup remains feasible. Predicted occupancy is shown in both panels but is used only by Semantic-On . Generated conceptual illustration; geometry, trajectories, and timing are schematic.
Fig. 4: Selected MuJoCo trials with physical-event prediction disabled (Semantic-Off) and enabled (Semantic-On). (a) Ceiling fixture: Semantic-Off continues beneath the falling fixture and collides, whereas Semantic-On brakes before full detachment and later navigates around the debris. The red region shows the unsafe occupancy used by the filter at 7.0 s. (b) Impact-induced loss of support: Semantic-Off continues as the stack begins to fall and is struck by a block, whereas Semantic-On brakes and subsequently reaches the goal. Insets show synchronized robot-view images. These qualitative trials are separate from the aggregate benchmark.
Recent vision-language-action (VLA) models are promising for general-purpose manipulation, but long-horizon execution remains fragile. Small state-estimation or control errors can lead to irreversible failures (e.g., collisions and object drops). Avoiding these risks requires a proactive safety mechanism capable of anticipating hazards. In this paper, we introduce SafeLoop, a non-invasive external wrapper that adds hazard prediction and rollback-based recovery to a VLA model without changing its parameters. SafeLoop trains a risk predictor from vision and proprioception to output four values: the probability and time-to-hazard for body collisions and for object failures. A lightweight controller then chooses one of three actions based on the predicted risk: continue execution (noop), save a safety checkpoint (record), or retreat in joint space (rollback). Rollback moves the robot back to a recent safe waypoint and queries the base policy again, which may yield an alternative continuation. Across 24 LIBERO tasks (16 random seeds each) and three real-robot tasks (25 rollouts each), SafeLoop achieves a stronger overall safety-success trade-off than alternative methods, reducing hazard cases by roughly 70% while preserving task success and the base-policy control rate. Project code is available at https://github.com/Loule0-0/SafeLoop/tree/release/safeloop.
Zeyu Lou, Tianran Zhang, Xinquan Yue +2
Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangdong, China · Beijing University of Technology, Beijing, China
Traditional safety-critical control methods, such as control barrier functions, suffer from semantic blindness, exhibiting the same behavior around obstacles regardless of contextual significance. This limitation leads to the uniform treatment of all obstacles, despite their differing semantic meanings. We present Safe-SAGE (Social-Semantic Adaptive Guidance for Safe Engagement), a unified framework that bridges the gap between high-level semantic understanding and low-level safety-critical control through a Poisson safety function (PSF) modulated using a Laplace guidance field. Our approach perceives the environment by fusing multi-sensor point clouds with vision-based instance segmentation and persistent object tracking to maintain up-to-date semantics beyond the camera's field of view. A multi-layer safety filter is then used to modulate system inputs to achieve safe navigation using this semantic understanding of the environment. This safety filter consists of both a model predictive control layer and a control barrier function layer. Both layers utilize the PSF and flux modulation of the guidance field to introduce varying levels of conservatism and multi-agent passing norms for different obstacles in the environment. Our framework enables legged robots to safely navigate semantically rich, dynamic environments with context-dependent safety margins.
As robots are increasingly deployed in everyday environments, ensuring their safety has become a central challenge. Existing methods often encode safety requirements as opaque mathematical/logical formulations or dense cost functions. While effective in specific tasks, they remain difficult to interpret, tightly coupled to individual tasks, and offer limited insight into why a robot action is considered safe or unsafe. To address this limitation, we propose ``Neuro-Symbolic Predicate Learning for Semantic Safe Robot Control'' (NEUPRO), which leverages a differentiable reasoner that can learn reusable safety representations from human-specified safety knowledge. NEUPRO allows practitioners to express task-related safety requirements as transparent symbolic rules, while enabling gradients to propagate through these rules to a feature extractor that maps raw observations to safety-relevant concepts. As a result, the learned feature extractor is (softly) grounded in human-understandable semantics, supports transparent constraint evaluation, and is transferable across tasks. By coupling interpretability with differentiability, NEUPRO moves beyond opaque cost design toward reusable safety reasoning. To evaluate NEUPRO's capability, we collect and release REASON, the first real robot benchmark dataset for interpretable robot safety specification. Experiments on REASON show that NEUPRO learns safety-critical features that generalize across tasks, mitigate the interpretability limitations of conventional black-box cost formulations, and provide explicit explanations of safety violation.