Vision-language-action (VLA) models enable increasingly general-purpose robotic manipulation, but such learned policies do not provide safety guarantees for collision avoidance---especially in environments outside of training distributions. This work presents Vision-Language-Poisson-Safe Actions (VLPSA), a safety filtering framework that provides full-body safety for VLA policies in cluttered and dynamic environments without retraining. VLPSA synthesizes Poisson Safety Functions (PSF) online from perception data, yielding a Control Barrier Function (CBF) that is enforced through a CBF-QP safety filter over the full body and any grasped object, treated as an extension of the final robot link. To enable real-time deployment while maintaining fine spatial resolution in critical task regions, VLPSA combines dual resolutions of this PSF using Boolean CBF compositions. We evaluate VLPSA on SafeLIBERO against safety-filtering baselines, where it achieves the highest collision avoidance rate among the evaluated methods, increasing collision avoidance from 23.1% for the base π0.5 policy to 91.2% while surpassing its task success rate. We further deploy VLPSA on a Franka FR3 in cluttered scenes with dynamic obstacles and human interference, demonstrating real-time full-body safety during manipulation tasks.
Figures & tables
Fig. 1 : Pipeline for VLPSA: Sensory and semantic input and robot state are sent to the VLA for a nominal trajectory and gripper state. Depth sensors are fused with robot state information to create an occupancy map for the synthesis of PSFs at varying resolutions for safety filtering. When grasping, live samples are generated and updated to protect the held object, incorporated into the same safety filter.
Fig. 2 : Multigrid Usage. (Left) Fine-grid usage (% of sample points) for the grasped object during a trial in static clutter and with dynamic obstacles. The grasped object shows consistent moderate usage under clutter but sparser, higher peaks under moving obstacles. (Right) Zero-level set of the fine PSF hf in a cluttered environment showing which grid wins at each point.
Grid ( ε , N3 )
Poisson [ms]
QP [ms]
Total [ms]
Single uniform
3.0cm , 803
3.44±0.23
0.72±0.02
6.5±1.9
2.0cm , 1203
9.12±0.44
0.76±0.04
23.1±6.6
1.5cm , 1603
20.20±0.46
0.78±0.12
38.5±5.7
1.0cm , 2403
91.80±0.95
0.61±0.07
261.6±9.0
0.5cm , 4803
724.19±3.71
0.59±0.03
2059±33
TABLE I : Solve cost for a fixed workspace extent ( 2.4m per side) and a fixed 100 -iteration budget. Mean ± std over 27 – 1708 solves per configuration on an RTX 4090, with ∣Yaug∣=679 . The total time includes all numerical processing. The multigrid scheme resolves 1.0cm over the region of interest at coarse-grid cost; it doubles the number of QP constraints, but latency is dominated by the Poisson solve.
Method
SR ↑
CAR ↑
SSR ↑
Steps ↓
Spatial
Nominal
57.8 ± 4.8
20.8 ± 4.0
19.2 ± 3.9
119.5 ± 6.3
VLSA ∗
73.8 ± 4.3
70.5 ± 4.5
59.8 ± 4.8
133.4 ± 4.2
VLPSA-NI
71.5 ± 4.4
75.8 ± 4.2
54.5 ± 4.9
123.0 ± 5.9
VLPSA
72.2 ± 4.4
89.8 ± 3.0
66.5 ± 4.6
136.9 ± 5.9
Goal
TABLE II : VLPSA vs. VLSA ∗ [ 11 ] (full action space) and nominal π0.5 on SafeLIBERO (Levels I+II). VLPSA-NI ablates injection of held-object samples into the safe set. Success (SR), Avoidance (CAR, 100− collision rate), and Safe Success (SSR) rates in %; Steps is the mean number of steps to reach success, over successful episodes. ± denotes a 95% confidence interval (normal approximation; n=400 episodes per suite). Bold marks the best method where its interval does not overlap that of the runner-up.
Fig. 3 : (A) Hardware validation of the VLA policy with and without the proposed safety layer in a cluttered tabletop environment for the task “Pick up the banana.” (Top) The cluttered environment is out-of-distribution for the nominal VLA policy, which completes the task but collides with multiple objects in the scene. (Middle) With the proposed full-body safety architecture, the VLA policy completes the same task while avoiding all collisions. (Bottom Left) The minimum active Poisson Safety Function value per link remains positive through-out the trial. (Bottom Right) The nominal trajectory versus the safety filtered trajectory throughout the trial. (B) The minimum value of the PSF per-link throughout a long horizon trial with the same text prompt as above. This dictates two successful executions of the task in clutter with dynamical obstacles.
Vision-Language-Action (VLA) models have demonstrated impressive end-to-end performance across a variety of robotic manipulation tasks. However, these policies offer no guarantees against collisions with task-irrelevant objects in the scene. Existing safety filters sidestep this problem by querying a vision-language model (VLM) to identify obstacles and their locations. This, however, is too slow to run in the control loop and can only be invoked at episode initialization, leaving the filter unable to track moving obstacles. We discover that a small number of attention heads within a VLA model reliably localize the object the policy intends to approach. These heads can be exploited within a training-free safety framework that obtains the active target from the attention heads at every step, treats the remainder of the scene as obstacles, and feeds these into a Control Barrier Function (CBF) filter. Together with a lightweight real-time object tracker, this allows for collision avoidance for non-static obstacles. We evaluate our framework on SafeLIBERO, which we extend with moving obstacles. On the original static benchmark, our method performs comparably to an oracle that uses privileged simulator state to identify the target, emulating a VLM-based identification step run once at episode initialization. On the dynamic variant, where the oracle's init-time target assignment becomes stale, our method substantially outperforms it by 43%, on average. Our findings suggest that the perceptual signals needed for real-time safety filtering are already present within VLA policies and can be exploited without additional training or heavy auxiliary models.
Seongbin Park, Fan Zhang, Baharan Mirzasoleiman +2
University of California Los Angeles United States
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in generalizing across diverse robotic manipulation tasks. However, deploying these models in unstructured environments remains challenging due to the critical need for simultaneous task compliance and safety assurance, particularly in preventing potential collisions during physical interactions. In this work, we introduce a Vision-Language-Safe Action (VLSA) architecture, named AEGIS, which contains a plug-and-play safety constraint (SC) layer formulated via control barrier functions. AEGIS integrates directly with existing VLA models to improve safety with theoretical guarantees, while maintaining their original instruction-following performance. To evaluate the efficacy of our architecture, we construct a comprehensive safety-critical benchmark SafeLIBERO, spanning distinct manipulation scenarios characterized by varying degrees of spatial complexity and obstacle intervention. Extensive experiments demonstrate the superiority of our method over state-of-the-art baselines. Notably, AEGIS achieves over 50% improvement in obstacle avoidance rate while substantially increasing the task success rate by nearly 10%. All benchmark datasets, code, and supplementary materials are publicly available at https://vlsa-aegis.github.io/.
Songqiao Hu, Zeyi Liu, Shuang Liu +5
Department of Automation, Tsinghua University, Beijing 100084, China. · Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China. · 3TetraBOT. +1
Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety violations along the trajectory: a policy may reach the goal while applying excessive contact, disturbing bystander objects, destabilizing a held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks that reveals violations missed by success-only evaluation. It encodes task-aware safety requirements as Signal Temporal Logic (STL) invariants with quantitative robustness semantics. Alongside native success, it reports the safety rate and the success-but-unsafe rate (SBU) used in prior safety evaluations, and introduces the Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating twenty-seven policy-benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: the fifteen tabletop policies above 90% mean success still have 18-28% unsafe-episode rates, and 38-56% of successful RoboCasa-365 rollouts violate at least one active safety clause. A post-training case study further shows that SafeVLA-Bench can be used to improve policy safety. Project page: https://safevla.org
Jialiang Fan, Weizhe Xu, Zijun Wang +3
University of Notre Dame · University of Pennsylvania