Beyond Behaviour: A Stimulus-Response Safety Evaluation of Vision-Language-Action Models

Vy Vo; Dinh Phung

cs.AIOct 26, 2026

Beyond Behaviour: A Stimulus-Response Safety Evaluation of Vision-Language-Action Models

Authors: Vy Vo, Dinh Phung

Organizations: Monash University, Australia

Abstract

Humans exhibit biological sensitivity to safety-critical stimuli that can trigger defensive action. Across three models and two video benchmarks, we investigate whether similar responses emerge in vision-language-action models by examining their internal representations and action tendencies in response to visible hazards and harmful consequences. Our findings suggest that task execution may remain the default response even when danger is internally registered.

Figures & tables

Figure 1: Transition surprise at the vision encoder. Each curve measures the transition difference from calm frames : the

Q

score of frames at time

x

from the consequence minus the score of calm frames (Eq. 7 ). Grey refers to the comparisons with all calm frames. Blue refers to the comparisons only with calm frames at the same second into the clip controlling for the progression of the video over time. Detailed metrics are presented in Appendix A.1.4 .

Panel of Figure 2 — Figure 2: Action direction relative to danger over the onset and final frames across categories. (a) Median distance-reduction ratio $R$ with $95\%$ frame-bootstrap confidence intervals; (b) proportion of frames across three types of behaviour.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Symbol	HomeSafe-Bench	RoboShackles
Frame groups and preparation
Calm start	$t_{\min}$	$0.5\,\mathrm{s}$	$0.5\,\mathrm{s}$
Calm cutoff	$c_{e}$	$t_{\mathrm{intent},e}$	$t_{e}^{\star}-1\,\mathrm{s}$
Near / just-after window widths	$w_{-},w_{+}$	$2\,\mathrm{s},2\,\mathrm{s}$	$1\,\mathrm{s},1\,\mathrm{s}$
Donor fraction	$\rho$	$0.20$	$0.20$
Donor / evaluation clips	—	$61/245$	$240/960$

Appendix

Table 1: Settings for the representational sensitivity analysis. Image-change bin counts apply separately to object motion and appearance change. A dash denotes an analysis not performed.


Setting	Value
Annotation image long side	$768$ px
RoboShackles focal length	$526.7$ px
HomeSafe-Bench focal length	Depth Pro’s per-frame estimate
Replay steps $J$ / frequency $\nu$	$8$ / $20$ Hz ( $0.4$ s)
Controller position / rotation scales	$0.05$ m / $0.5$ rad per step

Appendix

Table 2: Settings for avoidance behaviour analysis. Controller scales apply to delta-action replay.

Figure 3: HomeSafe-Bench : transition surprise

Q

relative to calm frames along the time from the impact (marked zero on the x-axis), without matching: each time bin is compared with all calm frames. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the impact.

Figure 4: HomeSafe-Bench : transition surprise

Q

relative to calm frames along the time from the impact (marked zero on the x-axis), at the same elapsed time: each time bin is compared only with calm frames at the same time into the clip; bins with too few matched frames are not drawn. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the impact.

Figure 5: RoboShackles : transition surprise

Q

relative to calm frames along the time from the consequence onset (marked zero on the x-axis), without matching: each time bin is compared with all calm frames. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the consequence onset.

Figure 6: RoboShackles : transition surprise

Q

relative to calm frames along the time from the consequence onset (marked zero on the x-axis), at the same elapsed time: each time bin is compared only with calm frames at the same time into the clip; bins with too few matched frames are not drawn. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the consequence onset.

Figure 7: HomeSafe-Bench : difference in

Q

between groups of frames, per model and site group, without matching and under each control (same elapsed time; same image change). Columns: near vs calm, just after vs calm, just after vs near. Units: clip-to-clip spreads of the reference group (calm frames, or near frames in the last column); zero means no difference. Filled markers with thick lines are significant after Benjamini–Hochberg correction. HomeSafe-Bench has no joint elapsed-time and image-change control.

Figure 8: RoboShackles : difference in

Q

between groups of frames, per model and site group, without matching and under each control (same elapsed time; same image change; same elapsed time and image change). Columns: near vs calm, just after vs calm, just after vs near. Units: clip-to-clip spreads of the reference group (calm frames, or near frames in the last column); zero means no difference. Filled markers with thick lines are significant after Benjamini–Hochberg correction.

Reason	RoboShackles	HomeSafe-Bench

Standing-board layout	260	0
Assumed gripper	494	3
Gripper out of view (hand-placed)	0	32
Gripper at the danger (hand-placed)	0	13
Gripper borrowed from the other frame	330	41

Appendix

Table 3: Action-evaluation exclusions after annotation and support-plane screening. Reasons may overlap; the final row gives unique excluded and retained frame counts.

Panel of Figure 9 — Figure 9: Examples of reconstructed digital-cousin scenes in LIBERO . Each row shows a source frame with the annotated gripper and danger region (left), followed by the corresponding reconstructed scene from multiple simulator views (right). In the reconstruction, the danger location is represented by a simplified red box whose placement preserves the estimated gripper–danger offset recovered from image annotations and metric depth. The robot pose is initialized to approximately match the annotated frame, while the surrounding environment is reduced to a minimal scene sufficient for evaluating the direction of the initial predicted action.

Panel of Figure 10 — Figure 10: HomeSafe-Bench: action tendencies by danger level (the higher the level, the more dangerous the event): L1 and L2 in the top row, and L3 and L4 in the bottom row.

Panel of Figure 11 — Figure 11: RoboShackles: action tendencies by danger type: hand and human in the top row; fire hazard and electrical hazard in the middle row; water safety and falling risk in the bottom row.

Safety Evaluation of VLA Models

Safety evaluation of vision-language-action (VLA) models measures whether embodied agents follow instructions without causing semantic or physical harm.

See all 33 papers

Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms
Apr 26, 2026Qi Li, Bo Yin, Weiqi Huang +6AI Agent Safety Adversarial Attacks on VLA Models
National University of Singapore · Peking University · Monash University
Are Vision-Language-Action Models Robust to One-Step Observation Perturbations?
Sep 26, 2026Shojiro Yamabe, Jun SakumaRobustness of VLA Models Safety Evaluation of VLA Models
Science Tokyo · Science Tokyo, RIKEN
HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models
Apr 14, 2026Zixing Chen, Yifeng Gao, Li Wang +8Robotics AI Safety Evaluation
Fudan University · City University of Hong Kong · Shanghai Innovation Institute

Apr 26, 2026J/K move · Enter open · S save

Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms

Qi Li, Bo Yin, Weiqi Huang, Ruhao Liu+5

National University of Singapore · Peking University · Monash University

AI Agent Safety Adversarial Attacks on VLA Models Vision-Language-Action Models Robustness of VLA Models Safety Evaluation of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are emerging as a unified substrate for embodied intelligence. This shift raises a new class of safety challenges, stemming from the embodied nature of VLA systems, including irreversible physical consequences, a multimodal attack surface across vision, language, and state, real-time latency constraints on defense, error propagation over long-horizon trajectories, and vulnerabilities in the data supply chain. Yet the literature remains fragmented across robotic learning, adversarial machine learning, AI alignment, and autonomous systems safety. This survey provides a unified and up-to-date overview of safety in Vision-Language-Action models. We organize the field along two parallel timing axes, attack timing (training-time vs. inference-time and defense timing (training-time vs. inference-time, linking each class of threat to the stage at which it can be mitigated. We first define the scope of VLA safety, distinguishing it from text-only LLM safety and classical robotic safety, and review the foundations of VLA models, including architectures, training paradigms, and inference mechanisms. We then examine the literature through four lenses: Attacks, Defenses, Evaluation, and Deployment. We survey training-time threats such as data poisoning and backdoors, as well as inference-time attacks including adversarial patches, cross-modal perturbations, semantic jailbreaks, and freezing attacks. We review training-time and runtime defenses, analyze existing benchmarks and metrics, and discuss safety challenges across six deployment domains. Finally, we highlight key open problems, including certified robustness for embodied trajectories, physically realizable defenses, safety-aware training, unified runtime safety architectures, and standardized evaluation.

Abstract

Safety Evaluation of VLA Models

Explore similar work

Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms

Are Vision-Language-Action Models Robust to One-Step Observation Perturbations?

HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models