Safety Evaluation of VLA Models

VLA: Vision-Language-Action

This field tests whether robots and driving agents can interpret instructions and act safely in interactive environments. Benchmarks use matched safe and unsafe scenarios, trajectory-level safety rules, counterfactual tests, and simulation-to-physical evaluation to separate safe behavior from task completion. Methods include adversarial red teaming, runtime action screening and abstention, semantic guards, robustness checks, and safety-aware training. Strong hazard recognition or task success often fails to ensure safe execution, leaving open challenges in cross-domain generalization, standardized metrics, and balancing safety with useful performance.

Latest papers

33 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models
Apr 14, 2026Zixing Chen, Yifeng Gao, Li Wang +8Robotics AI Safety Evaluation
Fudan University · City University of Hong Kong · Shanghai Innovation Institute
Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming
Apr 7, 2026Baoshun Tong, Haoran He, Ling Pan +2Adversarial Attacks on VLA Models Robustness of VLA Models
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Safety Case Patterns for VLA-based driving systems: Insights from SimLingo
Mar 16, 2026Gerhard Yu, Fuyuki Ishikawa, Oluwafemi Odu +1Autonomous Driving Safety Evaluation Autonomous Driving
York University, North York ON M3J 1P3, Canada · National Institute of Informatics, Chiyoda City Tokyo 101-8430, Japan

Apr 14, 2026J/K move · Enter open · S save

HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models

Zixing Chen, Yifeng Gao, Li Wang, Yunhan Zhao+7

Fudan University · City University of Hong Kong · Shanghai Innovation Institute

Robotics AI Safety Evaluation LLM Safety Vision-Language-Action Models Safety Evaluation of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models inherit rich world knowledge from vision-language backbones and acquire executable skills from action demonstrations. Yet current evaluations primarily measure task completion, leaving the semantic safety of learned action policies underexplored. This gap creates a critical vulnerability: a policy may execute the intended action correctly while producing unsafe outcomes when the surrounding visual-linguistic context changes. We present HazardArena, a benchmark for stress-testing semantic safety in VLA systems. Its core design is a set of safe/unsafe twin scenarios: paired environments with matched objects, layouts, and action requirements, but different semantic risk contexts. This controlled contrast isolates safety judgment from motor capability and directly tests whether a VLA can recognize when an otherwise valid action becomes hazardous. HazardArena includes over 2,000 assets and 51 risk-sensitive tasks across seven safety categories grounded in robotic safety standards. Across four representative VLA backbones, we observe a consistent and alarming pattern: safe-only fine-tuning improves benign task success while also increasing hazardous execution on matched unsafe scenarios. Physical-world experiments confirm that this failure transfers beyond simulation. These results show that stronger action execution does not imply safer behavior, and motivate semantic-risk-aware evaluation and enforcement as first-class requirements for real-world VLA deployment. Code released at https://github.com/HazardArena-Team/HazardArena ; updated code availability information.

Latest papers

HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models

Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming

Safety Case Patterns for VLA-based driving systems: Insights from SimLingo