Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming

Baoshun Tong; Haoran He; Ling Pan; Yang Liu; Liang Lin

cs.ROApr 7, 2026

Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming

Authors: Baoshun Tong, Haoran He, Ling Pan, Yang Liu, Liang Lin

Organizations: School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China

Abstract

Vision-Language-Action (VLA) models have achieved remarkable success in robotic manipulation. However, their robustness to linguistic nuances remains a critical, under-explored safety concern, posing a significant safety risk to real-world deployment. Red teaming, or identifying environmental scenarios that elicit catastrophic behaviors, is an important step in ensuring the safe deployment of embodied AI agents. Reinforcement learning (RL) has emerged as a promising approach in automated red teaming that aims to uncover these vulnerabilities. However, standard RL-based adversaries often suffer from severe mode collapse due to their reward-maximizing nature, which tends to converge to a narrow set of trivial or repetitive failure patterns, failing to reveal the comprehensive landscape of meaningful risks. To bridge this gap, we propose a novel Diversity-Aware Embodied Red Teaming (DAERT) framework, to expose the vulnerabilities of VLAs against linguistic variations. Our design is based on evaluating a uniform policy, which is able to generate a diverse set of challenging instructions while ensuring its attack effectiveness, measured by execution failures in a physical simulator. We conduct extensive experiments across different robotic benchmarks against two state-of-the-art VLAs, including $π_0$ and OpenVLA. Our method consistently discovers a wider range of more effective adversarial instructions that reduce the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents and exposing critical safety blind spots before real-world deployment.

Figures & tables

Figure 1: Overall architecture of our framework.

Method	Spatial	Object	Goal	Long	Average
$\pi_{0}$	22.0	30.4	1.6	16.6	17.65
$\pi_{0.5}$	62.8	69.0	12.6	75.2	54.90

Table 1: Inference success rates of VLA models on the LIBERO benchmark under the “no action” instruction.

Method	Spatial			Object			Goal			Long			Average
	Succ $\downarrow$	Cos $\uparrow$	LLM $\uparrow$	Succ $\downarrow$	Cos $\uparrow$	LLM $\uparrow$	Succ $\downarrow$	Cos $\uparrow$	LLM $\uparrow$	Succ $\downarrow$	Cos $\uparrow$	LLM $\uparrow$	Succ $\downarrow$	Cos $\uparrow$	LLM $\uparrow$
$\pi_{0}$	96.4	–	–	92.6	–	–	97.0	–	–	87.3	–	–	93.33	–	–
$\pi_{0}$ +ERT	78.4	10.5	6.8	80.4	10.2	6.3	29.2	12.1	6.6	74.0	7.8	5.7	65.50	10.15	6.35
$\pi_{0}$ +GRPO	23.8	8.5	5.3	18.8	9.0	5.3	6.6	6.1	3.7	32.6	4.6	4.0	20.45	7.05	4.58
$\pi_{0}$ +GRPO w/o KL	17.0	4.2	3.2	13.8	6.1	3.8	3.4	6.5	4.8	11.2	4.3	4.0	11.35	5.28	3.95
$\pi_{0}$ + DAERT (Ours)	7.4	14.3	8.7	8.8	13.2	8.6	3.0	11.3	7.8	4.2	10.1	8.8	5.85	12.23	8.48

Table 2: Quantitative results of

\pi_{0}

and its variants on the LIBERO benchmark. We evaluate attack effectiveness and instructional diversity across different task suites. Metrics include: Succ (

\downarrow

), denoting the average task success rate where lower values indicate effective attacks; Cos (

\uparrow

), measuring semantic diversity based on the cosine distance of CLIP embeddings; and LLM (

\uparrow

), representing the holistic linguistic diversity score evaluated via our LLM-as-Judge protocol.

Method	Spatial	Object	Goal	Long	Average
OpenVLA	84.7	88.4	79.2	53.7	76.50
OpenVLA+ERT	42.2	31.6	23.4	31.4	32.15
OpenVLA+GRPO	29.0	15.8	10.6	12.6	17.00
OpenVLA+ DAERT (Ours)	8.4	6.8	4.0	5.8	6.25

Table 3: Success rates on the LIBERO benchmark of OpenVLA-finetuned.

Task	Method	Instruction
Object	Original	Pick up the milk and place it in the basket.
	ERT	Move towards the red milk carton, secure it with your grip, and set it in the basket on the left.
	DAERT	Retrieve the milk carton from its current position, orient it correctly for insertion, and gently deposit it into the woven basket without disturbing any other objects on the floor.
Spatial	Original	Pick up the black bowl next to the ramekin and place it on the plate.
	ERT	Identify the black bowl adjacent to the silver container, grasp it, and rest it on the plate with stripes.
	DAERT	Retrieve the dark-colored bowl positioned adjacent to the small metallic ramekin, precisely orient it, and gently set it atop the circular plate with concentric red stripes.

Table 4: Qualitative Comparison of Adversarial Instructions. Adversarial instructions generated by the baseline (ERT) and our method (DAERT) are compared for Object and Spatial tasks. ERT primarily relies on lexical substitution, whereas DAERT introduces richer visual descriptions and additional constraints.

Panel of Figure 2 — Figure 2: Evaluation of robustness and instruction diversity under red-teaming. (a) Cross-model transferability attack on 3D-Diffuser in the CALVIN benchmark. (b) Performance of OpenVLA-7B on the SimplerEnv benchmark under different red-teaming settings. Attack performance is defined as $1-\text{task\_success\_rate}$ , where higher values indicate stronger attacks .

Panel of Figure 3 — Figure 3: PCA visualization of 100 rewritten instructions generated for two different LIBERO tasks, comparing three methods.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Qualitative visualizations of four representative LIBERO tasks. Each row illustrates a single task with four sequential observations, highlighting spatial reasoning, object manipulation, goal-directed placement, and multi-step interaction under cluttered environments.

Adversarial Attacks on VLA Models

Adversarial attacks on vision-language-action (VLA) models study how crafted inputs, poisoned data, or physical cues can disrupt robot decisions and safety.

See all 32 papers

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies
Mar 12, 2026Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu +9Robustness of VLA Models Adversarial Attacks on VLA Models
Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.
RedVLA: Physical Red Teaming for Vision-Language-Action Models
Apr 24, 2026Yuhao Zhang, Borong Zhang, Jiaming Fan +4Robotics Adversarial Attacks on VLA Models
Institute for AI, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University
Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms
Apr 26, 2026Qi Li, Bo Yin, Weiqi Huang +6AI Agent Safety Adversarial Attacks on VLA Models
National University of Singapore · Peking University · Monash University

Mar 12, 2026J/K move · Enter open · S save

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies

Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu, Varun Bhatt+8

Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.

Robustness of VLA Models Adversarial Attacks on VLA Models Quality-Diversity Optimization Instruction Generalization in VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have significant potential to enable general-purpose robotic systems for a range of vision-language tasks. However, the performance of VLA-based robots is highly sensitive to the precise wording of language instructions, and it remains difficult to predict when such robots will fail. We propose Quality Diversity (QD) optimization as a natural framework for red-teaming embodied models, and present Q-DIG (Quality Diversity for Diverse Instruction Generation), which performs red-teaming by scalably identifying diverse, natural language task descriptions that induce failures while remaining task-relevant. Q-DIG integrates QD techniques with Vision-Language Models (VLMs) to generate a broad spectrum of adversarial instructions that expose meaningful vulnerabilities in VLA behavior. Our results across multiple simulation benchmarks show that Q-DIG finds more diverse and meaningful failure modes compared to baseline methods, and that fine-tuning VLAs on the generated instructions improves task success rates. Furthermore, results from a user study highlight that Q-DIG generates prompts judged to be more natural and human-like than those from baselines. Finally, real-world evaluations of Q-DIG prompts show results consistent with simulation, and fine-tuning VLAs on the generated prompts further success rates on unseen instructions. Together, these findings suggest that Q-DIG is a promising approach for identifying vulnerabilities and improving the robustness of VLA-based robots. Our anonymous project website is at qdigvla.github.io.

Abstract

Adversarial Attacks on VLA Models

Explore similar work

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies

RedVLA: Physical Red Teaming for Vision-Language-Action Models

Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms