Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies

Siddharth Srikanth; Freddie Liang; Ya-Chuan Hsu; Varun Bhatt; Shihan Zhao; Henry Chen; Bryon Tjanaka; Minjune Hwang

cs.ROMar 12, 2026

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies

Authors: Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu, Varun Bhatt, Shihan Zhao, Henry Chen, Bryon Tjanaka, Minjune Hwang,

Organizations: Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.

Abstract

Vision-Language-Action (VLA) models have significant potential to enable general-purpose robotic systems for a range of vision-language tasks. However, the performance of VLA-based robots is highly sensitive to the precise wording of language instructions, and it remains difficult to predict when such robots will fail. We propose Quality Diversity (QD) optimization as a natural framework for red-teaming embodied models, and present Q-DIG (Quality Diversity for Diverse Instruction Generation), which performs red-teaming by scalably identifying diverse, natural language task descriptions that induce failures while remaining task-relevant. Q-DIG integrates QD techniques with Vision-Language Models (VLMs) to generate a broad spectrum of adversarial instructions that expose meaningful vulnerabilities in VLA behavior. Our results across multiple simulation benchmarks show that Q-DIG finds more diverse and meaningful failure modes compared to baseline methods, and that fine-tuning VLAs on the generated instructions improves task success rates. Furthermore, results from a user study highlight that Q-DIG generates prompts judged to be more natural and human-like than those from baselines. Finally, real-world evaluations of Q-DIG prompts show results consistent with simulation, and fine-tuning VLAs on the generated prompts further success rates on unseen instructions. Together, these findings suggest that Q-DIG is a promising approach for identifying vulnerabilities and improving the robustness of VLA-based robots. Our anonymous project website is at qdigvla.github.io.

Figures & tables

Fig. 2 : Overview of Q-DIG. Q-DIG leverages previously generated instructions as in-context examples to generate new adversarial instructions in target attack styles (green arrows). The generated instructions are evaluated (red arrows) to obtain the variance of failure rates they induce in the VLA as well as their actual attack style. Instructions inducing high failure rates with different attack styles ( z0 to z7 from Table I ) are stored in an archive, providing high-quality and diverse examples for future iterations.

Step-by-step instructions: unnecessary breakdown of the task into multiple sequential steps (z0)	Uncommon vocabulary: use of rare, technical, or overly formal words instead of simple ones (z1)
Human-centric tone: instruction phrased as if addressing a human instead of a robot (z2)	Use of adverbs: adding vague modifiers like “carefully” or “gently” that don’t add clarity (z3)
Unnecessary action specification: extra details about how to execute the task, beyond the core instruction (z4)	Overly verbose reformulation: inflated or wordy phrasing without changing the meaning (z5)
Colloquial/casual style: use of slang or conversational tone inappropriate for robotic tasks (z6)	Mixed modality references: adding sensory references (e.g., sight, sound) that the robot may not have (z7)

TABLE I : Categories of failure in task instructions, partially obtained from [ 11 ] . For ease of reference, we index the attack styles using “z0” through “z7” (later used in Figure 5 ). Refer to Section V-A for details on how these failure modes were selected.

Panel of Figure 3 — Fig. 3 : Diversity of our generated data compared to the Rephrase and ERT [ 11 ] baselines on OpenVLA-OFT. Each experiment was repeated 4 times, with error bars representing the standard error of the measurements. “Variance of Failure” is rescaled to be between 0 and 1. “Distance to Original” represents the average sentence embedding dissimilarity (1 - cosine similarity) for each domain (see Sec. V-A ). Q-DIG obtains the highest diversity metric in all cases.

Panel of Figure 4 — Fig. 4 : An example of task images shown in our user study. Users were asked two sets of questions. First, they were asked to enter two instructions, one with natural wording and one attempting to be adversarial. Second, they were asked to rank and rate the human-likeness of instructions generated by Q-DIG, ERT, and Rephrase.

Fig. 5 : Example archive heatmap from Q-DIG on the LIBERO-Goal “put the bowl on top of the cabinet” task for OpenVLA-OFT. The “z0” through “z7” labels refer to the failure modes outlined in Table I . For example, “z0” corresponds to “step-by-step instructions.” The value in the cell corresponds to the failure variance of the discovered instruction, from 0 to 1. Q-DIG discovers instructions of diverse attack styles with high failure variance.

Unseen Prompts	OpenVLA-OFT				$\mathbf{\pi_{0.5}}$				GR00T N1.6
Unseen Prompts	Orig	Reph	ERT	Q-DIG	Orig	Reph	ERT	Q-DIG	Orig	Reph	ERT	Q-DIG
Original	97.4	62.2	94.6	93.8	96.8	96.8	96.4	96.4	77.0	51.6	46.6	64.8
Rephrase	90.3	63.4	90.8	84.7	90.7	96.0	95.9	95.7	28.2	44.4	30.1	47.7
ERT	66.2	53.3	91.1	66.9	62.3	72.7	69.6	70.8	18.1	36.1	41.9	37.6
Q-DIG	63.9	47.6	76.4	88.9	67.5	71.9	74.8	73.4	33.7	42.4	33.6	55.1
Avg	76.9	55.8	87.3	82.1	76.8	82.6	82.4	82.3	33.9	42.5	36.8	49.4

TABLE II : Fine-tuned VLA results on the LIBERO-Goal task suite, using (i) original and (ii) adversarial plus original instructions across three VLAs: OpenVLA-OFT,

\pi_{0.5}

, and GR00T N1.6. “Original” represents evaluation on the original task instruction. Each instruction was evaluated 50 times. “Rephrase,” “ERT,” and “Q-DIG” represent evaluation on 2 unseen adversarial instructions. Columns correspond to the original VLA and models fine-tuned with instructions from the corresponding algorithm, and rows correspond to unseen instructions during training. Models trained on adversarial instructions (ERT and Q-DIG rows) perform better on unseen adversarial instructions.

Metric (Domain)	Rephrase	ERT	Q-DIG
BLEU (LIBERO)	$\textbf{0.963}_{0.003}$	$0.951_{0.01}$	$0.947_{0.01}$
BLEU (SimplerEnv)	$0.928_{0.01}$	$0.936_{0.01}$	$\textbf{0.946}_{0.01}$
Coverage (LIBERO)	$0.363_{0.02}$	$0.325_{0.01}$	$\textbf{0.972}_{0.02}$
Coverage (SimplerEnv)	$0.388_{0.03}$	$0.325_{0.02}$	$\textbf{0.913}_{0.02}$

TABLE III : Archive Coverage of failure modes (Table I ) and pairwise BLEU diversity results across all 3 methods and 2 benchmarks for our OpenVLA-OFT fine-tuned model. We show the values along with their standard errors in subscripts.

	Original	P1	P2	P3	P4
Real-World	9/10	0/10	8/10	5/10	10/10
Simulation	10/10	4/10	7/10	4/10	10/10

TABLE IV : Comparison of Original and Q-DIG instruction variations for the “push the coke can” real-world task.

	Unseen P1	Unseen P2
Original	0/10	8/10
Q-DIG	7/10	9/10

TABLE V : Unseen prompt evaluated on Original vs. Original and Q-DIG augmented models for “push the coke can” real-world task.

Robustness of VLA Models

Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.

See all 221 papers

Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation
Oct 10, 2026Shihan Zhao, Henry Chen, Ya-Chuan Hsu +7Robustness of VLA Models Adversarial Attacks on VLA Models
Thomas Lord Department of Computer Science, University of Southern California. · Columbia University. · Adobe Research +1
Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming
Apr 7, 2026Baoshun Tong, Haoran He, Ling Pan +2Adversarial Attacks on VLA Models Robustness of VLA Models
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
FATE-VLA:Failue-aware test generation for vision-language-action models
Jun 1, 2026Arusa Kanwal, Pablo Valle, Shaukat Ali +1VLM Evaluation Vision-Language-Action Models
Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway

Oct 10, 2026J/K move · Enter open · S save

Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

Shihan Zhao, Henry Chen, Ya-Chuan Hsu, Minjune Hwang+6

Thomas Lord Department of Computer Science, University of Southern California. · Columbia University. · Adobe Research+1

Robustness of VLA Models Adversarial Attacks on VLA Models

Open paper AskPDF arXiv

A long-standing goal in robotics is to develop generalist robots that can perform tasks across a wide range of settings, flexibly specified with images and text. Vision-Language-Action (VLA) models offer a promising path toward this goal by training on large collections of robot demonstrations. However, while VLAs perform well in-distribution, they often fail under minor test-time shifts, such as changes in object positions, lighting, textures, and camera poses. To address this, we propose Quality Diversity for Diverse Environment Generation (Q-DEG), a method for improving VLA robustness by generating adversarial environments and fine-tuning the VLA on its own successful rollouts. Q-DEG uses Quality Diversity (QD) optimization to find a set of high-failure environments, while promoting diversity using policy embeddings extracted from the VLA. These environments expose vulnerabilities that can be mitigated through self-imitation fine-tuning, thereby improving the VLA's robustness. In simulation, we conduct experiments across three representative VLA models, and our results indicate that Q-DEG improves robustness to out-of-distribution (OOD) adversarial environments compared to the base VLA and VLAs fine-tuned using data collected from two red-teaming baselines. Furthermore, we demonstrate that improving VLA robustness through Q-DEG facilitates sim-to-real transfer. Our anonymous project website is at: https://qd-env.github.io/qdeg/.

Abstract

Robustness of VLA Models

Explore similar work

Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming

FATE-VLA:Failue-aware test generation for vision-language-action models