Organizations: Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.
Vision-Language-Action (VLA) models have significant potential to enable general-purpose robotic systems for a range of vision-language tasks. However, the performance of VLA-based robots is highly sensitive to the precise wording of language instructions, and it remains difficult to predict when such robots will fail. We propose Quality Diversity (QD) optimization as a natural framework for red-teaming embodied models, and present Q-DIG (Quality Diversity for Diverse Instruction Generation), which performs red-teaming by scalably identifying diverse, natural language task descriptions that induce failures while remaining task-relevant. Q-DIG integrates QD techniques with Vision-Language Models (VLMs) to generate a broad spectrum of adversarial instructions that expose meaningful vulnerabilities in VLA behavior. Our results across multiple simulation benchmarks show that Q-DIG finds more diverse and meaningful failure modes compared to baseline methods, and that fine-tuning VLAs on the generated instructions improves task success rates. Furthermore, results from a user study highlight that Q-DIG generates prompts judged to be more natural and human-like than those from baselines. Finally, real-world evaluations of Q-DIG prompts show results consistent with simulation, and fine-tuning VLAs on the generated prompts further success rates on unseen instructions. Together, these findings suggest that Q-DIG is a promising approach for identifying vulnerabilities and improving the robustness of VLA-based robots. Our anonymous project website is at qdigvla.github.io.
Figures & tables
Fig. 2 : Overview of Q-DIG. Q-DIG leverages previously generated instructions as in-context examples to generate new adversarial instructions in target attack styles (green arrows). The generated instructions are evaluated (red arrows) to obtain the variance of failure rates they induce in the VLA as well as their actual attack style. Instructions inducing high failure rates with different attack styles ( z0 to z7 from Table I ) are stored in an archive, providing high-quality and diverse examples for future iterations.
Step-by-step instructions: unnecessary breakdown of the task into multiple sequential steps (z0)
Uncommon vocabulary: use of rare, technical, or overly formal words instead of simple ones (z1)
Human-centric tone: instruction phrased as if addressing a human instead of a robot (z2)
Use of adverbs: adding vague modifiers like “carefully” or “gently” that don’t add clarity (z3)
Unnecessary action specification: extra details about how to execute the task, beyond the core instruction (z4)
Overly verbose reformulation: inflated or wordy phrasing without changing the meaning (z5)
Colloquial/casual style: use of slang or conversational tone inappropriate for robotic tasks (z6)
Mixed modality references: adding sensory references (e.g., sight, sound) that the robot may not have (z7)
TABLE I : Categories of failure in task instructions, partially obtained from [ 11 ] . For ease of reference, we index the attack styles using “z0” through “z7” (later used in Figure 5 ). Refer to Section V-A for details on how these failure modes were selected.
Fig. 3 : Diversity of our generated data compared to the Rephrase and ERT [ 11 ] baselines on OpenVLA-OFT. Each experiment was repeated 4 times, with error bars representing the standard error of the measurements. “Variance of Failure” is rescaled to be between 0 and 1. “Distance to Original” represents the average sentence embedding dissimilarity (1 - cosine similarity) for each domain (see Sec. V-A ). Q-DIG obtains the highest diversity metric in all cases.
Fig. 4 : An example of task images shown in our user study. Users were asked two sets of questions. First, they were asked to enter two instructions, one with natural wording and one attempting to be adversarial. Second, they were asked to rank and rate the human-likeness of instructions generated by Q-DIG, ERT, and Rephrase.
Fig. 5 : Example archive heatmap from Q-DIG on the LIBERO-Goal “put the bowl on top of the cabinet” task for OpenVLA-OFT. The “z0” through “z7” labels refer to the failure modes outlined in Table I . For example, “z0” corresponds to “step-by-step instructions.” The value in the cell corresponds to the failure variance of the discovered instruction, from 0 to 1. Q-DIG discovers instructions of diverse attack styles with high failure variance.
Unseen Prompts
OpenVLA-OFT
π0.5
GR00T N1.6
Orig
Reph
ERT
Q-DIG
Orig
Reph
ERT
Q-DIG
Orig
Reph
ERT
Q-DIG
Original
97.4
62.2
94.6
93.8
96.8
96.8
96.4
96.4
77.0
51.6
46.6
64.8
Rephrase
90.3
63.4
90.8
84.7
90.7
96.0
95.9
95.7
28.2
44.4
30.1
47.7
ERT
66.2
53.3
91.1
66.9
62.3
72.7
69.6
70.8
18.1
36.1
41.9
37.6
Q-DIG
63.9
47.6
76.4
88.9
67.5
71.9
74.8
73.4
33.7
42.4
33.6
55.1
Avg
76.9
55.8
87.3
82.1
76.8
82.6
82.4
82.3
33.9
42.5
36.8
49.4
TABLE II : Fine-tuned VLA results on the LIBERO-Goal task suite, using (i) original and (ii) adversarial plus original instructions across three VLAs: OpenVLA-OFT, π0.5 , and GR00T N1.6. “Original” represents evaluation on the original task instruction. Each instruction was evaluated 50 times. “Rephrase,” “ERT,” and “Q-DIG” represent evaluation on 2 unseen adversarial instructions. Columns correspond to the original VLA and models fine-tuned with instructions from the corresponding algorithm, and rows correspond to unseen instructions during training. Models trained on adversarial instructions (ERT and Q-DIG rows) perform better on unseen adversarial instructions.
Metric (Domain)
Rephrase
ERT
Q-DIG
BLEU (LIBERO)
0.9630.003
0.9510.01
0.9470.01
BLEU (SimplerEnv)
0.9280.01
0.9360.01
0.9460.01
Coverage (LIBERO)
0.3630.02
0.3250.01
0.9720.02
Coverage (SimplerEnv)
0.3880.03
0.3250.02
0.9130.02
TABLE III : Archive Coverage of failure modes (Table I ) and pairwise BLEU diversity results across all 3 methods and 2 benchmarks for our OpenVLA-OFT fine-tuned model. We show the values along with their standard errors in subscripts.
Original
P1
P2
P3
P4
Real-World
9/10
0/10
8/10
5/10
10/10
Simulation
10/10
4/10
7/10
4/10
10/10
TABLE IV : Comparison of Original and Q-DIG instruction variations for the “push the coke can” real-world task.
Unseen P1
Unseen P2
Original
0/10
8/10
Q-DIG
7/10
9/10
TABLE V : Unseen prompt evaluated on Original vs. Original and Q-DIG augmented models for “push the coke can” real-world task.
A long-standing goal in robotics is to develop generalist robots that can perform tasks across a wide range of settings, flexibly specified with images and text. Vision-Language-Action (VLA) models offer a promising path toward this goal by training on large collections of robot demonstrations. However, while VLAs perform well in-distribution, they often fail under minor test-time shifts, such as changes in object positions, lighting, textures, and camera poses. To address this, we propose Quality Diversity for Diverse Environment Generation (Q-DEG), a method for improving VLA robustness by generating adversarial environments and fine-tuning the VLA on its own successful rollouts. Q-DEG uses Quality Diversity (QD) optimization to find a set of high-failure environments, while promoting diversity using policy embeddings extracted from the VLA. These environments expose vulnerabilities that can be mitigated through self-imitation fine-tuning, thereby improving the VLA's robustness. In simulation, we conduct experiments across three representative VLA models, and our results indicate that Q-DEG improves robustness to out-of-distribution (OOD) adversarial environments compared to the base VLA and VLAs fine-tuned using data collected from two red-teaming baselines. Furthermore, we demonstrate that improving VLA robustness through Q-DEG facilitates sim-to-real transfer. Our anonymous project website is at: https://qd-env.github.io/qdeg/.
Shihan Zhao, Henry Chen, Ya-Chuan Hsu +7
Thomas Lord Department of Computer Science, University of Southern California. · Columbia University. · Adobe Research +1
Vision-Language-Action (VLA) models have achieved remarkable success in robotic manipulation. However, their robustness to linguistic nuances remains a critical, under-explored safety concern, posing a significant safety risk to real-world deployment. Red teaming, or identifying environmental scenarios that elicit catastrophic behaviors, is an important step in ensuring the safe deployment of embodied AI agents. Reinforcement learning (RL) has emerged as a promising approach in automated red teaming that aims to uncover these vulnerabilities. However, standard RL-based adversaries often suffer from severe mode collapse due to their reward-maximizing nature, which tends to converge to a narrow set of trivial or repetitive failure patterns, failing to reveal the comprehensive landscape of meaningful risks. To bridge this gap, we propose a novel Diversity-Aware Embodied Red Teaming (DAERT) framework, to expose the vulnerabilities of VLAs against linguistic variations. Our design is based on evaluating a uniform policy, which is able to generate a diverse set of challenging instructions while ensuring its attack effectiveness, measured by execution failures in a physical simulator. We conduct extensive experiments across different robotic benchmarks against two state-of-the-art VLAs, including π0 and OpenVLA. Our method consistently discovers a wider range of more effective adversarial instructions that reduce the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents and exposing critical safety blind spots before real-world deployment.
Baoshun Tong, Haoran He, Ling Pan +2
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We reframe VLA evaluation as an active failure-discovery problem and propose a failure-aware test-generation approach that combines diversity-driven exploration with surrogate models learned from observed executions. The method steers testing toward high-risk yet diverse scene regions. Across four state-of-the-art VLA models, it uncovers substantially more failures (up to +29.7 % over selected baselines) while revealing more diverse failure modes. This mean that, for instance, in the case of GR00T-N1.6, success rate dropped from 64.4% to 34.7%. More broadly, our findings call for a shift in VLA evaluation: from passive measurement on fixed task suites to adaptive, failure-seeking test generation that exposes the structure of model weaknesses before deployment.
Arusa Kanwal, Pablo Valle, Shaukat Ali +1
Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway