Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

Shihan Zhao; Henry Chen; Ya-Chuan Hsu; Minjune Hwang; Siddharth Srikanth; Freddie Liang; Akanksha Saran; Aaquib Tabrez

cs.ROOct 10, 2026

Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

Authors: Shihan Zhao, Henry Chen, Ya-Chuan Hsu, Minjune Hwang, Siddharth Srikanth, Freddie Liang, Akanksha Saran, Aaquib Tabrez,

Organizations: Thomas Lord Department of Computer Science, University of Southern California. · Columbia University. · Adobe Research · Cornell University.

Abstract

A long-standing goal in robotics is to develop generalist robots that can perform tasks across a wide range of settings, flexibly specified with images and text. Vision-Language-Action (VLA) models offer a promising path toward this goal by training on large collections of robot demonstrations. However, while VLAs perform well in-distribution, they often fail under minor test-time shifts, such as changes in object positions, lighting, textures, and camera poses. To address this, we propose Quality Diversity for Diverse Environment Generation (Q-DEG), a method for improving VLA robustness by generating adversarial environments and fine-tuning the VLA on its own successful rollouts. Q-DEG uses Quality Diversity (QD) optimization to find a set of high-failure environments, while promoting diversity using policy embeddings extracted from the VLA. These environments expose vulnerabilities that can be mitigated through self-imitation fine-tuning, thereby improving the VLA's robustness. In simulation, we conduct experiments across three representative VLA models, and our results indicate that Q-DEG improves robustness to out-of-distribution (OOD) adversarial environments compared to the base VLA and VLAs fine-tuned using data collected from two red-teaming baselines. Furthermore, we demonstrate that improving VLA robustness through Q-DEG facilitates sim-to-real transfer. Our anonymous project website is at: https://qd-env.github.io/qdeg/.

Figures & tables

Fig. 1: Q-DEG generates diverse failure-inducing environments by perturbing parameters such as object placement, background, and camera and light source poses. New perturbations are generated from previously discovered environments, which act as “stepping stones” that facilitate exploration.

Fig. 2: Overview of Q-DEG . a) CMA-ES [ 37 ] proposes environment parameters such as object locations, background, lighting, and camera pose; invalid proposals are repaired to restore task-relevant spatial relations. b) The VLA is rolled out in each proposed environment to obtain feasibility-aware failure rate $\hat{P}^{\text{fail}}$ and policy embeddings. c) Q-DEG adds each environment to the QD archive based on its policy embeddings, and rewards CMA-ES with its failure rate improvement $\Delta\hat{P}^{\text{fail}}$ over predecessors with similar embeddings. d) Successful trajectories from archived environments are collected into a fine-tuning dataset for improving the VLA’s robustness.

	$\pi_{0}$ -FAST				OpenVLA-OFT				$\pi_{0.5}$
Eval. set	base	rand	CMA-ES	Q-DEG	base	rand	CMA-ES	Q-DEG	base	rand	CMA-ES	Q-DEG
default	96.80	68.60	82.60	89.60	96.20	94.80	86.70	93.80	98.40	85.60	93.60	95.60
rand	29.15	39.40	37.79	48.73	56.25	82.29	68.75	78.13	77.38	77.98	83.33	88.09
CMA-ES	27.90	38.46	24.81	29.82	45.75	52.52	48.27	75.79	54.98	49.39	69.17	58.00
Q-DEG	54.83	51.28	36.89	56.13	47.19	50.77	55.61	69.13	38.03	61.82	70.45	81.81
average	52.17	49.44	45.52	56.07	61.35	70.10	64.83	79.21	67.20	68.70	79.14	80.88

TABLE I: VLA success rates on the default LIBERO-Spatial tasks and adversarial evaluation sets generated by randomization (rand), CMA-ES, and Q-DEG. Rows denote evaluation sets and columns denote VLA checkpoints.

Fig. 3: Number of feasibly adversarial environments (top) found by randomization , CMA-ES , and Q-DEG, and the corresponding average pairwise DINOv2 cosine distance among environments (bottom), plotted against the number of VLA rollouts. Shaded regions denote $95\%$ confidence intervals aggregated over three independent trials.

Fig. 4: Success counts and final gripper positions when evaluating the base VLA and VLAs fine-tuned with randomization ( rand ), CMA-ES , and Q-DEG on real-world tasks. Successful trials are shown in green, and failed trials are shown in red.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Fig. 5: Variance explained by the first 10 principal components of raw policy embeddings extracted from $\pi_{0}$ -FAST, OpenVLA-OFT, and $\pi_{0.5}$ . Results are aggregated over three independent trials, each sampling $320$ environments. Error bars denote $95\%$ confidence intervals.

Robustness of VLA Models

Robustness of vision-language-action (VLA) models concerns reliable robot behavior under changes to language, vision, sensors, tasks, and hardware.

See all 221 papers

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies
Mar 12, 2026Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu +9Robustness of VLA Models Adversarial Attacks on VLA Models
Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.
Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming
Apr 7, 2026Baoshun Tong, Haoran He, Ling Pan +2Adversarial Attacks on VLA Models Robustness of VLA Models
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models
Aug 11, 2026Jiahui Han, Yuhui Yao, Xin Wang +6Adversarial Attacks Adversarial Attacks on VLA Models
1Xi’an Jiaotong University · 2Shanghai AI Laboratory · University of Science and Technology of China

Mar 12, 2026J/K move · Enter open · S save

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies

Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu, Varun Bhatt+8

Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.

Robustness of VLA Models Adversarial Attacks on VLA Models Quality-Diversity Optimization Instruction Generalization in VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have significant potential to enable general-purpose robotic systems for a range of vision-language tasks. However, the performance of VLA-based robots is highly sensitive to the precise wording of language instructions, and it remains difficult to predict when such robots will fail. We propose Quality Diversity (QD) optimization as a natural framework for red-teaming embodied models, and present Q-DIG (Quality Diversity for Diverse Instruction Generation), which performs red-teaming by scalably identifying diverse, natural language task descriptions that induce failures while remaining task-relevant. Q-DIG integrates QD techniques with Vision-Language Models (VLMs) to generate a broad spectrum of adversarial instructions that expose meaningful vulnerabilities in VLA behavior. Our results across multiple simulation benchmarks show that Q-DIG finds more diverse and meaningful failure modes compared to baseline methods, and that fine-tuning VLAs on the generated instructions improves task success rates. Furthermore, results from a user study highlight that Q-DIG generates prompts judged to be more natural and human-like than those from baselines. Finally, real-world evaluations of Q-DIG prompts show results consistent with simulation, and fine-tuning VLAs on the generated prompts further success rates on unseen instructions. Together, these findings suggest that Q-DIG is a promising approach for identifying vulnerabilities and improving the robustness of VLA-based robots. Our anonymous project website is at qdigvla.github.io.

Abstract

Robustness of VLA Models

Explore similar work

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies

Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming

Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models