Organizations: Thomas Lord Department of Computer Science, University of Southern California. · Columbia University. · Adobe Research · Cornell University.
A long-standing goal in robotics is to develop generalist robots that can perform tasks across a wide range of settings, flexibly specified with images and text. Vision-Language-Action (VLA) models offer a promising path toward this goal by training on large collections of robot demonstrations. However, while VLAs perform well in-distribution, they often fail under minor test-time shifts, such as changes in object positions, lighting, textures, and camera poses. To address this, we propose Quality Diversity for Diverse Environment Generation (Q-DEG), a method for improving VLA robustness by generating adversarial environments and fine-tuning the VLA on its own successful rollouts. Q-DEG uses Quality Diversity (QD) optimization to find a set of high-failure environments, while promoting diversity using policy embeddings extracted from the VLA. These environments expose vulnerabilities that can be mitigated through self-imitation fine-tuning, thereby improving the VLA's robustness. In simulation, we conduct experiments across three representative VLA models, and our results indicate that Q-DEG improves robustness to out-of-distribution (OOD) adversarial environments compared to the base VLA and VLAs fine-tuned using data collected from two red-teaming baselines. Furthermore, we demonstrate that improving VLA robustness through Q-DEG facilitates sim-to-real transfer. Our anonymous project website is at: https://qd-env.github.io/qdeg/.
Figures & tables
Fig. 1: Q-DEG generates diverse failure-inducing environments by perturbing parameters such as object placement, background, and camera and light source poses. New perturbations are generated from previously discovered environments, which act as “stepping stones” that facilitate exploration.
Fig. 2: Overview of Q-DEG . a) CMA-ES [ 37 ] proposes environment parameters such as object locations, background, lighting, and camera pose; invalid proposals are repaired to restore task-relevant spatial relations. b) The VLA is rolled out in each proposed environment to obtain feasibility-aware failure rate P^fail and policy embeddings. c) Q-DEG adds each environment to the QD archive based on its policy embeddings, and rewards CMA-ES with its failure rate improvement ΔP^fail over predecessors with similar embeddings. d) Successful trajectories from archived environments are collected into a fine-tuning dataset for improving the VLA’s robustness.
π0 -FAST
OpenVLA-OFT
π0.5
Eval. set
base
rand
CMA-ES
Q-DEG
base
rand
CMA-ES
Q-DEG
base
rand
CMA-ES
Q-DEG
default
96.80
68.60
82.60
89.60
96.20
94.80
86.70
93.80
98.40
85.60
93.60
95.60
rand
29.15
39.40
37.79
48.73
56.25
82.29
68.75
78.13
77.38
77.98
83.33
88.09
CMA-ES
27.90
38.46
24.81
29.82
45.75
52.52
48.27
75.79
54.98
49.39
69.17
58.00
Q-DEG
54.83
51.28
36.89
56.13
47.19
50.77
55.61
69.13
38.03
61.82
70.45
81.81
average
52.17
49.44
45.52
56.07
61.35
70.10
64.83
79.21
67.20
68.70
79.14
80.88
TABLE I: VLA success rates on the default LIBERO-Spatial tasks and adversarial evaluation sets generated by randomization (rand), CMA-ES, and Q-DEG. Rows denote evaluation sets and columns denote VLA checkpoints.
Fig. 3: Number of feasibly adversarial environments (top) found by randomization , CMA-ES , and Q-DEG, and the corresponding average pairwise DINOv2 cosine distance among environments (bottom), plotted against the number of VLA rollouts. Shaded regions denote 95% confidence intervals aggregated over three independent trials.
Fig. 4: Success counts and final gripper positions when evaluating the base VLA and VLAs fine-tuned with randomization ( rand ), CMA-ES , and Q-DEG on real-world tasks. Successful trials are shown in green, and failed trials are shown in red.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Fig. 5: Variance explained by the first 10 principal components of raw policy embeddings extracted from π0 -FAST, OpenVLA-OFT, and π0.5 . Results are aggregated over three independent trials, each sampling 320 environments. Error bars denote 95% confidence intervals.
Vision-Language-Action (VLA) models have significant potential to enable general-purpose robotic systems for a range of vision-language tasks. However, the performance of VLA-based robots is highly sensitive to the precise wording of language instructions, and it remains difficult to predict when such robots will fail. We propose Quality Diversity (QD) optimization as a natural framework for red-teaming embodied models, and present Q-DIG (Quality Diversity for Diverse Instruction Generation), which performs red-teaming by scalably identifying diverse, natural language task descriptions that induce failures while remaining task-relevant. Q-DIG integrates QD techniques with Vision-Language Models (VLMs) to generate a broad spectrum of adversarial instructions that expose meaningful vulnerabilities in VLA behavior. Our results across multiple simulation benchmarks show that Q-DIG finds more diverse and meaningful failure modes compared to baseline methods, and that fine-tuning VLAs on the generated instructions improves task success rates. Furthermore, results from a user study highlight that Q-DIG generates prompts judged to be more natural and human-like than those from baselines. Finally, real-world evaluations of Q-DIG prompts show results consistent with simulation, and fine-tuning VLAs on the generated prompts further success rates on unseen instructions. Together, these findings suggest that Q-DIG is a promising approach for identifying vulnerabilities and improving the robustness of VLA-based robots. Our anonymous project website is at qdigvla.github.io.
Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.
Vision-Language-Action (VLA) models have achieved remarkable success in robotic manipulation. However, their robustness to linguistic nuances remains a critical, under-explored safety concern, posing a significant safety risk to real-world deployment. Red teaming, or identifying environmental scenarios that elicit catastrophic behaviors, is an important step in ensuring the safe deployment of embodied AI agents. Reinforcement learning (RL) has emerged as a promising approach in automated red teaming that aims to uncover these vulnerabilities. However, standard RL-based adversaries often suffer from severe mode collapse due to their reward-maximizing nature, which tends to converge to a narrow set of trivial or repetitive failure patterns, failing to reveal the comprehensive landscape of meaningful risks. To bridge this gap, we propose a novel Diversity-Aware Embodied Red Teaming (DAERT) framework, to expose the vulnerabilities of VLAs against linguistic variations. Our design is based on evaluating a uniform policy, which is able to generate a diverse set of challenging instructions while ensuring its attack effectiveness, measured by execution failures in a physical simulator. We conduct extensive experiments across different robotic benchmarks against two state-of-the-art VLAs, including π0 and OpenVLA. Our method consistently discovers a wider range of more effective adversarial instructions that reduce the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents and exposing critical safety blind spots before real-world deployment.
Baoshun Tong, Haoran He, Ling Pan +2
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China
Vision-Language-Action (VLA) models have shown strong capabilities in controlling robots across diverse manipulation tasks. However, their adversarial robustness remains largely underexplored, and exploiting this weakness can lead to physical-world harm. Existing attacks on VLA models often rely on pixel-space perturbations or white-box access, resulting in noticeable artifacts and limited deployability in real-world robotic systems. In this work, we propose DURA, a diffusion-based unrestricted robotic attack that generates visually natural adversarial patches for VLA models. DURA supports both white-box and black-box attack settings, where the black-box setting requires only the predicted actions of the victim model. By optimizing along the latent trajectory of a pretrained diffusion model, DURA generates visually natural patches while steering the robot toward attacker-specified target actions. Extensive experiments in both simulation and the real physical world show that DURA consistently outperforms existing methods. Our findings expose a safety risk for physically deployed VLA models and call for stronger defenses.
Jiahui Han, Yuhui Yao, Xin Wang +6
1Xi’an Jiaotong University · 2Shanghai AI Laboratory · University of Science and Technology of China