SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Models

Xiyang Wu; Guangyao Shi; Qingzi Wang; Zongxia Li; Amrit Singh Bedi; Dinesh Manocha

cs.ROMar 26, 2026

SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Models

Authors: Xiyang Wu, Guangyao Shi, Qingzi Wang, Zongxia Li, Amrit Singh Bedi, Dinesh Manocha

Organizations: University of Maryland, College Park, MD, USA · University of Southern California, Los Angeles, CA, USA · University of Central Florida, Orlando, FL, USA

Abstract

Vision-language-action (VLA) models enable robots to follow natural-language instructions grounded in visual observations, but the instruction channel also introduces a critical vulnerability: small textual perturbations can alter downstream robot behavior. Systematic robustness evaluation therefore requires a black-box attacker that can generate minimal yet effective instruction edits across diverse VLA models. To this end, we present SABER, an agent-centric approach for automatically generating instruction-based adversarial attacks on VLA models under bounded edit budgets. SABER uses a GRPO-trained ReAct attacker to generate small, plausible adversarial instruction edits using character-, token-, and prompt-level tools under a bounded edit budget that induces targeted behavioral degradation, including task failure, unnecessarily long execution, and increased constraint violations. On the LIBERO benchmark across six state-of-the-art VLA models, SABER reduces task success by 20.6%, increases action-sequence length by 55%, and raises constraint violations by 33%, while requiring 21.1% fewer tool calls and 54.7% fewer character edits than strong GPT-based baselines. These results show that small, plausible instruction edits are sufficient to substantially degrade robot execution, and that an agentic black-box pipeline offers a practical, scalable, and adaptive approach for red-teaming robotic foundation models. The codebase is publicly available at https://github.com/wuxiyang1996/SABER.

Figures & tables

Fig. 1 : SABER: An agent-centric black-box pipeline for stealthy, automated instruction-based attacks on VLAs. VLA models for robot manipulation are expected to achieve high task success, efficient action planning and execution, and safe behavior under physical constraints. However, even small instruction perturbations can induce VLA malfunctions. SABER ( Dashed Box ) applies stealthy edits to manipulation instructions through a ReAct-style tool-calling protocol ( Red Box ) with a two-stage FIND $\rightarrow$ APPLY workflow, using a perturbation toolbox ( Blue Box ) spanning character-, token-, and prompt-level attacks. After the perturbed instruction is fed to the target VLA, the robot exhibits degraded behaviors aligned with the attack objective, including task failure, action inflation, and increased constraint violations.

Fig. 2 : Overview of SABER. For each LIBERO task, we maintain two contrastive rollouts under a frozen target VLA. A clean baseline rollout ( Green Box ) is first executed and cached as reference. For the attack rollout, the instruction is passed to a red-team agent ( Red Box ), which uses an LLM backbone to reason over the instruction and available tools, then performs multi-turn FIND $\rightarrow$ APPLY edits in a ReAct-style loop. The perturbation toolbox ( Blue Box ) returns edited instructions from target positions and local context. The target VLA then executes the perturbed instruction to produce the attack rollout ( Yellow Box ). The reward function ( Purple Box ) compares the clean and attack rollouts, together with the agent’s tool-use traces, to compute rewards from task outcome, action inflation, constraint violations, and stealth signals, including character edits and tool calls.

Fig. 3 : Two-stage training procedure. We cold-start by caching clean baseline rollouts from target VLAs ( Orange ) and collecting initial attack trajectories with a frozen red-team agent ( Red ) via lightweight random exploration over tool-calling chains. These rollouts form the cold-start dataset for SFT before GRPO training. We then perform agentic RL in interactive scenarios, where the red-team agent attacks target VLAs through tool calling and learns from reward feedback ( Purple ) computed by comparing clean and attack rollouts, together with the agent’s tool-use traces, under different attack objectives.

	LIBERO-Spatial			LIBERO-Object			LIBERO-Goal			LIBERO-Long			Overall
Victim VLA	Base TER $\uparrow$	Attack TER $\downarrow$	ASR $\uparrow$	Base TER $\uparrow$	Attack TER $\downarrow$	ASR $\uparrow$	Base TER $\uparrow$	Attack TER $\downarrow$	ASR $\uparrow$	Base TER $\uparrow$	Attack TER $\downarrow$	ASR $\uparrow$	Base TER $\uparrow$	Attack TER $\downarrow$	ASR $\uparrow$
$\pi_{0}$ -LIBERO [ 22 ]	100.0	86.7	13.3	100.0	80.0	20.0	100.0	53.3	46.7	66.7	40.0	26.7	91.7	65.0	26.7
$\pi_{0.5}$ [ 22 ]	100.0	93.3	6.7	93.3	93.3	0.0	100.0	53.3	46.7	93.3	80.0	13.3	96.7	80.0	16.7
GR00T-N1.5 [ 38 ]	100.0	93.3	6.7	100.0	100.0	0.0	100.0	53.3	46.7	93.3	86.7	6.6	98.3	83.3	15.0
X-VLA [ 39 ]	93.3	80.0	13.3	93.3	73.3	20.0	100.0	66.7	33.3	60.0	46.7	13.3	86.7	66.7	20.0
InternVLA-M1 [ 40 ]	93.3	86.7	6.6	100.0	93.3	6.7	100.0	46.7	53.3	86.7	73.3	13.4	95.0	75.0	20.0

TABLE I : LIBERO category-level attack results across victim VLA models (Task Failure). Columns are grouped by the four LIBERO suites ( Spatial , Object , Goal , Long ) and Overall . We report Base TER (no-attack task execution rate), Attack TER (task execution rate under attack), and ASR (attack success rate for task failure , computed as

\text{Base TER}-\text{Attack TER}

, in %). The Average row reports the mean over victim models for each column.

	LIBERO-Spatial			LIBERO-Object			LIBERO-Goal			LIBERO-Long			Overall
Victim VLA	Base $\|\mathbf{a}\|$ $\downarrow$	Attack $\|\mathbf{a}\|$ $\uparrow$	AIR $\Delta\|\mathbf{a}\|$ $\uparrow$	Base $\|\mathbf{a}\|$ $\downarrow$	Attack $\|\mathbf{a}\|$ $\uparrow$	AIR $\Delta\|\mathbf{a}\|$ $\uparrow$	Base $\|\mathbf{a}\|$ $\downarrow$	Attack $\|\mathbf{a}\|$ $\uparrow$	AIR $\Delta\|\mathbf{a}\|$ $\uparrow$	Base $\|\mathbf{a}\|$ $\downarrow$	Attack $\|\mathbf{a}\|$ $\uparrow$	AIR $\Delta\|\mathbf{a}\|$ $\uparrow$	Base $\|\mathbf{a}\|$ $\downarrow$	Attack $\|\mathbf{a}\|$ $\uparrow$	AIR $\Delta\|\mathbf{a}\|$ $\uparrow$
$\pi_{0}$ -LIBERO [ 22 ]	119.3	220.7	1.85	139.2	233.9	1.68	101.3	230.0	2.27	363.0	457.4	1.26	180.7	285.5	1.58
$\pi_{0.5}$ [ 22 ]	112.2	173.9	1.55	151.1	226.7	1.50	105.1	196.5	1.87	346.5	391.5	1.13	178.7	247.2	1.38
GR00T-N1.5 [ 38 ]	133.7	514.7	3.85	129.7	220.5	1.70	98.3	378.5	3.85	343.5	346.9	1.01	176.3	365.2	2.07
X-VLA [ 39 ]	157.8	189.4	1.20	189.7	187.8	0.99	126.5	261.9	2.07	431.3	504.6	1.17	226.3	285.9	1.26
InternVLA-M1 [ 40 ]	114.3	192.0	1.68	143.8	204.2	1.42	95.1	255.8	2.69	320.9	327.3	1.02	168.5	244.8	1.45

TABLE II : LIBERO action-length statistics under attack across victim VLA models. Columns are grouped by the four LIBERO suites ( Spatial , Object , Goal , Long ) and Overall . We report Base

|\mathbf{a}|

(average baseline action-sequence length), Attack

|\mathbf{a}|

(average action-sequence length under attack), and AIR via action inflation ratio

\Delta|\mathbf{a}|=|\mathbf{a}_{\text{attack}}|/|\mathbf{a}_{\text{base}}|

. The Average row reports the mean over victim models for each column.

	LIBERO-Spatial			LIBERO-Object			LIBERO-Goal			LIBERO-Long			Overall
Victim VLA	Base CV $\downarrow$	Attack CV $\uparrow$	CVI $\Delta\text{CV}$ $\uparrow$	Base CV $\downarrow$	Attack CV $\uparrow$	CVI $\Delta\text{CV}$ $\uparrow$	Base CV $\downarrow$	Attack CV $\uparrow$	CVI $\Delta\text{CV}$ $\uparrow$	Base CV $\downarrow$	Attack CV $\uparrow$	CVI $\Delta\text{CV}$ $\uparrow$	Base CV $\downarrow$	Attack CV $\uparrow$	CVI $\Delta\text{CV}$ $\uparrow$
$\pi_{0}$ -LIBERO [ 22 ]	550.7	326.2	0.59	711.5	838.3	1.18	309.6	624.5	2.02	1039.6	1269.1	1.22	652.9	764.5	1.17
$\pi_{0.5}$ [ 22 ]	570.9	549.8	0.96	681.4	699.0	1.03	260.3	439.1	1.69	863.9	1336.2	1.55	594.1	756.0	1.27
GR00T-N1.5 [ 38 ]	599.9	702.7	1.17	644.1	595.7	0.92	258.9	862.3	3.33	918.8	778.1	0.85	605.4	734.7	1.21
X-VLA [ 39 ]	838.3	1356.3	1.62	828.3	725.7	0.88	347.1	885.0	2.55	1145.3	1171.0	1.02	789.8	1034.5	1.31
InternVLA-M1 [ 40 ]	475.9	130.3	0.27	639.3	493.0	0.77	232.9	495.3	2.13	681.9	1550.3	2.27	507.5	667.2	1.31

TABLE III : LIBERO category-level constraint violation results across victim VLA models. Columns are grouped by the four LIBERO suites ( Spatial , Object , Goal , Long ) and Overall . We report Base CV (average constraint violations per episode without attack), Attack CV (average constraint violations per episode under attack), and CVI (constraint violation inflation)

\Delta\text{CV}=\text{CV}_{\text{attack}}/\text{CV}_{\text{base}}

. The Average row reports the mean over victim models for each column.

	Spatial		Object		Goal		Long		Overall
Objective	Tool Calls $\downarrow$	Char Edits $\downarrow$	Tool Calls $\downarrow$	Char Edits $\downarrow$	Tool Calls $\downarrow$	Char Edits $\downarrow$	Tool Calls $\downarrow$	Char Edits $\downarrow$	Tool Calls $\downarrow$	Char Edits $\downarrow$
Task Failure	2.97	13.4	3.34	13.2	2.80	10.3	2.97	15.1	3.02	13.0
Action Inflation	2.98	126.7	3.26	114.0	3.36	130.3	3.05	117.3	3.16	122.1
Constraint Violation	3.34	89.3	2.34	65.0	1.67	50.0	2.34	51.7	2.42	64.0

TABLE IV : Average tool usage by objective. Mean Tool Calls and Char Edits per episode for each LIBERO suite and averaged over victim VLA models.

Models	Stealthy		Objective
	Tool Calls $\downarrow$	Char Edit $\downarrow$	ASR $\uparrow$	AIR $\Delta\|\mathbf{a}\|$ $\uparrow$	CVI $\Delta\text{CV}$ $\uparrow$
GPT-5 mini	3.93	168.8	14.5	1.37	1.25
SABER	3.10	76.46	16.7	1.38	1.27

TABLE V : Objective-level summary of attack effectiveness and stealth. We compare a frozen GPT-5 mini attacker (same tool-calling interface) against SABER on stealthy and Objective metrics. SABER achieves comparable or better objective performance while using substantially fewer tool calls and character edits, indicating more efficient, high-leverage perturbations learned via RL.

Models	Stealthy		Objective
	Tool Calls $\downarrow$	Char Edit $\downarrow$	Base TER $\uparrow$	Attack TER $\downarrow$	ASR $\uparrow$
GRPO Only	2.76	11.78	96.7	88.0	8.7
SFT + GRPO	3.10	76.46	96.7	80.0	16.7

TABLE VI : Necessity of cold-start. We ablate agent training with cold-start data ( SFT + GRPO ) versus without cold-start data ( GRPO only) under the Task Failure objective. Cold-start data is crucial for stabilizing RL training and preventing policy degradation.

Adversarial Attacks on VLA Models

Adversarial attacks on vision-language-action (VLA) models study how crafted inputs, poisoned data, or physical cues can disrupt robot decisions and safety.

See all 32 papers

Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models
Aug 11, 2026Jiahui Han, Yuhui Yao, Xin Wang +6Adversarial Attacks Adversarial Attacks on VLA Models
1Xi’an Jiaotong University · 2Shanghai AI Laboratory · University of Science and Technology of China
Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics
Sep 30, 2026Songhua Yang, Ziyu Liu, Yuanwei Liu +5Adversarial Attacks on VLA Models Robotic Manipulation
School of Computer Science, Wuhan University. · School of Robotics, Wuhan University. · School of Power and Mechanical Engineering, Wuhan University.
Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies
Mar 12, 2026Siddharth Srikanth, Freddie Liang, Ya-Chuan Hsu +9Robustness of VLA Models Adversarial Attacks on VLA Models
Thomas Lord Department of Computer Science, University of Southern California · Sony AI · Sibley School of Mechanical and Aerospace Engineering, Cornell University.

Aug 11, 2026J/K move · Enter open · S save

Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models

Jiahui Han, Yuhui Yao, Xin Wang, Jiafei Cao+5

1Xi’an Jiaotong University · 2Shanghai AI Laboratory · University of Science and Technology of China

Adversarial Attacks Adversarial Attacks on VLA Models Black-Box Adversarial Attacks Physical Adversarial Attacks

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have shown strong capabilities in controlling robots across diverse manipulation tasks. However, their adversarial robustness remains largely underexplored, and exploiting this weakness can lead to physical-world harm. Existing attacks on VLA models often rely on pixel-space perturbations or white-box access, resulting in noticeable artifacts and limited deployability in real-world robotic systems. In this work, we propose DURA, a diffusion-based unrestricted robotic attack that generates visually natural adversarial patches for VLA models. DURA supports both white-box and black-box attack settings, where the black-box setting requires only the predicted actions of the victim model. By optimizing along the latent trajectory of a pretrained diffusion model, DURA generates visually natural patches while steering the robot toward attacker-specified target actions. Extensive experiments in both simulation and the real physical world show that DURA consistently outperforms existing methods. Our findings expose a safety risk for physically deployed VLA models and call for stronger defenses.

Abstract

Adversarial Attacks on VLA Models

Explore similar work

Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models

Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics

Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policies