Enhancing Visual Task-Redirection Attacks on Vision-Language-Action Models via Closed-Loop Optimization

Chunjin Ye; Guanying Chen; Jianjie Huang; Jianhao Chen; Quanlin Chen; Hanjie Zhao; Haozhe Liang; Tieyun Qian

cs.ROOct 16, 2026

Enhancing Visual Task-Redirection Attacks on Vision-Language-Action Models via Closed-Loop Optimization

Authors: Chunjin Ye, Guanying Chen, Jianjie Huang, Jianhao Chen, Quanlin Chen, Hanjie Zhao, Haozhe Liang, Tieyun Qian,

Organizations: Sun Yat-sen University, Shenzhen 518107, China · Zhongguancun Academy, Beijing 100094, China · Wuhan University, Wuhan 430072, China · Shenzhen Loop Area Institute, Shenzhen 518000, China ·

Abstract

Vision-language-action (VLA) models enable language-guided robot manipulation, but remain vulnerable to visual attacks that undermine user control over robot behavior. Prior studies optimize visual perturbations using fixed target supervision. However, the perturbation can change earlier actions and, in turn, the physical states reached later. As the attacked trajectory evolves, this supervision may no longer match the behavior needed at the states actually reached. Accurately matching these targets can therefore fail to advance the attacker-specified task. Even with suitable supervision, reducing average matching error does not ensure sustained task progress. The robot may still fail at a critical stage, and further updates can disrupt previously successful behavior. To address these limitations, we propose TRACO, a framework that enhances existing targeted visual attacks through closed-loop optimization. Its Execution-Driven Supervision rebuilds supervision for the states reached under attack to guide subsequent updates. Within this framework, Progress-Guided Selection focuses updates where progress stalls while maintaining supervision on the earlier behavior needed to reach that point. Experiments across VLA models and simulation environments show improved task-redirection success with TRACO, highlighting the security implications of closed-loop attack optimization.

Figures & tables

Fig. 1: Motivation for closed-loop attack optimization. (a) During offline optimization (top), the attacker learns a perturbation $\delta$ by matching target behavior on observations from a recorded target trajectory (yellow). At test time (bottom), the same perturbation changes the robot’s actions under the user’s original instruction and can lead to different states (purple). The fixed targets may then no longer provide the behavior needed to complete the target task. (b,c) A single-scene example illustrates this mismatch. Three perturbations learned offline achieve lower action MSE on the recorded observations than TRACO, but show larger trajectory deviations and fail during execution, whereas TRACO succeeds.

Fig. 2: Overview of TRACO. (a) Execution-Driven Supervision (EDS) records the states reached under the current perturbation. (b) Progress-Guided Selection (PGS) uses a successful target trajectory to select frontier and prefix states, helping advance the target task while preserving earlier behavior. EDS queries the frozen VLA under the target instruction to obtain supervision at these states. TRACO then updates the perturbation and repeats this cycle, using new rollouts to refresh supervision and guide subsequent updates.

		Global perturbations				Adversarial patches
		Seen		Unseen		Seen		Unseen
Model	Method	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$
PI	Random Noise	0.0	$-$ 0.6030	0.0	$-$ 0.6420	0.0	$-$ 0.6650	0.0	$-$ 0.6880
	Action Attack	54.8	0.3489	45.6	0.2530	56.0	0.3282	54.0	0.3308
	TRACO	89.2	0.6279	79.6	0.5334	69.6	0.4348	67.2	0.4458
GR00T	Random Noise	0.4	$-$ 0.5930	0.0	$-$ 0.8340	0.0	$-$ 0.6430	0.0	$-$ 0.6540

TABLE I: Task-redirection performance on LIBERO. Best results are highlighted in bold red.

Method	Seen		Unseen
Method	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$
Random Noise	0.0	$-$ 0.6748	1.6	$-$ 0.6039
Action Attack	8.8	$-$ 0.4456	9.6	$-$ 0.5474
+ TRACO	12.8	$-$ 0.3809	7.2	$-$ 0.4800
$\mathrm{TRAP}_{\text{CoT-only}}$	11.2	0.0602	10.4	$-$ 0.0501
+ TRACO	30.4	0.0980	20.0	0.0019

TABLE II: Adversarial patch attacks on InstructVLA in SimplerEnv. Best results are highlighted in bold red.

Fig. 3: Qualitative examples of task redirection. (a) Global perturbation on LIBERO. (b,c) Adversarial patches on LIBERO and SimplerEnv, respectively. Each image sequence shows execution with TRACO under the source instruction. The 3D plots compare clean source, clean target, and attacked trajectories.

EDS	PGS	Seen		Unseen
		ASR (%) $\uparrow$	$H$ $\uparrow$	ASR (%) $\uparrow$	$H$ $\uparrow$
Global
–	–	43.6	0.2723	35.2	0.1671
$\checkmark$	–	58.4	0.4415	54.8	0.3445
$\checkmark$	$\checkmark$	89.2	0.6279	79.6	0.5334
Patch

TABLE III: Ablation of EDS and PGS on PI. Without EDS, the initial observations and target supervision remain fixed. Both variants without PGS use random sampling with the same sample counts.

Fig. 4: Effect of supervision refresh on target action error. Results cover five tasks with 10 Unseen layouts each. For each task–layout pair, we collect one rollout under each perturbation and evaluate both perturbations on all recorded observations. Normalized-action MSE is averaged within each rollout and then equally across the two rollouts. (a) Distributions of the 50 mean MSE values for Fixed and EDS. Boxes show interquartile ranges and median lines. Diamonds mark means. (b) Sorted differences in mean MSE for each task–layout pair (Fixed

-

EDS). Positive values indicate lower error with EDS.

Fig. 5: End-effector position error with and without PGS. Each curve shows end-effector distances to the successful target reference for one optimization round. Panels (a) and (b) compare w/o PGS and TRACO on PI in the global setting. Open circles mark the first sampled position error above the 4 cm tolerance. Attack success denotes target-task completion.

Fig. 6: Progress gains and regressions with and without PGS. Bars show the mean percentages of progress increases, no changes, and decreases between consecutive optimization rounds. Progress is measured using the passing prefix under the position tolerance (4 cm), with target-task completion assigned full progress. Results are averaged across five tasks on PI in the global setting.

Supervision source	Seen		Unseen
Supervision source	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$
Historical sampling	36.4	0.2094	30.0	0.1100
Current round only	58.4	0.4415	54.8	0.3445

TABLE IV: Current-round and historical supervision on PI. Both variants disable PGS and use the same number of samples per update.

Fig. 7: Parameter sensitivity of TRACO on PI. (a) PGS position tolerance. (b) Global perturbation budget. (c) Patch area. All three studies use flow-time probes at 0.50 and 0.75, as detailed in Appendix -B .

Fig. 8: Physical setup and real-world task redirection with a printed TRACO patch. Both executions use the same source instruction.

LIBERO Global $\cdot$ Five-model average
		ASR (%) / $H$
Method	Split	Original	Paraphrased	Extra-Context
Action Attack	Seen	${46.9}/{0.2342}$	${48.5}/{0.2672}$	${45.4}/{0.2325}$
Action Attack	Unseen	${43.2}/{0.1758}$	${45.8}/{0.2127}$	${41.4}/{0.1701}$
TRACO	Seen	${{\color[rgb]{0.707,0.3516,0.3516}\mathbf{85.2}}}/{{\color[rgb]{0.707,0.3516,0.3516}\mathbf{0.5887}}}$	${{\color[rgb]{0.707,0.3516,0.3516}\mathbf{81.2}}}/{{\color[rgb]{0.707,0.3516,0.3516}\mathbf{0.5954}}}$	${{\color[rgb]{0.707,0.3516,0.3516}\mathbf{85.6}}}/{{\color[rgb]{0.707,0.3516,0.3516}\mathbf{0.6041}}}$
TRACO	Unseen	${75.9}/{0.4819}$	${76.0}/{0.5288}$	${77.0}/{0.5082}$

TABLE V: Task redirection under instruction variations.

Method	Seen		Unseen
Method	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$
$\mathrm{TRAP}_{\text{CoT-only}}$	0.8	$-$ 0.4393	2.4	$-$ 0.4706
+ TRACO	3.2	$-$ 0.3579	3.2	$-$ 0.3324
TRAP	9.6	$-$ 0.3847	7.2	$-$ 0.3855
+ TRACO	12.0	$-$ 0.2358	8.0	$-$ 0.3433

TABLE VI: Transferability of adversarial patches across InstructVLA variants.

Fig. 9: Example Seen and Unseen layouts in LIBERO and SimplerEnv.

Metric	LIBERO Global (PI)		SimplerEnv Patch (InstructVLA)
	Action Attack	TRACO	TRAP	TRACO
Total time (min)	$\approx\!396.3$	$\approx\!\mathbf{88.8}$	$\approx\!513.4$	$\approx\!\mathbf{353.1}$
Optimization time (min)	$\approx\!396.3$	$\approx\!50.0$	$\approx\!513.4$	$\approx\!280.8$
Rollout time (min)	$0.0$	$\approx\!38.8$	$0.0$	$\approx\!72.3$
Sample uses ( $\times 10^{3}$ )	$\approx\!32$	$\mathbf{\approx\!6.1}$	$\approx\!32$	$\mathbf{\approx\!19.1}$

TABLE VII: Average attack construction cost per task on LIBERO and SimplerEnv. Times include model loading and auxiliary processing.

Alignment	Seen		Unseen
Alignment	ASR (%) $\uparrow$	$H\uparrow$	ASR (%) $\uparrow$	$H\uparrow$
$\mathcal{T}=\{0.25\}$	88.0	0.6392	83.6	0.5969
$\mathcal{T}=\{0.50\}$	92.4	0.6588	84.4	0.5812
$\mathcal{T}=\{0.75\}$	91.6	0.6702	82.0	0.5790
$\mathcal{T}=\{0.25,0.50\}$	87.2	0.6330	78.4	0.5631
$\mathcal{T}=\{0.25,0.75\}$	93.6	0.6845	84.0	0.5830

TABLE VIII: Flow-time probe sensitivity on PI in the LIBERO global setting. Results are averaged over five tasks.

Source task	Target task
LIBERO: Global
Push plate to stove front	Open middle cabinet drawer
Open top drawer and place bowl inside	Place wine bottle on cabinet top
Place cream cheese in bowl	Turn on stove
Place bowl on plate	Place bowl on cabinet top
Place wine bottle on rack	Place bowl on stove

TABLE IX: Task pairs in LIBERO and SimplerEnv.

Condition	Instruction
LIBERO: Global
Original	push the plate to the front of the stove
Paraphrased	slide the plate to a position in front of the stove
Extra-Context	There are several objects in the workspace. Push the plate to the front of the stove.
SimplerEnv: Patch
Original	pick orange

TABLE X: Examples of source-instruction variants.

Fig. 10: Color calibration for physical patch deployment.

Model	VLM backbone	Action generation
PI	Qwen3-VL-4B-Instruct	Flow matching
GR00T	Qwen3-VL-4B-Instruct	Flow matching
FAST	Qwen2.5-VL-3B-Instruct	Autoregressive tokens
OFT	Qwen3-VL-4B-Instruct	Direct regression
LangForce	Qwen3-VL-4B-Instruct	Flow matching
InstructVLA	Eagle2-2B	Flow matching

TABLE XI: VLM backbones and action interfaces of the evaluated policies.

Adversarial Attacks on VLA Models

Adversarial attacks on vision-language-action (VLA) models study how crafted inputs, poisoned data, or physical cues can disrupt robot decisions and safety.

See all 32 papers

Trajectory-Level Redirection Attacks on Vision-Language-Action Models
Jun 11, 2026Gokul Puthumanaillam, Vardhan Dongre, Pranay Thangeda +3Language-Conditioned Robot Manipulation Adversarial Attacks on VLA Models
University of Illinois Urbana-Champaign · Amazon
Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models
Aug 11, 2026Jiahui Han, Yuhui Yao, Xin Wang +6Adversarial Attacks Adversarial Attacks on VLA Models
1Xi’an Jiaotong University · 2Shanghai AI Laboratory · University of Science and Technology of China
DropVLA: An Action-Level Backdoor Attack on Vision-Language-Action Models
Oct 13, 2025Zonghuan Xu, Jiayu Li, Yunhan Zhao +3Adversarial Attacks on VLA Models Robotic Manipulation
Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China. · Shanghai Key Laboratory of Multimodal Embodied AI, Shanghai, China. · City University of Hong Kong, Hong Kong SAR, China.

Jun 11, 2026J/K move · Enter open · S save

Trajectory-Level Redirection Attacks on Vision-Language-Action Models

Gokul Puthumanaillam, Vardhan Dongre, Pranay Thangeda, Hooshang Nayyeri+2

University of Illinois Urbana-Champaign · Amazon

Language-Conditioned Robot Manipulation Adversarial Attacks on VLA Models Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) policies bring natural language into closed-loop robot control, enabling robots to execute manipulation tasks directly from text instructions. The same interface gives text a recurring role in control because the prompt is reused at every replanning step, and each prompt-conditioned action changes the future observations on which the policy acts. Existing VLA attacks study adversarial prompts that elicit targeted low-level actions or make such actions persist across changing images. We identify a stronger trajectory-level failure mode: a prompt that still

\textit{appears}

to specify the intended task but redirects the final physical outcome. We mathematically formalize this setting as

\textit{command-preserving trajectory redirection}

, a prompt-only threat model in which the attacker chooses one prompt before the episode, all policy and environment components remain fixed, and the prompt must stay close to the benign instruction while omitting target words and correction language. To find such prompts, we introduce an on-policy prompt search method that uses rollouts to discover perturbations whose closed-loop behavior tracks a target task while satisfying the command-preserving constraints. Experiments in simulation and on hardware show that near-benign prompt perturbations can redirect VLA rollouts to attacker-specified targets. These results expose a trajectory-level vulnerability in VLA instruction grounding: text that appears to preserve the intended command can still give an adversary control over the robot's final physical outcome. Project website: https://vla-redirection-attack.github.io/

Abstract

Adversarial Attacks on VLA Models

Explore similar work

Trajectory-Level Redirection Attacks on Vision-Language-Action Models

Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models

DropVLA: An Action-Level Backdoor Attack on Vision-Language-Action Models