TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches

Zhengxian Huang; Wenjun Zhu; Haoxuan Qiu; Xiaoyu Ji; Wenyuan Xu

cs.CRMar 24, 2026

TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches

Authors: Zhengxian Huang, Wenjun Zhu, Haoxuan Qiu, Xiaoyu Ji, Wenyuan Xu

Organizations: Zhejiang University, Hangzhou, China · Harbin Institute of Technology, Harbin, China

Abstract

By integrating Chain-of-Thought (CoT) reasoning, Vision-Language-Action (VLA) models have demonstrated strong capabilities in robotic manipulation, particularly by improving generalization and interpretability. However, the security of CoT-based reasoning mechanisms remains largely unexplored. In this paper, we show that CoT reasoning introduces a novel attack vector for targeted behavior hijacking--for example, causing a robot to mistakenly deliver a knife to a person instead of an apple--without modifying the user's instruction. We first provide empirical evidence that CoT strongly governs action generation, even when it is semantically misaligned with the input instructions. Building on this observation, we propose TRAP, the first targeted behavior-hijacking adversarial attack against CoT-reasoning VLA models. By targeting the reasoning-to-action pathway, TRAP uses an adversarial patch (e.g., a tablecloth placed on the table) to steer intermediate CoT reasoning and downstream actions toward adversary-defined behaviors. Extensive evaluations on three representative reasoning VLAs, spanning distinct CoT reasoning mechanisms, demonstrate the effectiveness of TRAP. Notably, we implemented the patch by printing it on paper in a real-world setting. Our findings highlight the urgent need to secure CoT reasoning in VLA systems. The project page is available at https://zhengxian-huang.github.io/TRAP-website/.

Figures & tables

Figure 1: Illustration of existing representative paradigms of reasoning VLAs.

Model	$\text{TSR}_{\text{ori}}$	$\text{TSR}_{m}$	$\text{Sim}_{m}$	$\text{TSR}_{s^{i}}$	$\text{TSR}_{s^{j}}$	Score
InstructVLA	45.23%	39.95%	0.3169	16.42%	13.43%	0.0508 $\pm$ 0.2392
MolmoAct	50.40%	17.35%	0.1805	12.94%	16.26%	0.0520 $\pm$ 0.1964
GraspVLA	94.20%	92.60%	0.3990	94.20%	0.00%	0.5278 $\pm$ 0.1696

Table 1: Instruction masking and CoT shuffling analysis on reasoning VLAs. TSR denotes Task Success Rate. Subscripts

m,s^{i},s^{j}

denote mask and shuffling settings respectively.

Figure 2: Overview of the TRAP attack framework. The adversary places an adversarial patch in the scene to corrupt the VLA’s intermediate CoT, causing the model to execute an attacker-specified behavior while the user instruction remains benign. The patch is optimized with two groups of objectives: attack-effectiveness losses, including CoT hijacking and action losses, and stealthiness losses, including content and TV losses.

Model	Arch.	CoT Type
MolmoAct	Integrated (Disc.)	Trace, Depth
GraspVLA	Integrated (Cont.)	Bbox, Grasp pose
InstructVLA	Hierarchical (Cont.)	Subtasks

Table 2: Description of victim VLAs. “Disc.” denotes Discrete, “Cont.” denotes Continuous.

Figure 3: Qualitative results of TRAP attack. We visualize the hijacked CoTs and actions across different VLA models: (a) MolmoAct: blue lines, (b) InstructVLA: red texts, and (c) GraspVLA: green boxes. The visualized 3D trajectories demonstrate that TRAP effectively hijacks various VLAs to execute the attacker’s target behaviors.

Method	MolmoAct		InstructVLA		GraspVLA		Average
Method	ASR(%)	Score	ASR(%)	Score	ASR(%)	Score	ASR(%)	Score
Random Noise	0.97	-0.3769	3.39	-0.3280	0.32	-0.3064	1.56	-0.3371
Action Attack	9.68	0.1282	6.77	-0.2736	0.00	-0.2948	5.48	-0.1467
$\text{TRAP}_{\text{CoT-only}}$	49.52	0.3422	4.03	-0.0332	69.04	0.3898	40.86	0.2329
TRAP	48.06	0.3904	33.71	0.1724	75.84	0.4253	52.54	0.3294
$\Lsh$ Unseen layouts	48.00	0.1828	31.60	0.1307	75.20	0.4024	51.60	0.2386

Table 3: Main results: Overall performance of TRAP attacks on reasoning VLAs across all tasks.

\text{TRAP}_{\text{CoT-only}}

denotes our reasoning-only variant that optimizes only the CoT hijacking loss, while TRAP jointly optimizes CoT and action objectives.

Panel of Figure 4 — Figure 4: Illustration of the attention shift. The $19^{th}$ layer attention of MolmoAct is visualized. The patch is optimized to redirect the model’s execution from the benign instruction “pick orange” to the adversarial target “pick coke can”.

	MolmoAct		InstructVLA
	ASR (%)	Score	ASR (%)	Score
Original	72.0	0.34	67.4	0.56
Paraphrasing	70.6	0.32	25.1	0.09
Extra-Context	60.0	0.35	44.8	0.32

Table 4: Instruction analysis: Performance under Paraphrasing and Extra-Context instruction variations.

Figure 5: Impact of patch size on attack effectiveness.

Model Pair	Variant	ASR (%) $\uparrow$	Score $\uparrow$
MolmoAct	Optimized	48.05	0.329
	Transfer	18.39	0.170
InstructVLA	Optimized	33.10	0.160
	Transfer	9.20	-0.170

Table 5: Transferability results. Performance comparison between source models (optimized) and target variants (transfer).

Panel of Figure 6 — Figure 6: Physical deployment settings for real-world evaluation. We evaluate (a) an occlusion-free tabletop patch and (b) an object-occluded tablecloth patch. The benign instruction is “pick up carrot”, while the target is “pick up knife”; red waypoints denote the hijacked trajectory. Videos of the real-world experiments are available on our project homepage .

Figure 7: Impact of optimization methods on real-world adversarial patches.

CoT	BCR	ARR	Time
Bbox	87.00% (+12.00%)	92.50% (-1.30%)	201.67 ms
Trace	77.01% (-5.79%)	87.42% (+8.02%)	114.35 ms
Subtask	86.07% (-9.43%)	91.94% (-8.06%)	2.62 ms

Table 6: Lightweight defense performance. Parentheses indicate changes relative to the GPT-5-based method. BCR: Benign Consistent Rate; ARR: Attack Reject Rate. Parentheses indicate relative changes compared with GPT-5.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Visualization of hijacking score distribution across different VLAs under the shuffling setting.

Task	Object Pairs	Instruction	Selected for Adv.	Motion Primitive
1	green cube & yellow cube	pick green cube		Pick
		pick yellow cube	✓
2	coke can & pepsi can	put coke can on plate		Place
		put pepsi can on plate
3	apple & orange	pick apple	✓	Pick
		pick orange	✓

Appendix

Table 7: Overview of curated SimplerEnv tasks

Task ID	Object Pairs	Instruction	Selected for Adv.
libero_10_0	tomato sauce can & milk	pick up tomato sauce can	✓
		pick up milk
libero_10_3	akita black bowl & wine bottle	pick up akita black bowl
		pick up wine bottle
libero_10_4	black book & white yellow mug	pick up black book
		pick up white yellow mug

Appendix

Table 8: Overview of curated LIBERO tasks

Figure 9: Representative layout variations across tasks.

Setting	Instructions
Extra-Context	carefully pick the orange; please pick orange; go ahead and pick orange; pick the orange on the table; pick orange in front of you
Paraphrasing	could you pick up the orange?; grasp the orange; grab the orange; pick the orange; pick up the orange

Appendix

Table 9: Detailed instructions used for instruction analysis.

Type	Ratio	MolmoAct		InstructVLA		GraspVLA
Type	Ratio	ASR (%) $\uparrow$	Score $\uparrow$	ASR (%) $\uparrow$	Score $\uparrow$	ASR (%) $\uparrow$	Score $\uparrow$
Edge	10%	44.4	0.221	62.9	0.505	8.6	-0.251
	25%	0.0	-0.363	17.1	-0.188	0.0	-0.406
	50%	0.0	-0.369	0.0	-0.358	0.0	-0.405
Center	10%	2.8	-0.320	20.0	-0.094	0.0	-0.410
	25%	0.0	-0.379	8.6	-0.162	0.0	-0.427

Appendix

Table 10: Performance under different occlusion settings.

Model	Setting	Instructions
InstructVLA	Scene-Irrelevant	move the yellow object; pick up the red block; place the item on the left; put the object in the container
InstructVLA	Scene-Relevant	grasp coke can; move orange near coke can; pick cube; pick orange
MolmoAct	Scene-Irrelevant	move the yellow object; pick up the red block; place the item on the left; put the object in the container
MolmoAct	Scene-Relevant	grasp apple; move orange near coke can; pick orange; pick pepsi can
GraspVLA	Scene-Irrelevant	pick up apple; pick up box; pick up nutcracker; pick up pineapple
GraspVLA	Scene-Relevant	pick up carrot; pick up drill; pick up mouse; pick up pumpkin

Appendix

Table 11: Detailed instruction settings for universal patch analysis across different models.

Victim Model	Scene-Rel.		Scene-Irrel.
	ASR (%)	Sim.	ASR (%)	Sim.
MolmoAct	0.7	0.10	35.7	0.25
InstructVLA	25.0	0.23	65.0	0.45
GraspVLA	6.0	0.08	21.0	0.14

Appendix

Table 12: Universal patch analysis: Performance under Scene-Relevant and Scene-Irrelevant instruction variations.

Figure 10: Illustration of the CLIP-based objective for adversarial patch optimization. (a) The CLIP loss aligns the image embedding of an adversarial patch with the embedding of a target text prompt. (b,c) Examples of optimized adversarial patches for the prompts “a realistic flower” and “a photo of tree”, respectively.

Figure 11: t -SNE visualization of latent features.

Layer	Type	Input Dim.	Output Dim.
1	Linear	3	16
2	LeakyReLU	16	16
3	Linear	16	16
4	LeakyReLU	16	16
5	Linear	16	3

Appendix

Table 13: Architecture of the color calibration network.

Panel of Figure 13 — Figure 13: Illustration of the effect of color calibration. We compare the real image, the output without color calibration, and the output with color calibration. Color calibration reduces the visual gap between the generated output and the real observation, which is critical for real-world effectiveness.

TV Loss weight $\lambda_{3}$	0.5	1	10	100
High-frequency ratio $\downarrow$	0.283	0.253	0.016	0.004
Relative drop(%) $\uparrow$	0.0	10.6	94.3	98.7

Appendix

Table 14: Results of TV regularization on high-frequency artifacts. Relative drop is computed with respect to

\lambda_{3}=0.5

Figure 14: Detailed occlusion-free deployment result.

Figure 15: Illustration of representative failure cases. (a) Out-of-distribution observations reduce the transferability of the optimized adversarial patch, especially when large robot-arm motions introduce substantial visual changes. (b) The adversarial patch may induce an attraction effect on CoT prediction, causing GraspVLA to localize the patch itself rather than the intended target object.

Panel of Figure 16 — Figure 16: Illustration of evaluation under different deployment variants.

Figure 17: Illustration of another hazardous task. The benign user instruction is “pick up mouse”, while the adversary aims to hijack the robot toward “pick up scissors”.

Adversarial Attacks on VLA Models

Adversarial attacks on vision-language-action (VLA) models study how crafted inputs, poisoned data, or physical cues can disrupt robot decisions and safety.

See all 32 papers

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2Reasoning for VLA Models ICL for Robotics
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)
TrapVLA: Trapping Vision-Language-Action Models in Configured Failure Modes
Aug 27, 2026Jun-Hui Liu, Kun-Yu Lin, Yi-Lin Wei +9Backdoor Attacks on VLA Models Adversarial Attacks on VLA Models
School of Computer Science and Engineering, Sun Yat-sen University, China · Pengcheng Laboratory, China · The University of Hong Kong, China +1
Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models
Aug 11, 2026Jiahui Han, Yuhui Yao, Xin Wang +6Adversarial Attacks Adversarial Attacks on VLA Models
1Xi’an Jiaotong University · 2Shanghai AI Laboratory · University of Science and Technology of China

Aug 6, 2026J/K move · Enter open · S save

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Jiarui Yang, Wen Huang, Jiale Zhang, Maowei Hu+1

Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

Reasoning for VLA Models ICL for Robotics Continual Learning for VLA Models Robotic Manipulation Vision-Language-Action Models VLMs for Robotics+4

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have become the dominant recipe for generalist manipulation, yet they are almost universally trained by behavior cloning: a policy imitates expert action chunks conditioned on a static image and a fixed instruction. A natural remedy is to inject explicit reasoning through textual chain-of-thought (CoT). We show, both empirically and analytically, that free-form textual CoT degrades low-level control: the reasoning it produces is ungrounded, its latency breaks closed-loop timing, and, crucially, the reasoning and action tokens are optimized against conflicting objectives so that the policy learns to narrate rather than to act. We argue that what a VLA needs is not the ability to generate language, but the ability to consume grounded language. To this end we introduce \ourmethod{}, a framework that endows a VLA with language competence through (i) in-context post-training, in which perceptual evidence is injected as structured context and the model is supervised only on actions, and (ii) an agentic tool-use interface, in which the policy queries open-vocabulary detectors, monocular depth, and a vision--language model to actively acquire task-relevant information. Rather than emitting a single templated caption, our data engine produces diverse, paraphrased, and evidence-conditioned spatial descriptions, so that the policy learns to interpret language it has never seen verbatim. Across the RoboCasa-GR1, SimplerEnv, and LIBERO simulation benchmarks, together with 8 real-world robot manipulation tasks, our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches under matched configurations.

Abstract

Adversarial Attacks on VLA Models

Explore similar work

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

TrapVLA: Trapping Vision-Language-Action Models in Configured Failure Modes

Hidden in Plain Sight: Diffusion-Based Unrestricted Robotic Attacks on Vision-Language-Action Models