Beyond the Patch: Exploring Vulnerabilities of Visuomotor Policies via Viewpoint-Consistent 3D Adversarial Object

Chanmi Lee; Minsung Yoon; Woojae Kim; Sebin Lee; Sung-eui Yoon

cs.ROMar 5, 2026

Beyond the Patch: Exploring Vulnerabilities of Visuomotor Policies via Viewpoint-Consistent 3D Adversarial Object

Authors: Chanmi Lee, Minsung Yoon, Woojae Kim, Sebin Lee, Sung-eui Yoon

Organizations: The School of Computing at the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, 34141, Republic of Korea

Abstract

Neural network-based visuomotor policies enable robots to perform manipulation tasks but remain susceptible to perceptual attacks. For example, conventional 2D adversarial patches are effective under fixed-camera setups, where appearance is relatively consistent; however, their efficacy often diminishes under dynamic viewpoints from moving cameras, such as wrist-mounted setups, due to perspective distortions. To proactively investigate potential vulnerabilities beyond 2D patches, this work proposes a viewpoint-consistent adversarial texture optimization method for 3D objects through differentiable rendering. As optimization strategies, we employ Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, exploiting distance-dependent frequency characteristics to induce textures effective across varying camera-object distances. We further integrate saliency-guided perturbations to redirect policy attention and design a targeted loss that persistently drives robots toward adversarial objects. Our comprehensive experiments show that the proposed method is effective under various environmental conditions, while confirming its black-box transferability and real-world applicability.

Figures & tables

Fig. 1: Visuomotor policy deception using a 3D adversarial object. (a) The policy successfully guides the robot to its target $O_{\text{goal}}$ . (b) Our adversarial object $O_{\text{adv}}$ manipulates the visual input, compelling the policy to misguide the robot towards itself instead of the true target $O_{\text{goal}}$ .

Fig. 2: Apparent size comparison of a 3D adversarial object and a 2D patch. The 2D patch significantly shrinks and distorts, especially at large viewing angles $\phi$ , unlike the more stable 3D object.

Fig. 3: Overview of the proposed method. (a) Coarse-to-Fine (C2F) Pose Scheduling: From a set of task-feasible initial configurations, poses where the original policy succeeds, we schedule viewpoint sampling using a distance-based Beta distribution. The scheduler progressively shifts focus from distant (Coarse stage) to closer (Fine stage) viewpoints. (b) 3D Adversarial Object Optimization Pipeline: Guided by the Expectation over Transformation (EOT) framework, the pipeline optimizes the adversarial texture $T$ through short policy rollouts from each initial pose $\tau_{i}$ . In each step, the policy $\pi_{\omega}$ processes a composite image $I_{\text{adv}}$ (formed from $I_{\text{diff}}$ and $I_{\text{sim}}$ ) to output an action. A targeted adversarial loss is then computed from the resulting action, guiding the robot toward the adversarial object $O_{\text{adv}}$ . The total loss, reflecting actual image-action pairs from the rollout, is backpropagated to update the texture $T$ .

Fig. 4: Visualization of texture update patterns under different scheduling strategies: (a) Coarse-to-Fine, (b) Fine-to-Coarse, (c) Non-staged, (d) Coarse-only, (e) Fine-only.

Panel of Figure 5 — Fig. 5: Comparison of policy saliency maps: (a) before vs. (b) after the 3D adversarial attack.

Sec.	Condition	T-ASR	ASR	Sec.	Condition	T-ASR	ASR
	Ours	79.80	81.12	D.3	Light: Bright	77.04	84.70
D.1	Shape: Dog	60.04	65.80		Light: Dim	67.80	72.20
D.1	Shape: Duck	63.20	68.32		Add Noise	75.57	80.41
D.2	Stereo Cam	63.28	78.91		Bkg Varied	77.91	85.64

TABLE III: Attack Generalization and Robustness Analysis (%)

Panel of Figure 7 — Fig. 6: Visualization of Sec. IV-D configurations: (a) D.1 . Dog shape, (b) D.1 . Duck shape, (c) D.2 . Stereo camera setup.

Policy	T-ASR	ASR	Policy	T-ASR	ASR
ResNet18 †	79.08	81.12	Inception-v3	56.68	70.28
VGG16	59.68	70.08	ResNet34	71.20	78.32

TABLE IV: Attack Transferability Across Visual Policy Architectures (Source: ResNet18 † , (%))

Panel of Figure 9 — Fig. 7: Visualization of adversarial textures on real-world objects: (a) Cube in EnvA; (b) Cylinder in EnvA; (c) Cube in EnvB; (d) Cylinder in EnvB.

Env	Shape of $O_{\text{adv}}$	Simulation		Real-World		Note
Env		T-ASR	ASR	T-ASR	ASR	Note
A	Cube	71.60	80.60	60.00	73.33	Fig. 1 (b)
A	Cylinder	76.20	84.60	73.33	76.67	Fig. 8 (a)
B	Cube	72.80	83.00	50.00	60.00	Fig. 8 (b)
B	Cylinder	75.40	85.40	53.33	63.33	Fig. 8 (c)

TABLE V: Attack Sim-to-Real Transferability Evaluation (%)

Fig. 8: Policy rollout with (a) a cylinder-shaped adversarial object in EnvA, (b) a cube-shaped adversarial object in EnvB, and (c) a cylinder-shaped adversarial object in EnvB.

Env	Scenario	T-ASR	ASR	Env	Scenario	T-ASR	ASR
A	Dynamic object (a)	46.67	60.00	A	Occluded scene (b)	43.33	56.67
B	Dynamic object	40.00	50.00	B	Occluded scene	36.67	50.00

TABLE VI: Attack Robustness (%) Under Challenging Scenarios

Fig. 9: Policy rollout in EnvA: (a) with the dynamically moved adversarial object; (b) with the partially occluded adversarial object.

Env	Lighting	T-ASR	ASR	Env	Lighting	T-ASR	ASR
A	Dim1	50.33	66.67	A	Bright1	53.33	73.33
A	Dim2	53.33	66.67	A	Bright2	56.67	73.33
A	Dim3	53.33	70.00	A	Dynamic	36.67	50.00
B	Dim1	46.67	60.00	B	Bright1	50.00	66.67
B	Dim2	50.00	60.00	B	Bright2	53.33	66.67
B	Dim3	50.00	63.33	B	Dynamic	33.33	50.00

TABLE VII: Attack Robustness (%) Under Different Lighting Variations

Fig. 10: (a) Setups for assessing robustness to lighting in EnvA (from left: dim1, dim2, dim3, bright1, and bright2); (b) policy rollout in dynamic lighting projected via an LED monitor.

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 447 papers

Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models
Apr 2, 2026Jiawei Chen, Simin Huang, Jiawei Du +5Adversarial Attacks on VLA Models Adversarial Examples
East China Normal University · Zhongguancun Academy · CFAR, A*STAR, Singapore +2
UniTexture: Cross-Task Universal Adversarial Textures for Vision-Language-Action Models
Aug 13, 2026Yukun Dai, Mingzhe Dai, Tianshi Wang +3Adversarial Attacks on VLA Models Vision-Language-Action Models
Tongji University · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China
ARGUS: Aligning Robot Scene Geometry Under Shifting Views with Large 3D Vision Models
Aug 6, 2026Rishik Sathua, Haonan Chen, Katherine Driggs-CampbellGeneralization in Robotic Manipulation Visuomotor Policy Learning
University of Illinois at Urbana-Champaign · Harvard University

Apr 2, 2026J/K move · Enter open · S save

Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models

Jiawei Chen, Simin Huang, Jiawei Du, Shuaihang Chen+4

East China Normal University · Zhongguancun Academy · CFAR, A*STAR, Singapore+2

Adversarial Attacks on VLA Models Adversarial Examples Physical Adversarial Attacks Robustness of VLA Models VLMs for Robotics

Open paper AskPDF arXiv

Vision-language-action (VLA) models have shown strong performance in robotic manipulation, yet their robustness to physically realizable adversarial attacks remains underexplored. Existing studies reveal vulnerabilities through language perturbations and 2D visual attacks, but these attack surfaces are either less representative of real deployment or limited in physical realism. In contrast, adversarial 3D textures pose a more physically plausible and damaging threat, as they are naturally attached to manipulated objects and are easier to deploy in physical environments. Bringing adversarial 3D textures to VLA systems is nevertheless nontrivial. A central obstacle is that standard 3D simulators do not provide a differentiable optimization path from the VLA objective function back to object appearance, making it difficult to optimize through an end-to-end manner. To address this, we introduce Foreground-Background Decoupling (FBD), which enables differentiable texture optimization through dual-renderer alignment while preserving the original simulation environment. To further ensure that the attack remains effective across long-horizon and diverse viewpoints in the physical world, we propose Trajectory-Aware Adversarial Optimization (TAAO), which prioritizes behaviorally critical frames and stabilizes optimization with a vertex-based parameterization. Built on these designs, we present Tex3D, the first framework for end-to-end optimization of 3D adversarial textures directly within the VLA simulation environment. Experiments in both simulation and real-robot settings show that Tex3D significantly degrades VLA performance across multiple manipulation tasks, achieving task failure rates of up to 96.7%. Our empirical results expose critical vulnerabilities of VLA systems to physically grounded 3D adversarial attacks and highlight the need for robustness-aware training.

Abstract

Adversarial Attacks

Explore similar work

Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models

UniTexture: Cross-Task Universal Adversarial Textures for Vision-Language-Action Models

ARGUS: Aligning Robot Scene Geometry Under Shifting Views with Large 3D Vision Models