Neural network-based visuomotor policies enable robots to perform manipulation tasks but remain susceptible to perceptual attacks. For example, conventional 2D adversarial patches are effective under fixed-camera setups, where appearance is relatively consistent; however, their efficacy often diminishes under dynamic viewpoints from moving cameras, such as wrist-mounted setups, due to perspective distortions. To proactively investigate potential vulnerabilities beyond 2D patches, this work proposes a viewpoint-consistent adversarial texture optimization method for 3D objects through differentiable rendering. As optimization strategies, we employ Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, exploiting distance-dependent frequency characteristics to induce textures effective across varying camera-object distances. We further integrate saliency-guided perturbations to redirect policy attention and design a targeted loss that persistently drives robots toward adversarial objects. Our comprehensive experiments show that the proposed method is effective under various environmental conditions, while confirming its black-box transferability and real-world applicability.
Figures & tables
Fig. 1: Visuomotor policy deception using a 3D adversarial object. (a) The policy successfully guides the robot to its target Ogoal . (b) Our adversarial object Oadv manipulates the visual input, compelling the policy to misguide the robot towards itself instead of the true target Ogoal .
Fig. 2: Apparent size comparison of a 3D adversarial object and a 2D patch. The 2D patch significantly shrinks and distorts, especially at large viewing angles ϕ , unlike the more stable 3D object.
Fig. 3: Overview of the proposed method. (a) Coarse-to-Fine (C2F) Pose Scheduling: From a set of task-feasible initial configurations, poses where the original policy succeeds, we schedule viewpoint sampling using a distance-based Beta distribution. The scheduler progressively shifts focus from distant (Coarse stage) to closer (Fine stage) viewpoints. (b) 3D Adversarial Object Optimization Pipeline: Guided by the Expectation over Transformation (EOT) framework, the pipeline optimizes the adversarial texture T through short policy rollouts from each initial pose τi . In each step, the policy πω processes a composite image Iadv (formed from Idiff and Isim ) to output an action. A targeted adversarial loss is then computed from the resulting action, guiding the robot toward the adversarial object Oadv . The total loss, reflecting actual image-action pairs from the rollout, is backpropagated to update the texture T .
Fig. 4: Visualization of texture update patterns under different scheduling strategies: (a) Coarse-to-Fine, (b) Fine-to-Coarse, (c) Non-staged, (d) Coarse-only, (e) Fine-only.
Fig. 5: Comparison of policy saliency maps: (a) before vs. (b) after the 3D adversarial attack.
Sec.
Condition
T-ASR
ASR
Sec.
Condition
T-ASR
ASR
Ours
79.80
81.12
D.3
Light: Bright
77.04
84.70
D.1
Shape: Dog
60.04
65.80
Light: Dim
67.80
72.20
Shape: Duck
63.20
68.32
Add Noise
75.57
80.41
D.2
Stereo Cam
63.28
78.91
Bkg Varied
77.91
85.64
TABLE III: Attack Generalization and Robustness Analysis (%)
Fig. 6: Visualization of Sec. IV-D configurations: (a) D.1 . Dog shape, (b) D.1 . Duck shape, (c) D.2 . Stereo camera setup.
Fig. 8: Policy rollout with (a) a cylinder-shaped adversarial object in EnvA, (b) a cube-shaped adversarial object in EnvB, and (c) a cylinder-shaped adversarial object in EnvB.
Env
Scenario
T-ASR
ASR
Env
Scenario
T-ASR
ASR
A
Dynamic object (a)
46.67
60.00
A
Occluded scene (b)
43.33
56.67
B
Dynamic object
40.00
50.00
B
Occluded scene
36.67
50.00
TABLE VI: Attack Robustness (%) Under Challenging Scenarios
Fig. 9: Policy rollout in EnvA: (a) with the dynamically moved adversarial object; (b) with the partially occluded adversarial object.
Env
Lighting
T-ASR
ASR
Env
Lighting
T-ASR
ASR
A
Dim1
50.33
66.67
A
Bright1
53.33
73.33
A
Dim2
53.33
66.67
A
Bright2
56.67
73.33
A
Dim3
53.33
70.00
A
Dynamic
36.67
50.00
B
Dim1
46.67
60.00
B
Bright1
50.00
66.67
B
Dim2
50.00
60.00
B
Bright2
53.33
66.67
B
Dim3
50.00
63.33
B
Dynamic
33.33
50.00
TABLE VII: Attack Robustness (%) Under Different Lighting Variations
Fig. 10: (a) Setups for assessing robustness to lighting in EnvA (from left: dim1, dim2, dim3, bright1, and bright2); (b) policy rollout in dynamic lighting projected via an LED monitor.
Vision-language-action (VLA) models have shown strong performance in robotic manipulation, yet their robustness to physically realizable adversarial attacks remains underexplored. Existing studies reveal vulnerabilities through language perturbations and 2D visual attacks, but these attack surfaces are either less representative of real deployment or limited in physical realism. In contrast, adversarial 3D textures pose a more physically plausible and damaging threat, as they are naturally attached to manipulated objects and are easier to deploy in physical environments. Bringing adversarial 3D textures to VLA systems is nevertheless nontrivial. A central obstacle is that standard 3D simulators do not provide a differentiable optimization path from the VLA objective function back to object appearance, making it difficult to optimize through an end-to-end manner. To address this, we introduce Foreground-Background Decoupling (FBD), which enables differentiable texture optimization through dual-renderer alignment while preserving the original simulation environment. To further ensure that the attack remains effective across long-horizon and diverse viewpoints in the physical world, we propose Trajectory-Aware Adversarial Optimization (TAAO), which prioritizes behaviorally critical frames and stabilizes optimization with a vertex-based parameterization. Built on these designs, we present Tex3D, the first framework for end-to-end optimization of 3D adversarial textures directly within the VLA simulation environment. Experiments in both simulation and real-robot settings show that Tex3D significantly degrades VLA performance across multiple manipulation tasks, achieving task failure rates of up to 96.7%. Our empirical results expose critical vulnerabilities of VLA systems to physically grounded 3D adversarial attacks and highlight the need for robustness-aware training.
Jiawei Chen, Simin Huang, Jiawei Du +5
East China Normal University · Zhongguancun Academy · CFAR, A*STAR, Singapore +2
Vision-Language-Action (VLA) models have emerged as generalist robotic policies capable of following diverse language instructions and performing a wide range of manipulation tasks. However, their direct control over embodied agents also exposes them to adversarial interference that may cause unsafe physical behaviors. Existing attacks on robotic policies are typically optimized for a single task or instruction, leaving the cross-task vulnerabilities of multitask VLAs largely unexplored. We introduce UniTexture, a cross-task universal adversarial texture attack that uses a single textured 3D object to induce targeted deviations in VLA action predictions across multiple tasks. UniTexture backpropagates gradients from the policy's action outputs to surface texture parameters through a differentiable renderer. It jointly optimizes the shared texture over a distribution of tasks, instructions, states, and viewpoints using a targeted action-space objective, steering predicted actions toward attacker-defined targets without optimizing a separate texture for each task. We evaluate UniTexture on OpenVLA and π0.5 across diverse manipulation tasks and multiple evaluation settings. UniTexture reduces the mean task success rate from 90.0% under benign conditions to 48.4% under attack, induces target-aligned action shifts, and further exhibits cross-suite and cross-model transfer without re-optimization. Together, these findings reveal shared cross-task vulnerabilities in multitask VLAs that can be systematically exploited through a single adversarial surface texture.
Yukun Dai, Mingzhe Dai, Tianshi Wang +3
Tongji University · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China
Large-scale visuomotor policies have demonstrated impressive performance across a wide range of robot manipulation tasks. However, despite this success, manipulation polices often entangle scene geometry with the corresponding viewpoint, learning where objects lie in an image rather than where it lies in the task space. This entanglement inherently limits the corresponding policy's ability to learn from viewpoint-diverse datasets (ex. DROID, BridgeV2) and generalize beyond the viewpoints captured in their training data. In this work, we present ARGUS, an observation pre-processing pipeline that uses large-scale 3D vision models to align image observations from arbitrary camera viewpoints into a canonical viewpoint before passing it to downstream visuomotor policies. Experiments across training datasets with varying levels of viewpoint diversity, from fixed multi-view camera configurations to highly varied camera placements, show that our method consistently outperforms prior approaches across both limited-view and view-diverse training regimes. In efficiency comparisons, ARGUS demonstrates an ability to learn from view-diverse data, converging to high success rates 4-6x faster than previous methods by leveraging a simplified observation space. Overall, our findings show that leveraging large-scale 3D vision models reduces the learning burden on visuomotor policies, enabling more efficient learning from large-scale, viewpoint-diverse robot datasets.