RefCloth: Transferable Adversarial Tablecloths for Vision-Language-Action Models
Authors: Shaozhen Liu, Zhiwei Li, Jiayang Sun, Shuangjun Du, Yi Zhong, Shenrui Ma, Yaozu Liu, Qi Li, Zhenan Sun
Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Closed-loop vision-language-action (VLA) policies rely continuously on multi-view cameras, yet existing visual attacks typically hijack sensor streams with 2D patches or alter target object meshes that require retraining when manipulation objects change. We introduce RefCloth, a framework that optimizes one persistent adversarial tablecloth on a frozen source policy and deploys it zero-shot across unseen tasks and target models without modifying task objects or cameras. Throughout manipulation, static third-person and moving wrist views observe this shared world-bound texture under evolving occlusions. To generalize across shifting surface views while bridging the gap between differentiable and native renderers, RefCloth combines bounded texture parameterization with shared surface UV transformations and a same-path clean reference that removes static rendering offsets at initialization. Across four LIBERO suites in native simulation, one frozen texture transfers from OpenVLA-OFT to π0 and π0.5, reducing success relative to an unoptimized cloth by 62.50 and 43.75 percentage points on average and far exceeding amplitude- and variation-matched random controls. Dual-view audits and target closed-loop camera interventions show that moving wrist exposure drives over twice the standalone drop of the main view with strong synergy, while ablations confirm the roles of UV diversification and path calibration.
Figures & tables
Figure 2: RefCloth optimization and native closed-loop transfer. (a) Both branches share scene state x , UV transformation u , the differentiable rasterization path, and frozen source VLA weights. Detached path-consistent feature displacement and amplitude/TV penalties optimize texture grid θ . Images are 224×224 source-path renderings. (b) Bounded texture generation via barycentric interpolation and clipping. (c) Fixed-PNG zero-shot deployment in native simulation across unseen tasks and target policies, shown with Object camera views.
Method
Carrier
Obj.-agn.
3D-Dual
Reuse
No-Tgt.
Ctrl.
UPA [ 32 ] , EDPA [ 53 ]
2D patch
✓
–
✓
✓
–
Tex3D [ 7 ]
3D mesh
–
–
–
✓
–
UniTexture [ 10 ]
3D mesh
–
✓
✓
–
–
TRAP [ 19 ]
CoT cloth
✓
–
–
–
–
RefCloth (Ours)
3D cloth
✓
✓
✓
✓
✓
Table 1: Comparison with representative VLA visual attacks. Columns indicate object-agnostic carrier (Obj.-agn.), 3D-consistent dual-view projection (3D-Dual), single frozen texture reuse (Reuse), optimization without prescribed action targets or CoT traces (No-Tgt.), and de-confounded carrier/random controls (Ctrl.).
Figure 3: Carrier footprint and clean-reference offset. (a) Visible pixel fractions (median and IQR at 224×224 ) of the cloth and target bowl in both views across 32 fixed LIBERO-Spatial states, compared with the nominal footprint of a 50×50 UPA-RFAS patch on a 256×256 frame (dashed line). (b) Relative Plain feature offset 100∥zR0−zref∥2/∥zN0∥2 across the 32 Spatial states, where zR0 and zN0 denote differentiable- and native-path Plain features. Blue dots and lines show states and task means under zref=zN0 , while red lines show the zero offset under zref=zR0 .
Suite
Original
Plain
RefCloth
G↑
π0
Spatial
94
89
12
77
Object
96
88
30
58
Goal
90
83
9
74
Long
79
43
2
41
Mean
89.75
75.75
13.25
62.50
Table 2: One frozen texture across tasks and policies. Success rates (%) and net texture reduction G=P−A (pp) over 100 episodes per suite. Original and Plain separate carrier placement from texture optimization.
Texture
π0 : A/G
π0.5 : A/G
Random 101
57.75/18.00
87.75/5.00
Random 102
53.50/22.25
85.00/7.75
Random 103
56.75/19.00
86.50/6.25
Random mean
56.00/19.75
86.42/6.33
RefCloth
13.25/62.50
49.00/43.75
Table 3: Matched-statistics random controls. Four-suite mean success rate A (%) / net drop G (pp) over 800 episodes per texture.
Figure 4: Visual response and closed-loop behavior. (a) Final-layer source action-token attention over 16×16 visual patches at the first fixed state (Spatial task 0, initial state 21, step 10). Color is normalized by the within-view mean and clipped at four for display. Cyan contours show marker-derived cloth support. Random uses texture 101. (b) Full main and wrist frames from late Plain and RefCloth rollouts on two π0.5 Object tasks at initial state 10. Plain succeeds. RefCloth places a green bottle and a milk carton in the basket, respectively. (c) Recorded XY end-effector paths on π0 -Spatial task 0, initial state 10. The numbers give executed actions.
π0
π0.5
Suite
Main
Wrist
Main
Wrist
Spatial
23.5
41.3
23.7
39.7
Object
11.9
35.7
11.9
34.4
Goal
22.4
35.6
22.4
37.9
Long
17.9
33.4
18.7
22.8
Table 4: Dual-view cloth support across suites. Percentage of native pixels where Plain and cloth-hidden renders differ by >10 RGB levels (median over 30 episode medians).
Condition
Succ. (%)
G↑ (pp)
Steps
pH vs. Plain
Plain
98
0
32.4
—
Main-only (M)
88
10
36.8
.00885
Wrist-only (W)
75
23
44.3
4.7×10−6
Both (M+W)
34
64
66.1
<10−18
Table 5: Closed-loop camera interventions on π0.5 -Object. 100 paired episodes per condition under synchronized noise, showing success (%), net drop G (pp), mean query steps, and Holm-adjusted McNemar pH vs. Plain.
Method
S
O
G
L
Mean
π0
Native ref.
9/80
28/60
21/62
4/39
15.50/60.25
EDPA
23/66
14/74
18/65
2/41
14.25/61.50
UPA-RFAS
27/62
35/53
32/51
7/36
25.25/50.50
RefCloth
12/77
30/58
9/74
2/41
13.25/62.50
π0.5
Table 6: Cloth-carrier transfer. Success A (%) / net drop G (pp) over 100 episodes per suite (S/O/G/L: Spatial/Object/Goal/Long). Bold marks the strongest attack per suite.
Method
Surface
π0
π0.5
Original
None
359 (89.75)
376 (94.00)
Random
Patches
290 (72.50)
346 (86.50)
EDPA
Patches
197 (49.25)
273 (68.25)
UPA-RFAS
Patch
253 (63.25)
334 (83.50)
RefCloth
Cloth
53 (13.25)
196 (49.00)
Table 7: Native-surface deployment. Successful episodes out of 400 and success rate (%) across four suites under each method’s native surface and training configuration.
Tasks
UV
Spatial
Object
Goal
Long
π0
One
Off
49/40
65/23
63/20
19/24
One
On
14/75
6/82
17/66
1/42
Four
Off
45/44
61/27
40/43
13/30
Four
On
12/77
30/58
9/74
2/41
π0.5
Table 8: Source-pool size and UV augmentation. Cells are successes out of 100 / G (pp). Plain counts are in Tab. 2 .
Figure 5: Reference-dependent update directions. Gradient-weighted sign disagreement for 20 paired batches of the full objective. Points show batches; diamonds show means.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Surface
Source
Training pool
Updates
Batch
Optimizer / step
RefCloth
Cloth
OpenVLA-OFT
296 frames
5,000
8
Sign / 0.05
Native reference
Cloth
OpenVLA-OFT
Same pool
5,000
8
Sign / 0.05
EDPA
Cloth
OpenVLA-OFT
Same pool
5,000
8
Sign / 0.05
UPA-RFAS
Cloth
OpenVLA-OFT
Same pool
5,000
8
Sign / 0.05
EDPA
Two 57×57 patches
OpenVLA-OFT
303 no-cloth frames
1,200
2
Sign / 2/255
UPA-RFAS
One 50×50 patch
OpenVLA
Spatial RLDS
2,000
2
AdamW / 10−3
Appendix
Table S1: Training configurations across methods. Cloth rows share a common observation pool, perturbation budget ϵ=64/255 , Gaussian initialization N(0,0.012I) , and parameter bounds [−5,5] . “Sign” denotes sign-gradient ascent. Native EDPA settings apply independently to each camera patch.
Target
Suite
Original
Plain
Random cloth
Native ref.
EDPA
UPA-RFAS
RefCloth
π0
Spatial
94
89
62
9
23
27
12
Object
96
88
73
28
14
35
30
Goal
90
83
79
21
18
32
9
Long
79
43
36
4
2
7
2
π0.5
Spatial
98
98
97
67
77
83
60
Object
99
100
92
80
57
55
38
Appendix
Table S2: Complete cloth-carrier performance across suites. Successful episodes out of 100. The lower-amplitude random cloth has RMS 24.011, whereas main RefCloth has RMS 37.499, and calibrated amplitude-matched controls appear in Tab. S7 .
Suite
Original
Random
EDPA
UPA-RFAS
π0
Spatial
94
87
59
74
Object
96
81
62
74
Goal
90
74
50
61
Long
79
48
26
44
Total
359
290
197
253
Appendix
Table S3: Native-patch outcomes across suites. Successful episodes out of 100, and totals out of 400. Original denotes the baseline unperturbed scene. Random and optimized EDPA patches share recorded placement sequences.
Target
Suite
ΔG
Pairs
pH
π0
Spatial
11
18/7
.216
Object
−16
5/21
.020
Goal
9
13/4
.216
Long
0
2/2
1.000
π0.5
Spatial
17
26/9
.038
Object
19
31/12
.038
Appendix
Table S4: Paired comparison between RefCloth and cloth-adapted EDPA. ΔG=GRefCloth−GEDPA (pp). Discordant pairs count RefCloth stronger / EDPA stronger, and pH denotes Holm-adjusted exact McNemar p -values.
Target
Suite
ΔG
Pairs
pH
π0
Spatial
15
22/7
.0407
Object
5
19/14
.974
Goal
23
27/4
.000272
Long
5
6/1
.375
π0.5
Spatial
23
30/7
.00134
Object
17
29/12
.0460
Appendix
Table S5: RefCloth versus cloth-adapted UPA-RFAS. Pairs count RefCloth stronger / UPA-RFAS stronger. ΔG=GRefCloth−GUPA , and pH uses Holm adjustment across eight tests.
Method / Comparator
Configuration
Successes / 100
G (pp)
Pairs
pH
RefCloth (Ours)
Attenuated grid ( ϵtanh )
71
29
—
—
Native reference
Batch 8, spatial TV
92
8
21/0
3.81×10−6
Native reference
Batch 1, spatial TV
84
16
14/1
.00195
Random cloth
Fixed, unoptimized
92
8
23/2
5.83×10−5
EDPA
ℓ1 amplitude penalty
77
23
15/9
.307
Appendix
Table S6: RMS-controlled comparisons on π0.5 -Object. All conditions utilize the cloth carrier with texture RMS held near 24. Plain baseline achieves 100/100 successes. Pairs count RefCloth stronger / comparator stronger. Adjusted pH values use a predefined four-test Holm-corrected family comparing RefCloth against each baseline.
Target
Suite
Plain
RefCloth
Random 101
Random 102
Random 103
Maximum pH
π0
Spatial
89
12
65
62
65
4.88×10−12
Object
88
30
67
64
68
9.31×10−7
Goal
83
9
73
65
71
1.84×10−12
Long
43
2
26
23
23
1.14×10−5
π0.5
Spatial
98
60
99
99
98
3.73×10−10
Object
100
38
90
93
90
4.27×10−14
Appendix
Table S7: Complete matched-statistics random comparison across all suites. Successes out of 100. Each row compares identical task–initial-state keys. The final column reports the maximum adjusted p -value among the three RefCloth–random comparisons per condition.
Figure S1 : Exported textures in the main-amplitude random control. Decoded 8-bit PNGs of RefCloth and three independent random controls. All patterns share RMS 37.499 and L∞=64 , with calibrated TV labels indicating closely matched spatial variation.
Figure S2 : Additional native-rendered deployments. Plain, RefCloth, and cloth-adapted EDPA at matched states in Object task 5 and Goal task 5. Spatial is shown in the main text, and full texture maps appear in Fig. S3 .
Figure S3 : Gallery of deployed textures and baselines. Exported 8-bit PNGs in UV coordinates. RMS and TV are computed directly on decoded pixel arrays. The main benchmark uses Random 101 (RMS 37.50), and the RMS 24.01 control is evaluated in Tab. S6 .
Spatial tasks
Plain
Native ref.
RefCloth
All ten
99/100
64/100
59/100
Six held out
59/60
35/60
28/60
Appendix
Table S8: Closed-loop evaluation on OpenVLA-OFT. Successful episodes out of total under native source rendering. Held-out tasks are excluded from texture training.
Seed
Native
Cloth-EDPA
RefCloth
ΔNat
ΔEDPA
42
25.75
24.00
35.50
+9.75
+11.50
7
27.00
27.25
45.50
+18.50
+18.25
66
27.25
30.00
45.25
+18.00
+15.25
88
30.50
32.25
32.00
+1.50
−0.25
Mean
27.62
28.38
39.56
+11.94
+11.19
SD
2.03
3.56
6.86
8.03
8.11
Appendix
Table S9 : Four-seed common-carrier comparison on π0.5 . Four-suite macro-averaged net reduction Gˉ (pp) across Spatial, Object, Goal, and Long (400 episodes per cell). ΔNat=GˉRefCloth−GˉNative , and ΔEDPA=GˉRefCloth−GˉEDPA . SD is computed across the four independent training seeds.
Seed
Scope
Native
M → N
N → M
Matched
7
Spatial
31.0
44.0
62.0
38.0
Object
20.0
29.0
31.0
62.0
8
Spatial
29.0
18.0
28.0
63.0
Object
38.0
35.0
39.0
40.0
9
Spatial
45.0
62.0
44.0
38.0
Object
43.0
40.0
40.0
50.0
Appendix
Table S10 : Step-500 reference-switching schedules on π0.5 -Spatial/Object. Net reduction G (pp, 100 episodes per suite, where Plain is 98/100 on Spatial and 100/100 on Object). M → N and N → M switch references at step 500 of 5,000 while preserving parameters, optimizer state, and sampling order. The middle block reports the two-suite mean per seed, and bottom rows report the mean and SD across seeds ( n=3 ) and across all seed–suite cells ( n=6 ).
Native
Matched
Batch
Updates
Visits
S
O
S
O
8
5,000
40,000
67/31
80/20
60/38
38/62
1
40,000
40,000
85/13
72/28
54/44
43/57
Appendix
Table S11 : Update granularity at 40,000 source visits. Cells are successes out of 100 / G (pp) on π0.5 -Spatial/Object (S/O), with Plain counts in Tab. 2 . Source visits equal outer updates times batch size, and all other settings match Tab. S1 .
Suite
Plain ( P )
Random 101
RefCloth ( Tab. 2 )
G↑ ( pPlain )
X-VLA-Self
VLM-only
Spatial
98.20 (491/500)
97.40 (487/500)
97.80 (489/500)
+0.40 (.688)
0.40 (2/500)
16.00 (16/100)
Object
99.60 (498/500)
98.60 (493/500)
89.80 (449/500)
+9.80 ( 4.62×10−14 )
12.40 (62/500)
7.00 (7/100)
Goal
97.80 (489/500)
97.60 (488/500)
94.80 (474/500)
+3.00 (.0107)
18.40 (92/500)
13.00 (13/100)
Long
95.00 (475/500)
95.20 (476/500)
83.20 (416/500)
+11.80 ( 1.79×10−9 )
51.80 (259/500)
41.00 (41/100)
Mean
97.65 (1953/2000)
97.20 (1944/2000)
91.40 (1828/2000)
+6.25
20.75 (415/2000)
19.25 (77/400)
Appendix
Table S12 : Cross-backbone evaluation on X-VLA across four LIBERO suites. Success rates (%) over N=500 paired episodes per suite ( 2,000 total, with N=100 per suite for VLM-only ). Net drop G=P−A ( pp ) and exact paired McNemar p -values ( pPlain ) compare frozen OpenVLA-OFT RefCloth ( Tab. 2 ) against Plain . X-VLA-Self ( N=500 ) and VLM-only ( N=100 ) optimize directly against X-VLA and its Florence-2 VLM backbone, respectively, reducing four-suite mean success by 76.90 and 78.40pp ( p≤6.89×10−53 vs. Plain and p≤1.25×10−36 vs. RefCloth across every suite for X-VLA-Self ).
Condition
Succ.
Rate (%)
G↑ (pp)
pPlain
Original ( O )
118
92.19
—
—
Plain ( P )
105
82.03
0.00
—
Random 101
92
71.88
10.16
.00443
Random 102
93
72.66
9.38
.00754
RefCloth ( Tab. 2 )
86
67.19
14.84
5.46×10−4
RefCloth-RT
84
65.62
16.41
4.92×10−5
Appendix
Table S13 : RoboTwin 2.0 adjust_bottle evaluation ( N=128 ) on RLinf- π0.5 -PPO. Net drop G=P−A ( pp ) is measured relative to Plain ( 82.03% ), and pPlain reports two-sided exact paired McNemar tests against Plain . X-VLA-Self further significantly outperforms both Random 101 ( p=8.21×10−4 ) and Random 102 ( p=4.92×10−5 ).
adjust_bottle ( N=128 )
beat_block_hammer ( N=100 )
place_empty_cup ( N=100 )
Condition
Rate (%)
G↑ (pp)
pPlain
Rate (%)
G↑ (pp)
pPlain
Rate (%)
G↑ (pp)
pPlain
Original ( O )
85.16
—
—
75.00
—
—
53.00
—
—
Plain ( P )
53.12
0.00
—
57.00
0.00
—
19.00
0.00
—
Random 101
46.09
7.03
.136
42.00
15.00
.0275
6.00
13.00
9.77×10−4
RefCloth ( Tab. 2 )
14.06
39.06
7.64×10−11
2.00
55.00
8.05×10−16
3.00
16.00
4.02×10−4
RefCloth-RT
31.25
21.88
8.36×10−6
26.00
31.00
3.35×10−7
3.00
16.00
1.45×10−4
Appendix
Table S14 : Evaluation on bimanual X-VLA across three RoboTwin 2.0 tasks ( N=128 for adjust_bottle , and N=100 for beat_block_hammer and place_empty_cup ). Net texture reduction G=P−A ( pp ) and exact paired McNemar p -values ( pPlain ) are measured against Plain , separating the baseline carrier shift ( D=O−P=32.03,18.00, and 34.00pp ). On adjust_bottle and beat_block_hammer , RefCloth , RefCloth-RT , and X-VLA-Self all significantly outperform Random 101 ( p≤.00904 , with p≤2.72×10−8 for RefCloth and p≤1.00×10−11 for X-VLA-Self ), while on adjust_bottle , an additional control Random 102 attains 54.69% ( pPlain=.845 ) and all three learned textures surpass it ( p≤6.94×10−8 ).
Target
Suite
Successes / 30
Queries / episode
Main support (%)
Wrist (%)
q>1%
π0
Spatial
26
29.5
23.48 [22.86,24.48]
41.27
1.000
Object
26
36.9
11.86 [11.42,12.19]
35.72
1.000
Goal
22
39.4
22.41 [21.42,22.50]
35.59
1.000
Long
11
71.1
17.93 [15.03,23.57]
33.38
1.000
π0.5
Spatial
30
22.2
23.72 [23.12,24.61]
39.66
1.000
Object
29
31.2
11.95 [11.55,12.15]
34.36
1.000
Appendix
Table S15: Native Plain–hidden RGB support over complete rollouts. Thirty Plain episodes per target–suite. Support reflects the median of episode medians (%), with interquartile ranges in brackets. q>1% is the median fraction of queries with main-view support >1% .
Target
View
Marker support
Plain–hidden
RefCloth–Plain
RMS
π0
Main
14.58 [14.14,14.91]
11.75 [11.27,12.05]
11.98 [11.51,12.17]
10.70
Wrist
39.04 [32.50,52.38]
36.00 [30.49,48.07]
36.01 [31.02,47.02]
23.20
π0.5
Main
14.63 [14.29,14.88]
11.85 [11.41,12.03]
12.01 [11.67,12.17]
10.69
Wrist
37.79 [31.99,45.06]
34.67 [29.81,41.13]
34.47 [29.77,39.39]
22.92
Appendix
Table S16: Same-state measurements at deployed Object policy inputs. Summarizing 30 complete Plain trajectories per target at 224×224 resolution with threshold 10 on the 0–255 RGB scale. RMS measures median RefCloth–Plain pixel deviation.
Target
Plain
RefCloth
G (pp)
Induced
Recovered
pH
π0
26/30
11/30
50.0
15
0
6.10×10−5
π0.5
29/30
9/30
66.7
20
0
3.81×10−6
Appendix
Table S17: Paired Object evaluation under synchronized sampling noise. Thirty paired rollouts per target model. Induced/recovered counts measure Plain-success/RefCloth-failure transitions and the converse.
Pairwise contrast
Δ (pp)
Pairs
Exact p
pH
Plain vs. Main
+10
11 / 1
.00635
.00885
Plain vs. Wrist
+23
24 / 1
1.6×10−6
4.7×10−6
Plain vs. Both
+64
64 / 0
1.1×10−19
6.5×10−19
Main vs. Wrist
+13
16 / 3
.00443
.00885
Main vs. Both
+54
55 / 1
1.6×10−15
7.9×10−15
Wrist vs. Both
+41
41 / 0
9.1×10−13
3.6×10−12
Appendix
Table S18 : Pairwise contrasts for closed-loop interventions on π0.5 -Object. 100 paired episodes per condition (Plain 98, Main 88, Wrist 75, Both 34). Δ denotes success difference (pp), Pairs counts discordant episodes (first / second succeeds), and pH is the Holm-adjusted exact McNemar p -value across all 6 contrasts.
Texture
Main
Wrist
Plain
0.678
0.376
Random 101
0.775
0.583
Random 102
0.785
0.693
Random 103
0.775
0.557
Native ref.
1.467
0.692
UV off
0.884
0.516
Appendix
Table S19: Source attention allocation across eight frozen states. Median area-normalized cloth attention density ρv ( ρv=1 corresponds to uniform spatial allocation within that view).
Figure S4 : Source visual attention heatmaps at a representative state. Spatial task 0, initial state 21, step 10. Native RGB inputs (rows 1 and 3) and corresponding 16×16 patch attention densities (rows 2 and 4). Cyan contours indicate visible cloth support.
Step
∥gm∥2
∥gw∥2
cos(gm,gw)
sw (%)
sw>50%
0
0.00089
0.00097
0.005
53.7
19 / 32
1,875
0.04923
0.05757
0.054
57.2
23 / 32
3,750
0.06104
0.06564
0.030
52.7
18 / 32
5,000
0.06222
0.06337
0.046
50.4
16 / 32
Appendix
Table S20 : Dual-view source gradient decomposition across training checkpoints. Medians over 32 batches (4 seeds × 8 batches) of main-view ( ∥gm∥2 ) and wrist-view ( ∥gw∥2 ) control-grid gradient norms, inter-view cosine similarity cos(gm,gw) , wrist projection share sw=⟨gw,gm+gw⟩/∥gm+gw∥22 , and number of batches with sw>50% .
Figure S5 : Two-view action attribution across eight frozen states. Pose-action RMS drift relative to Plain under single-view or dual-view adversarial substitution. Connecting lines trace individual task steps.
π0
π0.5
Suite
Task
Initial
Plain
Random
RefCloth
Plain
Random
RefCloth
Spatial
0
10
85 S
77 S
400 F
79 S
75 S
95 S
Spatial
2
10
114 S
110 S
400 F
92 S
100 S
348 S
Spatial
0
12
75 S
166 S
400 F
90 S
82 S
103 S
Goal
5
10
125 S
311 S
400 F
169 S
126 S
400 F
Goal
7
10
81 S
100 S
400 F
75 S
78 S
76 S
Appendix
Table S21: Continuous behavior recordings across representative tasks. Executed action steps and outcomes (S denotes success, F denotes failure). Episodes share initial states across conditions.
Figure S6 : Three-dimensional continuous end-effector trajectories. Continuous XYZ paths for Spatial task 0 under π0 (a) and π0.5 (b). Open circles mark shared starting positions, and filled diamonds indicate termination points.
Direction cos(d,d∗)
Rel. Error ∥d−d∗∥/∥d∗∥
Snapshot (32 states)
Matched
Native ref.
Matched
Native ref.
Archived ( Fig. 3 b)
0.8612
0.9393
0.5029
0.3437
Fresh (PNG)
0.8644
0.9382
0.4900
0.3459
Fresh (float)
0.8643
0.9381
0.4899
0.3460
Appendix
Table S22 : Displacement fidelity across 32 Spatial states. Comparing matched ( dM ) and native ( dN ) displacements against true native displacement d∗ for frozen RefCloth. Rel. error is ∥d−d∗∥2/∥d∗∥2 (which equals ∥rθ∥2/∥d∗∥2 for Matched).
Figure S7 : Trained-texture displacement fidelity and target closed-loop camera interventions. (a) Cosine similarity and (b) relative displacement error ( ∥d−d∗∥2/∥d∗∥2 ) of matched ( dM ) and native-reference ( dN ) source feature displacements relative to true native displacement ( d∗ ) across 32 Spatial states for the same frozen RefCloth PNG. (c) Closed-loop task success (%) on π0.5 -Object across 100 paired episodes per camera-intervention condition.
Adversarial attacks on vision-language-action (VLA) models study how crafted inputs, poisoned data, or physical cues can disrupt robot decisions and safety.
We present Cloak, a training recipe that endows a Vision-Language-Action (VLA) model with zero-shot cross-embodiment transfer by cloaking the end-effector from its own wrist camera. The end-effector occupies a large and consistent region of the wrist view and masking it allows for embodiment-agnostic visual reasoning. Cloak renders a mask in simulation from the robot's known geometry, accurately and in real time, with no segmentation or generative models. During training, we augment the mask so the model generalizes to embodiments unseen at training time. We demonstrate the recipe with Cloak-VLA, a VLA trained with Cloak on a single parallel-jaw gripper dataset. No data of new embodiments is ever collected. Cloak-VLA transfers zero-shot to various unseen embodiments, including another gripper, another arm, and a five-fingered hand, while preserving the source embodiment's performance. By decoupling the wrist view from its own embodiment, Cloak allows data to outlive the hardware it was collected on.
Vision-Language-Action (VLA) models have emerged as generalist robotic policies capable of following diverse language instructions and performing a wide range of manipulation tasks. However, their direct control over embodied agents also exposes them to adversarial interference that may cause unsafe physical behaviors. Existing attacks on robotic policies are typically optimized for a single task or instruction, leaving the cross-task vulnerabilities of multitask VLAs largely unexplored. We introduce UniTexture, a cross-task universal adversarial texture attack that uses a single textured 3D object to induce targeted deviations in VLA action predictions across multiple tasks. UniTexture backpropagates gradients from the policy's action outputs to surface texture parameters through a differentiable renderer. It jointly optimizes the shared texture over a distribution of tasks, instructions, states, and viewpoints using a targeted action-space objective, steering predicted actions toward attacker-defined targets without optimizing a separate texture for each task. We evaluate UniTexture on OpenVLA and π0.5 across diverse manipulation tasks and multiple evaluation settings. UniTexture reduces the mean task success rate from 90.0% under benign conditions to 48.4% under attack, induces target-aligned action shifts, and further exhibits cross-suite and cross-model transfer without re-optimization. Together, these findings reveal shared cross-task vulnerabilities in multitask VLAs that can be systematically exploited through a single adversarial surface texture.
Yukun Dai, Mingzhe Dai, Tianshi Wang +3
Tongji University · Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China
Vision-Language-Action (VLA) policies are commonly adapted to new manipulation settings through additional gradient updates, which limits rapid deployment when task-specific data or compute is scarce. We present ICI-VLA, a training and retrieval framework that equips a text-action VLM with few-shot test-time adaptation through in-context demonstrations. Unlike mainstream VLA designs based on action-specific multimodal fusion, ICI-VLA retains the native text-generation interface. ICI-VLA updates its parameters only during offline training; at inference, the policy remains fixed and conditions action generation on retrieved micro-demonstrations. The framework decomposes long trajectories into short, semantically labeled examples and trains an RD-Encoder with positives mined by Dynamic Time Warping (DTW), aligning the retrieved context with the phase and geometry of the current subtask. We further introduce Target Action Masking, a context-corruption objective designed to reduce direct action copying and increase reliance on the current observation. ICI-VLA reaches average success rates of 97.7% on LIBERO and 60.4% on RoboTwin 2.0, exceeding the highest reported baseline average on RoboTwin 2.0 by 19.3 percentage points. It also achieves 83.2% across four physical tasks. These results indicate that a fixed VLA policy can benefit from conditioning on spatiotemporally aligned demonstrations at test time.
Songhua Yang, Ziyu Liu, Xuetao Li +3
Wuhan University · Institute of Technological Sciences, Wuhan University