Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · ACE Robotics · Nanyang Technological University · School of Computer Science, The University of Sydney
World action models (WAMs) increasingly couple internal visual or world representations with action generation, making their world-to-action pathways a critical yet poorly understood source of robustness. We investigate when perturbations to these internal representations actually become behaviorally consequential, and uncover a pronounced mismatch between representation perturbability and closed-loop vulnerability: attack effectiveness varies sharply across visual K/V routes, while larger representation displacement does not reliably imply stronger control failure. Our analysis attributes this mismatch to the action-conditioned readout, where keys determine where action queries read and values determine what content they receive. This motivates an action-facing view of vulnerability, in which the effect of a perturbation depends on how it is routed into action generation rather than on representation-space magnitude alone. We instantiate this view with Tail-AV, a short-probe signal for identifying vulnerable routes, and operationalize it in PASA-KV, which selects action-facing K/V targets before training a fresh task-shared attack. Experiments across multiple WAM systems and a physical robot validate readout-informed target selection, demonstrate strong closed-loop attacks, and reveal substantial system-level differences in resistance.
Figures & tables
Figure 1: Layerwise evidence for action-facing vulnerability. All panels use the evaluated FastWAM checkpoint. (a) Policy SR for 30 singleton joint K/V targets after 10k exposures; dashed: clean reference. (b) Tail-AV scores from short probes; rings mark Top-3. (c) Budget–SR trajectories at 5k/10k/20k exposures. Colors denote retrospective response groups, unused by the selector.
Figure 2: PASA-KV construction pipeline. (a) Action-facing visual K/V interface. (b) Candidate routes are first probed with a small optimization budget and ranked by Tail-AV; a fresh task-shared patch is then trained only on the selected Top- k routes. Victim weights remain fixed throughout, and rollout outcomes are excluded from layer selection.
Constant learning rate; no warm-up, scheduler, gradient clipping or accumulation in this patch loop
Initialization
Uniform [0,1] initializer; primary attack suite indexed by protocol/run label 42 ; singleton training configurations record seed 20260801 ; final training starts from fresh pixels
Projection and loss
Clamp pixels to [0,1] after every update; minimize −JKV (singleton) or −Jjoint ; image-L1 weight zero
Updates and batch
BS1, one update per exposure; 100 per singleton probe, 10,000 per final patch
Table 1: FastWAM PASA-KV attack-construction recipe. These settings optimize RGB patch pixels with the victim weights frozen. Probe and final training share the optimizer; only the loss-target set and update budget change.
System
Scoring interface
Attention normalization and visual slice
FastWAM
Fixed-clip cache; first action-denoising step; 30 layers
Masked softmax over available keys, then current-frame visual columns; no within-slice renormalization.
LingBot-VA
One video step (endpoint 0.6), followed by the first of ten action steps; 30 layers
Action queries attend to full video-cache and action keys; retain current primary-view cache columns and values, conditional batch only.
Cosmos Policy
One joint-noise level; action-slot 4 queries; blocks 0–27
Softmax over all joint-slot keys; retain current-primary slot 3 columns and values. Slot indices are zero-based.
Table 2: Architecture-specific selector capture interfaces. The common influence reduction is specified in Section 4.2.1 .
Victim
Inference graph and visual route
Patch (px @ x,y )
Evaluation solver
Clean
FastWAM
Direct; current-primary action-facing video K/V
50×50 at (5,160)
10 action steps
462/500 (92.4%)
LingBot-VA
Cascaded; current agent-view video-to-action route
29×29 at (3,91)
1 video + 10 action steps
462/500 (92.4%)
Cosmos Policy
Joint; current-primary conditioning slot
50×50 at (5,160)
1 joint denoising step
484/500 (96.8%)
Table 3: Architecture-specific main threat models. Historical frames and cached states are not retroactively modified.
Figure 3: FastWAM patch geometry on LIBERO-10. Tasks T0–T9 share (x,y,w,h)=(5,160,50,50) on the 224×224 primary view (4.98% of that view; 2.49% of the two-view input). The secondary view is unpatched.
System
Solver
Use
Success
Clean SR [95% CI]
Drop
LingBot-VA
v20/a50
High ref.
199/200
99.5% [97.22, 99.91]
—
LingBot-VA
v3/a10
Mid.
191/200
95.5% [91.67, 97.61]
−4.0 pp
LingBot-VA
v1/a10
Unified
187/200
93.5% [89.20, 96.16]
−6.0 pp
Cosmos
j5
High ref.
198/200
99.0% [96.43, 99.73]
—
Cosmos
j3
Mid.
191/200
95.5% [91.67, 97.61]
−3.5 pp
Cosmos
j1
Unified
194/200
97.0% [93.61, 98.62]
−2.0 pp
Table 4: Reduced-step clean controls. Each condition has 200 episodes; bracketed values are two-sided Wilson 95% intervals. Drops use the high-step reference.
FastWAM
LingBot-VA
Cosmos Policy
Method
SR ↓
GPU-h ↓
SR ↓
GPU-h ↓
SR ↓
GPU-h ↓
Protocol
Shared clean reference
92.4
–
92.4
–
96.8
–
Shared Eval500 reference
DP-Attacker
81.0
4.23
94.2
1.97
95.2
2.88
Diffusion
DRIFT
53.0
4.78
92.4
2.46
89.2
3.33
Flow/EDM
UPA-RFAS
90.8
10.75
91.0
8.94
89.2
23.74
No PSM
UPA
77.4
6.45
94.2
8.65
89.6
3.14
Action
Table 5: Fixed-patch comparisons with WAM-adapted baselines. Policy SR (%, lower is stronger). All conditions use LIBERO-10 Eval500 (10 tasks × 50 trials). Each PASA-KV Top- k condition uses a separately trained, task-shared patch (10k exposures); bold marks the lowest observed SR per architecture. GPU-h reports recorded construction cost; PASA-KV values include Probe100, selection, and final training for all three WAMs (Section 6.2 ).
Method
Probe
Full construction
Tail-AV Top-3
0.568632
2.3297
Table 6: FastWAM Tail-AV Top-3 construction cost (GPU-h). The total includes probing, selection, and final patch training.
Figure 4: Matched FastWAM route-selection controls. Bars show observed Policy SR on Eval200 (ten tasks, 20 trials each); lower is a stronger attack. Labels give successes/200. All methods share the victim, final joint K/V objective, three targets, patch geometry, and 10k exposures. The complete layer sets and retrospective classes are retained in Appendix Table 16 .
Method
Seed 42
Seed 43
Seed 44
Mean SR (%)
SD (pp)
Tail-AV Top-3
3
3
3
1.50
0.00
Probe- JKV
95
94
89
46.33
1.61
Evenly spaced
37
53
43
22.17
4.04
Random-1
173
170
169
85.33
1.04
Random-2
127
131
11
44.83
34.08
Random-3
50
19
25
15.67
8.22
Table 7: Fixed-layer final-training repeats on FastWAM. Counts are successes per 200 episodes; lower is stronger attack. Seeds 42/43/44 vary training RNG and shuffled sample order with identical saved initial pixels and layer sets. Evaluation uses seed 42 on a common Eval200 grid. Mean/SD are across runs. The seed-42 Tail-AV result uses the Eval200 subset of the primary Eval500 evaluation.
Run
Probe seed
Spearman
Top-3
Top-4
Top-5
Canonical
42
1.000000
Ref.
Ref.
Ref.
Seed42-A
42
0.998665
Yes
Yes
Yes
Seed42-B
42
0.997775
Yes
Yes
Yes
Seed43-A
43
0.998665
Yes
Yes
Yes
Seed43-B
43
0.996885
Yes
Yes
Yes
Seed43-C
43
0.994216
Yes
Yes
Yes
Table 8: FastWAM Tail-AV ranking repeatability on fixed scoring observations. A/B/C are separate probe-and-score executions at one seed, sharing initial pixels and scoring inputs. Spearman correlations compare eight 30-layer rankings with the canonical ranking; Top- k columns give exact set agreement. Only Seed44-B swaps L03 for L01 at Top-3.
Score
Early-group AUC ↑
ρ10k
ρ20k
Early@Top-3
Early@Top-5
Tail-AV
0.8624
0.521
0.508
3/3
4/5
Attention-only-tail
0.7672
0.506
0.537
3/3
3/5
Mean-AV
0.7831
0.229
0.173
2/3
3/5
Value-only-tail
0.6931
0.100
0.143
2/3
3/5
Table 9: Selector-score component ablation on the evaluated FastWAM checkpoint. All four score variants use the same Probe100 component-ablation run (30 layers and ten task-balanced scoring pairs). ρ10k and ρ20k are Spearman correlations with later layer AttackStrength; Early-group AUC and Early@Top- k are retrospective diagnostics using the nine Early layers as positives.
Figure 5: Complete selector-score component rankings on the evaluated FastWAM checkpoint. Thirty-layer rankings for Tail-AV, Attention-only-tail, Value-only-tail, and Mean-AV from the shared Probe100 component-ablation run. Colors denote retrospective Early, Intermediate, and Other groups; rings mark each ranking’s Top-3.
Figure 6: Action-response diagnostics. (a) FastWAM Action-L2 versus SR across six patches ( r=−0.956 , ρ=−0.714 ). (b) One-shot and persistent exposure; dashed/solid denote local/trajectory responses. (c) LingBot-VA response over history length (log scale). (d) Cosmos σ -normalized cumulative Action-L2. P/W/B denote primary/wrist/both views; suffixes give patched-frame and joint-step counts. Protocols: Sections 7.1 , 7.2 – 7.4 .
Rank
Layer
Normalized paired Action-L2
95% bootstrap CI
1
L02
0.47284
[0.36650,0.58085]
2
L01
0.25876
[0.19552,0.33376]
3
L06
0.12833
[0.09933,0.15727]
4
L00
0.09233
[0.07501,0.11057]
5
L12
0.08447
[0.06727,0.10072]
Table 10: Frozen high-response candidates selected from all 30 FastWAM layers by normalized paired Action-L2. Each layer has ten task-balanced paired records from an equal-budget Probe1000.
Deployment
Clean SR
Attacked SR
Recovery
No defense
93.50
1.33
–
Guard (30 layers)
53.00
49.67
52.44
Guard targeted (5)
93.50
87.50
93.49
Table 11: Pooled Guard results on fixed Tail-AV attacks. SR and recovery are percentages. Attacked SR pools three patches (600 episodes); clean SR uses 200 episodes per deployment. Recovery is the fraction of the attack-induced SR loss restored, relative to the 93.5% undefended clean reference. Per-patch results: Table 12 .
No defense
Guard (30)
Guard targeted (5)
Input
SR
Count
SR
Count
SR
Count
Clean
93.50
187/200
53.00
106/200
93.50
187/200
Tail-AV Top-3
0.00
0/200
52.50
105/200
85.50
171/200
Tail-AV Top-4
0.00
0/200
40.50
81/200
89.00
178/200
Tail-AV Top-5
4.00
8/200
56.00
112/200
88.00
176/200
Pooled attacks
1.33
8/600
49.67
298/600
87.50
525/600
Table 12: Per-patch Guard results. Policy SR (%) and success counts, with 200 episodes per patch/deployment. No-defense results use the fixed patch family evaluated for Guard.
Figure 7: Piper FastWAM task scenes and reference rollouts. Each pair shows the initial scene (left) and a representative annotated frame from the provided ground-truth rollout (right). The manipulated object is marked in red and its target slot or placement site in blue. Tasks are pen-in-cup, second-drawer placement, carrot-in-bowl, and bowl stacking; closed-loop attack results appear in Figure 8 .
Figure 8: Piper real-robot attack effects across three tasks. Bars show observed closed-loop task success; labels give successes/10. Error bars show two-sided 95% Wilson binomial confidence intervals over ten trials per condition. Top-3 and Top-5 use separately trained task-shared patches. The drawer task, whose clean SR is 0/10, is excluded from attack comparison. The input perturbation is a digital head-camera overlay.
Figure 9: Bowl-stacking success under PASA-KV and baseline attacks. Each method has ten in-distribution physical trials. Bars show observed task success; labels give successes/10. Each method uses its own patch geometry, as specified in Section 9 . Lower SR means a stronger attack.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Rank
Layer
Score
Rank
Layer
Score
Rank
Layer
Score
1
L29
0.049694
11
L14
0.009106
21
L28
−0.008567
2
L02
0.043262
12
L19
0.007954
22
L10
−0.017586
3
L03
0.032456
13
L16
0.004701
23
L26
−0.029588
4
L01
0.027921
14
L17
0.004437
24
L22
−0.034651
5
L24
0.026162
15
L15
0.002879
25
L09
−0.038692
6
L00
0.023169
16
L18
0.002561
26
L06
−0.043447
Appendix
Table 13: Production Tail-AV selector scores and ranking on the evaluated FastWAM checkpoint. Descending Eq. 5 , with layer-ID tie-breaking; scores rounded to six decimals. Top-3/4/5 prefixes give the selected attack sets.
Evidence
Patch relationship
Endpoint and role
Primary comparison
Separately trained Top-3, Top-4, and Top-5 patches per model
Eval500; baseline comparison and target-count tradeoff
Matched selection
Primary FastWAM Top-3 plus separately trained controls
Common Eval200; isolates layer-set choice
Training repeats
Same layer sets and exact initial pixels; varied final-training seed and shuffled sample order
Common Eval200; fixed-set training sensitivity
Layerwise analysis
Separate singleton-target patches
SR trajectories and paired internal responses; target characterization
Input diagnostics
Patch specified per panel
Paired action/representation responses; FastWAM also has 30 diagnostic rollouts
Guard
Separate fixed Top-3/4/5 family
200 episodes per patch and deployment; mitigation
Appendix
Table 14: Experiment map. Patch lineage is listed for each evidence block. Eval200/500 denote 20/50 online trials per task.
AdamW; learning rate 10−4 ; betas (0.9,0.95) ; epsilon 10−8 ; weight decay 0.01; gradient clipping 1.0; 1,085-update 5% warm-up; cosine decay to 10−6 ; no EMA or early stopping
Model
Wan-AI/Wan2.2-TI2V-5B; approximately 5.00B video-expert and 1.02B action-expert parameters; 30-layer MoT; video/action/proprioception trainable, VAE and other modules frozen; both loss weights 1.0
ModelScope revision
c2272aadfec2d95cea869a4e748d5e28275621c6
Appendix
Table 15: FastWAM victim training configuration. Values describe the frozen victim model used for the FastWAM experiments.
Selection rule
Layer set
Class
Successes
Policy SR (%)
Probe- JKV
L27/L23/L28
∙ / ∙ / ∙
95/200
47.5
Evenly spaced
L00/L15/L29
∙ / ∙ / ∙
37/200
18.5
Random-1
L27/L18/L16
∙ / ∙ / ∙
173/200
86.5
Random-2
L12/L23/L00
∙ / ∙ / ∙
127/200
63.5
Random-3
L20/L22/L00
∙ / ∙ / ∙
50/200
25.0
Tail-AV
L29/L02/L03
∙ / ∙ / ∙
3/200
1.5
Appendix
Table 16: Matched FastWAM selection controls. Same joint K/V loss, three targets, geometry, and 10k exposures; Eval200 (10 tasks × 20 trials). Dots denote retrospective response classes.
Paired layer statistic
Estimate
95% interval
K+V strictly stronger than K-only
17/30 (56.7%)
[40.0, 73.3]%
K+V strictly stronger than V-only
17/30 (56.7%)
[40.0, 73.3]%
Median SRK+V−SRK
−1.25 pp
[ −3.75 , 0.50] pp
Median SRK+V−SRV
−1.00 pp
[ −3.50 , 0.25] pp
Strong-attack layer coverage: K-only
7/30 (23.3%)
[10.0, 40.0]%
Strong-attack layer coverage: V-only
11/30 (36.7%)
[20.0, 53.3]%
Appendix
Table 17: Paired objective comparisons across the 30 layers of the evaluated FastWAM checkpoint. Lower Policy SR is a stronger attack. “Strong attack” denotes the descriptive coverage threshold SR ≤50% . Percentile intervals resample the 30 evaluated layers 20,000 times with seed 20260905.
Model
Attn.-map
K only
V only
K+V
FastWAM
90.68
65.77
66.43
58.87
Appendix
Table 18: All-layer component-objective attacks on the evaluated FastWAM checkpoint. Values are pooled closed-loop Policy SR (%; lower is stronger) over independently trained single-layer patches. K-only and V-only denote end-to-end patch optimization targets. Bold marks the lowest SR.
Figure 10: Action-facing visual K/V pathways. A fixed action query reads visual keys through attention and aggregates visual values into the action-facing readout. Key and value perturbations can alter routing and received content, respectively.
Figure 11: Representative component-objective attack cases on the evaluated FastWAM checkpoint. (a) L21, K-dominant. (b) L29, V-dominant. (c) L17, joint-beneficial. (d) L20, joint non-dominance. Bars show the objective contrasts discussed in the text; labels report singleton Policy SR.
Figure 12: Probe100 Tail-AV scores versus later attack strength. Each point is a FastWAM layer: Tail-AV from the shared 100-exposure component-ablation run versus later AttackStrength =1−SR/100 from the matched BS=1 singleton patches at 10k or 20k exposures.
Quantity
Mean ± SD
Observed range
Spearman ρ to 5k AttackStrength
0.3550±0.0074
[0.3465,0.3655]
Spearman ρ to 10k AttackStrength
0.5198±0.0080
[0.5067,0.5297]
Spearman ρ to 20k AttackStrength
0.5065±0.0084
[0.4953,0.5194]
Canonical Top-3 overlap
0.9583±0.1179
[0.6667,1.0000]
Canonical Top-5 overlap
1.0000±0.0000
[1.0000,1.0000]
Appendix
Table 19: Probe100 probe-seed stability across eight additional runs on the evaluated FastWAM checkpoint. Each repeat uses the fixed scoring pairs and a complete 30-layer ranking; values report mean ± sample SD and observed range. The canonical ranking is used only as the Top- k set reference.
Layer
SR (%)
AS
DK
DV
DA
∥ΔK∥
∥ΔV∥
∥ΔKV∥
Tail-AV
L00
1.5
0.985
0.377
1.433
0.429
9.4
105.8
112.9
0.425
L01
30.5
0.695
0.646
2.241
0.723
33.2
331.8
327.5
0.760
L02
10.0
0.900
0.553
1.967
0.566
37.5
324.7
310.8
0.474
L03
15.5
0.845
0.561
1.452
0.587
37.3
222.0
205.9
0.291
L04
95.5
0.045
0.565
1.357
0.687
58.3
214.6
208.1
0.187
L05
94.5
0.055
0.736
1.599
0.746
63.9
260.9
272.5
0.371
Appendix
Table 20: Complete 30-layer K/V mechanism summary. All rows use the joint K+V sweep with 50 observations from ten task-balanced tasks. DK , DV , and DA are representation/readout displacement summaries; AS is 1−SR/100 . The decomposition and heatmaps provide per-observation detail.
Figure 13: Spatial changes in action-facing attention on the evaluated FastWAM checkpoint. Each panel shows the task-macro signed mean of ΔAij=Aijp−Aij0 , averaged over action queries and heads and mapped to the 7×14 visual-token grid after full-key softmax. Rows show L21/L29/L17/L20; columns show patches trained with K-only, V-only, and joint K + V objectives. K-only and V-only denote training objectives, not isolated cache-tensor interventions. Red/blue indicate increases/decreases.
Figure 14: Spatial changes in action-facing visual contribution on the evaluated FastWAM checkpoint. Each panel shows the signed task-macro mean over 50 observations of ΔIij=Aijp∥Vjp∥2−Aij0∥Vj0∥2 , averaged over action queries and heads on the 7×14 visual-token grid. Rows show L21/L29/L17/L20; columns show K-only, V-only, and joint K + V optimization objectives. K-only and V-only name training objectives, not isolated cache-tensor interventions. Red/blue indicate increases/decreases. These maps complement the quantitative case summary in Figure 11 .
Architecture
Attn.-map
K only
V only
K+V
FastWAM
90.68
65.77
66.43
58.87
LingBot-VA
92.45
93.37
92.63
93.18
Cosmos Policy
91.34
89.91
89.79
90.04
Appendix
Table 21: All-layer component-objective attacks across three WAMs. One evaluated checkpoint per architecture; values are pooled closed-loop Policy SR (%; lower is stronger) over independently trained single-layer patches. K-only and V-only denote end-to-end patch optimization targets. Bold marks the lowest SR among matched-10k cells in each architecture.
FastWAM
LingBot-VA
Layer
Attn.
K
V
K+V
Layer
Attn.
K
V
K+V
L00
89.5
83.5
31.5
1.5
L00
94.0
96.0
93.0
91.5
L01
87.5
31.5
28.0
30.5
L01
90.0
93.5
90.0
89.5
L02
91.0
9.5
10.0
10.0
L02
95.0
93.0
92.0
95.0
L03
91.5
87.5
6.0
15.5
L03
90.0
94.0
95.0
94.0
L04
91.5
67.5
91.0
95.5
L04
93.0
94.0
90.5
93.0
Appendix
Table 22: Complete layerwise component attacks for the evaluated FastWAM and LingBot-VA checkpoints. Each independently trained singleton uses BS1, 10k exposures and 200 evaluation episodes; values are Policy SR (%). The FastWAM attention-map controls record training seed 20260801; other cells retain the formal protocol label 42 where reported.
Matched 10k component objectives
Historical 5k target-scope objectives
Layer
Attn.-map
K only
V only
K+V
C-KV
A-KV
W-KV
J-KV
L00
89.5
92.0
92.0
93.0
92.0
90.5
92.5
91.5
L01
92.5
89.0
92.0
91.0
90.5
90.5
94.0
93.0
L02
95.0
88.5
88.0
88.5
89.5
91.0
91.5
89.5
L03
91.0
91.0
91.0
88.5
90.5
91.0
90.0
90.0
L04
90.5
90.0
89.5
89.0
87.5
92.5
90.5
91.0
Appendix
Table 23: Complete layerwise component attacks for the evaluated Cosmos Policy checkpoint. The first four columns are matched BS=1, formal 42-labeled, joint=1, 10k-exposure objectives; K+V targets current-primary slot 3. The last four columns report historical 5k C/A/W/J objectives. Every cell uses 200 episodes; Table 21 summarizes the matched 10k columns.
Layer
5k
10k
20k
Response
Layer
5k
10k
20k
Response
L00
64.5
1.5
0.0
Early
L15
60.5
41.0
28.5
Intermediate
L01
34.5
30.5
30.5
Early
L16
92.0
62.5
47.5
Intermediate
L02
10.0
10.0
10.0
Early
L17
91.0
11.5
6.0
Early
L03
91.0
15.5
9.5
Early
L18
46.5
39.0
35.0
Intermediate
L04
92.5
95.5
91.5
Other
L19
38.5
6.5
0.0
Early
L05
92.5
94.5
92.0
Other
L20
90.0
91.5
68.5
Other
Appendix
Table 24: Complete FastWAM singleton K/V exposure trajectories. Policy SR is in percent; labels are post-hoc reporting groups under the documented objective, exposure checkpoints, and rollout protocol.
Figure 15: Layerwise intervention diagnostics on the evaluated FastWAM checkpoint. (a) Tail-AV-selected targets and Rescue peaks. (b) K/V-footprint growth relative to each target’s 5k value. Colors denote retrospective response groups. Detailed protocols and analyses are provided in Appendices C.2 and C.3 .
Target
Group
5k
10k
15k
20k
20k/5k
L00
Early
13.768
15.131
15.638
16.121
1.1709
L01
Early
15.310
15.577
15.678
15.722
1.0269
L02
Early
16.536
16.939
17.014
17.032
1.0300
L03
Early
10.294
14.971
15.792
15.969
1.5513
L04
Other
8.533
8.886
9.026
9.119
1.0687
L05
Other
7.459
9.079
9.279
9.353
1.2539
Appendix
Table 25: Independent all-target exact-checkpoint K/V footprints. Absolute task-mean footprint Fℓ(s) and target-normalized 20k/5k ratio Rℓ(20k) . All thirty targets use training seed 20260801, four checkpoints from the same run per target, ten paired tasks per point, and thirty observed layers.
Class mean
20k across-layer distribution
Group
n
10k
15k
20k
Median
SD
Range
Early
9
1.3269
1.3889
1.4390
1.2390
0.4795
[1.0269, 2.4966]
Intermediate
5
1.0758
1.1077
1.1220
1.1060
0.0910
[1.0327, 1.2418]
Other
16
1.0618
1.0937
1.0975
1.0309
0.2246
[0.9086, 1.8927]
Appendix
Table 26: Class summaries of target-normalized footprints. Class means average Rℓ(s) after normalization within each target. Groups are retrospective; SD uses n−1 across target layers. All 5k ratios equal one.
Figure 16: Visual-state replacement profiles. Each layer averages 30 target patches and four fixed samples (tasks 0/1; frames 0/10); negative gains are clipped before across-layer normalization. Shading marks L22–L28. Five-k/20k profile correlations are 0.983/0.991 with unchanged Top-3 sets.
Attack target
Clean
Attack
Set A (3 layers)
Set B (3 layers)
A ∪ B (6)
L18
94
44
88
52
86
L29
96
16
26
32
58
L25 (weak control)
94
88
88
94
98
Appendix
Table 27: Historical pilot of oracle K/V replacement. Closed-loop Policy SR (%; higher means better control here), 50 episodes per cell. These 10k singleton patches provide supplementary mechanism evidence.
Feature deviation, token contrastive repulsion, and action-query attention dominance; no explicit K/V intervention.
UPA
Conceptual adaptation
Same geometry as UPA-RFAS.
Continuous action-chunk objective.
TMA
Conceptual adaptation
Same geometry as UPA-RFAS.
Continuous action-chunk objective.
UADA
Conceptual adaptation
Same geometry as UPA-RFAS.
Continuous action-chunk objective.
Appendix
Table 28: Baseline fidelity under our documented WAM adaptations. Areas refer to the affected resized view, not the complete input.
Accounting field
Record and comparison boundary
Updates / exposures
All six: BS=1, seed 42, 10,000 optimizer updates and 10,000 sample exposures.
GPU-hour basis
GPU-active or card-wall construction records, excluding victim pretraining, smoke tests, queueing, and rollout evaluation. LingBot DRIFT records training-loop time only (2.462206 GPU-h), also excluding loading.
Observation pool
DRIFT uses its historical Spatial pool, whereas DP-Attacker uses a task-balanced pool.
Appendix
Table 29: Construction-cost accounting and observation-pool boundaries for the baseline adaptations.
Layer
Action-L2
SR (%)
Layer
Action-L2
SR (%)
Layer
Action-L2
SR (%)
L00
2.1042
0.0
L10
0.9184
88.0
L20
1.3762
68.5
L01
2.1661
30.5
L11
0.4562
88.5
L21
2.5344
0.0
L02
2.8538
10.0
L12
0.3653
81.5
L22
0.2111
89.5
L03
2.9605
9.5
L13
1.2415
34.0
L23
0.2490
92.5
L04
0.3385
91.5
L14
1.9948
17.0
L24
0.2514
90.5
L05
0.8464
92.0
L15
1.9247
28.5
L25
0.1863
91.0
Appendix
Table 30: Matched BS=1, 20k singleton Action-L2 and Policy SR across 30 FastWAM layers. Each row is a separately trained single-layer joint K/V patch. Action-L2 is the mean unnormalized FP32 L2 norm of the full action-chunk difference over four fixed inputs (tasks 0/1, frames 0/10). Eval200 SR is matched by layer to Table 24 .
World-action models (WAMs) are emerging as a promising foundation for embodied control: rather than predicting actions alone, they learn representations that couple action generation with future world prediction. This coupling is often viewed as a source of robustness, interpretability, and safety, as a robot's action can in principle be checked against its imagined future. In this paper, we show that this assumption is fragile. We introduce BadWAM, a unified framework for modeling and evaluating World-Action Drift Attacks: a new class of WAM-specific adversarial attacks that use small visual perturbations to break the alignment between what a WAM imagines and what it executes. BadWAM characterizes this attack surface along two natural criteria: attack strength and stealthiness. When the adversary prioritizes disruption, BadWAM instantiates an action-only adversarial attack, which directly drives the model toward task-failing actions. When the adversary additionally prioritizes stealth, BadWAM instantiates an imagination-preserving adversarial attack, which seeks to induce harmful action shifts while keeping the model's predicted future close to its clean imagination. Together, these two attacks capture a spectrum of WAM-specific failures: from overt action hijacking to stealthier cases where the model appears to imagine a plausible future but executes a desynchronized action. We evaluate BadWAM across different variants of WAMs. Results show that our attacks substantially reduce task success rates under closed-loop execution. For example, our action-only attack reduces the model performance from 96.5% to 43.1% success. The results of our imagination-preserving attack further exposes a WAM-specific vulnerability: moderate future-preserving regularization can maintain strong attack performance while reducing future imagination drift.
Qi Li, Xingyi Yang, Xinchao Wang
National University of Singapore · The Hong Kong Polytechnic University
World models learn to predict how their environment will evolve, making them an important foundation for general-purpose robotic control. Yet world action models depend on camera inputs whose manipulation can corrupt the visual representations used across tasks and action policies. Existing attacks on these models optimize against the victim's actions or predicted futures and therefore require access to target-model outputs. In this paper, we propose an attack, TAPDreamer, against world action models that instead uses a public encoder alone to construct a fixed local perturbation that transfers across tasks and action architectures. TAPDreamer requires no target-policy queries. Our key insight is that interactions between patch-induced changes in attention weights and value vectors broadcast a nearly identical representation shift far beyond the patch footprint, and this shift remains stable across task observations. Guided by this insight, TAPDreamer uses six frames from one source task to maximize the global L1 distance between clean and patched encoder representations. In closed-loop evaluation, one frozen patch per benchmark, covering about 6.5% of the input, reduces FastWAM's success rate from 97.7% to 0.0% across 40 LIBERO tasks and from 90.86% to 0.0% across 50 RoboTwin tasks; matched random patches retain 81.5% and 79.2% success. The same patches reduce success to 1.45% and 1.00% on two DreamWAM configurations and to 10.60% on Motus. These results show that protecting downstream action generation alone is insufficient: defenses for world action models must also secure shared visual encoders against persistent local perturbations.
Xuanyu Lu, Fengqing Jiang, Kaiyuan Zheng +8
University of Washington · University of Georgia · Western Washington University +2
World Action Models (WAMs) inherit powerful priors from pretrained video generation models, but their variational autoencoder (VAE) representations are optimized for visual reconstruction rather than manipulation. External visual representations can improve manipulation, but existing methods couple representation choices with different integration mechanisms, obscuring their individual effects. We systematically compare visual representations, five integration mechanisms, and feature combinations under controlled settings. Our study yields three main findings. First, strong general-purpose visual representations provide broader robustness gains and can match or outperform representations pretrained for specific visual capabilities. Second, channel-wise joint prediction achieves the highest average success rate; further analyses suggest that future-feature prediction and the action expert's effective use of added visual information contribute to its gains. Third, additional sources should be selected for the task-relevant information they contribute beyond existing representations, as the same source can help one combination but harm another. However, even the best evaluated composition provides a smaller additional gain than the initial improvement from adding a strong general-purpose representation to the VAE-only baseline. We instantiate the resulting Representation Recipe as R2-WAM. It achieves state-of-the-art performance on RoboCasa GR1 Tabletop with an 84.7% success rate and reaches 92.8% on RoboTwin 2.0, improving over the VAE-only baselines by 5.9 and 2.2 percentage points, respectively. Taken together, these findings suggest that future work should prioritize stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition serving as a supplementary route to improvement.
Kai Liu, Jiangran Lyu, Tingyu Deng +9
Institute of Automation, Chinese Academy of Sciences · Galbot · Peking University +2