PASA-KV: Attacking Action-Facing Visual Routes in World Action Models

Zhongbo Wu; Jiakai Huang; Zheng Zhang; Siyu Xu; Kaixiang Chen; Xiaohong Liu; Shan You; Chang Xu

cs.ROOct 29, 2026

PASA-KV: Attacking Action-Facing Visual Routes in World Action Models

Authors: Zhongbo Wu, Jiakai Huang, Zheng Zhang, Siyu Xu, Kaixiang Chen, Xiaohong Liu, Shan You, Chang Xu,

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · ACE Robotics · Nanyang Technological University ·

Abstract

World action models (WAMs) increasingly couple internal visual or world representations with action generation, making their world-to-action pathways a critical yet poorly understood source of robustness. We investigate when perturbations to these internal representations actually become behaviorally consequential, and uncover a pronounced mismatch between representation perturbability and closed-loop vulnerability: attack effectiveness varies sharply across visual K/V routes, while larger representation displacement does not reliably imply stronger control failure. Our analysis attributes this mismatch to the action-conditioned readout, where keys determine where action queries read and values determine what content they receive. This motivates an action-facing view of vulnerability, in which the effect of a perturbation depends on how it is routed into action generation rather than on representation-space magnitude alone. We instantiate this view with Tail-AV, a short-probe signal for identifying vulnerable routes, and operationalize it in PASA-KV, which selects action-facing K/V targets before training a fresh task-shared attack. Experiments across multiple WAM systems and a physical robot validate readout-informed target selection, demonstrate strong closed-loop attacks, and reveal substantial system-level differences in resistance.

Figures & tables

Figure 1: Layerwise evidence for action-facing vulnerability. All panels use the evaluated FastWAM checkpoint. (a) Policy SR for 30 singleton joint K/V targets after 10k exposures; dashed: clean reference. (b) Tail-AV scores from short probes; rings mark Top-3. (c) Budget–SR trajectories at 5k/10k/20k exposures. Colors denote retrospective response groups, unused by the selector.

Panel of Figure 2 — Figure 2: PASA-KV construction pipeline. (a) Action-facing visual K/V interface. (b) Candidate routes are first probed with a small optimization budget and ranked by Tail-AV; a fresh task-shared patch is then trained only on the selected Top- $k$ routes. Victim weights remain fixed throughout, and rollout outcomes are excluded from layer selection.

Setting	Documented implementation
Trainable variable	FP32 RGB patch, $3\times 50\times 50$ ; frozen victim, BF16 forward
Optimizer	AdamW; learning rate $5\times 10^{-4}$ , betas $(0.9,0.95)$ , epsilon $10^{-8}$ , weight decay $0.01$
Schedule and gradients	Constant learning rate; no warm-up, scheduler, gradient clipping or accumulation in this patch loop
Initialization	Uniform $[0,1]$ initializer; primary attack suite indexed by protocol/run label $42$ ; singleton training configurations record seed $20260801$ ; final training starts from fresh pixels
Projection and loss	Clamp pixels to $[0,1]$ after every update; minimize $-J^{KV}$ (singleton) or $-J_{\rm joint}$ ; image-L1 weight zero
Updates and batch	BS1, one update per exposure; 100 per singleton probe, 10,000 per final patch

Table 1: FastWAM PASA-KV attack-construction recipe. These settings optimize RGB patch pixels with the victim weights frozen. Probe and final training share the optimizer; only the loss-target set and update budget change.

System	Scoring interface	Attention normalization and visual slice
FastWAM	Fixed-clip cache; first action-denoising step; 30 layers	Masked softmax over available keys, then current-frame visual columns; no within-slice renormalization.
LingBot-VA	One video step (endpoint 0.6), followed by the first of ten action steps; 30 layers	Action queries attend to full video-cache and action keys; retain current primary-view cache columns and values, conditional batch only.
Cosmos Policy	One joint-noise level; action-slot 4 queries; blocks 0–27	Softmax over all joint-slot keys; retain current-primary slot 3 columns and values. Slot indices are zero-based.

Table 2: Architecture-specific selector capture interfaces. The common influence reduction is specified in Section 4.2.1 .

Victim	Inference graph and visual route	Patch (px @ $x,y$ )	Evaluation solver	Clean
FastWAM	Direct; current-primary action-facing video K/V	$50\times 50$ at $(5,160)$	10 action steps	462/500 (92.4%)
LingBot-VA	Cascaded; current agent-view video-to-action route	$29\times 29$ at $(3,91)$	1 video + 10 action steps	462/500 (92.4%)
Cosmos Policy	Joint; current-primary conditioning slot	$50\times 50$ at $(5,160)$	1 joint denoising step	484/500 (96.8%)

Table 3: Architecture-specific main threat models. Historical frames and cached states are not retroactively modified.

Figure 3: FastWAM patch geometry on LIBERO-10. Tasks T0–T9 share $(x,y,w,h)=(5,160,50,50)$ on the $224\times 224$ primary view (4.98% of that view; 2.49% of the two-view input). The secondary view is unpatched.

System	Solver	Use	Success	Clean SR [95% CI]	Drop
LingBot-VA	v20/a50	High ref.	199/200	99.5% [97.22, 99.91]	—
LingBot-VA	v3/a10	Mid.	191/200	95.5% [91.67, 97.61]	$-4.0$ pp
LingBot-VA	v1/a10	Unified	187/200	93.5% [89.20, 96.16]	$-6.0$ pp
Cosmos	j5	High ref.	198/200	99.0% [96.43, 99.73]	—
Cosmos	j3	Mid.	191/200	95.5% [91.67, 97.61]	$-3.5$ pp
Cosmos	j1	Unified	194/200	97.0% [93.61, 98.62]	$-2.0$ pp

Table 4: Reduced-step clean controls. Each condition has 200 episodes; bracketed values are two-sided Wilson 95% intervals. Drops use the high-step reference.

	FastWAM		LingBot-VA		Cosmos Policy
Method	SR $\downarrow$	GPU-h $\downarrow$	SR $\downarrow$	GPU-h $\downarrow$	SR $\downarrow$	GPU-h $\downarrow$	Protocol
Shared clean reference	92.4	–	92.4	–	96.8	–	Shared Eval500 reference
DP-Attacker	81.0	4.23	94.2	1.97	95.2	2.88	Diffusion
DRIFT	53.0	4.78	92.4	2.46	89.2	3.33	Flow/EDM
UPA-RFAS	90.8	10.75	91.0	8.94	89.2	23.74	No PSM
UPA	77.4	6.45	94.2	8.65	89.6	3.14	Action

Table 5: Fixed-patch comparisons with WAM-adapted baselines. Policy SR (%, lower is stronger). All conditions use LIBERO-10 Eval500 (10 tasks

\times

50 trials). Each PASA-KV Top-

k

condition uses a separately trained, task-shared patch (10k exposures); bold marks the lowest observed SR per architecture. GPU-h reports recorded construction cost; PASA-KV values include Probe100, selection, and final training for all three WAMs (Section 6.2 ).

Method	Probe	Full construction
Tail-AV Top-3	0.568632	2.3297

Table 6: FastWAM Tail-AV Top-3 construction cost (GPU-h). The total includes probing, selection, and final patch training.

Figure 4: Matched FastWAM route-selection controls. Bars show observed Policy SR on Eval200 (ten tasks, 20 trials each); lower is a stronger attack. Labels give successes/200. All methods share the victim, final joint K/V objective, three targets, patch geometry, and 10k exposures. The complete layer sets and retrospective classes are retained in Appendix Table 16 .

Method	Seed 42	Seed 43	Seed 44	Mean SR (%)	SD (pp)
Tail-AV Top-3	3	3	3	1.50	0.00
Probe- $J^{KV}$	95	94	89	46.33	1.61
Evenly spaced	37	53	43	22.17	4.04
Random-1	173	170	169	85.33	1.04
Random-2	127	131	11	44.83	34.08
Random-3	50	19	25	15.67	8.22

Table 7: Fixed-layer final-training repeats on FastWAM. Counts are successes per 200 episodes; lower is stronger attack. Seeds 42/43/44 vary training RNG and shuffled sample order with identical saved initial pixels and layer sets. Evaluation uses seed 42 on a common Eval200 grid. Mean/SD are across runs. The seed-42 Tail-AV result uses the Eval200 subset of the primary Eval500 evaluation.

Run	Probe seed	Spearman	Top-3	Top-4	Top-5
Canonical	42	1.000000	Ref.	Ref.	Ref.
Seed42-A	42	0.998665	Yes	Yes	Yes
Seed42-B	42	0.997775	Yes	Yes	Yes
Seed43-A	43	0.998665	Yes	Yes	Yes
Seed43-B	43	0.996885	Yes	Yes	Yes
Seed43-C	43	0.994216	Yes	Yes	Yes

Table 8: FastWAM Tail-AV ranking repeatability on fixed scoring observations. A/B/C are separate probe-and-score executions at one seed, sharing initial pixels and scoring inputs. Spearman correlations compare eight 30-layer rankings with the canonical ranking; Top-

k

columns give exact set agreement. Only Seed44-B swaps L03 for L01 at Top-3.

Score	Early-group AUC $\uparrow$	$\rho_{10\mathrm{k}}$	$\rho_{20\mathrm{k}}$	Early@Top-3	Early@Top-5
Tail-AV	0.8624	0.521	0.508	3/3	4/5
Attention-only-tail	0.7672	0.506	0.537	3/3	3/5
Mean-AV	0.7831	0.229	0.173	2/3	3/5
Value-only-tail	0.6931	0.100	0.143	2/3	3/5

Table 9: Selector-score component ablation on the evaluated FastWAM checkpoint. All four score variants use the same Probe100 component-ablation run (30 layers and ten task-balanced scoring pairs).

\rho_{10\mathrm{k}}

and

\rho_{20\mathrm{k}}

are Spearman correlations with later layer AttackStrength; Early-group AUC and Early@Top-

k

are retrospective diagnostics using the nine Early layers as positives.

Figure 5: Complete selector-score component rankings on the evaluated FastWAM checkpoint. Thirty-layer rankings for Tail-AV, Attention-only-tail, Value-only-tail, and Mean-AV from the shared Probe100 component-ablation run. Colors denote retrospective Early, Intermediate, and Other groups; rings mark each ranking’s Top-3.

Panel of Figure 6 — Figure 6: Action-response diagnostics. (a) FastWAM Action-L2 versus SR across six patches ( $r=-0.956$ , $\rho=-0.714$ ). (b) One-shot and persistent exposure; dashed/solid denote local/trajectory responses. (c) LingBot-VA response over history length (log scale). (d) Cosmos $\sigma$ -normalized cumulative Action-L2. P/W/B denote primary/wrist/both views; suffixes give patched-frame and joint-step counts. Protocols: Sections 7.1 , 7.2 – 7.4 .

Rank	Layer	Normalized paired Action-L2	95% bootstrap CI
1	L02	0.47284	$[0.36650,0.58085]$
2	L01	0.25876	$[0.19552,0.33376]$
3	L06	0.12833	$[0.09933,0.15727]$
4	L00	0.09233	$[0.07501,0.11057]$
5	L12	0.08447	$[0.06727,0.10072]$

Table 10: Frozen high-response candidates selected from all 30 FastWAM layers by normalized paired Action-L2. Each layer has ten task-balanced paired records from an equal-budget Probe1000.

Deployment	Clean SR	Attacked SR	Recovery
No defense	93.50	1.33	–
Guard (30 layers)	53.00	49.67	52.44
Guard targeted (5)	93.50	87.50	93.49

Table 11: Pooled Guard results on fixed Tail-AV attacks. SR and recovery are percentages. Attacked SR pools three patches (600 episodes); clean SR uses 200 episodes per deployment. Recovery is the fraction of the attack-induced SR loss restored, relative to the 93.5% undefended clean reference. Per-patch results: Table 12 .

	No defense		Guard (30)		Guard targeted (5)
Input	SR	Count	SR	Count	SR	Count
Clean	93.50	187/200	53.00	106/200	93.50	187/200
Tail-AV Top-3	0.00	0/200	52.50	105/200	85.50	171/200
Tail-AV Top-4	0.00	0/200	40.50	81/200	89.00	178/200
Tail-AV Top-5	4.00	8/200	56.00	112/200	88.00	176/200
Pooled attacks	1.33	8/600	49.67	298/600	87.50	525/600

Table 12: Per-patch Guard results. Policy SR (%) and success counts, with 200 episodes per patch/deployment. No-defense results use the fixed patch family evaluated for Guard.

Figure 7: Piper FastWAM task scenes and reference rollouts. Each pair shows the initial scene (left) and a representative annotated frame from the provided ground-truth rollout (right). The manipulated object is marked in red and its target slot or placement site in blue. Tasks are pen-in-cup, second-drawer placement, carrot-in-bowl, and bowl stacking; closed-loop attack results appear in Figure 8 .

Figure 8: Piper real-robot attack effects across three tasks. Bars show observed closed-loop task success; labels give successes/10. Error bars show two-sided 95% Wilson binomial confidence intervals over ten trials per condition. Top-3 and Top-5 use separately trained task-shared patches. The drawer task, whose clean SR is 0/10, is excluded from attack comparison. The input perturbation is a digital head-camera overlay.

Figure 9: Bowl-stacking success under PASA-KV and baseline attacks. Each method has ten in-distribution physical trials. Bars show observed task success; labels give successes/10. Each method uses its own patch geometry, as specified in Section 9 . Lower SR means a stronger attack.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Rank	Layer	Score	Rank	Layer	Score	Rank	Layer	Score
1	L29	$0.049694$	11	L14	$0.009106$	21	L28	$-0.008567$
2	L02	$0.043262$	12	L19	$0.007954$	22	L10	$-0.017586$
3	L03	$0.032456$	13	L16	$0.004701$	23	L26	$-0.029588$
4	L01	$0.027921$	14	L17	$0.004437$	24	L22	$-0.034651$
5	L24	$0.026162$	15	L15	$0.002879$	25	L09	$-0.038692$
6	L00	$0.023169$	16	L18	$0.002561$	26	L06	$-0.043447$

Appendix

Table 13: Production Tail-AV selector scores and ranking on the evaluated FastWAM checkpoint. Descending Eq. 5 , with layer-ID tie-breaking; scores rounded to six decimals. Top-3/4/5 prefixes give the selected attack sets.

Evidence	Patch relationship	Endpoint and role
Primary comparison	Separately trained Top-3, Top-4, and Top-5 patches per model	Eval500; baseline comparison and target-count tradeoff
Matched selection	Primary FastWAM Top-3 plus separately trained controls	Common Eval200; isolates layer-set choice
Training repeats	Same layer sets and exact initial pixels; varied final-training seed and shuffled sample order	Common Eval200; fixed-set training sensitivity
Layerwise analysis	Separate singleton-target patches	SR trajectories and paired internal responses; target characterization
Input diagnostics	Patch specified per panel	Paired action/representation responses; FastWAM also has 30 diagnostic rollouts
Guard	Separate fixed Top-3/4/5 family	200 episodes per patch and deployment; mitigation

Appendix

Table 14: Experiment map. Patch lineage is listed for each evidence block. Eval200/500 denote 20/50 online trials per task.

Item	Documented record
Hardware and distributed training	8 MetaX GPUs; DeepSpeed ZeRO-2; FP32; seed 42; per-GPU batch 8; gradient accumulation 2; global batch 128
Data	LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-10; 40 tasks; 1,712 demonstrations; 277,713 training frames
Schedule	10 planned epochs; 21,700 planned optimizer updates; save/eval/log frequencies 2,000/200/10 updates
Optimizer and schedule	AdamW; learning rate $10^{-4}$ ; betas $(0.9,0.95)$ ; epsilon $10^{-8}$ ; weight decay 0.01; gradient clipping 1.0; 1,085-update 5% warm-up; cosine decay to $10^{-6}$ ; no EMA or early stopping
Model	Wan-AI/Wan2.2-TI2V-5B; approximately 5.00B video-expert and 1.02B action-expert parameters; 30-layer MoT; video/action/proprioception trainable, VAE and other modules frozen; both loss weights 1.0
ModelScope revision	c2272aadfec2d95cea869a4e748d5e28275621c6

Appendix

Table 15: FastWAM victim training configuration. Values describe the frozen victim model used for the FastWAM experiments.

Selection rule	Layer set	Class	Successes	Policy SR (%)
Probe- $J^{KV}$	L27/L23/L28	$\bullet$ / $\bullet$ / $\bullet$	95/200	47.5
Evenly spaced	L00/L15/L29	$\bullet$ / $\bullet$ / $\bullet$	37/200	18.5
Random-1	L27/L18/L16	$\bullet$ / $\bullet$ / $\bullet$	173/200	86.5
Random-2	L12/L23/L00	$\bullet$ / $\bullet$ / $\bullet$	127/200	63.5
Random-3	L20/L22/L00	$\bullet$ / $\bullet$ / $\bullet$	50/200	25.0
Tail-AV	L29/L02/L03	$\bullet$ / $\bullet$ / $\bullet$	3/200	1.5

Appendix

Table 16: Matched FastWAM selection controls. Same joint K/V loss, three targets, geometry, and 10k exposures; Eval200 (10 tasks

\times

20 trials). Dots denote retrospective response classes.

Paired layer statistic	Estimate	95% interval
K+V strictly stronger than K-only	17/30 (56.7%)	[40.0, 73.3]%
K+V strictly stronger than V-only	17/30 (56.7%)	[40.0, 73.3]%
Median $\mathrm{SR}_{K+V}-\mathrm{SR}_{K}$	$-1.25$ pp	[ $-3.75$ , 0.50] pp
Median $\mathrm{SR}_{K+V}-\mathrm{SR}_{V}$	$-1.00$ pp	[ $-3.50$ , 0.25] pp
Strong-attack layer coverage: K-only	7/30 (23.3%)	[10.0, 40.0]%
Strong-attack layer coverage: V-only	11/30 (36.7%)	[20.0, 53.3]%

Appendix

Table 17: Paired objective comparisons across the 30 layers of the evaluated FastWAM checkpoint. Lower Policy SR is a stronger attack. “Strong attack” denotes the descriptive coverage threshold SR

\leq 50\%

. Percentile intervals resample the 30 evaluated layers 20,000 times with seed 20260905.

Model	Attn.-map	K only	V only	K+V
FastWAM	90.68	65.77	66.43	58.87

Appendix

Table 18: All-layer component-objective attacks on the evaluated FastWAM checkpoint. Values are pooled closed-loop Policy SR (%; lower is stronger) over independently trained single-layer patches. K-only and V-only denote end-to-end patch optimization targets. Bold marks the lowest SR.

Figure 10: Action-facing visual K/V pathways. A fixed action query reads visual keys through attention and aggregates visual values into the action-facing readout. Key and value perturbations can alter routing and received content, respectively.

Figure 11: Representative component-objective attack cases on the evaluated FastWAM checkpoint. (a) L21, K-dominant. (b) L29, V-dominant. (c) L17, joint-beneficial. (d) L20, joint non-dominance. Bars show the objective contrasts discussed in the text; labels report singleton Policy SR.

Figure 12: Probe100 Tail-AV scores versus later attack strength. Each point is a FastWAM layer: Tail-AV from the shared 100-exposure component-ablation run versus later AttackStrength

=1-\mathrm{SR}/100

from the matched BS=1 singleton patches at 10k or 20k exposures.

Quantity	Mean $\pm$ SD	Observed range
Spearman $\rho$ to 5k AttackStrength	$0.3550\pm 0.0074$	$[0.3465,0.3655]$
Spearman $\rho$ to 10k AttackStrength	$0.5198\pm 0.0080$	$[0.5067,0.5297]$
Spearman $\rho$ to 20k AttackStrength	$0.5065\pm 0.0084$	$[0.4953,0.5194]$
Canonical Top-3 overlap	$0.9583\pm 0.1179$	$[0.6667,1.0000]$
Canonical Top-5 overlap	$1.0000\pm 0.0000$	$[1.0000,1.0000]$

Appendix

Table 19: Probe100 probe-seed stability across eight additional runs on the evaluated FastWAM checkpoint. Each repeat uses the fixed scoring pairs and a complete 30-layer ranking; values report mean

\pm

sample SD and observed range. The canonical ranking is used only as the Top-

k

set reference.

Layer	SR (%)	AS	$D_{K}$	$D_{V}$	$D_{A}$	$\\|\Delta_{K}\\|$	$\\|\Delta_{V}\\|$	$\\|\Delta_{KV}\\|$	Tail-AV
L00	1.5	0.985	0.377	1.433	0.429	9.4	105.8	112.9	0.425
L01	30.5	0.695	0.646	2.241	0.723	33.2	331.8	327.5	0.760
L02	10.0	0.900	0.553	1.967	0.566	37.5	324.7	310.8	0.474
L03	15.5	0.845	0.561	1.452	0.587	37.3	222.0	205.9	0.291
L04	95.5	0.045	0.565	1.357	0.687	58.3	214.6	208.1	0.187
L05	94.5	0.055	0.736	1.599	0.746	63.9	260.9	272.5	0.371

Appendix

Table 20: Complete 30-layer K/V mechanism summary. All rows use the joint K+V sweep with 50 observations from ten task-balanced tasks.

D_{K}

D_{V}

, and

D_{A}

are representation/readout displacement summaries; AS is

1-\mathrm{SR}/100

. The decomposition and heatmaps provide per-observation detail.

Figure 13: Spatial changes in action-facing attention on the evaluated FastWAM checkpoint. Each panel shows the task-macro signed mean of $\Delta A_{ij}=A^{p}_{ij}-A^{0}_{ij}$ , averaged over action queries and heads and mapped to the $7\times 14$ visual-token grid after full-key softmax. Rows show L21/L29/L17/L20; columns show patches trained with K-only, V-only, and joint K $+$ V objectives. K-only and V-only denote training objectives, not isolated cache-tensor interventions. Red/blue indicate increases/decreases.

Figure 14: Spatial changes in action-facing visual contribution on the evaluated FastWAM checkpoint. Each panel shows the signed task-macro mean over 50 observations of $\Delta I_{ij}=A^{p}_{ij}\lVert V^{p}_{j}\rVert_{2}-A^{0}_{ij}\lVert V^{0}_{j}\rVert_{2}$ , averaged over action queries and heads on the $7\times 14$ visual-token grid. Rows show L21/L29/L17/L20; columns show K-only, V-only, and joint K $+$ V optimization objectives. K-only and V-only name training objectives, not isolated cache-tensor interventions. Red/blue indicate increases/decreases. These maps complement the quantitative case summary in Figure 11 .

Architecture	Attn.-map	K only	V only	K+V
FastWAM	90.68	65.77	66.43	58.87
LingBot-VA	92.45	93.37	92.63	93.18
Cosmos Policy	91.34	89.91	89.79	90.04

Appendix

Table 21: All-layer component-objective attacks across three WAMs. One evaluated checkpoint per architecture; values are pooled closed-loop Policy SR (%; lower is stronger) over independently trained single-layer patches. K-only and V-only denote end-to-end patch optimization targets. Bold marks the lowest SR among matched-10k cells in each architecture.

	FastWAM					LingBot-VA
Layer	Attn.	K	V	K+V	Layer	Attn.	K	V	K+V
L00	89.5	83.5	31.5	1.5	L00	94.0	96.0	93.0	91.5
L01	87.5	31.5	28.0	30.5	L01	90.0	93.5	90.0	89.5
L02	91.0	9.5	10.0	10.0	L02	95.0	93.0	92.0	95.0
L03	91.5	87.5	6.0	15.5	L03	90.0	94.0	95.0	94.0
L04	91.5	67.5	91.0	95.5	L04	93.0	94.0	90.5	93.0

Appendix

Table 22: Complete layerwise component attacks for the evaluated FastWAM and LingBot-VA checkpoints. Each independently trained singleton uses BS1, 10k exposures and 200 evaluation episodes; values are Policy SR (%). The FastWAM attention-map controls record training seed 20260801; other cells retain the formal protocol label 42 where reported.

	Matched 10k component objectives				Historical 5k target-scope objectives
Layer	Attn.-map	K only	V only	K+V	C-KV	A-KV	W-KV	J-KV
L00	89.5	92.0	92.0	93.0	92.0	90.5	92.5	91.5
L01	92.5	89.0	92.0	91.0	90.5	90.5	94.0	93.0
L02	95.0	88.5	88.0	88.5	89.5	91.0	91.5	89.5
L03	91.0	91.0	91.0	88.5	90.5	91.0	90.0	90.0
L04	90.5	90.0	89.5	89.0	87.5	92.5	90.5	91.0

Appendix

Table 23: Complete layerwise component attacks for the evaluated Cosmos Policy checkpoint. The first four columns are matched BS=1, formal 42-labeled, joint=1, 10k-exposure objectives; K+V targets current-primary slot 3. The last four columns report historical 5k C/A/W/J objectives. Every cell uses 200 episodes; Table 21 summarizes the matched 10k columns.

Layer	5k	10k	20k	Response	Layer	5k	10k	20k	Response
L00	64.5	1.5	0.0	Early	L15	60.5	41.0	28.5	Intermediate
L01	34.5	30.5	30.5	Early	L16	92.0	62.5	47.5	Intermediate
L02	10.0	10.0	10.0	Early	L17	91.0	11.5	6.0	Early
L03	91.0	15.5	9.5	Early	L18	46.5	39.0	35.0	Intermediate
L04	92.5	95.5	91.5	Other	L19	38.5	6.5	0.0	Early
L05	92.5	94.5	92.0	Other	L20	90.0	91.5	68.5	Other

Appendix

Table 24: Complete FastWAM singleton K/V exposure trajectories. Policy SR is in percent; labels are post-hoc reporting groups under the documented objective, exposure checkpoints, and rollout protocol.

Figure 15: Layerwise intervention diagnostics on the evaluated FastWAM checkpoint. (a) Tail-AV-selected targets and Rescue peaks. (b) K/V-footprint growth relative to each target’s 5k value. Colors denote retrospective response groups. Detailed protocols and analyses are provided in Appendices C.2 and C.3 .

Target	Group	5k	10k	15k	20k	20k/5k
L00	Early	13.768	15.131	15.638	16.121	1.1709
L01	Early	15.310	15.577	15.678	15.722	1.0269
L02	Early	16.536	16.939	17.014	17.032	1.0300
L03	Early	10.294	14.971	15.792	15.969	1.5513
L04	Other	8.533	8.886	9.026	9.119	1.0687
L05	Other	7.459	9.079	9.279	9.353	1.2539

Appendix

Table 25: Independent all-target exact-checkpoint K/V footprints. Absolute task-mean footprint

F_{\ell}(s)

and target-normalized 20k/5k ratio

R_{\ell}(20\mathrm{k})

. All thirty targets use training seed 20260801, four checkpoints from the same run per target, ten paired tasks per point, and thirty observed layers.

Group	$n$	10k	15k	20k	Median	SD	Range
		Class mean			20k across-layer distribution
Early	9	1.3269	1.3889	1.4390	1.2390	0.4795	[1.0269, 2.4966]
Intermediate	5	1.0758	1.1077	1.1220	1.1060	0.0910	[1.0327, 1.2418]
Other	16	1.0618	1.0937	1.0975	1.0309	0.2246	[0.9086, 1.8927]

Appendix

Table 26: Class summaries of target-normalized footprints. Class means average

R_{\ell}(s)

after normalization within each target. Groups are retrospective; SD uses

n-1

across target layers. All 5k ratios equal one.

Figure 16: Visual-state replacement profiles. Each layer averages 30 target patches and four fixed samples (tasks 0/1; frames 0/10); negative gains are clipped before across-layer normalization. Shading marks L22–L28. Five-k/20k profile correlations are 0.983/0.991 with unchanged Top-3 sets.

Attack target	Clean	Attack	Set A (3 layers)	Set B (3 layers)	A $\cup$ B (6)
L18	94	44	88	52	86
L29	96	16	26	32	58
L25 (weak control)	94	88	88	94	98

Appendix

Table 27: Historical pilot of oracle K/V replacement. Closed-loop Policy SR (%; higher means better control here), 50 episodes per cell. These 10k singleton patches provide supplementary mechanism evidence.

Method	Fidelity	Recorded geometry	Target / accessible representation
DP-Attacker	Source-principle adaptation	FastWAM: $45\times 45$ , main, 4.04%, EOT. Cosmos: $50\times 50$ , primary, 4.98%. LingBot: $29\times 29$ , agent, $\sim 5.13\%$ .	Action-denoising target / diffusion surrogate; no explicit K/V intervention.
DRIFT	Source-principle adaptation; Cosmos EDM variant	FastWAM: $32\times 32$ , wrist, 2.04%, no EOT. Cosmos: $50\times 50$ , primary. LingBot: $18\times 18$ , wrist, $\sim 1.98\%$ , at $(55,110)$ .	Initial flow/velocity for FastWAM and LingBot; joint denoised prediction for Cosmos.
UPA-RFAS	Conceptual adaptation; no PSM	FastWAM/Cosmos: $50\times 50$ , primary. LingBot: $29\times 29$ , agent.	Feature deviation, token contrastive repulsion, and action-query attention dominance; no explicit K/V intervention.
UPA	Conceptual adaptation	Same geometry as UPA-RFAS.	Continuous action-chunk objective.
TMA	Conceptual adaptation	Same geometry as UPA-RFAS.	Continuous action-chunk objective.
UADA	Conceptual adaptation	Same geometry as UPA-RFAS.	Continuous action-chunk objective.

Appendix

Table 28: Baseline fidelity under our documented WAM adaptations. Areas refer to the affected resized view, not the complete input.

Accounting field	Record and comparison boundary
Updates / exposures	All six: BS=1, seed 42, 10,000 optimizer updates and 10,000 sample exposures.
GPU-hour basis	GPU-active or card-wall construction records, excluding victim pretraining, smoke tests, queueing, and rollout evaluation. LingBot DRIFT records training-loop time only (2.462206 GPU-h), also excluding loading.
Observation pool	DRIFT uses its historical Spatial pool, whereas DP-Attacker uses a task-balanced pool.

Appendix

Table 29: Construction-cost accounting and observation-pool boundaries for the baseline adaptations.

Layer	Action-L2	SR (%)	Layer	Action-L2	SR (%)	Layer	Action-L2	SR (%)
L00	2.1042	0.0	L10	0.9184	88.0	L20	1.3762	68.5
L01	2.1661	30.5	L11	0.4562	88.5	L21	2.5344	0.0
L02	2.8538	10.0	L12	0.3653	81.5	L22	0.2111	89.5
L03	2.9605	9.5	L13	1.2415	34.0	L23	0.2490	92.5
L04	0.3385	91.5	L14	1.9948	17.0	L24	0.2514	90.5
L05	0.8464	92.0	L15	1.9247	28.5	L25	0.1863	91.0

Appendix

Table 30: Matched BS=1, 20k singleton Action-L2 and Policy SR across 30 FastWAM layers. Each row is a separately trained single-layer joint K/V patch. Action-L2 is the mean unnormalized FP32 L2 norm of the full action-chunk difference over four fixed inputs (tasks 0/1, frames 0/10). Eval200 SR is matched by layer to Table 24 .

Adversarial Attacks on World Models

See all 18 papers

BadWAM: When World-Action Models Dream Right but Act Wrong
Jul 16, 2026Qi Li, Xingyi Yang, Xinchao WangWorld Models for Robotics Adversarial Attacks
National University of Singapore · The Hong Kong Polytechnic University
TAPDreamer: Transferable Adversarial Patches for World Action Models
Oct 5, 2026Xuanyu Lu, Fengqing Jiang, Kaiyuan Zheng +8Adversarial Patch Attacks Adversarial Attacks
University of Washington · University of Georgia · Western Washington University +2
What Visual Representations Do World Action Models Need?
Oct 9, 2026Kai Liu, Jiangran Lyu, Tingyu Deng +9World Action Models World Models for Robotics
Institute of Automation, Chinese Academy of Sciences · Galbot · Peking University +2

Jul 16, 2026J/K move · Enter open · S save

BadWAM: When World-Action Models Dream Right but Act Wrong

Qi Li, Xingyi Yang, Xinchao Wang

National University of Singapore · The Hong Kong Polytechnic University

World Models for Robotics Adversarial Attacks World Action Models Adversarial Perturbations Adversarial Attacks on World Models

Open paper AskPDF arXiv

World-action models (WAMs) are emerging as a promising foundation for embodied control: rather than predicting actions alone, they learn representations that couple action generation with future world prediction. This coupling is often viewed as a source of robustness, interpretability, and safety, as a robot's action can in principle be checked against its imagined future. In this paper, we show that this assumption is fragile. We introduce BadWAM, a unified framework for modeling and evaluating World-Action Drift Attacks: a new class of WAM-specific adversarial attacks that use small visual perturbations to break the alignment between what a WAM imagines and what it executes. BadWAM characterizes this attack surface along two natural criteria: attack strength and stealthiness. When the adversary prioritizes disruption, BadWAM instantiates an action-only adversarial attack, which directly drives the model toward task-failing actions. When the adversary additionally prioritizes stealth, BadWAM instantiates an imagination-preserving adversarial attack, which seeks to induce harmful action shifts while keeping the model's predicted future close to its clean imagination. Together, these two attacks capture a spectrum of WAM-specific failures: from overt action hijacking to stealthier cases where the model appears to imagine a plausible future but executes a desynchronized action. We evaluate BadWAM across different variants of WAMs. Results show that our attacks substantially reduce task success rates under closed-loop execution. For example, our action-only attack reduces the model performance from 96.5% to 43.1% success. The results of our imagination-preserving attack further exposes a WAM-specific vulnerability: moderate future-preserving regularization can maintain strong attack performance while reducing future imagination drift.

Abstract

Adversarial Attacks on World Models

Explore similar work

BadWAM: When World-Action Models Dream Right but Act Wrong

TAPDreamer: Transferable Adversarial Patches for World Action Models

What Visual Representations Do World Action Models Need?