Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

Kunal Bhosikar; Nicole Meng; Charu Sharma; Yingjie Lao

cs.CVOct 30, 2026

Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

Authors: Kunal Bhosikar, Nicole Meng, Charu Sharma, Yingjie Lao

Organizations: IIIT Hyderabad · Miko · Tufts University

Abstract

Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding $3\times$ further than an equivalent agent shift, and swapping the block pose alone accounts for $99\%$ of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box $\ell_\infty$ perturbation of just $ε{=}4/255$ applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in $73$ -- $95\%$ of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from $87\%$ to $32\%$ , whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At $ε{=}8/255$ , the attack bridges $82$ -- $97\%$ of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.

Figures & tables

Figure 2: Overview. Top: LeWM encodes the camera frame $o_{t}$ and the goal image $o_{g}$ with a frozen ViT-Tiny encoder $E_{\theta}$ , forecasts the latent $\hat{z}_{H}$ under candidate actions with an action-conditioned transformer predictor $P_{\psi}$ , and a CEM planner searches for the actions that minimize the latent cost $C$ . Bottom: our attacker (orange) leaves all of this untouched. It (1) renders a counterfactual scene $\tilde{o}$ in which the task object is displaced, (2) encodes it with the public encoder to a target $z^{\star}$ , (3) optimizes an $L_{\infty}$ -bounded perturbation $\delta$ with $E_{\theta}(o{+}\delta)\approx z^{\star}$ by gradient steps through the encoder, and (4) adds $\delta$ to the goal image (i) or to the camera frames (ii). The planner then pursues the fake goal.

Figure 3: Latent geometry of task state in LeWM on Push-T. (a) Embedding displacement for a single-variable shift as a percentage of the median distance between unrelated scenes (

19.7

), with interquartile bands across scenes. Block position displaces the embedding far more than agent position and saturates near

16

–

32

pixels. (b) Cosine similarity between the actual finite displacement and the local tangent predicted from an infinitesimal shift. Alignment decays rapidly for the block, indicating a curved manifold, whereas the agent trajectory remains nearly linear.

Counterfactual scene	$\epsilon$	spoof succ.	non-target drift
Block $+9$ px	$2$ / $4$	$64\%$ / $95\%$	agent $3$ px
Block $+17$ px	$4$ / $8$	$73\%$ / $98\%$	agent $6$ px
Block $+35$ px	$4$ / $8$	$24\%$ / $90\%$	agent $16$ / $12$ px
Block rot. $45^{\circ}$	$4$ / $8$	$81\%$ / $95\%$	agent $9$ / $7$ px
Agent $+9$ px	$1$ / $2$ / $4$	$54\%$ / $79\%$ / $90\%$	block $<1$ px
Uniform noise	$8$	$<1\%$	all $<1$ px

Table 1: Spoof success on Push-T (

520

–

600

scenes per row). Share of scenes where the independent probe places the target object closer to the counterfactual pose than the true pose, and median drift of the non-targeted object (selectivity).

Noise $\epsilon{=}8$	goal $87.7$ camera $88.3$
Attacked input	clean	$\epsilon{=}1$	$2$	$4$	$8$	perfect
Goal image	$87.0$	$79.0$	$62.0$	$32.3$	$21.3$	$18.3$
Camera frame	$87.3$	–	$49.3$	$35.3$	$25.3$	$27.0$

Table 2: Planning under attack on Push-T (success %,

300

episodes,

+17

px block target). “Perfect” feeds the planner a clean render of the counterfactual target; matched-budget noise is harmless.

Counterfactual scene	$\epsilon{=}1$	$2$	$4$	$8$	perfect	nearer
Block $+17$ px	$79.0$	$62.0$	$32.3$	$21.3$	$18.3$	$65\%$
Block $+35$ px	$76.3$	$57.7$	$25.0$	$7.7$	$7.7$	$64\%$
Block rot. $45^{\circ}$	$82.7$	$67.3$	$32.0$	$25.3$	$28.0$	$47\%$

Table 3: Other counterfactual targets on Push-T (goal-image attack, success %,

300

episodes). “Nearer” is the share of episodes at

\epsilon{=}8/255

where the block ends closer to the spoofed pose than the true goal.

Figure 4: Clean versus attacked goal frames across the four LeWM environments ( $\epsilon{=}4/255$ ; selected by a fixed rule, Appendix G ). Residuals are amplified $30\times$ . Circles mark the true object pose (block, agent, fingertip, cube) and crosses mark the spoofed target pose; the attack closes $86\%$ , $92\%$ , $82\%$ , and $95\%$ of the squared distance to the target embedding (top to bottom).

Environment	clean	$\epsilon{=}4$	$8$	$16$	perfect	noise
Push-T	$87.0$	$32.3$	$21.3$	$19.0$	$18.3$	$87.7$
Two-Room	$86.0$	$38.4$	$30.0$	$32.4$	$38.4$	$87.2$
Reacher	$61.2$	$29.6$	$26.0$	$20.4$	$17.6$	$60.4$
Cube	$70.0$	$53.6$	$53.2$	$50.0$	$55.6$	$70.8$

Table 4: Goal-image attack across all four LeWM environments (success %; Push-T:

6

seeds, others

5

\times 50

episodes). “Perfect” uses a dataset-faithful counterfactual render; “Noise” is

\epsilon{=}8/255

uniform noise.

Environment	met at start	clean	$\epsilon{=}8$	perfect
Push-T	$0.3\%$	$87.0$	$21.1$	$18.1$
Two-Room	$8.4\%$	$84.7$	$23.6$	$32.8$
Reacher	$0.4\%$	$61.0$	$25.7$	$17.3$
Cube	$73.2\%$	$38.8$	$28.4$	$26.9$
already met	–	$81.4$	$62.3$	$66.1$

Table 5: Task sensitivity. Share of episodes already satisfying the goal at initialization, and goal-image success (%) on the remaining episodes (

n{=}299/229/249/67

). Last row: the

183

Cube episodes already met at start.

Attacked input ( $\epsilon{=}2/4/8$ )	Surprise ( $\times$ )	AUROC	TPR@5%
Single frame	$23$ / $70$ / $116\times$	$1.00$	$100\%$
All frames	$3.4$ / $5.4$ / $5.6\times$	$.84$ / $.89$ / $.91$	$44$ / $57$ / $59\%$
Goal image (any $\epsilon$ )	not predicted (no surprise)

Table 6: Surprise as a runtime detector (Push-T,

498

clips,

+17

px block target; TPR reported at

5\%

FPR). An isolated spoofed frame is reliably flagged, a temporally consistent camera attack is missed roughly half the time, and goal-image attacks trigger no surprise by construction.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Attacker controls	Backdoor [ 35 ]	Ours
Released checkpoint (weights)	✓	✗
Training data / pipeline	✗	✗
Downstream controller / planner	✗	✗
Single input image at test time	✗	✓
Passes clean checkpoint audit	partial	by constr.

Appendix

Table A1 : Comparison of threat models. We contrast our test-time, input-only adversary against the supply-chain checkpoint backdoor of Riaño et al . [ 35 ] . A “ ✓ ” indicates components modified or controlled by the adversary.

Steps	passes	GPU (s)	batch (ms)	CPU (s)	spoof succ.
$100+20$	$121$	$0.46$	$57$	$3.39$	$73\%$
$50+20$	$71$	$0.27$	$35$	$1.83$	$73\%$
$20+20$	$41$	$0.19$	$20$	$1.21$	$71\%$
$10+20$	$31$	$0.12$	$15$	$0.91$	$70\%$
$20+5$	$26$	$0.10$	$13$	$0.76$	$70\%$
$10+5$	$16$	$0.07$	$8$	$0.44$	$62\%$

Appendix

Table A2 : Computational cost versus PGD iteration budget (Push-T,

+17

px block target,

\epsilon{=}4/255

599

scenes). “Steps” denotes continuous sign-gradient steps in the

\epsilon

-ball plus discrete unit steps on the integer lattice (top row corresponds to our default configuration). Runtime is reported for a single image on one GPU (NVIDIA RTX PRO 5000), across

8

CPU threads, and amortized per image over a batch of

250

on the GPU. “Spoof succ.” matches the definition in Tab. 1 .

Scene pair subset	block sep.	latent dist.	Spearman $\rho_{S}$
All pairs	51 px	19.7	0.16
Closest $0.1\%$	0 4 px	0 8.0	0.57
Closest $5\%$	14 px	18.5	0.41
Farthest $95\%$	53 px	19.7	0.11

Appendix

Table A3 : Distance saturation in LeWM’s latent metric. We report the Spearman rank correlation (

\rho_{S}

) between squared latent distance and squared ground-truth state distance across

6.3

M clean Push-T scene pairs, stratified by physical state separation (alongside median block separation and median Euclidean latent distance). Results are consistent across random seeds.

Figure A1 : Closed-loop planning success versus perturbation budget

\epsilon

on Push-T (

300

episodes; error bars denote

95\%

Wilson confidence intervals). Both the goal-image and camera-frame attacks degrade performance monotonically toward the unconstrained counterfactual floor, while uniform random noise of equal magnitude has no effect.

Environment	entity	tolerance	target shift	target source
Push-T	block	$8.8$ px	$17.5$ px	render
Two-Room	agent	$16$ px	$32$ px	render
Reacher	arm	$0.05$ rad	$0.4$ rad	real frame
Cube	cube	$4$ cm	$8$ cm	goal state

Appendix

Table A4 : Cross-environment attack protocol. Counterfactual target sources: render uses our standalone simulator renderer; real frame selects an on-manifold dataset frame from the same episode with a displaced arm configuration; goal state re-renders the goal frame’s exact simulator state after translating only the cube.

Environment	re-render vs. dataset	clean goal $\to$ target
Push-T	$0.34$	$278$
Two-Room	$<10^{-4}$	$390$
Cube	$0.005$	$222$
Reacher	$\mathbf{223}$	$193$

Appendix

Table A5 : Simulator rendering fidelity. Median squared Euclidean distance in latent space between a dataset frame and our standalone re-render of the identical state, compared against the median squared distance between the clean goal frame and the displaced counterfactual target (

50

goal frames per benchmark).

$\epsilon$	nearer target	err. to target	err. to true	gap closed
$2$	$56.3\%$	$15.6^{\circ}$	$17.6^{\circ}$	$0.75$
$4$	$58.8\%$	$13.4^{\circ}$	$16.0^{\circ}$	$0.82$
$8$	$72.5\%$	$11.9^{\circ}$	$20.1^{\circ}$	$0.80$
$16$	$98.8\%$	$4.9^{\circ}$	$20.8^{\circ}$	$0.95$

Appendix

Table A6 : Reacher joint-angle probe evaluation on real dataset frames (

80

goal frames across

\ell_{\infty}

budgets

\epsilon

). Angular errors denote the maximum per-joint deviation in degrees (baseline clean probe error is

0.26^{\circ}

Environment	$\epsilon{=}2$	$4$	$8$	$16$
Push-T	$.52$	$.86$	$.97$	$.99$
Two-Room	$.86$	$.91$	$.86$	$.90$
Reacher	$.79$	$.82$	$.82$	$.95$
Cube	$.93$	$.96$	$.97$	$.99$

Appendix

Table A7 : Fraction of squared latent distance closed by the goal-image attack , defined as

1-\lVert z_{\mathrm{adv}}-z^{\star}\rVert_{2}^{2}/\lVert z_{0}-z^{\star}\rVert_{2}^{2}

(median across episodes per seed, averaged across seeds;

6

seeds on Push-T,

5

on others).

Adversarial Attacks on World Models

See all 19 papers

Keeping JEPA World Models Plannable When Little of the Frame Moves
Oct 2, 2026Florian Strohm, Patrick Wagner, Jannik Schwab +1Language-Conditioned Robot Planning World Model Learning
Fraunhofer IPA, Stuttgart, Germany · University of Stuttgart, Germany
Reperesentation Geometry Matters for Planning with JEPA World Models
Aug 17, 2026Jiaming Hu, Yan Zheng, Shi Bo +9Representation Learning Neural Representation Geometry
Boston University · Unity Technologies
LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
Mar 13, 2026Lucas Maes, Quentin Le Lidec, Damien Scieur +2World Model Learning Joint-Embedding Predictive Architecture
Mila & Université de Montréal · New York University · Samsung SAIL +1

Oct 2, 2026J/K move · Enter open · S save

Keeping JEPA World Models Plannable When Little of the Frame Moves

Florian Strohm, Patrick Wagner, Jannik Schwab, Marco Huber

Fraunhofer IPA, Stuttgart, Germany · University of Stuttgart, Germany

Language-Conditioned Robot Planning World Model Learning World Model-Based Planning Action-Conditioned World Models Robot Task Planning Robotic Manipulation

Open paper AskPDF arXiv

Specifying a goal in language rather than as a goal frame is a natural interface for planning with a latent world model, but testing it needs scenes in which language must discriminate between several objects. We build SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes. On SLIM a LeWM world model that solves PushT succeeds on under 1% of trials, although a scripted controller with simulator state solves every tier. Probes locate the failure in the encoder: its latent is nearly action-insensitive, neither pusher nor object positions can be decoded from it, and rollouts are no better than copying the current latent forward. One inverse-dynamics auxiliary loss, applied to encoder latents and to predicted latents through a shared head discarded at test time, restores every probe and raises success from 0.003 to 0.35 (0.16 on the hard pushing tier, where a goal-agnostic policy scores zero), and improves PushT at twice the trained horizon. Controls attribute the repair to the gradient into the encoder, and a response sweep shows that the vanilla model plans once enough of the frame responds to actions. A cheap action-sensitivity probe, computable without environment access, acts as an empirical necessary condition: all configurations below its threshold failed to plan. On the repaired latent, a small language-goal head plans from sentences without retraining the world model: it reaches 0.84 on navigation (visual-goal oracle 1.00), follows the named zone when it is swapped with a decoy, and degrades gracefully to unseen nouns. A single goal sentence rarely completes a push, but given the push as a sequence of stage sentences the head raises success on the medium and hard pushing tiers from 0.04 to 0.25, on par with the goal-frame oracle, also when the switch between stages is read from the latent alone.

Abstract

Adversarial Attacks on World Models

Explore similar work

Keeping JEPA World Models Plannable When Little of the Frame Moves

Reperesentation Geometry Matters for Planning with JEPA World Models

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels