Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

Kunal Bhosikar; Nicole Meng; Charu Sharma; Yingjie Lao

cs.CVOct 30, 2026

Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

Authors: Kunal Bhosikar, Nicole Meng, Charu Sharma, Yingjie Lao

Organizations: IIIT Hyderabad · Miko · Tufts University

Abstract

Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding $3\times$ further than an equivalent agent shift, and swapping the block pose alone accounts for $99\%$ of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box $\ell_\infty$ perturbation of just $ε{=}4/255$ applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in $73$ -- $95\%$ of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from $87\%$ to $32\%$ , whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At $ε{=}8/255$ , the attack bridges $82$ -- $97\%$ of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.

Figures & tables

Figure 2: Overview. Top: LeWM encodes the camera frame $o_{t}$ and the goal image $o_{g}$ with a frozen ViT-Tiny encoder $E_{\theta}$ , forecasts the latent $\hat{z}_{H}$ under candidate actions with an action-conditioned transformer predictor $P_{\psi}$ , and a CEM planner searches for the actions that minimize the latent cost $C$ . Bottom: our attacker (orange) leaves all of this untouched. It (1) renders a counterfactual scene $\tilde{o}$ in which the task object is displaced, (2) encodes it with the public encoder to a target $z^{\star}$ , (3) optimizes an $L_{\infty}$ -bounded perturbation $\delta$ with $E_{\theta}(o{+}\delta)\approx z^{\star}$ by gradient steps through the encoder, and (4) adds $\delta$ to the goal image (i) or to the camera frames (ii). The planner then pursues the fake goal.

Figure 3: Latent geometry of task state in LeWM on Push-T. (a) Embedding displacement for a single-variable shift as a percentage of the median distance between unrelated scenes (

19.7

), with interquartile bands across scenes. Block position displaces the embedding far more than agent position and saturates near

16

–

32

pixels. (b) Cosine similarity between the actual finite displacement and the local tangent predicted from an infinitesimal shift. Alignment decays rapidly for the block, indicating a curved manifold, whereas the agent trajectory remains nearly linear.

Counterfactual scene	$\epsilon$	spoof succ.	non-target drift
Block $+9$ px	$2$ / $4$	$64\%$ / $95\%$	agent $3$ px
Block $+17$ px	$4$ / $8$	$73\%$ / $98\%$	agent $6$ px
Block $+35$ px	$4$ / $8$	$24\%$ / $90\%$	agent $16$ / $12$ px
Block rot. $45^{\circ}$	$4$ / $8$	$81\%$ / $95\%$	agent $9$ / $7$ px
Agent $+9$ px	$1$ / $2$ / $4$	$54\%$ / $79\%$ / $90\%$	block $<1$ px
Uniform noise	$8$	$<1\%$	all $<1$ px

Table 1: Spoof success on Push-T (

520

–

600

scenes per row). Share of scenes where the independent probe places the target object closer to the counterfactual pose than the true pose, and median drift of the non-targeted object (selectivity).

Noise $\epsilon{=}8$	goal $87.7$ camera $88.3$
Attacked input	clean	$\epsilon{=}1$	$2$	$4$	$8$	perfect
Goal image	$87.0$	$79.0$	$62.0$	$32.3$	$21.3$	$18.3$
Camera frame	$87.3$	–	$49.3$	$35.3$	$25.3$	$27.0$

Table 2: Planning under attack on Push-T (success %,

300

episodes,

+17

px block target). “Perfect” feeds the planner a clean render of the counterfactual target; matched-budget noise is harmless.

Counterfactual scene	$\epsilon{=}1$	$2$	$4$	$8$	perfect	nearer
Block $+17$ px	$79.0$	$62.0$	$32.3$	$21.3$	$18.3$	$65\%$
Block $+35$ px	$76.3$	$57.7$	$25.0$	$7.7$	$7.7$	$64\%$
Block rot. $45^{\circ}$	$82.7$	$67.3$	$32.0$	$25.3$	$28.0$	$47\%$

Table 3: Other counterfactual targets on Push-T (goal-image attack, success %,

300

episodes). “Nearer” is the share of episodes at

\epsilon{=}8/255

where the block ends closer to the spoofed pose than the true goal.

Figure 4: Clean versus attacked goal frames across the four LeWM environments ( $\epsilon{=}4/255$ ; selected by a fixed rule, Appendix G ). Residuals are amplified $30\times$ . Circles mark the true object pose (block, agent, fingertip, cube) and crosses mark the spoofed target pose; the attack closes $86\%$ , $92\%$ , $82\%$ , and $95\%$ of the squared distance to the target embedding (top to bottom).

Environment	clean	$\epsilon{=}4$	$8$	$16$	perfect	noise
Push-T	$87.0$	$32.3$	$21.3$	$19.0$	$18.3$	$87.7$
Two-Room	$86.0$	$38.4$	$30.0$	$32.4$	$38.4$	$87.2$
Reacher	$61.2$	$29.6$	$26.0$	$20.4$	$17.6$	$60.4$
Cube	$70.0$	$53.6$	$53.2$	$50.0$	$55.6$	$70.8$

Table 4: Goal-image attack across all four LeWM environments (success %; Push-T:

6

seeds, others

5

\times 50

episodes). “Perfect” uses a dataset-faithful counterfactual render; “Noise” is

\epsilon{=}8/255

uniform noise.

Environment	met at start	clean	$\epsilon{=}8$	perfect
Push-T	$0.3\%$	$87.0$	$21.1$	$18.1$
Two-Room	$8.4\%$	$84.7$	$23.6$	$32.8$
Reacher	$0.4\%$	$61.0$	$25.7$	$17.3$
Cube	$73.2\%$	$38.8$	$28.4$	$26.9$
already met	–	$81.4$	$62.3$	$66.1$

Table 5: Task sensitivity. Share of episodes already satisfying the goal at initialization, and goal-image success (%) on the remaining episodes (

n{=}299/229/249/67

). Last row: the

183

Cube episodes already met at start.

Attacked input ( $\epsilon{=}2/4/8$ )	Surprise ( $\times$ )	AUROC	TPR@5%
Single frame	$23$ / $70$ / $116\times$	$1.00$	$100\%$
All frames	$3.4$ / $5.4$ / $5.6\times$	$.84$ / $.89$ / $.91$	$44$ / $57$ / $59\%$
Goal image (any $\epsilon$ )	not predicted (no surprise)

Table 6: Surprise as a runtime detector (Push-T,

498

clips,

+17

px block target; TPR reported at

5\%

FPR). An isolated spoofed frame is reliably flagged, a temporally consistent camera attack is missed roughly half the time, and goal-image attacks trigger no surprise by construction.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Attacker controls	Backdoor [ 35 ]	Ours
Released checkpoint (weights)	✓	✗
Training data / pipeline	✗	✗
Downstream controller / planner	✗	✗
Single input image at test time	✗	✓
Passes clean checkpoint audit	partial	by constr.

Appendix

Table A1 : Comparison of threat models. We contrast our test-time, input-only adversary against the supply-chain checkpoint backdoor of Riaño et al . [ 35 ] . A “ ✓ ” indicates components modified or controlled by the adversary.

Steps	passes	GPU (s)	batch (ms)	CPU (s)	spoof succ.
$100+20$	$121$	$0.46$	$57$	$3.39$	$73\%$
$50+20$	$71$	$0.27$	$35$	$1.83$	$73\%$
$20+20$	$41$	$0.19$	$20$	$1.21$	$71\%$
$10+20$	$31$	$0.12$	$15$	$0.91$	$70\%$
$20+5$	$26$	$0.10$	$13$	$0.76$	$70\%$
$10+5$	$16$	$0.07$	$8$	$0.44$	$62\%$

Appendix

Table A2 : Computational cost versus PGD iteration budget (Push-T,

+17

px block target,

\epsilon{=}4/255

599

scenes). “Steps” denotes continuous sign-gradient steps in the

\epsilon

-ball plus discrete unit steps on the integer lattice (top row corresponds to our default configuration). Runtime is reported for a single image on one GPU (NVIDIA RTX PRO 5000), across

8

CPU threads, and amortized per image over a batch of

250

on the GPU. “Spoof succ.” matches the definition in Tab. 1 .

Scene pair subset	block sep.	latent dist.	Spearman $\rho_{S}$
All pairs	51 px	19.7	0.16
Closest $0.1\%$	0 4 px	0 8.0	0.57
Closest $5\%$	14 px	18.5	0.41
Farthest $95\%$	53 px	19.7	0.11

Appendix

Table A3 : Distance saturation in LeWM’s latent metric. We report the Spearman rank correlation (

\rho_{S}

) between squared latent distance and squared ground-truth state distance across

6.3

M clean Push-T scene pairs, stratified by physical state separation (alongside median block separation and median Euclidean latent distance). Results are consistent across random seeds.

Figure A1 : Closed-loop planning success versus perturbation budget

\epsilon

on Push-T (

300

episodes; error bars denote

95\%

Wilson confidence intervals). Both the goal-image and camera-frame attacks degrade performance monotonically toward the unconstrained counterfactual floor, while uniform random noise of equal magnitude has no effect.

Environment	entity	tolerance	target shift	target source
Push-T	block	$8.8$ px	$17.5$ px	render
Two-Room	agent	$16$ px	$32$ px	render
Reacher	arm	$0.05$ rad	$0.4$ rad	real frame
Cube	cube	$4$ cm	$8$ cm	goal state

Appendix

Table A4 : Cross-environment attack protocol. Counterfactual target sources: render uses our standalone simulator renderer; real frame selects an on-manifold dataset frame from the same episode with a displaced arm configuration; goal state re-renders the goal frame’s exact simulator state after translating only the cube.

Environment	re-render vs. dataset	clean goal $\to$ target
Push-T	$0.34$	$278$
Two-Room	$<10^{-4}$	$390$
Cube	$0.005$	$222$
Reacher	$\mathbf{223}$	$193$

Appendix

Table A5 : Simulator rendering fidelity. Median squared Euclidean distance in latent space between a dataset frame and our standalone re-render of the identical state, compared against the median squared distance between the clean goal frame and the displaced counterfactual target (

50

goal frames per benchmark).

$\epsilon$	nearer target	err. to target	err. to true	gap closed
$2$	$56.3\%$	$15.6^{\circ}$	$17.6^{\circ}$	$0.75$
$4$	$58.8\%$	$13.4^{\circ}$	$16.0^{\circ}$	$0.82$
$8$	$72.5\%$	$11.9^{\circ}$	$20.1^{\circ}$	$0.80$
$16$	$98.8\%$	$4.9^{\circ}$	$20.8^{\circ}$	$0.95$

Appendix

Table A6 : Reacher joint-angle probe evaluation on real dataset frames (

80

goal frames across

\ell_{\infty}

budgets

\epsilon

). Angular errors denote the maximum per-joint deviation in degrees (baseline clean probe error is

0.26^{\circ}

Environment	$\epsilon{=}2$	$4$	$8$	$16$
Push-T	$.52$	$.86$	$.97$	$.99$
Two-Room	$.86$	$.91$	$.86$	$.90$
Reacher	$.79$	$.82$	$.82$	$.95$
Cube	$.93$	$.96$	$.97$	$.99$

Appendix

Table A7 : Fraction of squared latent distance closed by the goal-image attack , defined as

1-\lVert z_{\mathrm{adv}}-z^{\star}\rVert_{2}^{2}/\lVert z_{0}-z^{\star}\rVert_{2}^{2}

(median across episodes per seed, averaged across seeds;

6

seeds on Push-T,

5

on others).

Adversarial Attacks on World Models

See all 19 papers

Oct 2, 2026

Keeping JEPA World Models Plannable When Little of the Frame Moves

Specifying a goal in language rather than as a goal frame is a natural interface for planning with a latent world model, but testing it needs scenes in which language must discriminate between several objects. We build SLIM, a pushing benchmark with several small objects and paired visual and language goals on identical scenes. On SLIM a LeWM world model that solves PushT succeeds on under 1% of trials, although a scripted controller with simulator state solves every tier. Probes locate the failure in the encoder: its latent is nearly action-insensitive, neither pusher nor object positions can be decoded from it, and rollouts are no better than copying the current latent forward. One inverse-dynamics auxiliary loss, applied to encoder latents and to predicted latents through a shared head discarded at test time, restores every probe and raises success from 0.003 to 0.35 (0.16 on the hard pushing tier, where a goal-agnostic policy scores zero), and improves PushT at twice the trained horizon. Controls attribute the repair to the gradient into the encoder, and a response sweep shows that the vanilla model plans once enough of the frame responds to actions. A cheap action-sensitivity probe, computable without environment access, acts as an empirical necessary condition: all configurations below its threshold failed to plan. On the repaired latent, a small language-goal head plans from sentences without retraining the world model: it reaches 0.84 on navigation (visual-goal oracle 1.00), follows the named zone when it is swapped with a decoy, and degrades gracefully to unseen nouns. A single goal sentence rarely completes a push, but given the push as a sequence of stage sentences the head raises success on the medium and hard pushing tiers from 0.04 to 0.25, on par with the goal-frame oracle, also when the switch between stages is read from the latent alone.

Florian Strohm, Patrick Wagner, Jannik Schwab +1

Fraunhofer IPA, Stuttgart, Germany · University of Stuttgart, Germany

Language-Conditioned Robot Planning World Model Learning World Model-Based Planning Action-Conditioned World Models Robot Task Planning Robotic Manipulation

Ask PDF

Aug 17, 2026

Reperesentation Geometry Matters for Planning with JEPA World Models

Joint-embedding predictive world models support planning through latent predictions, but unconstrained joint training can collapse distinct observations to identical embeddings. Two prominent strategies for avoiding collapse are to inherit pretrained features, as in DINO-WM, or to learn representations end-to-end with anti-collapse regularization, as in LeWorldModel (LeWM). Yet avoiding collapse does not ensure that latent distances distinguish outcomes in ways that matter for the task. In object manipulation, for example, success depends on the object's position and orientation relative to the goal. Such task-relevant state information can remain accurately decodable while barely influencing latent distance. The resulting planning cost may fail to reflect how close a predicted outcome is to the task goal. In this paper, we propose SCALE (State-CAlibrated Latent Embeddings), a method that correlates sampled pairwise latent distances with distances in task-relevant state space. Added to LeWM's existing objective, SCALE preserves its architecture, requires privileged state only during training, and adds no planning-time computation. We show that SCALE improves planning success over LeWM across manipulation and navigation tasks with multiple solvers and provide a comprehensive analysis of how SCALE reshapes representation geometry to support planning.

Jiaming Hu, Yan Zheng, Shi Bo +9

Boston University · Unity Technologies

Representation Learning Neural Representation Geometry World Models for Robotics World Model-Based Planning Joint-Embedding Predictive Architecture Model-Based Planning

Ask PDF

Mar 13, 2026

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels

Joint Embedding Predictive Architectures (JEPAs) offer a compelling framework for learning world models in compact latent spaces, yet existing methods remain fragile, relying on complex multi-term losses, exponential moving averages, pre-trained encoders, or auxiliary supervision to avoid representation collapse. In this work, we introduce LeWorldModel (LeWM), the first JEPA that trains stably end-to-end from raw pixels using only two loss terms: a next-embedding prediction loss and a regularizer enforcing Gaussian-distributed latent embeddings. This reduces tunable loss hyperparameters from six to one compared to the only existing end-to-end alternative. With ~15M parameters trainable on a single GPU in a few hours, LeWM plans up to 48x faster than foundation-model-based world models while remaining competitive across diverse 2D and 3D control tasks. Beyond control, we show that LeWM's latent space encodes meaningful physical structure through probing of physical quantities. Surprise evaluation confirms that the model reliably detects physically implausible events.

Lucas Maes, Quentin Le Lidec, Damien Scieur +2

Mila & Université de Montréal · New York University · Samsung SAIL +1

World Model Learning Joint-Embedding Predictive Architecture Latent World Models

Ask PDF

Abstract

Adversarial Attacks on World Models

Explore similar work

Keeping JEPA World Models Plannable When Little of the Frame Moves

Reperesentation Geometry Matters for Planning with JEPA World Models

LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels