Goal-conditioned planning with image observations and goals requires predictions that distinguish visual states and action consequences. Spiking neural networks offer sparse computation, but single-step spikes can obscure motion cues and merge actions with different physical outcomes. We propose the Spiking Latent World Model (SpikWM), an end-to-end joint-embedding predictive architecture (JEPA) with one simulation step per image and one neural update per latent position. Spiking Regional Encoding retains spatial cues through regional pooling. Bi-Spike Action Encoding combines positive and negative binary responses, while Causal Membrane Prediction carries subthreshold evidence across the latent history with learned group-wise time constants. The encoder and predictor are trained jointly on reward-free image--action trajectories using next-embedding prediction and SIGReg. With a common planner, SpikWM achieves mean success rates of 88.67%, 97.33%, 99.33%, and 86.67% on Push-T, Two-Room, Reacher, and OGBench-Cube, respectively, outperforming two spiking baselines. Push-T ablations support each module, and controlled replays examine responses to appearance and physical state changes. Under a matched planning budget, SpikWM reduces estimated arithmetic energy by 75.6% relative to LeWM. Code will be released at https://github.com/ShawnYE618/SpikWM.
Figures & tables
Figure 1: Lost motion and action distinctions on Push-T. LeWM (ANN) and Spiking-LeWM (SNN). (a) Motion direction readout errors on 305 shared moving windows. (b) Replayed endpoints and predicted goal costs. Dashed outlines mark the goal. (c) Distinct predictor conditions for one pool of action blocks.
Figure 2: SpikWM architecture and planning. Spiking Regional Encoding (SRE) combines spatial spike processing, regional pooling and an output projection. Bi-Spike Action Encoding (BAE) uses learned group thresholds and combines two binary streams through separate maps. Causal Membrane Prediction (CMP) includes conditional causal blocks, final normalization and an output projection. Within each block, LayerNorm precedes action-conditioned scaling and shifting. Shared SRE branches learn from next-latent prediction and SIGReg. The cross-entropy method replans using terminal goal distance.
Success (%) ↑
Method
Spiking
Updates
Features
Params (M)
Push-T
Two-Room
Reacher
OGBench-Cube
PLDM † [ 27 ]
×
1
Learned
18.03
71.8±6.1
93.0±2.0
69.5±5.8
51.2±1.6
DINO-WM † [ 43 ]
×
1
DINOv2
41.75
65.2±6.6
99.6±0.9
67.2±1.8
74.4±2.6
Sub-JEPA [ 42 ]
×
1
Learned
18.03
89.00±5.33
95.00±2.76
84.00±4.00
76.33±5.99
Fast-LeWM [ 15 ]
×
1
Learned
17.91
91.7±4.1
95.4±3.2
84.0±5.9
71.7±8.4
Σ -LeWM [ 46 ]
×
1
Learned
18.03
94.5±3.0
92.4±4.4
86.0±4.8
82.6±5.0
Table 1: Goal-conditioned planning success. SNN results are mean ± sample SD across three evaluation seeds. † denotes Anchor-WM’s five-seed results [ 32 ] . ‡ marks our Spiking-WM adaptation. Updates count ANN passes or SNN steps per latent position. Parameters cover the deployed Push-T modules. Bold marks the highest mean within each group.
Figure 3: Open-loop forecasts on four tasks. Frame pairs show steps 15/20 (Push-T), 30/35 (Two-Room), 20/35 (OGBench-Cube), and 25/30 (Reacher). We render state readouts of the latent forecasts under shared actions [ 20 ] .
Figure 4: Component ablations and prediction diversity. (a) Push-T success. (b) Distinct Cube forecasts under 300 shared action blocks. Bars show mean ± sample SD across three evaluation seeds (a) and 12 starts (b).
Model
Success (%) ↑
FLOPs (G)
SOPs (G)
Energy (J) ↓
Time (s) ↓
LeWM
90.40
2747.75
0.00
6.320
1.53
Spiking-LeWM
53.33
1771.70
65.82
4.134
2.21
Spiking-WM
37.33
2608.01
53.97
6.047
3.82
SpikWM
88.67
600.15
178.38
1.541
1.97
Table 2: Push-T success, planning time & arithmetic energy. Success means follow Tab. 1 . Times are means on one RTX 4090 D. Counts use one common initial-plan request.
Figure 5: Perturbation responses in Two-Room. At step 50, we recolor or displace the agent. We divide next-embedding MSE by each model’s pre-intervention mean. Bands show 95% pointwise bootstrap intervals over 16 paired starts.
Linear
MLP
Model
MSE ↓
r↑
MSE ↓
r↑
Gripper position
LeWM
0.855±1.019
0.633
0.742±0.805
0.515
Spiking-LeWM
1.007±1.625
0.604
0.557±0.720
0.666
SpikWM
0.309±0.395
0.847
0.394±0.544
0.789
Cube position
Table 3: Physical probing on OGBench-Cube. Normalized MSE (SD in subscripts) and mean Pearson r .
Visual world models enable robotic planning by predicting future observations, but dense latent-state propagation and sample-intensive trajectory optimization incur high inference latency and peak memory usage, limiting real-time deployment on resource-constrained platforms. Existing sparse world-model acceleration methods either rely on unguided token sparsification, which may discard planning-relevant information and restrict achievable sparsity, or introduce heavy auxiliary modules and cumbersome multi-stage training pipelines. In this work, we present Scope-WM, an efficient visual world model that scopes computation to prediction-relevant latent regions and promising action sequences. Scope-WM distills prediction relevance into a lightweight action-conditioned selector and applies full dynamics prediction only to a compact subset of selected tokens. It updates the remaining tokens using a compact summary of foreground states and their changes, allowing the background to perceive foreground dynamics without costly token-to-token interactions. During planning, Scope-WM preserves and reuses high-quality action sequences discovered during the initial MPC search, focusing subsequent search under reduced rollout budgets. The resulting pipeline requires only a one-off selector distillation followed by a single joint training stage for the sparse world model. On the challenging Push-T task, Scope-WM reduces peak GPU memory usage and planning time to 18.1% and 14.3% of those of dense DINO-WM, respectively, corresponding to a 6.97× planning speedup, while maintaining competitive task performance. Further evaluations across five diverse visual planning tasks demonstrate the general applicability of Scope-WM. Code is available at https://github.com/ChunZheng2022/Scope-WM.
A predictive model receives a self-supervised signal whenever the consequence of an action is observed. Using that signal after deployment is difficult when dynamics and semantics share parameters: freezing prevents adaptation, whereas weight updates require optimizer state and may alter the learned representation. Here we introduce SpikeWorld, a 1.45M-parameter sparse spiking model jointly trained for heterogeneous sensory prediction, semantics, image-text binding and action-conditioned dynamics. At deployment, all trained parameters are frozen. Delayed next-state residuals update two external paths: cumulative fixed-bank losses select the bounded action correction, while route-specific residual matrices refine next-state prediction. Neither path uses labels, teacher outputs, rewards, success signals or the true shift value. Joint optimization improves action next-state MSE by 17.10% while also improving multimodal prediction, semantic accuracy and image-text retrieval. On held-out shear and attenuation streams, the combined external state improves aggregate prediction by 5.48% and 30.01%; its fixed-bank action path improves tracking by 24.20% and 3.94%, respectively. In a six-arm study comprising 450 new Meta-World trajectories (75 per arm), SpikeWorld raises frozen-policy reward by 7.90 (95% CI [2.48, 14.06]); the 13.33-point success difference is descriptive (CI [0, 40]). For identical sensory inputs, model parameters and inherited semantic outputs remain bitwise unchanged. A 16-byte RLS estimator obtains the highest non-oracle reward on linear attenuation, showing that the contribution is not superior linear identification, but its integration with a frozen multimodal spiking checkpoint. Reference code is publicly available at https://github.com/Oooorca/SpikeWorld.
Planning with a learned latent world model is a promising route to control from raw pixels, but a strong world model alone is not enough. We show this experimentally: even with a perfect world model (operationalized by replacing the learned forward predictor with an idealized rollout of the true environment dynamics), a finite-budget sample-based planner still fails on some tasks, indicating that the bottleneck can lie in search rather than in world-model accuracy. Motivated by this gap, we propose IMWM (Intuition Model + World Model), which pairs the world model with an intuition model trained from demonstrations to recognize promising actions. The two models collaborate through three lightweight components: (i) Retrieval Initialization, which initializes the planner's action proposal from a retrieved demonstration; (ii) Hybrid Cost, which combines the intuition score with the world-model rollout cost; and (iii) a Reliability Gate, which adjusts how much the planner trusts intuition in each setting. Across four pixel-based goal-reaching tasks (Two-Room, Reacher, Push-T, and OGBench-Cube), IMWM has higher mean success than the world-model-only planner on all four, with the largest gains on Two-Room (99.2%, +11.5 percentage points) and OGBench-Cube (94.7%, +28.5 percentage points).
Baoqi Gao, Ruize Han, Miao Wang +1
Beihang University · Shenzhen University of Advanced Technology