Long-horizon world modeling in partially observable environments requires compact memories that preserve spatial structure despite accumulated localization drift. Inspired by hippocampal episodic and spatial memory and the multi-periodic coding of entorhinal grid cells, we propose HiSTFormer, a hierarchical spatiotemporal world model, which adopts a layer-wise organization of local temporal RoPE and multi-periodic spatial RoPE, together with content-conditioned address offsets, to capture recent temporal context and long-range spatial structure. A fixed-capacity memory keeps KV storage and per-step attention cost bounded as the sequence grows. On MemoryMaze and HM3D, HiSTFormer outperforms strong world-model baselines within the training sequence length and generalizes substantially better to prediction horizons far beyond those seen during training, while using a compact model. The hierarchical spatiotemporal design remains effective across deeper Transformer stacks, with consistent gains as depth increases. These results show that hierarchical spatiotemporal organization and content-adaptive spatial addressing provide effective inductive biases for efficient long-horizon world modeling.
Figures & tables
Figure 1: Overview of HiSTFormer. Observation frames are encoded into compact visual representations, while the motion encoder and coordinate transformation update a path-integrated grid-like spatial state. The temporal Transformer first extracts local context using temporal RoPE. Content-conditioned per-head offsets then adjust the grid-like spatial addresses used by the spatial Transformer to retrieve historical experience. The resulting latent representations are decoded autoregressively into future frames.
MemoryMaze
HM3D
Model
MSE ↓
SSIM ↑
LPIPS ↓
PSNR ↑
MSE ↓
LPIPS ↓
256 context frames → 64 predicted frames
Transformer
0.0100±0.0006
0.7092±0.0094
0.2886±0.0137
22.02±0.47
0.0137±0.0004
0.557±0.004
Mamba
0.0171±0.0060
0.6568±0.0348
0.3873±0.0813
19.01±2.14
0.0340±0.0004
0.739±0.003
RSSM
0.0184±0.0011
0.6269±0.0024
0.3914±0.0035
18.64±0.08
0.0423±0.0012
0.771±0.003
HPC–MEC
0.0082±0.0001
0.7392±0.0015
0.2447±0.0030
23.42±0.11
0.0137±0.0004
0.573±0.004
Table 1: Open-loop visual prediction on 9×9 MemoryMaze and HM3D. We report mean ± sample standard deviation over three training seeds. Best and second-best results are shown in bold and underlined, respectively, based on unrounded values.
Figure 2: Analysis of long-horizon visual prediction in Memory Maze. A : Qualitative predictions at selected rollout horizons under the 256→64 protocol. B : Per-step LPIPS over the 64-frame prediction horizon. C : LPIPS for the next 64 predicted frames as the available context length increases. D : MSE and LPIPS on spatially novel and revisited prediction frames.
Figure 3: Analysis of hierarchical temporal–spatial encoding. A : Comparison of positional encodings in two-layer models. B : Temporal–spatial layer arrangements at depths of 2, 4, and 6 layers. C : Effects of exchanging the default cache organizations of temporal and spatial layers. D : Combined comparison of positional encoding, hierarchy, and cache-organization variants. E : Zero-shot generalization of different spatial encodings from the 9×9 training environment to larger environments under the 256→64 and 500→500 protocols.
Figure 4: Analysis of content-conditioned learnable address offsets. A : LPIPS of the full HiSTFormer, a model trained without address offsets, and three test-time interventions applied after normal training: replacing the learned offsets with their mean, shuffling them, or setting them to zero. B : Distance from the base and offset-adjusted spatial addresses to the wall intersected by the current viewing direction. C : A qualitative example showing the ground-truth target, high-attention memories retrieved by spatial heads H0 and H2, the full prediction, and predictions obtained after shuffling the retrieval results of either head.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Protocol
Strategy
Long-term entries
MSE ↓
SSIM ↑
LPIPS ↓
PSNR ↑
256→64
Temporal coverage (default)
96
0.006216
0.7587
0.2205
24.400
Fixed interval ( k=4 )
56
0.006142
0.7582
0.2218
24.323
Random eviction ( 80% )
43.56
0.006480
0.7531
0.2294
24.095
500→500
Temporal coverage (default)
96
0.013600
0.6989
0.2926
21.166
Fixed interval ( k=8 )
59
0.013712
0.6961
0.2973
21.037
Random eviction ( 80% )
89.58
0.013623
0.6985
0.2945
21.100
Appendix
Table 2: Sensitivity to the test-time long-term memory-management strategy. All results use checkpoints trained with the default temporal-coverage strategy. The number of long-term entries is averaged over evaluation sequences.
Protocol
Training / inference cache
Peak KV (MiB) ↓
MSE ↓
SSIM ↑
LPIPS ↓
PSNR ↑
256→64
Full / Full
2.492
0.006856
0.747846
0.234693
23.844
R+C / R+C
1.000
0.006602
0.749573
0.231666
23.922
500→500
Full / Full
7.805
0.014121
0.689286
0.307988
20.648
R+C / R+C
1.000
0.014261
0.689550
0.304276
20.582
Appendix
Table 3: From-scratch comparison of full and bounded spatial KV caches using the SS model. R+C retains at most 128 entries during both training and inference. Peak spatial KV-cache memory is reported in MiB.
Figure 5: Representative layouts and sampled trajectories in 9×9 , 11×11 , and 13×13 Memory Maze environments. The dashed segment marks a 256-step ground-truth window, and the cross marks its endpoint.
Figure 6: Accumulation of path-integration error in the main HiSTFormer model on our collected and the official datasets. Position is decoded from the largest-period component of the grid-like representation and compared with the ground-truth position over the first 500 observed frames. The larger error on our collected trajectories indicates more severe localization drift and a more challenging setting for long-horizon visual prediction.
Figure 7: HM3D test trajectories and historical observation coverage. Twenty ground-truth trajectories from 10 test scenes are shown under the 500→500 protocol. Blue and orange lines indicate the observed 500-frame trajectories and the subsequent 500-frame ground-truth trajectories, respectively. Green circles, black diamonds, and red crosses mark the start, observation endpoint, and prediction endpoint. Each panel reports the position and pose matching rates against observed history, using thresholds of 0.5 m and 30∘ .
256→64
500→500
Matching criterion
MemoryMaze
HM3D
MemoryMaze
HM3D
Position
69.27%
87.27%
84.78%
99.27%
Position and heading
53.27%
71.95%
71.99%
96.06%
State-action transition
42.09%
65.39%
59.43%
92.32%
Appendix
Table 4: Historical observation coverage on MemoryMaze and HM3D. We report the percentage of predicted frames matching an observed position (distance ≤0.5 ), position and heading (angular difference ≤30∘ ), or state-action transition (matching start/end states and action). Only observed history is used for matching. Distance is measured in maze coordinate units for MemoryMaze and meters for HM3D.
Figure 8: Qualitative predictions on HM3D-based trajectories. HiSTFormer preserves major scene structure and viewpoint evolution across multiple test scenes. The 32- and 64-frame predictions extend beyond the 16-frame training horizon; frozen-VAE reconstructions are shown as a reference for representation quality.
MEC update
MSE ( 256→64 ) ↓
MSE ( 500→500 ) ↓
LPIPS ↓
SSIM ↑
PSNR ↑
CI-R
0.012053
0.018827
0.410430
0.647472
17.2523
SI
0.009785
0.015406
0.369950
0.662986
18.1231
Appendix
Table 5: Comparison of HPC–MEC update protocols. Both variants are trained from scratch for 10,000 gradient steps using identical configurations and the same random seed. LPIPS, SSIM, and PSNR are reported under the 500→500 protocol.
Protocol
Training setting
MSE ↓
PSNR ↑
SSIM ↑
LPIPS ↓
200→17
Matched budget
0.006815
21.672
0.6869
0.1143
Extended training
0.005252
22.797
0.7257
0.0927
256→64
Matched budget
0.017093
17.672
0.5738
0.2345
Extended training
0.015293
18.155
0.5954
0.2150
500→500
Matched budget
0.026328
15.797
0.4880
0.3169
Extended training
0.025996
15.851
0.4909
0.3120
Appendix
Table 6: CoME performance under the matched and extended training budgets. The extended setting trains Base, STM, and Spatial Base for 30,000 , 50,000 , and 50,000 gradient steps, respectively.
Model
MSE ( 256→64 ) ↓
LPIPS ( 256→64 ) ↓
MSE ( 500→500 ) ↓
LPIPS ( 500→500 ) ↓
HiSTFormer (ours)
0.005850
0.212404
0.013286
0.292006
Transformer
0.010017
0.288581
0.028135
0.406308
HEX96 + Full
0.009399
0.280305
0.025517
0.411359
XY + Full
0.009054
0.276214
0.025456
0.418372
Appendix
Table 7: Comparison of spatial RoPE with spatial representations injected as Transformer content. HEX96 + Full and XY + Full use relative temporal RoPE and a full KV cache. Values are means over three training seeds.
Figure 9: Visual Prediction Performance across Historical Revisit Frequencies. On HM3D, predicted frames are grouped by the number of historical pose-matching visits: 0, 1, 2–3, and 4 or more visits, using thresholds of 0.5 m in position and 30∘ in heading. (A–B) LPIPS and MSE under the 256→64 protocol; (C–D) the corresponding metrics under 500→500 . Both models are evaluated using the same training seed (Seed 42). Metrics are first averaged within each revisit group per test window, then averaged equally across windows containing that group. Prediction errors generally decrease with increasing revisit frequency for both models.
Protocol
Model
MSE ↓
LPIPS ↓
SSIM ↑
PSNR ↑
MemoryMaze
256→64
HiSTFormer
0.005850±0.000250
0.2124±0.0027
0.7639±0.0027
24.723±0.160
ESWM
0.008594±0.000532
0.2647±0.0068
0.7287±0.0083
22.997±0.356
500→500
HiSTFormer
0.013286±0.000454
0.2920±0.0037
0.7013±0.0047
21.146±0.168
ESWM
0.012702±0.000860
0.2973±0.0085
0.7023±0.0093
21.288±0.370
HM3D
Appendix
Table 8: Comparison of HiSTFormer and adapted ESWM on MemoryMaze and HM3D under two open-loop prediction protocols. Results are reported as mean ± standard deviation over three seeds. Best values are highlighted in bold.
Metric
Grid-like
Cartesian
Single-period
Observed memory / all attention (%)
53.13±2.16
54.20±3.18
55.75±1.80
Nearby + heading / all attention (%) ↑
4.79±0.10
4.64±0.33
4.37±0.34
Nearby + heading / observed attention (%) ↑
6.81±0.18
6.51±0.11
5.70±0.12
Nearby / observed attention (%) ↑
11.59±0.31
11.09±0.10
10.23±0.11
Attention-weighted distance (cells) ↓
2.93±0.07
3.00±0.09
3.12±0.08
Appendix
Table 9: Spatial relevance of memory attention on MemoryMaze ( 256→64 ). Nearby locations have a ground-truth distance below 0.5 environment units, and matching headings differ by less than 15∘ . All-attention fractions use the full attention distribution, while observed-memory fractions are normalized within ground-truth observed memories. Results are mean ± standard deviation over three seeds.
Figure 10: Distance-stratified prediction errors of single-period and multi-periodic spatial encodings. Under the 256-to-64 protocol, the horizontal axis shows the spatial distance from each predicted position to the last observed position. The vertical axis reports the error difference (Single-period minus HiSTFormer) in (A) MSE and (B) LPIPS. Orange curves show the mean over three training seeds, shaded regions indicate 95% confidence intervals, and gray curves show individual seeds. Positive values indicate lower prediction errors for HiSTFormer.
Figure 11: Motion Displacement Prediction Analysis. Comparison of forward displacements predicted by the action-only motion encoder and ground-truth forward steps across three training seeds. The horizontal and vertical axes represent ground-truth and predicted displacements, respectively, both in environment units. Blue hexagonal bins indicate sample density, with color intensity and opacity increasing logarithmically with sample count. Orange lines show linear fits. All three seeds exhibit approximately linear relationships, with differences in scale and sign.
Figure 12: Geometric Alignment and Memory Retrieval of Content-Conditioned Spatial Offsets. (A) Head-specific directional preferences across three training seeds and the relationship between offset magnitude and nearest-wall distance along each head’s preferred direction. Thin lines, thick lines, and white ticks indicate the 5th–95th percentile range, 25th–75th percentile range, and median, respectively. Black diamonds mark preferred directions estimated from separate calibration trajectories. (B) Head-specific spatial offsets in three representative scenes. (C) Spatial addresses and their relationships to the red and green walls in a representative case. (D) Historical memory frames receiving over 10% attention from H0 or H2, with their attention weights. (E) Predictions from the full model, No Offset, and targeted replacement of high-attention memory slots in H0, H2, or both.
World models aim to predict plausible futures consistent with past observations, a capability central to planning and decision-making in reinforcement learning. Yet, existing architectures face a fundamental memory trade-off: transformers preserve local detail but are bottlenecked by quadratic attention, while recurrent and state-space models scale more efficiently but compress history at the cost of fidelity. To overcome this trade-off, we suggest decoupling future-past consistency from any single architecture and instead leveraging a set of specialized experts. We introduce a diffusion-based framework that integrates heterogeneous memory models through a contrastive product-of-experts formulation. Our approach instantiates three complementary roles: a short-term memory expert that captures fine local dynamics, a long-term memory expert that stores episodic history in external diffusion weights via lightweight test-time finetuning, and a spatial long-term memory expert that enforces geometric and spatial coherence. This compositional design avoids mode collapse and scales to long contexts without incurring a quadratic cost. Across simulated and real-world benchmarks, our method improves temporal consistency, recall of past observations, and navigation performance, establishing a novel paradigm for building and operating memory-augmented diffusion world models.
Sebastian Stapf, Pablo Acuaviva Huertos, Aram Davtyan +1
Computer Vision Group Department of Computer Science University of Bern
Long-video generation and world models have shown strong potential for interactive entertainment and embodied simulation by predicting future observations conditioned on user actions and historical memory. However, as memory sequences grow longer and their structures become increasingly complex, managing long-range spatial context becomes increasingly challenging, calling for a more intelligent and systematic memory-management strategy. Building on the advancing spatial reasoning capabilities of multimodal large language models (MLLMs) and the broader vision of unified models, we propose Spatial Memory Intelligence (SMI), the first framework to systematically employ an understanding model for spatial-memory management in long-video world models. SMI introduces four coordinated atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering. Extensive experiments across multiple baselines, benchmarks, and world-model backbones demonstrate the effectiveness and generalizability of SMI, achieving comprehensive improvements in memory sparsity, generation stability, and spatial consistency.
Ying Yang, Guiyu Zhang, Lianghua Huang +5
The Chinese University of Hong Kong, Shenzhen · Alibaba Group · Nanjing University +3
World models trained via latent dynamics offer a scalable path for embodied planning, yet existing methods suffer from representational collapse, computational overhead, and limited long-horizon capability. We introduce Memory World Model (MemWM), an associative memory-enhanced latent world model that augments JEPA-style prediction with a content-addressable memory bank. MemWM learns high-level historical representations directly from observations and leverages them to condition predictions via adaptive layer normalization, complementing low-level action-conditioned dynamics within a shared latent space. To prevent memory slots from collapsing into identical representations, we propose two complementary losses that encourage structural diversity across slots and balanced utilization across timesteps. On PushT, TwoRoom, Reacher, and OGBCube, MemWM attains an average success rate of 89.9%, matching DINO-WM (89.3%) while using roughly 4× fewer trainable parameters and no pretrained encoder, and improving over LeWM by 4.1%. Additionally, we extend the horizon to 50 timesteps, double the 25 timesteps of the original evaluation, demonstrating MemWM's robustness to long-horizon generalization.
Yuwei Sun, Siyu Zhu
Shanghai Academy of AI for Science · Fudan University