Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced. Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive. We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory. PPMem augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion (DTF) module: each autoregressive chunk jointly produces RGB video and planar geometry, which merges into a persistent plane memory that conditions subsequent chunks via rendered depth. The resulting memory is three orders of magnitude more compact than per-pixel point clouds (~100K primitives, 5MB for a full scene), geometrically explicit (metric depth, normals, exportable surfaces), cross-view consistent (jointly predicted by the shared backbone), and adaptive (refined by the merging procedure as generation proceeds). On DL3DV and RealEstate10K, PPMem outperforms prior 3D-grounded world models in video quality while jointly producing metric-scale depth, surface normals, and exportable planar geometry, sustaining coherence across hundreds of frames under large viewpoint changes. Project page: https://hyzcluster.github.io/ppmem.
Figures & tables
Figure 1: Given a single input image and a camera trajectory, PPMem autoregressively generates video while accumulating a persistent plane-based 3D memory. We show two DL3DV scenes (indoor, outdoor) with generated frames at selected indices. Insets visualize the plane pool at each frame, colored by the current view frustum. The plane geometry grows to cover the full explored scene, serving as persistent spatial memory for subsequent generation.
Dense PC [ 24 ]
Surfels [ 13 ]
TSDF → PC [ 36 ]
Planes (PPMem)
3D source
External (MoGe [ 32 ] )
External (CUT3R [ 31 ] )
External (DA3 [ 15 ] )
Internal (joint head)
Cross-view consistent
✗
✗
partial
✓
Primitives (accumulated)
∼ 210M †
∼ 250K
∼ 1–2M
∼ 100K
Structured surfaces
✗
✗
✗
✓
Table 1: Properties of 3D representations for cross-chunk video memory. We compare along four aspects: whether geometry is predicted internally by the generation model or externally, whether predictions are cross-view consistent, the number of persistent 3D primitives, and whether the representation captures structured surfaces. Primitive counts are estimates at ∼ 400 frames, 576×1024 .
Figure 2: PPMem architecture overview. The Wan DiT receives noisy video latents channel-concatenated with projected source-view RGB/masks and partial depth rendered from the plane pool; context frames are prepended temporally. The DiT produces two token streams: video tokens decoded by the Wan VAE into RGB outputs, and geometry tokens adapted by geometry LoRA and fused with partial depth tokens via a ViT-based Depth Token Fusion module. The fused tokens are decoded into per-frame plane primitives, which merge into the persistent plane pool. The pool is rendered at the next chunk’s target cameras and fed back as conditioning.
N=49
N=121
Dataset
Method
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
DL3DV
Gen3C [ 24 ]
15.74
0.474
0.847
0.710
13.59
0.565
0.798
0.585
Voyager [ 11 ]
12.84
0.568
0.796
0.615
11.35
0.649
0.760
0.516
VMem [ 13 ]
11.88
0.636
0.781
0.477
10.87
0.722
0.696
0.274
SPMem [ 36 ]
14.70
0.492
0.853
0.703
13.03
0.607
0.788
0.536
PPMem (ours)
16.99
0.335
0.909
0.861
13.98
0.464
0.863
0.769
Table 2: Short-context comparison on DL3DV and RealEstate10K. We report two frame budgets: N=49 (native chunk size for Voyager and SPMem; VMem uses 48) and N=121 (Gen3C native chunk). At N=49 most baselines are single-chunk; at N=121 Voyager, VMem, and SPMem all require autoregressive looping.
Figure 3: Perceptual similarity vs. frame index on DL3DV long-context generation ( ∼ 350 frames). Left: CLIP similarity. Right: DINOv2 similarity. PPMem (blue) maintains the highest perceptual fidelity across the full generation. VMem collapses early; Gen3C and SPMem degrade steadily; the gap widens with frame count, especially on DINOv2.
Figure 4: Qualitative long-context comparison. PPMem preserves scene structure and color consistency across the full sequence. VMem collapses early under large viewpoint change; SPMem grows vague and drifts; Gen3C and Voyager lose structural alignment as the viewpoint diverges.
Variant
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
AbsRel ↓
δ<1.25↑
Normal ∘↓
Without persistent memory
14.58
0.407
0.835
0.733
0.272
64.0
25.2
MoGe-2 point cloud
14.37
0.423
0.832
0.716
0.339
55.0
25.8
Post-hoc Plana3R
14.22
0.432
0.829
0.713
0.311
57.0
26.3
Joint plane pool (ours)
14.91
0.387
0.835
0.741
0.210
70.3
24.1
Table 3: 3D representation and prediction ablation. We compare our joint plane pool against three inference-only alternatives (same checkpoint, no retraining): a same-model rollout without persistent memory, Gen3C-style per-frame MoGe-2 point cloud memory, and post-hoc standalone Plana3R reconstruction.
Figure 5: Depth quality analysis. (a) Per-frame depth metrics (AbsRel ↓ , δ<1.25↑ ) on DL3DV over 121 frames. PPMem maintains consistently lower depth error than Voyager, the only other baseline with joint depth output. (b) We scale the input depth conditioning by a known factor and measure the predicted/GT depth ratio. Concat + DTF (ours) tracks input scale faithfully ( 1.00±0.01 at scale 1.0, mean ± s.e. over 56 scenes), while channel concatenation alone exhibits compressed response at extreme scales. Measured as deviation from the identity mapping over the full scale range, DTF also stays closer to it (natural-log RMSE 0.273 vs. 0.320 for concatenation alone; 0.131 vs. 0.293 with scale augmentation).
Mechanism
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
AbsRel ↓
δ<1.25↑
Normal ∘↓
No depth conditioning
14.19
0.430
0.826
0.709
0.436
30.7
25.8
Channel concat only
14.88
0.388
0.838
0.743
0.212
72.3
23.8
Concat + DTF (ours)
14.91
0.387
0.835
0.741
0.210
70.3
24.1
Table 4: Depth conditioning mechanism ablation. All variants share the same backbone and training data, varying only the depth conditioning mechanism.
Variant
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
AbsRel ↓
δ<1.25↑
Normal ∘↓
Plana3R upsampling head
14.68
0.395
0.832
0.737
0.256
59.2
26.2
VAE-init decoder (ours)
14.91
0.387
0.835
0.741
0.210
70.3
24.1
− rasterization loss
14.82
0.391
0.834
0.736
0.217
66.4
25.4
− normal loss
13.27
0.502
0.807
0.675
—
—
—
Table 5: Feature decoder and loss component ablations. Top: decoder architecture (all else fixed). Bottom: loss removal (full model as baseline).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
Value
Model
Base video DiT
Wan 2.2 TI2V 5B
VAE
Per-frame (1:1 frame mapping, zdim=48 )
Depth token fusion encoder
ViT-Base ( ∼ 87M params), DINOv2-Base init
Feature decoder
Initialized from Wan VAE upsample block
LoRA (Stage 2)
Rank 256, α=256 , self-attention, all 30 blocks
Trainable params (Stage 2)
∼ 377M / 5.4B (7.0%)
Appendix
Table S1 : Full training hyperparameters for Stage 2. Stage 2 freezes the DiT backbone and injects LoRA adapters on self-attention; only geometry-specific modules and LoRA are trained. Stage 1 hyperparameters are similar except no plane loss, no DTF, no freeze/LoRA.
Anchor
History
Overlap
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
AbsRel ↓
δ<1.25↑
Normal ∘↓
Component ablation
No context
✗
✗
✗
14.60
0.403
0.835
0.730
0.219
64.2
24.5
Overlap only
✗
✗
✓
14.73
0.394
0.834
0.738
0.208
69.2
24.3
History + Overlap
✗
✓
✓
14.77
0.393
0.832
0.739
0.218
70.2
24.3
All (default)
✓
✓
✓
14.91
0.387
0.835
0.741
0.210
70.3
24.1
History scaling (Anchor + Overlap + varying history count)
Appendix
Table S2 : Context frame configuration ablation, inference-only. We vary the context frame composition at inference time without retraining (the model was trained with context dropout 0.1). Anchor = frame 0 of the video; History = frames from preceding chunks; Overlap = past RGB frames selected by coverage of the target viewpoints. Top: component ablation. Bottom: history count scaling. Evaluated on 56 DL3DV indoor scenes at 288×512 .
Outdoor (8 scenes)
Indoor (8 scenes)
Mechanism (cumulative)
Abs Rel ↓
δ<1.25↑
Coverage
Abs Rel ↓
δ<1.25↑
Coverage
Planes
Baseline (no merging improvements)
0.479
41.1
0.865
0.311
46.6
0.948
518K
+ Within-chunk dedup (stage 3)
0.486
44.7
0.833
0.297
51.8
0.920
116K
+ Cross-chunk replacement (stage 5)
0.477
47.0
0.803
0.296
52.7
0.916
100K
+ Forward-looking prune (stage 6)
0.469
48.3
0.798
0.285
53.9
0.908
100K
+ Boundary rejection (stage 2)
0.472
49.6
0.786
0.284
56.5
0.900
97K
Appendix
Table S3 : Progressive build-up of the plane merging pipeline. Depth accuracy is measured by absolute relative error (Abs Rel) and the percentage of pixels with predicted/GT depth ratio within 1.25 ( δ<1.25 ). Coverage is the fraction of pixels receiving a valid rendered depth. Evaluated on 16 DL3DV scenes (8 outdoor + 8 indoor).
Figure S1 : Accumulated plane pool vs. Plana3R reconstruction. We render depth from our plane pool and from planes reconstructed by Plana3R [ 18 ] on the same generated video. The pool built online during generation stays consistent, whereas pairwise reconstructions conflict across views and fragment.
Planar coverage
Low
Middle
High
# Scenes
34
32
34
PSNR ↑
12.98
13.67
15.27
LPIPS ↓
0.482
0.466
0.443
AbsRel ↓
0.289
0.250
0.195
δ<1.25↑
64.2
72.5
79.0
Appendix
Table S4 : Performance by scene planarity. The 100 DL3DV scenes are grouped into within-domain tertiles of planar coverage; all metrics are computed at N=121 .
N=49
N=121
Method
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
PSNR ↑
LPIPS ↓
CLIP sim↑
DINOv2 sim↑
WorldPlay [ 25 ]
13.22
0.532
0.883
0.784
11.98
0.604
0.848
0.683
MemCam [ 4 ]
12.35
0.592
0.782
0.614
11.08
0.657
0.732
0.472
PPMem (ours)
16.99
0.335
0.909
0.861
13.98
0.464
0.863
0.769
Appendix
Table S5 : Comparison to non-3D-persistent baselines on DL3DV. WorldPlay and MemCam condition on historical frames without an explicit persistent 3D representation. The evaluation set and frame budgets match Table 2 .
Figure S2 : Failure case: fine text rendering under zoom-in. Scene structure and prominently visible text are preserved across 288 frames, but text that is barely legible in the initial viewpoint becomes garbled as the camera approaches. Faithful text synthesis from low-resolution observations remains an open challenge for video diffusion models.
Figure S3 : Additional qualitative comparison: outdoor rooftop scene. PPMem maintains structural consistency across 288 frames under large viewpoint change.
Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to 10.57× faster end-to-end video generation and 55× reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.
Weijie Wang, Haoyu Zhao, Yifan Yang +7
Zhejiang University · Microsoft Research · Adelaide University +1
Video world models require persistent scene memory to maintain consistency during long-horizon video generation. Existing spatial memories accumulate RGB observations or latent features, increasing storage requirements as generation proceeds. We introduce Honeycomb, a video world model built on HexMemory, our proposed low-rank representation for storing scene features in a fixed-size memory with a total of six spatial and spatiotemporal planes. A feed-forward writer maps each generated chunk into new plane features. As the spatial coverage or temporal range expands, we warp the previous planes while preserving their dimensions, then fuse them with the new features through confidence-weighted pooling and a learned residual correction. A reader retrieves latents from HexMemory to condition subsequent video generation. The writer processes only observations from the new chunk, avoiding per-scene optimization and repeated processing of the full history. Experiments on WorldScore and RealEstate10K demonstrate strong video generation quality and robust revisit consistency while keeping HexMemory feature storage constant throughout generation. Code and additional visualizations are available on our project page at https://jackswl.github.io/honeycomb/.
Jack Wei Lun Shi, Kaichen Zhou, Haoyu Chen +6
National University of Singapore · Harvard University · MIT +1
Video world models aim to simulate controllable visual environments, but long-horizon rollouts depend on what the model remembers after observations leave its native context window. Explicit memories retain frames or online 3D reconstructions, which can suffer from heuristic retrieval errors, redundant appearance storage, or reconstruction artifacts. Implicit memories compress history into a compact state, but existing designs are not explicitly constrained to encode cross-view scene geometry. We propose GIM-World, a geometry-aware implicit memory framework for video world models. A lightweight transformer encoder compresses variable-length history into fixed-size memory tokens, a camera-queryable geometry head distills 3D scene structure from a frozen foundation model into the memory during training, and an information-guided pruning rule keeps encoding cost bounded as history grows. The geometry teacher is discarded at inference, leaving a lightweight memory module. Experiments on MIND show that GIM-World better preserves long-horizon geometric and visual consistency than both explicit- and implicit-memory baselines.
Zhengxuan Wei, Xu Guo, Xinghui Li +8
School of Intelligence Science and Technology, Nanjing University · 2Kling Team, Kuaishou Technology · 3Tsinghua University