Persistent Planar Memory for Video World Models

Yuze He; Bin Tan; Zelin Gao; Yujun Shen; Yong-Jin Liu; Nan Xue; Yinghao Xu

cs.CVOct 16, 2026

Persistent Planar Memory for Video World Models

Authors: Yuze He, Bin Tan, Zelin Gao, Yujun Shen, Yong-Jin Liu, Nan Xue, Yinghao Xu

Organizations: Tsinghua University · Ant Group · HKUST

Abstract

Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced. Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive. We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory. PPMem augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion (DTF) module: each autoregressive chunk jointly produces RGB video and planar geometry, which merges into a persistent plane memory that conditions subsequent chunks via rendered depth. The resulting memory is three orders of magnitude more compact than per-pixel point clouds (~100K primitives, 5MB for a full scene), geometrically explicit (metric depth, normals, exportable surfaces), cross-view consistent (jointly predicted by the shared backbone), and adaptive (refined by the merging procedure as generation proceeds). On DL3DV and RealEstate10K, PPMem outperforms prior 3D-grounded world models in video quality while jointly producing metric-scale depth, surface normals, and exportable planar geometry, sustaining coherence across hundreds of frames under large viewpoint changes. Project page: https://hyzcluster.github.io/ppmem.

Figures & tables

Figure 1: Given a single input image and a camera trajectory, PPMem autoregressively generates video while accumulating a persistent plane-based 3D memory. We show two DL3DV scenes (indoor, outdoor) with generated frames at selected indices. Insets visualize the plane pool at each frame, colored by the current view frustum. The plane geometry grows to cover the full explored scene, serving as persistent spatial memory for subsequent generation.

	Dense PC [ 24 ]	Surfels [ 13 ]	TSDF $\to$ PC [ 36 ]	Planes (PPMem)
3D source	External (MoGe [ 32 ] )	External (CUT3R [ 31 ] )	External (DA3 [ 15 ] )	Internal (joint head)
Cross-view consistent	✗	✗	partial	✓
Primitives (accumulated)	${\sim}$ 210M †	${\sim}$ 250K	${\sim}$ 1–2M	${\sim}$ 100K
Structured surfaces	✗	✗	✗	✓

Table 1: Properties of 3D representations for cross-chunk video memory. We compare along four aspects: whether geometry is predicted internally by the generation model or externally, whether predictions are cross-view consistent, the number of persistent 3D primitives, and whether the representation captures structured surfaces. Primitive counts are estimates at

{\sim}

400 frames,

576{\times}1024

Figure 2: PPMem architecture overview. The Wan DiT receives noisy video latents channel-concatenated with projected source-view RGB/masks and partial depth rendered from the plane pool; context frames are prepended temporally. The DiT produces two token streams: video tokens decoded by the Wan VAE into RGB outputs, and geometry tokens adapted by geometry LoRA and fused with partial depth tokens via a ViT-based Depth Token Fusion module. The fused tokens are decoded into per-frame plane primitives, which merge into the persistent plane pool. The pool is rendered at the next chunk’s target cameras and fed back as conditioning.

		$N{=}49$				$N{=}121$
Dataset	Method	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$
DL3DV	Gen3C [ 24 ]	15.74	0.474	0.847	0.710	13.59	0.565	0.798	0.585
	Voyager [ 11 ]	12.84	0.568	0.796	0.615	11.35	0.649	0.760	0.516
	VMem [ 13 ]	11.88	0.636	0.781	0.477	10.87	0.722	0.696	0.274
	SPMem [ 36 ]	14.70	0.492	0.853	0.703	13.03	0.607	0.788	0.536
	PPMem (ours)	16.99	0.335	0.909	0.861	13.98	0.464	0.863	0.769

Table 2: Short-context comparison on DL3DV and RealEstate10K. We report two frame budgets:

N{=}49

(native chunk size for Voyager and SPMem; VMem uses 48) and

N{=}121

(Gen3C native chunk). At

N{=}49

most baselines are single-chunk; at

N{=}121

Voyager, VMem, and SPMem all require autoregressive looping.

Figure 3: Perceptual similarity vs. frame index on DL3DV long-context generation (

{\sim}

350 frames). Left: CLIP similarity. Right: DINOv2 similarity. PPMem (blue) maintains the highest perceptual fidelity across the full generation. VMem collapses early; Gen3C and SPMem degrade steadily; the gap widens with frame count, especially on DINOv2.

Figure 4: Qualitative long-context comparison. PPMem preserves scene structure and color consistency across the full sequence. VMem collapses early under large viewpoint change; SPMem grows vague and drifts; Gen3C and Voyager lose structural alignment as the viewpoint diverges.

Variant	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$	Normal ${}^{\circ}\downarrow$
Without persistent memory	14.58	0.407	0.835	0.733	0.272	64.0	25.2
MoGe-2 point cloud	14.37	0.423	0.832	0.716	0.339	55.0	25.8
Post-hoc Plana3R	14.22	0.432	0.829	0.713	0.311	57.0	26.3
Joint plane pool (ours)	14.91	0.387	0.835	0.741	0.210	70.3	24.1

Table 3: 3D representation and prediction ablation. We compare our joint plane pool against three inference-only alternatives (same checkpoint, no retraining): a same-model rollout without persistent memory, Gen3C-style per-frame MoGe-2 point cloud memory, and post-hoc standalone Plana3R reconstruction.

Panel of Figure 5 — Figure 5: Depth quality analysis. (a) Per-frame depth metrics (AbsRel $\downarrow$ , $\delta{<}1.25$ $\uparrow$ ) on DL3DV over 121 frames. PPMem maintains consistently lower depth error than Voyager, the only other baseline with joint depth output. (b) We scale the input depth conditioning by a known factor and measure the predicted/GT depth ratio. Concat + DTF (ours) tracks input scale faithfully ( $1.00\pm 0.01$ at scale 1.0, mean $\pm$ s.e. over 56 scenes), while channel concatenation alone exhibits compressed response at extreme scales. Measured as deviation from the identity mapping over the full scale range, DTF also stays closer to it (natural-log RMSE $0.273$ vs. $0.320$ for concatenation alone; $0.131$ vs. $0.293$ with scale augmentation).

Mechanism	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$	Normal ${}^{\circ}\downarrow$
No depth conditioning	14.19	0.430	0.826	0.709	0.436	30.7	25.8
Channel concat only	14.88	0.388	0.838	0.743	0.212	72.3	23.8
Concat + DTF (ours)	14.91	0.387	0.835	0.741	0.210	70.3	24.1

Table 4: Depth conditioning mechanism ablation. All variants share the same backbone and training data, varying only the depth conditioning mechanism.

Variant	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$	Normal ${}^{\circ}\downarrow$
Plana3R upsampling head	14.68	0.395	0.832	0.737	0.256	59.2	26.2
VAE-init decoder (ours)	14.91	0.387	0.835	0.741	0.210	70.3	24.1
$-$ rasterization loss	14.82	0.391	0.834	0.736	0.217	66.4	25.4
$-$ normal loss	13.27	0.502	0.807	0.675	—	—	—

Table 5: Feature decoder and loss component ablations. Top: decoder architecture (all else fixed). Bottom: loss removal (full model as baseline).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Value
Model	Base video DiT	Wan 2.2 TI2V 5B
	VAE	Per-frame (1:1 frame mapping, $z_{\text{dim}}{=}48$ )
	Depth token fusion encoder	ViT-Base ( ${\sim}$ 87M params), DINOv2-Base init
	Feature decoder	Initialized from Wan VAE upsample block
	LoRA (Stage 2)	Rank 256, $\alpha{=}256$ , self-attention, all 30 blocks
	Trainable params (Stage 2)	${\sim}$ 377M / 5.4B (7.0%)

Appendix

Table S1 : Full training hyperparameters for Stage 2. Stage 2 freezes the DiT backbone and injects LoRA adapters on self-attention; only geometry-specific modules and LoRA are trained. Stage 1 hyperparameters are similar except no plane loss, no DTF, no freeze/LoRA.

	Anchor	History	Overlap	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$	Normal ${}^{\circ}\downarrow$
Component ablation
No context	✗	✗	✗	14.60	0.403	0.835	0.730	0.219	64.2	24.5
Overlap only	✗	✗	✓	14.73	0.394	0.834	0.738	0.208	69.2	24.3
History + Overlap	✗	✓	✓	14.77	0.393	0.832	0.739	0.218	70.2	24.3
All (default)	✓	✓	✓	14.91	0.387	0.835	0.741	0.210	70.3	24.1
History scaling (Anchor + Overlap + varying history count)

Appendix

Table S2 : Context frame configuration ablation, inference-only. We vary the context frame composition at inference time without retraining (the model was trained with context dropout 0.1). Anchor = frame 0 of the video; History = frames from preceding chunks; Overlap = past RGB frames selected by coverage of the target viewpoints. Top: component ablation. Bottom: history count scaling. Evaluated on 56 DL3DV indoor scenes at

288{\times}512

	Outdoor (8 scenes)			Indoor (8 scenes)
Mechanism (cumulative)	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	Coverage	Abs Rel $\downarrow$	$\delta{<}1.25\uparrow$	Coverage	Planes
Baseline (no merging improvements)	0.479	41.1	0.865	0.311	46.6	0.948	518K
+ Within-chunk dedup (stage 3)	0.486	44.7	0.833	0.297	51.8	0.920	116K
+ Cross-chunk replacement (stage 5)	0.477	47.0	0.803	0.296	52.7	0.916	100K
+ Forward-looking prune (stage 6)	0.469	48.3	0.798	0.285	53.9	0.908	100K
+ Boundary rejection (stage 2)	0.472	49.6	0.786	0.284	56.5	0.900	97K

Appendix

Table S3 : Progressive build-up of the plane merging pipeline. Depth accuracy is measured by absolute relative error (Abs Rel) and the percentage of pixels with predicted/GT depth ratio within 1.25 (

\delta{<}1.25

). Coverage is the fraction of pixels receiving a valid rendered depth. Evaluated on 16 DL3DV scenes (8 outdoor + 8 indoor).

Figure S1 : Accumulated plane pool vs. Plana3R reconstruction. We render depth from our plane pool and from planes reconstructed by Plana3R [ 18 ] on the same generated video. The pool built online during generation stays consistent, whereas pairwise reconstructions conflict across views and fragment.

Planar coverage	Low	Middle	High
# Scenes	34	32	34
PSNR $\uparrow$	12.98	13.67	15.27
LPIPS $\downarrow$	0.482	0.466	0.443
AbsRel $\downarrow$	0.289	0.250	0.195
$\delta{<}1.25\uparrow$	64.2	72.5	79.0

Appendix

Table S4 : Performance by scene planarity. The 100 DL3DV scenes are grouped into within-domain tertiles of planar coverage; all metrics are computed at

N{=}121

	$N{=}49$				$N{=}121$
Method	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	CLIP ${}_{\text{sim}}\uparrow$	DINOv2 ${}_{\text{sim}}\uparrow$
WorldPlay [ 25 ]	13.22	0.532	0.883	0.784	11.98	0.604	0.848	0.683
MemCam [ 4 ]	12.35	0.592	0.782	0.614	11.08	0.657	0.732	0.472
PPMem (ours)	16.99	0.335	0.909	0.861	13.98	0.464	0.863	0.769

Appendix

Table S5 : Comparison to non-3D-persistent baselines on DL3DV. WorldPlay and MemCam condition on historical frames without an explicit persistent 3D representation. The evaluation set and frame budgets match Table 2 .

Figure S2 : Failure case: fine text rendering under zoom-in. Scene structure and prominently visible text are preserved across 288 frames, but text that is barely legible in the initial viewpoint becomes garbled as the camera approaches. Faithful text synthesis from low-resolution observations remains an open challenge for video diffusion models.

Panel of Figure 18 — Figure S3 : Additional qualitative comparison: outdoor rooftop scene. PPMem maintains structural consistency across 288 frames under large viewpoint change.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 245 papers

Latent Spatial Memory for Video World Models
Jun 8, 2026Weijie Wang, Haoyu Zhao, Yifan Yang +7Video Diffusion Models Memory-Augmented Video Generation
Zhejiang University · Microsoft Research · Adelaide University +1
Honeycomb: Constant-Size Scene Memory Representation for Video World Models
Sep 29, 2026Jack Wei Lun Shi, Kaichen Zhou, Haoyu Chen +6Long-Horizon Video Generation Memory-Augmented Video Generation
National University of Singapore · Harvard University · MIT +1
Geometry-Aware Implicit Memory for Video World Models
Jun 1, 2026Zhengxuan Wei, Xu Guo, Xinghui Li +8World Model Learning Memory-Augmented Video Generation
School of Intelligence Science and Technology, Nanjing University · 2Kling Team, Kuaishou Technology · 3Tsinghua University

Jun 8, 2026J/K move · Enter open · S save

Latent Spatial Memory for Video World Models

Weijie Wang, Haoyu Zhao, Yifan Yang, Feng Chen+6

Zhejiang University · Microsoft Research · Adelaide University+1

Video Diffusion Models Memory-Augmented Video Generation Video World Models

Open paper AskPDF arXiv

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to 10.57

\times

faster end-to-end video generation and 55

\times

reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.

Abstract

Video World Models

Explore similar work

Latent Spatial Memory for Video World Models

Honeycomb: Constant-Size Scene Memory Representation for Video World Models

Geometry-Aware Implicit Memory for Video World Models