Beyond Pixel Histories: World Models with Persistent 3D State

Samuel Garcin; Thomas Walker; Steven McDonagh; Tim Pearce; Hakan Bilen; Tianyu He; Kaixin Wang; Jiang Bian

cs.CVMar 3, 2026

Beyond Pixel Histories: World Models with Persistent 3D State

Authors: Samuel Garcin, Thomas Walker, Steven McDonagh, Tim Pearce, Hakan Bilen, Tianyu He, Kaixin Wang, Jiang Bian

Organizations: University of Edinburgh · Microsoft Research

Abstract

Interactive world models continually generate video by responding to a user's actions, enabling open-ended generation capabilities. However, existing models typically lack a 3D representation of the environment, meaning 3D consistency must be implicitly learned from data, and spatial memory is restricted to limited temporal context windows. This results in an unrealistic user experience and presents significant obstacles to downstream tasks such as training agents. To address this, we present PERSIST, a new paradigm of world model which simulates the evolution of a latent 3D scene: environment, camera, and renderer. This allows us to synthesise new frames with persistent spatial memory and consistent geometry. Both quantitative metrics and a qualitative user study show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods, enabling coherent, evolving 3D worlds. We further demonstrate novel capabilities, including synthesising diverse 3D environments from a single image, as well as enabling fine-grained, geometry-aware control over generated experiences by supporting environment editing and specification directly in 3D space. Project page: https://francelico.github.io/persist.github.io

Figures & tables

Figure 1: Initialized with a single pixel frame, PERSIST evolves in an autoregressive loop in response to user actions . We first denoise the 3D environment centred on the agent in the form of a latent world-frame . Next, camera parameters are predicted with a feed-forward transformer. We then project the world to the camera plane to form a depth-ordered stack of world latents . Finally, pixel latents are denoised, using pixel-aligned 3D information from the world latents stack as guidance.

Figure 2: PERSIST enables long-horizon spatial memory by modelling the dynamics of a 3D world-frame around the agent. Camera parameters then act as memory look-up key, fetching relevant features from the world frame via a geometric projection (here visualized as the coloured voxels).

Figure 3: PERSIST can be initialized with a single RGB frame ( , row 1), or with a single RGB and world frame ( + , row 2). We visualize the world-frames and videos produced by an autoregressive rollout of 600 timesteps. Even with a single RGB frame for initialization, PERSIST can generate cohesive and evolving worlds.

Figure 4: World frame ${\bm{w}}$ features are projected to screen-space to obtain the depth-ordered stack of features $\tilde{{\bm{w}}}_{2D}$ and linear depth information ${\bm{d}}$ .

Figure 5: Video frames generated over 600 timestep episodes by PERSIST (Ours), Oasis ( Decart et al., 2024 ) and WorldMem ( Xiao et al., 2025a ) .

Figure 6: FID scores compared to ground truth over 600 frame episodes. PERSIST configurations remains stable, while baselines relying on pixel-histories degrade rapidly.

Method	200 Frames	400 Frames	600 Frames
Oasis	409	687	875
WorldMem	358	–	–
No-3D-Upscale	216	231	247
Camera-GT	161	152	152
PERSIST-S	159	170	179
PERSIST	129	141	148

Table 1: FVD (

\downarrow

) evaluated across different episode lengths (200, 400, and 600 frames). We compare our base configuration (PERSIST) against pixel-history baselines and various model ablations. PERSIST

+{\bm{w}}_{0}

is discussed in Section 6.2 .

Method	VF $\uparrow$	3D $\uparrow$	Temp $\uparrow$	Overall $\uparrow$
Oasis	2.1 $\pm$ 0.1	1.9 $\pm$ 0.1	1.8 $\pm$ 0.1	1.9 $\pm$ 0.1
WorldMem	1.7 $\pm$ 0.09	1.7 $\pm$ 0.09	1.5 $\pm$ 0.08	1.5 $\pm$ 0.07
PERSIST-S	2.8 $\pm$ 0.1	2.7 $\pm$ 0.1	2.5 $\pm$ 0.1	2.6 $\pm$ 0.09
PERSIST	2.8 $\pm$ 0.09	2.5 $\pm$ 0.09	2.5 $\pm$ 0.09	2.6 $\pm$ 0.08
PERSIST $+{\bm{w}}_{0}$	3.2 $\pm$ 0.1	2.8 $\pm$ 0.1	2.8 $\pm$ 0.1	3.0 $\pm$ 0.1

Table 2: Mean user ratings (score range: 1-5) for videos generated from different PERSIST configurations and baselines. Bold numbers indicate the best performing methods. VF = Visual Fidelity, 3D = 3D Consistency, Temp = Temporal Consistency, Overall = Overall Score.

Figure 7: PERSIST’s explicit 3D representation allows us to edit the world-state during an episode, enabling intuitive fine grained control over generations. We demonstrate this above with both global edits to terrain and biome, as well as the placement of smaller assets such as trees.

Figure 8: PERSIST’s 3D state enables collision modelling with out-of-view elements (top: player moves backwards into a tree at $t=50$ ). Since the 3D state is dynamic, it evolves even when unobserved (middle: a cave filling with water). This allows off-screen events to produce emergent on-screen effects (bottom: water flowing onto the player at $t=150$ ).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Table 11

Figure 9: User Study Interface. A screenshot of our web-based platform where participants evaluate video pairs based on temporal and spatial consistency.

Panel of Figure 10 — Figure 10: Score differences during head-to-head comparisons across evaluation metrics.

Method	Per Frame Visual Fidelity $\uparrow$	3D Consistency $\uparrow$	Temporal Consistency $\uparrow$	Overall Score $\uparrow$
Oasis	2 $\pm$ 0.2	1.6 $\pm$ 0.2	1.5 $\pm$ 0.2	1.6 $\pm$ 0.2
WorldMem	1.7 $\pm$ 0.2	1.5 $\pm$ 0.1	1.3 $\pm$ 0.1	1.4 $\pm$ 0.1
PERSIST-S	2.8 $\pm$ 0.2	2.6 $\pm$ 0.2	2.3 $\pm$ 0.2	2.5 $\pm$ 0.2
PERSIST	3.0 $\pm$ 0.2	2.5 $\pm$ 0.1	2.2 $\pm$ 0.1	2.5 $\pm$ 0.1
PERSIST $+w_{0}$	3.4 $\pm$ 0.2	3.0 $\pm$ 0.2	2.7 $\pm$ 0.3	3.1 $\pm$ 0.2

Appendix

Table 10: Per set user study scores: circle around .

Method	Per Frame Visual Fidelity $\uparrow$	3D Consistency $\uparrow$	Temporal Consistency $\uparrow$	Overall Score $\uparrow$
Oasis	2.3 $\pm$ 0.2	2.2 $\pm$ 0.2	2.0 $\pm$ 0.2	2.0 $\pm$ 0.2
WorldMem	1.5 $\pm$ 0.2	1.5 $\pm$ 0.2	1.2 $\pm$ 0.1	1.3 $\pm$ 0.1
PERSIST-S	3.0 $\pm$ 0.2	2.9 $\pm$ 0.2	2.8 $\pm$ 0.2	3.0 $\pm$ 0.2
PERSIST	2.9 $\pm$ 0.2	2.8 $\pm$ 0.2	2.8 $\pm$ 0.2	2.8 $\pm$ 0.2
PERSIST $+w_{0}$	3.6 $\pm$ 0.2	3.1 $\pm$ 0.2	3.3 $\pm$ 0.2	3.4 $\pm$ 0.2

Appendix

Table 11: Per set user study scores: free play .

Method	Per Frame Visual Fidelity $\uparrow$	3D Consistency $\uparrow$	Temporal Consistency $\uparrow$	Overall Score $\uparrow$
Oasis	1.9 $\pm$ 0.2	2.0 $\pm$ 0.2	2.0 $\pm$ 0.2	1.9 $\pm$ 0.2
WorldMem	1.7 $\pm$ 0.2	1.8 $\pm$ 0.2	1.6 $\pm$ 0.2	1.6 $\pm$ 0.1
PERSIST-S	3.0 $\pm$ 0.2	2.7 $\pm$ 0.2	2.7 $\pm$ 0.2	2.6 $\pm$ 0.2
PERSIST	2.7 $\pm$ 0.2	2.4 $\pm$ 0.2	2.4 $\pm$ 0.2	2.5 $\pm$ 0.1
PERSIST $+w_{0}$	3.0 $\pm$ 0.2	2.6 $\pm$ 0.2	2.6 $\pm$ 0.2	2.8 $\pm$ 0.2

Appendix

Table 12: Per set user study scores: backwards and look around .

Method	Per Frame Visual Fidelity $\uparrow$	3D Consistency $\uparrow$	Temporal Consistency $\uparrow$	Overall Score $\uparrow$
Oasis	2.4 $\pm$ 0.2	2.0 $\pm$ 0.2	1.8 $\pm$ 0.2	1.9 $\pm$ 0.2
WorldMem	2.1 $\pm$ 0.2	2.2 $\pm$ 0.2	1.9 $\pm$ 0.2	1.9 $\pm$ 0.2
PERSIST-S	2.2 $\pm$ 0.2	2.3 $\pm$ 0.2	2.1 $\pm$ 0.2	2.3 $\pm$ 0.2
PERSIST	2.6 $\pm$ 0.2	2.3 $\pm$ 0.2	2.3 $\pm$ 0.2	2.4 $\pm$ 0.1
PERSIST $+w_{0}$	2.7 $\pm$ 0.2	2.4 $\pm$ 0.2	2.3 $\pm$ 0.2	2.4 $\pm$ 0.2

Appendix

Table 13: Per set user study scores: forward and look around .

Method	Time per frame (ms) $\downarrow$
PERSIST-S	2672.4
Oasis	3007.4
WorldMem	6387.1

Appendix

Table 14: Inference speed comparison against baselines. All tests were conducted on a single A100 GPU with a batch size of 1 and 20 denoising steps.

Component	Time per frame (ms)
World frame	1538.4
Pixel frame	1111.0
Camera state	21.9
Projection	0.63

Appendix

Table 15: Component-wise breakdown of generation time per frame for PERSIST-S (20 denoising steps).

Figure 11: Conditioning on the ground truth camera trajectory results in physical inconsistency with the world-frame, causing the agent to phase through a wall.

Figure 12: PERSIST generates diverse but coherent initial world frames from a single RGB conditioning observation. Each row corresponds to a specific input RGB frame, while each column depicts the initial world frame sampled from ${\mathcal{W}}_{\theta}$ during a specific generation episode.

Panel of Figure 13 — Figure 13: Generation artifacts and subsequent recoveries over a 2000-step (83 seconds) episode generated with PERSIST. The 3D representation drifts locally, causing individual blocks to appear and disappear from the agent’s view. However we find that the 3D representation remains globally coherent and has a net stabilising effect on generation, allowing ${\mathcal{P}}_{\theta}$ to recover from visual artifacts. This lets PERSIST generate rollouts lasting thousands of steps, even if glitches become more frequent over time due to autoregressive drift.

Explore similar work

World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
Oct 12, 2026Junyi Wu, Zekun Li, Chenhongyi Yang +9Video World Models Memory-Augmented Video Generation
Meta Reality Labs · University of Illinois Chicago · Brown University
WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory
Jul 2, 2026Hanlin Wang, Hao Ouyang, Qiuyu Wang +10Physical Consistency in Video Generation Camera-Controlled Video Generation
1HKUST · 2Ant Group · 3ZJU +1
Lyra 2.0: Explorable Generative 3D Worlds
Apr 14, 2026Tianchang Shen, Sherwin Bahmani, Kai He +123D Scene Generation Long-Horizon Video Generation
NVIDIA

Oct 12, 2026J/K move · Enter open · S save

World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

Junyi Wu, Zekun Li, Chenhongyi Yang, Chen Guo+8

Meta Reality Labs · University of Illinois Chicago · Brown University

Video World Models Memory-Augmented Video Generation Long-Horizon Video Generation Camera-Controlled Video Generation Real-Time Interactive Video Generation Test-Time Training

Open paper AskPDF arXiv

Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context. We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. On a long-horizon revisit benchmark, World-T3 raises revisit PSNR from 13.52 to 18.31 dB and reduces absolute rotation error from 3.344 to 1.075 relative to the strongest baseline on both metrics, while sustaining 20.6 FPS at 1280 x 704 on a single NVIDIA B200 GPU.

Abstract

Explore similar work

World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

Lyra 2.0: Explorable Generative 3D Worlds