Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to 38.6× fewer 3D primitives, 3.2× faster memory updates, and 359× faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.
Figures & tables
Figure 1: Teaser. L3G-World generates an explorable, spatially consistent world in real time from a single image and a camera trajectory. To this end, L3G-Mem maintains a latent 3D Gaussian memory, and L3G-Gen generates video conditioned on VAE latents rendered from the memory.
Method
Resolution
Memory Type
Memory Build
Refinement
Memory Scaling ↓
Peak GPU Memory ↓
Total FPS ↑
Real-time
WorldKV ( Yi et al., 2026 )
464×832
Temporal context
KV cache
–
O(TR)
OOM
OOM
✗
Yume 1.5 ( Mao et al., 2026 )
704×1,280
Temporal context
Frame compression
–
O(TR)
66.7 GiB
1.42
✗
Spatia ( Zhao et al., 2026 )
704×1,248
Point clouds
Feed-forward
✗
O(TR)
32.6 GiB
0.24
✗
NeoVerse ( Yang et al., 2026 )
336×560
3D Gaussians
Feed-forward
✗
O(TR)
36.8 GiB
1.99
✗
ArtiFixer ( De Lutio et al., 2026 )
540×960
3D Gaussians
Optimization
✓
O(S)
11.7 GiB
0.10
✗
L3G-World (Ours)
480×832
Latent 3D Gaussians
Feed-forward
✓
O(S)
7.4 GiB
9.02
✓
Table 1: Computational efficiency of memory analysis. We compare memory scaling across methods and report memory and throughput for constructing and rendering explicit scene representations. For memory scaling, T denotes the number of generated frames, R denotes the resolution size, and S denotes the spatial coverage of the scene. ‘OOM’ denotes out-of-memory error. All cost analyses are measured by 400-frame video generation on one H100 GPU.
Figure 2: L3G-Mem. (a) L3G-Mem takes the current window Wk and the memory Mk−1 . Rendering Mk−1 into the current cameras determines which geometry features are clustered by K-means into spawning queries, and identifies the visible Gaussians whose memory tokens are retrieved as refinement tokens. Both token sets are jointly processed by a transformer and decoded into new Gaussians and residual updates, respectively. (b) Given the input video chunk, RGB and opacity rendered from the previous memory identify regions requiring new Gaussians through a spawning mask. As a result, the updated memory can fully reconstruct the observed scene.
Figure 3: L3G-Gen. (a) L3G-Gen renders the VAE latents and opacity of the 3D memory to the target cameras, concatenates them with the noisy latents along the channel dimension, and generates the next chunk conditioned on the history latents. (b) We visualize the ground-truth and generated video, with target-view rendered VAE condition from the latent 3D Gaussian memory, illustrating that spatially aligned latent memory provides consistent conditioning for video generation.
Figure 4: L3G-World Pipeline. L3G-World couples video generation and memory updating in a chunk-wise loop, where each generated chunk updates the persistent 3D memory, which in turn conditions subsequent chunk generation.
3D Memory Efficiency
Self-Consistency
Generation Quality
Pose Error
3D Consistency
Method
# Primitives (K) ↓
Memory (MiB) ↓
Recon. FPS ↑
Render FPS (RGB) ↑
Render FPS (VAE) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
LPIPS ↓
FID ↓
IQ ↑
AQ ↑
SC ↑
BC ↑
TransErr ↓
RotErr ( ∘ ) ↓
MEt3R ↓
DL3DV
Temporal Context
Yume 1.5
–
–
–
–
–
13.71
0.487
0.655
0.699
50.88
0.601
0.514
0.805
0.888
0.904
11.09
0.050
Matrix Game 3.0
–
–
–
–
–
12.26
0.401
0.540
0.681
67.78
0.609
0.431
0.774
0.865
0.376
3.11
0.070
Explicit 3D Memory
Table 2: Video generation evaluation. We evaluate generation quality, self-consistency, and efficiency on DL3DV and RealEstate10K. The best and second-best results for each metric within each dataset are shown in bold and underline , respectively.
Figure 5: Qualitative results of L3G-World. (a) L3G-Mem corrects geometric errors in the 3D memory through residual refinement using video chunks generated by L3G-Gen. (b) L3G-World maintains scene consistency over long-horizon generation through its persistent 3D memory. (c) Rendering the memory’s VAE features from target camera views provides spatially aligned conditioning, enabling consistent video generation along diverse camera trajectories.
Figure 6: Qualitative results of video generation on DL3DV and RealEstate10K. For each scene, all methods start from the same input view (top), explore unseen regions (middle), and return to the initial camera location (bottom). L3G-World preserves scene structure and appearance upon loop closure, whereas the baselines exhibit drift, blur, or rendering artifacts.
Variant
VAE MSE ↓
VAE Cosine ↑
# Gaussians (K) ↓
Recon. FPS ↑
Render FPS (VAE) ↑
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Adaptive Spawning
0.182
0.776
75
39.08
742.4
16.26
0.409
0.535
w/o Refinement
0.213
0.743
45
43.82
937.5
15.52
0.382
0.571
w/o Global Self-Attn.
0.184
0.775
41
40.99
938.5
16.36
0.397
0.535
w/o RGB Loss
0.251
0.696
74
39.55
1199.0
10.26
0.272
0.751
w/o VAE Lifting
0.570
0.467
–
–
2.97
–
–
–
L3G-Mem (Full)
0.177
0.782
41
40.03
943.6
16.52
0.406
0.534
Table 3: Ablation study of L3G-Mem on DL3DV. We report RGB and VAE latent NVS quality and efficiency. Reconstruction FPS is computed from nine-frame update chunks. The best and second-best results are shown in bold and underline , respectively.
Figure 7: Qualitative results of ablation study of L3G-World on DL3DV. Memory refinement can not fix geometric error in 3D memory, leading blurry video due to memory rendering artifacts.
Self-Consistency
Generation Quality
Pose Error
3D Consistency
Method
PSNR ↑
SSIM ↑
LPIPS ↓
LPIPS ↓
FID ↓
IQ ↑
AQ ↑
SC ↑
BC ↑
TransErr ↓
RotErr ( ∘ ) ↓
MEt3R ↓
w/o Memory Refinement
14.82
0.427
0.464
0.592
50.63
0.605
0.475
0.800
0.880
0.314
7.85
0.091
w/o VAE Lifting
15.41
0.480
0.461
0.611
51.84
0.570
0.478
0.808
0.885
0.407
5.21
0.075
L3G-World (Full)
15.11
0.431
0.435
0.583
48.09
0.619
0.486
0.814
0.888
0.278
5.22
0.081
Table 4: Ablation study of L3G-World on DL3DV . The best and second-best results are shown in bold and underline , respectively.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Component
L3G-Mem
L3G-Gen
L3G-Gen Fast
Optimizer
AdamW
AdamW
AdamW
Learning rate
1×10−4
1×10−4
5×10−5 / 2×10−6
Optimizer betas
(0.9,0.95)
–
(0.9,0.95) / (0,0.999)
Weight decay
0.05
0.01
-
Batch size
8
16
16 / 8
EMA decay
–
–
0.999 / 0.99
Appendix
Table 5: Training configuration of L3G-Mem, L3G-Gen, and L3G-Gen Distilled. For L3G-Gen Distilled, values separated by “/” denote the flow-map and on-policy distillation stages, respectively.
Figure 8: Additional qualitative results of video generation on DL3DV and RealEstate10K. For each scene, all methods start from the same input view (top), explore unseen regions (middle), and return to the initial camera location (bottom). L3G-World preserves scene structure and appearance upon loop closure, whereas the baselines exhibit drift, blur, or rendering artifacts.
Method
Resolution (H×W)
VAE MSE ↓
VAE Cosine Sim. ↑
# Primitives (K) ↓
Memory (MiB) ↓
Recon. FPS ↑
VAE Render FPS ↑
PSNR ↑
SSIM ↑
LPIPS ↓
DL3DV
ReCoSplat
480×480
0.118
0.858
2,085
198.88
11.12
3.0
21.53
0.707
0.318
ReCoSplat
480×832
0.664
0.344
11,550
1101.48
2.15
2.7
12.41
0.438
0.711
OF 3 GS
405×832
0.452
0.522
8,987
479.98
4.01
3.0
14.03
0.418
0.606
L3G-Mem
480×832
0.142
0.823
39
14.11
28.66
735.3
17.52
0.526
0.537
RealEstate10K (Out-of-domain)
Appendix
Table 6: Online NVS evaluation. We evaluate online reconstruction on DL3DV and on RealEstate10K, reporting RGB and VAE latent novel view synthesis quality together with memory size, reconstruction FPS, and VAE rendering FPS. Reconstruction FPS is computed from nine-frame update chunks. The best and second-best results within each dataset are shown in bold and underline , respectively.
Figure 9: Visualization of latent 3D Gaussian memory conditioning for video generation. We visualize the ground-truth and generated video, with target-view rendered VAE condition from the latent 3D Gaussian memory, illustrating that spatially aligned latent memory provides consistent conditioning for video generation.
Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to 10.57× faster end-to-end video generation and 55× reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.
Weijie Wang, Haoyu Zhao, Yifan Yang +7
Zhejiang University · Microsoft Research · Adelaide University +1
Recent advances in video world models have enabled plausible interactive environments, yet long-horizon generation places a fundamental demand on visual memory: the previously observed world must remain consistent as the model continues to generate. Existing autoregressive approaches often implicitly rely on the generative model itself to retain such information, leading to spatial memory decay beyond the active context. We introduce World-T3, a camera-controlled video world model with a test-time training memory that consolidates past observations into dedicated model parameters. To robustly update this memory, we propose a recursive least-squares rule that incrementally absorbs novel visual features via online pose-keyed memory updates. Our TTT memory reads out previously observed scene features from the fast-weight state by camera pose, enabling consistent generation with constant storage and per-step computation regardless of rollout length. To facilitate real-time interactive generation with this robust memory, we further distill the autoregressive model into a few-step generator via dual-teacher distribution matching. On a long-horizon revisit benchmark, World-T3 raises revisit PSNR from 13.52 to 18.31 dB and reduces absolute rotation error from 3.344 to 1.075 relative to the strongest baseline on both metrics, while sustaining 20.6 FPS at 1280 x 704 on a single NVIDIA B200 GPU.
Junyi Wu, Zekun Li, Chenhongyi Yang +9
Meta Reality Labs · University of Illinois Chicago · Brown University
Interactive world models continually generate video by responding to a user's actions, enabling open-ended generation capabilities. However, existing models typically lack a 3D representation of the environment, meaning 3D consistency must be implicitly learned from data, and spatial memory is restricted to limited temporal context windows. This results in an unrealistic user experience and presents significant obstacles to downstream tasks such as training agents. To address this, we present PERSIST, a new paradigm of world model which simulates the evolution of a latent 3D scene: environment, camera, and renderer. This allows us to synthesise new frames with persistent spatial memory and consistent geometry. Both quantitative metrics and a qualitative user study show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods, enabling coherent, evolving 3D worlds. We further demonstrate novel capabilities, including synthesising diverse 3D environments from a single image, as well as enabling fine-grained, geometry-aware control over generated experiences by supporting environment editing and specification directly in 3D space. Project page: https://francelico.github.io/persist.github.io