L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Jinhyeok Choi; JoungBin Lee; Jongmin Lee; Minkyeong Jeon; Jiyoung Kim; Kaeun Han; Yoonjin Yoon; Seungryong Kim

cs.CVOct 29, 2026

L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

Authors: Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim

Organizations: KAIST AI · Ewha Womans University · KAIST Urban AI Institute

Abstract

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to $38.6\times$ fewer 3D primitives, $3.2\times$ faster memory updates, and $359\times$ faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.

Figures & tables

Figure 1: Teaser. L3G-World generates an explorable, spatially consistent world in real time from a single image and a camera trajectory. To this end, L3G-Mem maintains a latent 3D Gaussian memory, and L3G-Gen generates video conditioned on VAE latents rendered from the memory.

Method	Resolution	Memory Type	Memory Build	Refinement	Memory Scaling $\downarrow$	Peak GPU Memory $\downarrow$	Total FPS $\uparrow$	Real-time
WorldKV ( Yi et al., 2026 )	$464\times 832$	Temporal context	KV cache	–	$\mathcal{O}(TR)$	OOM	OOM	✗
Yume 1.5 ( Mao et al., 2026 )	$704\times 1,280$	Temporal context	Frame compression	–	$\mathcal{O}(TR)$	66.7 GiB	1.42	✗
Spatia ( Zhao et al., 2026 )	$704\times 1,248$	Point clouds	Feed-forward	✗	$\mathcal{O}(TR)$	32.6 GiB	0.24	✗
NeoVerse ( Yang et al., 2026 )	$336\times 560$	3D Gaussians	Feed-forward	✗	$\mathcal{O}(TR)$	36.8 GiB	1.99	✗
ArtiFixer ( De Lutio et al., 2026 )	$540\times 960$	3D Gaussians	Optimization	✓	$\mathcal{O}(S)$	11.7 GiB	0.10	✗
L3G-World (Ours)	$480\times 832$	Latent 3D Gaussians	Feed-forward	✓	$\mathcal{O}(S)$	7.4 GiB	9.02	✓

Table 1: Computational efficiency of memory analysis. We compare memory scaling across methods and report memory and throughput for constructing and rendering explicit scene representations. For memory scaling,

T

denotes the number of generated frames,

R

denotes the resolution size, and

S

denotes the spatial coverage of the scene. ‘OOM’ denotes out-of-memory error. All cost analyses are measured by 400-frame video generation on one H100 GPU.

Figure 2: L3G-Mem. (a) L3G-Mem takes the current window $\mathcal{W}_{k}$ and the memory $\mathcal{M}_{k-1}$ . Rendering $\mathcal{M}_{k-1}$ into the current cameras determines which geometry features are clustered by K-means into spawning queries, and identifies the visible Gaussians whose memory tokens are retrieved as refinement tokens. Both token sets are jointly processed by a transformer and decoded into new Gaussians and residual updates, respectively. (b) Given the input video chunk, RGB and opacity rendered from the previous memory identify regions requiring new Gaussians through a spawning mask. As a result, the updated memory can fully reconstruct the observed scene.

Figure 3: L3G-Gen. (a) L3G-Gen renders the VAE latents and opacity of the 3D memory to the target cameras, concatenates them with the noisy latents along the channel dimension, and generates the next chunk conditioned on the history latents. (b) We visualize the ground-truth and generated video, with target-view rendered VAE condition from the latent 3D Gaussian memory, illustrating that spatially aligned latent memory provides consistent conditioning for video generation.

Figure 4: L3G-World Pipeline. L3G-World couples video generation and memory updating in a chunk-wise loop, where each generated chunk updates the persistent 3D memory, which in turn conditions subsequent chunk generation.

3D Memory Efficiency

Self-Consistency

Generation Quality

Pose Error

3D Consistency

Method

# Primitives (K)

\downarrow

Memory (MiB)

\downarrow

Recon. FPS

\uparrow

Render FPS (RGB)

\uparrow

Render FPS (VAE)

\uparrow

PSNR

\uparrow

SSIM

\uparrow

LPIPS

\downarrow

LPIPS

\downarrow

FID

\downarrow

\uparrow

\uparrow

\uparrow

\uparrow

TransErr

\downarrow

RotErr ( ∘ )

\downarrow

MEt3R

\downarrow

DL3DV

Temporal Context

Yume 1.5

–

13.71

0.487

0.655

0.699

50.88

0.601

0.514

0.805

0.888

0.904

11.09

0.050

Matrix Game 3.0

–

12.26

0.401

0.540

0.681

67.78

0.609

0.431

0.774

0.865

0.376

3.11

0.070

Explicit 3D Memory

Table 2: Video generation evaluation. We evaluate generation quality, self-consistency, and efficiency on DL3DV and RealEstate10K. The best and second-best results for each metric within each dataset are shown in bold and underline , respectively.

Figure 5: Qualitative results of L3G-World. (a) L3G-Mem corrects geometric errors in the 3D memory through residual refinement using video chunks generated by L3G-Gen. (b) L3G-World maintains scene consistency over long-horizon generation through its persistent 3D memory. (c) Rendering the memory’s VAE features from target camera views provides spatially aligned conditioning, enabling consistent video generation along diverse camera trajectories.

Figure 6: Qualitative results of video generation on DL3DV and RealEstate10K. For each scene, all methods start from the same input view (top), explore unseen regions (middle), and return to the initial camera location (bottom). L3G-World preserves scene structure and appearance upon loop closure, whereas the baselines exhibit drift, blur, or rendering artifacts.

Variant	VAE MSE $\downarrow$	VAE Cosine $\uparrow$	# Gaussians (K) $\downarrow$	Recon. FPS $\uparrow$	Render FPS (VAE) $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
w/o Adaptive Spawning	0.182	0.776	75	39.08	742.4	16.26	0.409	0.535
w/o Refinement	0.213	0.743	45	43.82	937.5	15.52	0.382	0.571
w/o Global Self-Attn.	0.184	0.775	41	40.99	938.5	16.36	0.397	0.535
w/o RGB Loss	0.251	0.696	74	39.55	1199.0	10.26	0.272	0.751
w/o VAE Lifting	0.570	0.467	–	–	2.97	–	–	–
L3G-Mem (Full)	0.177	0.782	41	40.03	943.6	16.52	0.406	0.534

Table 3: Ablation study of L3G-Mem on DL3DV. We report RGB and VAE latent NVS quality and efficiency. Reconstruction FPS is computed from nine-frame update chunks. The best and second-best results are shown in bold and underline , respectively.

Figure 7: Qualitative results of ablation study of L3G-World on DL3DV. Memory refinement can not fix geometric error in 3D memory, leading blurry video due to memory rendering artifacts.

	Self-Consistency			Generation Quality						Pose Error		3D Consistency
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	LPIPS $\downarrow$	FID $\downarrow$	IQ $\uparrow$	AQ $\uparrow$	SC $\uparrow$	BC $\uparrow$	TransErr $\downarrow$	RotErr ( ∘ ) $\downarrow$	MEt3R $\downarrow$
w/o Memory Refinement	14.82	0.427	0.464	0.592	50.63	0.605	0.475	0.800	0.880	0.314	7.85	0.091
w/o VAE Lifting	15.41	0.480	0.461	0.611	51.84	0.570	0.478	0.808	0.885	0.407	5.21	0.075
L3G-World (Full)	15.11	0.431	0.435	0.583	48.09	0.619	0.486	0.814	0.888	0.278	5.22	0.081

Table 4: Ablation study of L3G-World on DL3DV . The best and second-best results are shown in bold and underline , respectively.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Component	L3G-Mem	L3G-Gen	L3G-Gen Fast
Optimizer	AdamW	AdamW	AdamW
Learning rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$5\times 10^{-5}$ / $2\times 10^{-6}$
Optimizer betas	$(0.9,0.95)$	–	$(0.9,0.95)$ / $(0,0.999)$
Weight decay	0.05	0.01	-
Batch size	8	16	16 / 8
EMA decay	–	–	0.999 / 0.99

Appendix

Table 5: Training configuration of L3G-Mem, L3G-Gen, and L3G-Gen Distilled. For L3G-Gen Distilled, values separated by “/” denote the flow-map and on-policy distillation stages, respectively.

Figure 8: Additional qualitative results of video generation on DL3DV and RealEstate10K. For each scene, all methods start from the same input view (top), explore unseen regions (middle), and return to the initial camera location (bottom). L3G-World preserves scene structure and appearance upon loop closure, whereas the baselines exhibit drift, blur, or rendering artifacts.

Method	Resolution $(H\times W)$	VAE MSE $\downarrow$	VAE Cosine Sim. $\uparrow$	# Primitives (K) $\downarrow$	Memory (MiB) $\downarrow$	Recon. FPS $\uparrow$	VAE Render FPS $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DL3DV
ReCoSplat	$480\!\times\!480$	0.118	0.858	2,085	198.88	11.12	3.0	21.53	0.707	0.318
ReCoSplat	$480\!\times\!832$	0.664	0.344	11,550	1101.48	2.15	2.7	12.41	0.438	0.711
OF 3 GS	$405\!\times\!832$	0.452	0.522	8,987	479.98	4.01	3.0	14.03	0.418	0.606
L3G-Mem	$480\!\times\!832$	0.142	0.823	39	14.11	28.66	735.3	17.52	0.526	0.537
RealEstate10K (Out-of-domain)

Appendix

Table 6: Online NVS evaluation. We evaluate online reconstruction on DL3DV and on RealEstate10K, reporting RGB and VAE latent novel view synthesis quality together with memory size, reconstruction FPS, and VAE rendering FPS. Reconstruction FPS is computed from nine-frame update chunks. The best and second-best results within each dataset are shown in bold and underline , respectively.

Figure 9: Visualization of latent 3D Gaussian memory conditioning for video generation. We visualize the ground-truth and generated video, with target-view rendered VAE condition from the latent 3D Gaussian memory, illustrating that spatially aligned latent memory provides consistent conditioning for video generation.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 264 papers

Latent Spatial Memory for Video World Models
Jun 8, 2026Weijie Wang, Haoyu Zhao, Yifan Yang +7Video Diffusion Models Memory-Augmented Video Generation
Zhejiang University · Microsoft Research · Adelaide University +1
World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models
Oct 12, 2026Junyi Wu, Zekun Li, Chenhongyi Yang +9Video World Models Memory-Augmented Video Generation
Meta Reality Labs · University of Illinois Chicago · Brown University
Beyond Pixel Histories: World Models with Persistent 3D State
Mar 3, 2026Samuel Garcin, Thomas Walker, Steven McDonagh +5World Models Interactive World Models
University of Edinburgh · Microsoft Research

Jun 8, 2026J/K move · Enter open · S save

Latent Spatial Memory for Video World Models

Weijie Wang, Haoyu Zhao, Yifan Yang, Feng Chen+6

Zhejiang University · Microsoft Research · Adelaide University+1

Video Diffusion Models Memory-Augmented Video Generation Video World Models

Open paper AskPDF arXiv

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to 10.57

\times

faster end-to-end video generation and 55

\times

reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.

Abstract

Video World Models

Explore similar work

Latent Spatial Memory for Video World Models

World-T3: Test-Time Training as Persistent Memory for Real-Time Video World Models

Beyond Pixel Histories: World Models with Persistent 3D State