4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling

Jianhui Zhang; Zuyao Lin; Pengxin Xu; Jingdong Wang; Xingyu Chen

cs.CVOct 13, 2026

4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling

Authors: Jianhui Zhang, Zuyao Lin, Pengxin Xu, Jingdong Wang, Xingyu Chen

Organizations: Shanghai Jiaotong University · Zhongguancun Academy · Institute of Automation, Chinese Academy of Science · Harbin Institute of Technology

Abstract

For video generation in dynamic world modeling, observed static and dynamic content should be efficiently reused to provide conditions for repeated generation at different times and viewpoints. We introduce 4LC, a 4D latent cache with composite visual representations and a flexible spatiotemporal access interface. As a latent cache, 4LC directly reuses visual representations of static and dynamic content to provide generation conditions, avoiding per-request RGB rendering and re-encoding. As a composite cache, 4LC binds visual representations from multiple visual encoders (e.g., VAE, DINO, V-JEPA, VGGT) to shared geometry and motion, allowing the composite representations to describe the dynamic world with rich information. Besides, we design a flexible spatiotemporal interface, which directly reads the cache at independently specified times and viewpoints and combines readouts across observations, providing a common conditioning format for diverse downstream tasks. Building on 4LC, we develop 4LC-Video to address multiple video generation tasks through different configurations of source observations, spatiotemporal requests, and historical reuse, including camera control, temporal control, novel-view synthesis, scene revisits, and long-term generation. Our experiments show more efficient condition acquisition, effective spatiotemporal controllability, and scene consistency across tasks using a single 4LC-Video model without task-specific fine-tuning.

Figures & tables

Figure 2: Architecture of 4LC. (a) Construction binds visual representations from multiple encoders to shared geometry and motion in one cache entry. (b) Readout propagates the entry to a requested time and projects it through a target camera, yielding readout features and coverage masks. (c) Composition fills uncovered regions with readouts from further entries.

Figure 4: Qualitative comparison for camera and temporal control. Left: dynamic scene; right: static scene. Top: Static camera. The camera is fixed while scene time follows the source timeline. Bottom: Frozen time. Scene time is fixed at source frame 0 while the camera follows the source trajectory. Input frame labels identify each column; red boxes highlight distortions.

Method	$r_{\mathrm{dist}}{\downarrow}$	$t_{\mathrm{dist}}{\downarrow}$	Subj. Consist.	Back. Consist.	Temp. Flick.	Motion Smooth.	Aesth. Quality	Imag. Quality
Camera control
SpaceTimePilot	0.179	0.321	89.79	93.43	95.26	98.69	47.60	57.23
TrajectoryCrafter	0.244	0.395	86.37	92.19	92.69	96.17	45.16	60.97
GEN3C	0.203	0.285	90.45	93.79	95.66	97.55	45.62	56.39
NeoVerse	0.059	0.120	88.63	92.77	94.43	98.17	50.38	67.84
4LC-Video (ours)	0.037	0.089	90.71	94.14	95.47	98.06	48.16	65.49

Table 2: Quantitative comparison for camera and temporal control. We evaluate 100 randomly sampled unseen videos under four camera trajectories (400 cases) and three temporal trajectories (300 cases).

r_{\mathrm{dist}}

measures rotation error in radians and

t_{\mathrm{dist}}

scale-normalized translation error; higher VBench scores are better. Best and second-best results per task are bold and underlined.

Method	Novel-view synthesis			Closed-loop consistency
Method	PSNR ${\uparrow}$	SSIM ${\uparrow}$	LPIPS ${\downarrow}$	PSNR ${}_{C}{\uparrow}$	SSIM ${}_{C}{\uparrow}$	LPIPS ${}_{C}{\downarrow}$
ViewCrafter	15.78	0.580	0.396	14.79	0.481	0.365
FlexWorld	16.25	0.593	0.370	12.20	0.428	0.598
Voyager	17.79	0.636	0.297	17.66	0.540	0.380
Spatia	18.58	0.646	0.254	19.38	0.579	0.213
Mirage	18.38	0.779	0.250	20.05	0.825	0.228

Table 3: Quantitative comparison for novel-view synthesis and scene revisits. We use the protocols and baseline results reported by Spatia [ 5 ] and Mirage [ 13 ] , and evaluate LiveWorld under the same protocols.

Method	Env. score	Causal score	Spatial		Rendering		Self-evolution		Interaction
Method	Env. score	Causal score	Epipolar	Reproj.	Lighting	Style	State	Correct.	Action
HY-WorldPlay	83.23	65.86	83.86	68.17	82.67	98.23	49.50	62.40	85.69
Infinite-World	73.76	70.92	74.04	61.51	62.63	96.87	48.00	78.40	86.37
Yume-1.5	54.92	85.03	51.86	24.55	51.21	92.05	97.90	95.00	62.20
LingBot-World	38.90	82.91	22.12	7.57	40.06	85.87	96.00	89.40	63.32
4LC-Video (ours)	85.56	89.06	71.11	75.98	97.52	97.66	85.00	93.20	88.97

Table 4: Quantitative comparison for long-term generation on MBench-A [ 58 ] . We compare Environment and Causal scores with published leaderboard results.

Figure 5: Efficiency and feature similarity across condition-acquisition paths. Left: Latency and peak GPU memory of 4LC readout and of RGB rendering and VAE encoding from NeoVerse’s 4DGS and LiveWorld’s RGB point clouds, with the end-to-end latency of each framework. Rendering + Encoding includes rendering time. Solid and dashed 4LC Readout bars use Wan2.1 VAE alone and with DINOv2, VGGT, and V-JEPA 2.1. Latency uses a square-root scale. Right: Feature similarity between direct readouts and re-encoded features, as per-case mean cosine similarity in the original (Absolute) and relative-representation (Relative) spaces; black ticks mark medians.

RGB rendering and re-encoding
Visual representations	$r_{\mathrm{dist}}{\downarrow}$	$t_{\mathrm{dist}}{\downarrow}$	Subj. Consist.	Back. Consist.
Wan2.1 VAE	0.046	0.106	89.86	93.69
FLUX.2 VAE	0.044	0.106	90.00	93.80
Direct readout
Wan2.1 VAE	0.056	0.128	89.03	93.03
FLUX.2 VAE	0.049	0.114	90.06	93.81

Table 5: Ablations of condition-acquisition paths and composite visual representations. All variants share a frozen Wan2.1-T2V-1.3B backbone and the same 100 unseen videos with all trajectories. ALL denotes the other three encoders.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	Features	$d_{r}$	$D_{r}$	Target interval
FLUX.2 VAE	Posterior mean of RGB and depth	8	32	1
DINOv2-L	Patch tokens	14	1024	1
VGGT-1B	Aggregator layer-11 tokens	14	2048	1
V-JEPA 2.1 ViT-L	Patch tokens	16	1024	2
Wan2.1 VAE	Latents of RGB and depth	8	16	4

Appendix

Table 6: Visual encoder settings.

d_{r}

is the spatial stride in pixels,

D_{r}

the feature dimension, and the target interval the number of frames between readouts.

Type	Parameter	Value
Geometric	Number of keyframes $K$	11–21
	Augmentation probability	0.6
	Maximum source-frame drop fraction	0.3
	Lateral translation	0.1–0.3
	Yaw and pitch	$5^{\circ}$ – $25^{\circ}$
	Zoom-in / zoom-out focal-length factor	1.1–1.3 / 0.7–0.9

Appendix

Table 7: Training augmentation parameters. Geometric augmentation applies outside first-frame sampling. Lateral translation is in world units; focal-length factors are relative to the source camera.

Figure 6: Qualitative comparison for long-term generation on MBench-A. Each method occupies two rows: the top row follows the outbound trajectory from left to right, and the bottom row follows the return trajectory from right to left, so vertically aligned frames share a camera pose. Red boxes mark structures that change between the first and final frames.

Static scene generation
Method	WorldScore			Controllability			Quality				Dynamics
Method	Avg.	Static	Dynamic	Camera	Object	Content	3D	Photo.	Style	Subj.	Acc.	Mag.	Smooth.
WonderJourney	54.19	63.75	44.63	84.60	37.10	35.54	80.60	79.03	62.82	66.56	–	–	–
InvisibleStitch	51.95	61.12	42.78	93.20	36.51	29.53	88.51	89.19	32.37	58.50	–	–	–
WonderWorld	61.79	72.69	50.88	92.98	51.76	71.25	86.87	85.56	70.57	49.81	–	–	–
Voyager	66.08	77.62	54.53	85.95	66.92	68.92	81.56	85.99	84.89	71.09	–	–	–

Appendix

Table 8: Quantitative comparison on WorldScore [ 53 ] . We compare against its published leaderboard results and, following Spatia [ 5 ] , report Avg. as the mean of the Static and Dynamic scores; missing motion metrics of static scene generators count as zero.

Threshold $\epsilon$	$r_{\mathrm{dist}}{\downarrow}$	$t_{\mathrm{dist}}{\downarrow}$	Subj. Consist.	Back. Consist.	Temp. Flick.	Motion Smooth.	Aesth. Quality	Imag. Quality
$0\phantom{.00}$	0.065	0.138	89.91	93.53	94.44	97.88	49.08	67.07
$0.05$	0.035	0.092	90.61	94.06	94.81	98.03	48.50	66.88
$0.1\phantom{0}$	0.037	0.089	90.71	94.14	95.47	98.06	48.16	65.49
$0.2\phantom{0}$	0.035	0.083	90.76	94.24	94.88	98.04	48.15	66.61
$0.4\phantom{0}$	0.036	0.083	90.74	94.23	94.88	98.04	48.13	66.61

Appendix

Table 9: Motion filtering and threshold robustness. The main model is evaluated on the same 100 unseen videos with camera trajectories.

\epsilon

is in pixels/frame;

\epsilon=0

disables filtering, and

\epsilon=0.1

is the default. Bold and red values mark the best and worst results, respectively.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 257 papers

Beyond Pixels: From Video Priors to 4D Worlds
Aug 11, 2026Zihao Liu, Xiaolong Shen, Zhenglin Zhou +2Video Diffusion Models 4D Scene Generation
ReLER, CCAI, Zhejiang University
CtrlCache: Accelerating Interactive Video World Models with Control-Aware Caching
Oct 6, 2026Shangye Song, Dong Gong, Hong Jia +2Action-Conditioned Video Generation Real-Time Interactive Video Generation
University of Auckland · UNSW Sydney
SplatFlux: Feed-Forward 4D Gaussian Reconstruction and Generation from Video Latents
Oct 20, 2026Runmin Zhang, Binglin Ren, Zhu Yu +64D Reconstruction Dynamic Scene Reconstruction
Zhejiang University · Ant Group

Aug 11, 2026J/K move · Enter open · S save

Beyond Pixels: From Video Priors to 4D Worlds

Zihao Liu, Xiaolong Shen, Zhenglin Zhou, Ruijie Quan+1

ReLER, CCAI, Zhejiang University

Video Diffusion Models 4D Scene Generation

Open paper AskPDF arXiv

4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.

Abstract

Video World Models

Explore similar work

Beyond Pixels: From Video Priors to 4D Worlds

CtrlCache: Accelerating Interactive Video World Models with Control-Aware Caching

SplatFlux: Feed-Forward 4D Gaussian Reconstruction and Generation from Video Latents