Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution

Feiran Wang; Bin Duan; Junyi Wu; Gaowen Liu; Yan Yan

cs.CVOct 2, 2026

Kepler4D: Controllable Future Video Generation via 4D Scene State Evolution

Authors: Feiran Wang, Bin Duan, Junyi Wu, Gaowen Liu, Yan Yan

Organizations: University of Illinois at Chicago · University of Michigan Ann Arbor · Cisco

Abstract

Video world models aim to preserve scene structure and predict how dynamic objects evolve beyond visual observations. We present Kepler4D, a framework for future video generation through explicit 4D scene state evolution. Given a monocular video, Kepler4D constructs a shared 3D representation of background geometry, object motion histories, coarse spatial supports, and semantic context. Chain-of-Motion summarizes observed motion and uses a vision-language model to select structured speed and heading decisions and decide whether to bound object-center height from below. A deterministic rollout converts these decisions into future object trajectories for inspection and editing before synthesis. We render the evolving proxies into geometric controls for a pretrained video generator, separating coarse object motion from the synthesis of appearance and articulation. Experiments on real-world videos demonstrate that Kepler4D enables controllable object motion and plausible future rollout while preserving scene consistency.

Figures & tables

Figure 1: Kepler4D enables future video generation through explicit trajectory evolution and control in a 4D scene state. Our method organizes static geometry, foreground motion histories, and semantic context into the scene state, predicting scene-aware 3D trajectories to guide generation. It supports auto matic future evolution, language-guided trajectory edit ing, and view-consistent rendering across viewpoints, producing plausible future motion while preserving the reconstructed scene structure.

Figure 2: Overview of Kepler4D. Given an observed video, Kepler4D constructs a structured 4D scene state comprising static scene geometry, foreground motion histories, and semantic context. Chain-of-Motion reasons over the observed motion and scene context to infer structured motion decisions, which a deterministic 3D rollout expands into future object trajectories. These trajectories are rendered with static geometry into depth maps, object control signals, and masks to guide a pretrained video generator. This design separates explicit object motion prediction from video synthesis, enabling controllable generation of future videos while preserving the scene structure.

Figure 3: Static novel view synthesis on DL3DV. We compare Kepler4D with GEN3C, TrajectoryCrafter, and SPMem on indoor and outdoor scenes under target camera motion. Each row shows an input view, the target camera path, the held-out reference view, and the generated results. In these examples, Kepler4D preserves the scene layout and appearance details across viewpoint changes.

	Reconstruction Fidelity			Generation Quality					Efficiency
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	Imag. $\uparrow$	MotSm. $\uparrow$	BgC. $\uparrow$	Overall $\uparrow$	Mem. (GB)	Time (min)
GEN3C Ren et al. (2025)	11.67	0.2963	0.5706	63.99	0.6991	0.9863	0.9314	0.8041	54.73	43
TrajectoryCrafter Yu et al. (2025)	11.90	0.3114	0.5791	74.23	0.6709	0.9712	0.9372	0.8005	37.81	10.4
SPMem Wu et al. (2025)	13.19	0.3611	0.5273	56.92	0.5819	0.9776	0.9134	0.8175	45.57	9.7
Ours	13.55	0.3675	0.3995	41.37	0.7606	0.9745	0.9448	0.8322	62.46	5.2

Table 1: Static view synthesis on DL3DV ( Ling et al., 2024 ) . We compare Kepler4D with three baselines using reference fidelity, FID, and VBench ( Huang et al., 2024 ) metrics. We also report peak GPU memory and per-clip inference time under the evaluated model and inference configurations.

Figure 4: Automatic future evolution. All methods use the input video without externally supplied object trajectories or motion commands. Methods with camera control are given the same target camera path. Kepler4D depicts the runner moving forward and vaulting over the railing, as in the reference. TrajectoryCrafter shows a viewpoint change without a vault. SPMem shows apparent intersections between the runner and railing. VerseCrafter shows a lateral deviation before the vault.

	Reconstruction Fidelity			Generation Quality				Motion Fidelity
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	Imag. $\uparrow$	DynDeg $\uparrow$	Overall $\uparrow$	Flow-EPE $\downarrow$	RotErr $\downarrow$
TrajectoryCrafter Yu et al. (2025)	14.349	0.3593	0.5145	214.13	0.6595	0.1622	0.7064	17.409	7.5400
SPMem Wu et al. (2025)	14.641	0.3837	0.5102	265.13	0.5870	0.7838	0.7832	20.138	18.549
VerseCrafter Zheng et al. (2026)	14.090	0.3649	0.4556	303.80	0.6896	0.6486	0.7773	15.400	8.9610
Ours	14.921	0.3924	0.4561	202.64	0.6970	0.7838	0.8029	15.220	5.0200

Table 2: Automatic evolution on DAVIS ( Pont-Tuset et al., 2017 ) . All methods receive the same observed video and camera path. We report reconstruction fidelity, generation, and motion fidelity.

Figure 5: Language-guided controllable future generation. Given the same input video, the user specifies the intended motion through a high-level language command. Kepler4D infers an object-level trajectory from the command, shown in the second column. Baselines receive the same language command but not the inferred trajectory or control maps. With explicit trajectory control, Kepler4D better realizes the intended object motion while preserving scene layout and consistency.

Table 8

Table 9

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Component or setting	Configuration
Video backbone	Wan2.1-T2V-14B
Geometric conditioning	Pretrained VerseCrafter GeoAdapter
Acceleration	LightX2V CFG/step-distillation LoRA; merge weight 1.0
Numerical precision	bfloat16
Sampler	Flow-matching Euler scheduler
Denoising steps	6

Appendix

Table A1: Default video-synthesis launcher settings; listed spatial dimensions are height

\times

width.

Stream	Information represented
Background RGB	Appearance of the rendered background geometry
Background depth	Depth of the rendered background geometry
Foreground Gaussian RGB	Projected object support and identity color
Foreground Gaussian depth	Depth of the coarse foreground support
Merged mask	Spatial mask used by the video-conditioning interface

Appendix

Table A2: Control streams supplied to the pretrained synthesis interface. The combined background/foreground visualization is diagnostic and distinct from the separate conditioning streams.

Dimension	Measured property
Aesthetic Quality	Visual aesthetic quality
Imaging Quality (Imag.)	Image quality, including sharpness and artifacts
Motion Smoothness (MotSm.)	Smoothness of temporal transitions
Subject Consistency	Consistency of foreground visual identity
Background Consistency (BgC.)	Consistency of background appearance
Dynamic Degree (DynDeg)	Presence of sufficiently strong motion

Appendix

Table A3: Interpretation of the VBench dimensions used for video evaluation in the main paper.

Variant	Text guidance	CoM trajectory guidance
Qwen prompt	Observed scene description	Absent
CoM w/o traj.	Predicted motion description	Not rendered into foreground controls
CoM w/ traj.	Predicted motion description	Rendered future object trajectories

Appendix

Table A4: Text and trajectory guidance supplied by the three tested motion-conditioning variants.

Figure 6: Schematic illustration of Chain-of-Motion inputs and outputs. CoM uses observed motion and semantic scene context to select speed and heading actions, decide whether to apply a lower bound to object-center height, and produce a motion description. The structured decisions guide deterministic 3D trajectory rollout, while the description provides text conditioning for video synthesis. The fields illustrate the interface and are not numerical logs from a benchmark sequence.

Information	Role
Endpoint images	Visual evidence for object identity and scene context
Object identifier and label	Associate masks, history, decisions, and controls
Observed motion summary	Numerical evidence for speed and heading decisions
Scene description	Context for a plausible continuation
Speed, heading, and height-bound decisions	Inputs to deterministic trajectory updates
Motion description	Semantic guidance for video synthesis

Appendix

Table A5: Roles of observed information and motion decisions in the structured CoM interface.

Explore similar work

Learning Long-term Motion Embeddings for Efficient Kinematics Generation
Apr 13, 2026Nick Stracke, Kolja Bauer, Stefan Andreas Baumann +3Motion Representation Learning Human Motion Generation
Munich Center for Machine Learning · Apple
WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory
Jul 2, 2026Hanlin Wang, Hao Ouyang, Qiuyu Wang +10Physical Consistency in Video Generation Camera-Controlled Video Generation
1HKUST · 2Ant Group · 3ZJU +1
UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation
Oct 2, 2026Daikun Liu, Xin Zhan, Teng Wang +24D Scene Generation Video Prediction
Southeast University, China

Apr 13, 2026J/K move · Enter open · S save

Learning Long-term Motion Embeddings for Efficient Kinematics Generation

Nick Stracke, Kolja Bauer, Stefan Andreas Baumann, Miguel Angel Bautista+2

Munich Center for Machine Learning · Apple

Motion Representation Learning Human Motion Generation

Open paper AskPDF arXiv

Understanding and predicting motion is a fundamental component of visual intelligence. Although modern video models exhibit strong comprehension of scene dynamics, exploring multiple possible futures through full video synthesis remains prohibitively inefficient. We model scene dynamics orders of magnitude more efficiently by directly operating on a long-term motion embedding that is learned from large-scale trajectories obtained from tracker models. This enables efficient generation of long, realistic motions that fulfill goals specified via text prompts or spatial pokes. To achieve this, we first learn a highly compressed motion embedding with a temporal compression factor of 64x. In this space, we train a conditional flow-matching model to generate motion latents conditioned on task descriptions. The resulting motion distributions outperform those of both state-of-the-art video models and specialized task-specific approaches.

Abstract

Explore similar work

Learning Long-term Motion Embeddings for Efficient Kinematics Generation

WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation