We propose UniDynamics, a diffusion-based framework for future 4D dynamic scenes (RGB, depth, and optical flow) generation from a single event-RGB pair, without requiring long histories or control priors as in existing methods, while explicitly modeling future motion fields. The core idea is to leverage event streams to offer an alternative motion prior for single-RGB extrapolation, and to enforce geometric and motion constraints throughout generation via multimodal modeling. Specifically, we design an Event Latent Enhancement (ELE) module to align and enhance event latents into diffusion-injectable conditioning features, providing robust initial motion priors and reliable texture/structure cues. We further introduce a Perceptual Dynamics Space (PDS) embedded in the multi-scale U-Net, which decouples and adaptively interacts depth and flow while continuously feeding back constraints to appearance features, improving geometric-motion consistency for physically plausible and spatiotemporally coherent prediction. Experiments on VKitti2 and DSEC demonstrate state-of-the-art performance, producing high-quality, temporally coherent, and 4D-consistent future predictions, especially under challenging high-speed motion blur.
Figures & tables
Figure 1: Our UniDynamics infers future 4D scenes using only the current event-RGB pair. Leveraging motion cues and texture priors of events, it jointly models appearance, geometry, and motion, producing more faithful and spatiotemporally coherent predicts, especially under motion blur, where UniFuture [ 24 ] with only a blurred RGB fails.
Figure 2: The training pipeline of UniDynamics. The event stream is encoded by the Event Latent Enhancement (ELE) module into diffusion-injectable features that provide initial motion and texture cues. RGB, depth, and optical flow are derived from the same RGB latent: RGB is generated via diffusion denoising, while depth and flow are jointly learned through an additional Perceptual Dynamics Space (PDS). These modalities interact and are refined continuously across multiple scales, enabling physically plausible and spatiotemporally coherent future scene prediction.
Figure 3: UniDynamics submodules: (a) ELE module, which exploits the complementarity of events and RGB to produce an event latent with effective initial momentum and rich texture cues; (b) PDS layer, which decouples the depth and flow spaces to enable independent learning with adaptive interaction; and (c) depth geometry and flow motion cues are injected back into the U-Net’s spatial and temporal layers, respectively, providing effective constraints on RGB evolution.
Figure 4: UniDynamics inference pipeline: given a single event–RGB pair, it simultaneously predicts future RGB, depth, and optical flow. The RGB latent is concatenated with noise and, together with the event latent, fed into the U-Net for denoising to obtain multimodal latents, which are finally decoded by a shared decoder into future RGB frames, depth maps, and optical flow.
Method
Generation
Depth
Optical Flow
FID ↓
FVD ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
RMSE ↓
EPE ↓
AE ↓
1PE ↓
2PE ↓
3PE ↓
Only future generation
SimVP [ 9 ]
137.8
1301.7
Unsupported
TAU [ 35 ]
127.8
1530.4
Unsupported
SVD [ 2 ]
114.5
1244.4
Unsupported
Vista [ 8 ]
101.8
1364.6
Unsupported
Table 1: Quantitative comparison on E-VKitti2 [ 7 ] . “ ↑ ” and “ ↓ ” indicate that higher and lower values are better, respectively. Best results are highlighted in bold. For unsupported tasks, we mark them as “Unsupported”.
Figure 5: Qualitative comparisons of future 4D scene prediction on E-VKitti2. UniDynamics preserves spatiotemporal coherence and yields more accurate perception results.
Method
Generation
Depth
Optical Flow
FID ↓
FVD ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
RMSE ↓
EPE ↓
AE ↓
1PE ↓
2PE ↓
3PE ↓
UniFuture-Flow [ 24 ]
61.4
644.3
Unsupported
5.67
32.3
78.4
62.6
51.4
UniFuture-Depth [ 24 ]
56.1
668.7
0.391
57.9
84.1
90.1
59.5
Unsupported
UniDynamics (w/o events)
59.8
617.7
0.288
50.2
79.9
84.2
58.5
5.40
33.8
78.6
60.9
48.7
UniDynamics (ours)
53.6
272.8
0.265
68.1
88.0
92.9
54.4
2.86
19.5
55.4
36.0
25.9
Table 2: Performance comparison under motion blur conditions on the E-VKitti2 [ 7 ] .
Figure 6: The reconstructed 3D point cloud and scene flow, obtained from the future RGB, depth, and optical flow, demonstrate our ability to predict future 4D scenes.
Dataset
Method
Generation
Depth
Optical Flow
FID ↓
FVD ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
RMSE ↓
EPE ↓
AE ↓
1PE ↓
2PE ↓
3PE ↓
DSEC
TAU [ 35 ]
126.5
1052.9
Unsupported
FLODCAST [ 4 ]
Unsupported
0.54
26.5
63.2
74.6
10.3
15.3
25.8
-
-
-
FUTURIST [ 20 ]
Unsupported
0.40
38.1
66.9
85.8
8.0
Unsupported
UniFuture-Depth
46.2
285.7
0.35
39.7
71.2
89.5
8.9
Unsupported
UniFuture-Flow
54.3
314.8
Unsupported
12.4
32.4
96.3
88.1
79.1
Table 3: Zero-shot results on the real-world datasets. “-” indicates that the result is either not reported or the dataset does not support the computation of the metric.
Setting
Generation
Depth
Optical Flow
FID ↓
FVD ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
RMSE ↓
EPE ↓
AE ↓
1PE ↓
2PE ↓
3PE ↓
UniDynamics (w/o events)
46.8
577.1
0.36
56.0
83.9
90.3
59.0
5.32
34.1
79.2
61.5
49.1
UniDynamics (hard in)
43.3
417.6
0.33
60.2
85.8
91.4
56.7
4.63
28.2
71.8
53.5
41.4
UniDynamics-Depth (w/o flow)
42.0
264.2
0.27
68.6
88.5
93.0
53.9
Unsupported
UniDynamics-Flow (w/o depth)
44.1
268.3
Unsupported
3.64
24.8
64.4
45.3
34.1
UniDynamics (w/o adapters)
50.8
384.9
0.30
62.6
85.6
91.6
56.0
4.60
30.3
72.3
53.5
41.9
Table 4: Ablation studies of four key components on E-VKitti2 [ 7 ] .
School of Information Science and Technology, University of Science and Technology of China · Zhongguancun Academy · the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences. +2