Event-based Scene Synthesis via Inter-Frame Residual Alignment

Jiyun Kong; Jun-Hyuk Kim; Jong-Seok Lee

cs.CVDec 19, 2025

Event-based Scene Synthesis via Inter-Frame Residual Alignment

Authors: Jiyun Kong, Jun-Hyuk Kim, Jong-Seok Lee

Organizations: Yonsei University, Korea · Chung-Ang University, Korea

Abstract

Event-based scene synthesis reconstructs target RGB frames from sparse image observations and asynchronous event streams, encompassing both video frame prediction and interpolation. Existing event-based synthesis methods commonly estimate optical flow to warp the observed frames toward the target time, but are vulnerable to inaccurate flow under large motion and occlusion and often rely on flow supervision or pretrained estimators. In this work, we propose EvFRA, an Event-based scene synthesis framework based on inter-Frame Residual Alignment. We identify a structural correspondence between event measurements and frame-to-frame scene changes, and exploit this correspondence for target frame synthesis. Our training pipeline consists of two stages: 1) an Event-to-Residual Alignment Variational Autoencoder (ER-VAE) aligns the event frame captured between the anchor and target frames with the corresponding inter-frame residual, and 2) a ControlNet-conditioned diffusion model is fine-tuned to denoise the residual latent using event data. Our method outperforms state-of-the-art methods by up to 2.61 dB and 1.85 dB in PSNR for frame prediction and interpolation, respectively, with consistent SSIM improvements. Code is available at https://github.com/jiyun-kong/EvFRA.

Figures & tables

Panel of Figure 1 — Figure 1 : Comparison of our method with other approaches on BS-ERGB [ 47 ] , including VFPSIE [ 59 ] and CBMNet-Large [ 25 ] . Ours produces more accurate motion prediction under fast movement, preserving object boundaries without ghosting artifacts.

Figure 2 : Overview of our proposed method EvFRA . During inference, Gaussian noise at $\sigma_{\max}$ is added to $z^{\text{ER}}_{\text{event}}$ to initialize the denoising trajectory. The diffusion U-net predicts the residual latent $\hat{z}_{\text{res}}$ , which is added to $z_{t}$ to obtain the predicted latent $\hat{z}_{t+1}$ . Finally, the decoder reconstructs the target frame $\hat{I}_{t+1}$ from $\hat{z}_{t+1}$ .

Figure 3 : Training pipeline of EvFRA . The framework consists of two stages: in stage 1, ER-VAE is optimized with the loss $\mathcal{L}_{\text{E2R}}$ , which jointly enforces latent alignment between the event latent $z^{\text{ER}}_{\text{event}}$ and the residual latent $z_{\text{res}}=z_{t+1}-z_{t}$ . In stage 2, the event-conditioned encoder is trained with the denoising objective $\mathcal{L}_{\text{DM}}$ to guide the diffusion model in predicting the denoised residual latent $\hat{z}_{\text{res}}$ .

Figure 4 : Structural relationship between event and residual representations. The decoded residual latent (d) is derived from the anchor (a) and target frame (b) as the difference between them. The event frame (c) reveals that event data captures inter-frame motion consistent with the decoded residual latent.

Task	Method	BS-ERGB [ 47 ]						HS-ERGB [ 48 ]
		1 frame			3 frames			7 frames
		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Frame Prediction	VFPSIE [ 59 ] (AAAI’24)	24.41	0.805	0.091	23.20	0.782	0.114	28.81	0.845	0.112
Frame Prediction	EvFRA(VFP)	25.27	0.895	0.100	24.20	0.804	0.119	29.74	0.858	0.101
Frame Interpolation	TimeLens [ 48 ] (CVPR’21)	23.08	0.694	0.121	21.93	0.658	0.172	25.98	0.793	0.167
	CBMNet-Large [ 25 ] (CVPR’23)	24.96	0.827	0.120	23.83	0.800	0.123	28.14	0.844	0.149

Table 1 : Quantitative comparison for video frame prediction and interpolation on the real-world event datasets: BS-ERGB [ 47 ] and HS-ERGB [ 48 ] . The best and second-best results are highlighted in bold and underline , respectively.

Task	Method	GoPro [ 31 ]
		7 frames			15 frames
		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Frame Prediction	VFPSIE [ 59 ] (AAAI’24)	18.26	0.557	0.386	17.53	0.523	0.438
Frame Prediction	EvFRA(VFP)	20.87	0.703	0.231	19.75	0.672	0.308
Frame Interpolation	TimeLens [ 48 ] (CVPR’21)	17.29	0.571	0.300	15.02	0.531	0.384
	CBMNet-Large [ 25 ] (CVPR’23)	17.98	0.592	0.327	17.44	0.532	0.372

Table 2 : Quantitative comparison for video frame prediction and interpolation on the synthetic event dataset: GoPro [ 31 ] . The best and second-best results are highlighted in bold and underline , respectively.

Figure 5 : Qualitative comparison for video frame prediction on HS-ERGB [ 48 ] and GoPro [ 31 ] . Compared with VFPSIE [ 59 ] , EvFRA better preserves moving object boundaries and text details while reducing blur and scattered motion artifacts.

Figure 6 : Qualitative comparison for video frame interpolation on BS-ERGB [ 47 ] . Compared with CBMNet-Large [ 25 ] and RE-VDM [ 6 ] , EvFRA more accurately reconstructs rapidly moving objects with fewer duplicated contours, geometric distortions, and motion blur.

Table 9

Table 10

Panel of Figure 7 — Figure 7 : Analysis of the effectiveness of the ER-VAE in both the latent and pixel spaces.

Panel of Figure 8 — Figure 8 : Qualitative direct-decoding analysis of ER-VAE alignment. $\mathcal{D}$ denotes the pretrained VAE decoder, and $z_{\mathrm{event}}^{\mathrm{VAE}}$ and $z_{\mathrm{event}}^{\mathrm{ER}}$ denote the VAE-encoded and ER-VAE-aligned event latents, respectively. The second row compares (d) the decoded ground-truth residual latent with reconstructions from (e) the unaligned and (f) aligned event latents, showing that ER-VAE alignment reduces structural artifacts and produces a result closer to the target frame.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Metrics	Seed #123	Seed #0	Seed #42	Mean $\pm$ Std
Video Frame Prediction	PSNR $\uparrow$	24.20	23.92	24.09	24.07 $\pm$ 0.14
	SSIM $\uparrow$	0.804	0.800	0.802	0.802 $\pm$ 0.002
	LPIPS $\downarrow$	0.119	0.119	0.120	0.119 $\pm$ 0.001
Video Frame Interpolation	PSNR $\uparrow$	24.28	24.23	24.28	24.26 $\pm$ 0.03
	SSIM $\uparrow$	0.831	0.827	0.827	0.828 $\pm$ 0.002
	LPIPS $\downarrow$	0.116	0.115	0.115	0.115 $\pm$ 0.001

Appendix

Table S1 : Quantitative comparison across different diffusion sampling seeds on BS-ERGB [ 47 ] under the 3-frame video frame prediction and interpolation settings.

ID	Target	Initialization	Event Cond.	Generator	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
A	$z_{t+1}$	$\mathcal{N}(0,I)$	✓	Diffusion	16.72	0.571	0.340
B	$z_{t+1}$	$z_{t}+z_{\rm event}^{\rm VAE}$	✓	Diffusion	19.51	0.703	0.315
C	$z_{t+1}$	$z_{t}+z_{\rm event}^{\rm ER}$	✓	Diffusion	19.96	0.711	0.313
D	$z_{\rm res}$	$\mathcal{N}(0,I)$	✓	Diffusion	23.67	0.762	0.161
E	$z_{\rm res}$	$z_{\rm event}^{\rm VAE}$	✓	Diffusion	21.43	0.734	0.204
F (EvFRA)	$\mathbf{z_{\rm res}}$	$\mathbf{z_{\rm event}^{\rm ER}}$	✓	Diffusion	24.20	0.804	0.119

Appendix

Table S2 : Matched ablation study on BS-ERGB [ 47 ] under the 3-frame video frame prediction setting, analyzing residual modeling, initialization, event conditioning, and generator design.

Steps	2	5 (default)	10	15	25	50
PSNR $\uparrow$	24.15	24.20	24.20	24.21	24.21	24.21
SSIM $\uparrow$	0.800	0.804	0.802	0.801	0.801	0.801
LPIPS $\downarrow$	0.119	0.119	0.120	0.120	0.120	0.120
Time (s)	7.1	10.3	17.7	22.9	35.4	67.5

Appendix

Table S3 : Ablation on the number of diffusion denoising steps on BS-ERGB [ 47 ] under the 3-frame video frame prediction setting. The current setting uses 5 denoising steps.

Order	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
$1\rightarrow 2\rightarrow 3$	25.27 / 24.20	0.895 / 0.804	0.100 / 0.119
$3\rightarrow 2\rightarrow 1$	22.58 / 22.07	0.775 / 0.735	0.150 / 0.181
$1\rightarrow 3\rightarrow 2$	21.39 / 20.79	0.755 / 0.701	0.159 / 0.179

Appendix

Table S4 : Ablation study of VLT Augmentation on BS-ERGB [ 47 ] . Results are reported for 1-frame / 3-frame video frame prediction. The best and second-best results are highlighted in bold and underlined , respectively.

Setting	1 frame			3 frames
Setting	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
VAE Scaling
w/o	25.20	0.882	0.100	24.13	0.801	0.119
with	25.27	0.895	0.100	24.20	0.804	0.119
KL Divergence
w/o	25.27	0.895	0.100	24.20	0.804	0.119

Appendix

Table S5 : Ablation on VAE scaling, KL divergence objective, and image loss weight for BS-ERGB [ 47 ] . Metrics are reported for 1-frame and 3-frame prediction tasks. The best results are highlighted in bold .

Metric	$\lambda=0.1$	$\lambda=0.3$	$\lambda=0.5$	$\lambda=0.7$	$\lambda=0.9$
PSNR $\uparrow$	22.51	23.01	23.02	22.47	22.42
SSIM $\uparrow$	0.725	0.746	0.738	0.735	0.698
LPIPS $\downarrow$	0.182	0.160	0.169	0.168	0.178

Appendix

Table S6 : Ablation on the LPIPS loss weight

\lambda

in stage 2 training. The ablation is performed on BS-ERGB [ 47 ] under the 3-frame video frame prediction setting. The best results are highlighted in bold .

Modality	Task	Method	BS-ERGB [ 47 ]						HS-ERGB [ 48 ]
			1 frame			3 frames			7 frames
			PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
2 Images	Frame Prediction	RaMViD [ 17 ] (TMLR’22)	14.40	0.451	0.400	14.32	0.450	0.392	19.28	0.691	0.293
2 Images	Frame Prediction	DMVFN [ 18 ] (CVPR’23)	23.67	0.675	0.093	22.08	0.652	0.125	28.14	0.788	0.122
Event	Reconstruction	E2HQV [ 34 ] (AAAI’24)	14.19	0.358	0.247	13.88	0.358	0.260	17.91	0.487	0.329
Event	Reconstruction	HyperE2VID [ 9 ] (TIP’24)	18.79	0.487	0.273	18.78	0.410	0.278	18.20	0.529	0.331

Appendix

Table S7 : Quantitative comparison for video frame prediction on the real event datasets BS-ERGB [ 47 ] and HS-ERGB [ 48 ] .

Modality	Task	Method	GoPro [ 31 ]
			7 frames			15 frames
			PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
2 Images	Frame Prediction	RaMViD [ 17 ] (TMLR’22)	13.50	0.487	0.457	13.32	0.484	0.560
2 Images	Frame Prediction	DMVFN [ 18 ] (CVPR’23)	14.65	0.442	0.451	13.12	0.413	0.559
Event	Reconstruction	E2HQV [ 34 ] (AAAI’24)	10.81	0.468	0.423	10.86	0.471	0.520
Event	Reconstruction	HyperE2VID [ 9 ] (TIP’24)	9.72	0.430	0.468	9.62	0.430	0.571

Appendix

Table S8 : Quantitative comparison for video frame prediction on the synthetic event dataset GoPro [ 31 ] .

Stage	Small	Medium	Large	Occlusion
VFPSIE Pred. Flow Warp	30.24	27.54	20.95	23.63
RAFT Flow Warp	31.26	29.98	25.03	25.01
VFPSIE Final	30.55	28.47	22.68	25.34
EvFRA	33.87	31.85	27.63	29.17

Appendix

Table S9 : Flow and warping error analysis of VFPSIE [ 59 ] on HS-ERGB using 1,500 samples (PSNR

\uparrow

). For the reference flow, RAFT is applied to the anchor and ground-truth target frames, followed by the same warping operator as VFPSIE.

Task	Method	GFLOPs	# Params (M)	Run Time (s)	Memory Usage (GB)	Notes
Frame Prediction	VFPSIE [ 59 ]	55.3	2.15	0.0147	0.457	Optical flow-based
Frame Prediction	EvFRA(VFP)	10275.0	2029.76	10.3	4.310	Diffusion Model
Frame Interpolation	TimeLens [ 48 ]	2456.4	79.21	0.133	1.73	Optical flow-based
	CBMNet-Large [ 25 ]	3201.1	22.23	1.163	10.161	Optical flow-based
	RE-VDM [ 6 ]	23921.9	2936.70	82.296	9.388	Video Diffusion Model
	EvFRA(VFI)	20953.1	2029.76	21.46	4.310	Diffusion Model

Appendix

Table S10 : Model GFLOPs, parameter count, run time, and memory usage comparison.

Metric	Win Count	Mean Improvement	Wilcoxon $p$	$t$ -test $p$
PSNR $\uparrow$	15 / 15	0.930	0.00051	0.00238
SSIM $\uparrow$	13 / 15	0.014	0.00098	0.00047
LPIPS $\downarrow$	13 / 15	0.011	0.00092	0.00044

Appendix

Table S11 : Sequence-level statistical significance tests on HS-ERGB [ 48 ] . We report win counts, mean improvements, Wilcoxon signed-rank test

p

-values, and paired

t

-test

p

-values over 15 test sequences.

Panel of Figure 24 — Figure S1 : Qualitative examples of ground-truth and predicted frame sequences.

Example	Ground Truth	Prediction
Basketball	0.968	0.994
Football	0.965	0.996

Appendix

Table S12 : Examples where VBench [ 19 ] assigns higher temporal consistency scores to predicted sequences than to ground-truth sequences.

Explore similar work

UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models
Feb 22, 2026Gang Xu, Zhiyu Zhu, Junhui HouVideo Diffusion Models Large Inter-Frame Displacements
Department of Computer Science, City University of Hong Kong, Hong Kong, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · Department of Computer Science, City University of Hong Kong (Dongguan), Dongguan, China
LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models
Jul 9, 2026Cheng-De Fan, Chun-Wei Tuan Mu, Chen-Wei Chang +4Video Diffusion Models Future Video Prediction
National Yang Ming Chiao Tung University, Taiwan
Bridging Event Streams and DiT: Event-Guided Video Frame Interpolation
Aug 11, 2026Guixu Lin, Yuyang Yu, Xiang Ji +6Large Inter-Frame Displacements Diffusion Transformers
The University of Tokyo · South China University of Technology · Wuhan University +1

Feb 22, 2026, cs.CVJ/K move · Enter open · S save

UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models

Gang Xu, Zhiyu Zhu, Junhui Hou

Department of Computer Science, City University of Hong Kong, Hong Kong, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China · Department of Computer Science, City University of Hong Kong (Dongguan), Dongguan, China

Video Diffusion Models Large Inter-Frame Displacements Diffusion Models Diffusion Sampling

Open paper AskPDF arXiv

Event cameras excel at high-speed, low-power, and high-dynamic-range scene perception. However, as they fundamentally record only relative intensity changes rather than absolute intensity, the resulting data streams suffer from a significant loss of spatial information and static texture details. In this paper, we address this limitation by leveraging the generative prior of a pre-trained video diffusion model to reconstruct high-fidelity video frames from sparse event data. Specifically, we first establish a baseline model by directly applying event data as a condition to synthesize videos. Then, based on the physical correlation between the event stream and video frames, we further introduce the event-based inter-frame residual guidance to enhance the accuracy of video frame reconstruction. Furthermore, we extend our method to video frame interpolation and prediction in a zero-shot manner by modulating the reverse diffusion sampling process, thereby creating a unified event-to-frame reconstruction framework. Experimental results on real-world and synthetic datasets demonstrate that our method outperforms previous reconstruction approaches on most quantitative metrics and achieves superior qualitative results, while achieving competitive zero-shot performance compared with dedicated methods explicitly trained for interpolation and prediction tasks. We also refer the reviewers to the video demo contained in the supplementary material for video results. The code will be publicly available at https://github.com/CS-GangXu/UniE2F.

Abstract

Explore similar work

UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models

LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models

Bridging Event Streams and DiT: Event-Guided Video Frame Interpolation