Diffusion Model-Based Video Editing: A Survey

Wenhao Sun; Rong-Cheng Tu; Jingyi Liao; Dacheng Tao

cs.CVJun 26, 2024

Diffusion Model-Based Video Editing: A Survey

Authors: Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, Dacheng Tao

Organizations: College of Computing and Data Science, Nanyang Technological University.

Abstract

The rapid development of diffusion models (DMs) has significantly advanced image and video applications, making "what you want is what you see" a reality. Among these, video editing has gained substantial attention and seen a swift rise in research activity, necessitating a comprehensive and systematic review of the existing literature. This paper reviews diffusion model-based video editing techniques, including theoretical foundations and practical applications. We begin by overviewing the mathematical formulation and image domain's key methods. Subsequently, we categorize video editing approaches by the inherent connections of their core technologies, depicting evolutionary trajectory. This paper also dives into novel applications, including point-based editing and pose-guided human video editing. Additionally, we present a comprehensive comparison using our newly introduced V2VBench. Building on the progress achieved to date, the paper concludes with ongoing challenges and potential directions for future research.

Figures & tables

Figure 1: This survey focuses on video editing tasks. The edited results are obtained using [ 40 , 57 , 18 , 239 , 63 , 221 ] . We will discuss these methods in Sec. 3 and present more experimental results in Sec. 4 .

Figure 2: The architecture of LDM [ 159 ] . It consists of an encoder

\mathcal{E}

, a decoder

\mathcal{D}

, and a UNet noise predictor

\epsilon_{\theta}

with cross-attention blocks to incorporate conditioning from domain encoders

\tau_{\theta}

. Figure adapted from LDM [ 159 ] .

Figure 3: Three representative feature injection methods for image editing. P2P [ 69 ] injects cross-attention maps from the reconstruction branch into the editing branch. PnP [ 188 ] injects self-attention maps. While MasaCtrl [ 16 ] injects self-attention query and key features.

Figure 4: The temporal extension from image to video models involves incorporating 1D temporal convolutions after each 2D spatial convolution (left). Additionally, each 2D spatial attention block is followed by a 1D temporal attention block (right). Figure adapted from Gen-1 [ 47 ] .

Figure 5: Self-attention variations. The query tokens are in red, and the key and value tokens are in blue.

W

H

, and

F

denote the input video’s width, height, and frame numbers.

Figure 6: Overview of diffusion-based video editing components.

Figure 7: Taxonomy of diffusion-based video editing methods.

Figure 8: The architecture of VACE [ 88 ] . The VACE [ 88 ] training strategies: (a) Fully Fine-tuning and (b) Context Adapter Tuning. Figure adapted from VACE [ 88 ] .

Figure 9: The t-SNE visualization illustrates the distribution of $\mathbf{z}_{T}$ . The samples are obtained from latent inversion on real video frames using a pre-trained T2I diffusion model. The green dots in the background represent samples drawn from i.i.d. Gaussian distribution for reference. Blue dots represent successive frames from the same video, clustered closely together. While frames from different videos (yellow and red dots) are scattered and exhibit no correlation. The plot is adapted from PYoCo [ 55 ] .

Figure 10: Layered Neural Atlases (LNA) [ 94 ] . Original video frames are mapped to flattened 2D atlases by warping models. Edits on the 2D atlases can be propagated to all frames of the video. The plot is adapted from LNA [ 94 ] .

Figure 11: Some source video examples of V2VBench.

	Method	Frames Quality	Video Quality	Semantic Consistency	Object Consistency	Frames Text Alignment	Video Text Alignment	Motion Alignment	Frames Pick Score
Network and Training	Tune-A-Video [ 210 ] †	5.016 ( $\pm$ 0.110)	0.449 ( $\pm$ 0.071)	0.934 ( $\pm$ 0.009)	0.916 ( $\pm$ 0.013)	27.774 ( $\pm$ 0.824)	0.257 ( $\pm$ 0.012)	-5.463 ( $\pm$ 0.939)	20.755 ( $\pm$ 0.157)
	SimDA [ 217 ] †	4.987 ( $\pm$ 0.104)	0.480 ( $\pm$ 0.080)	0.940 ( $\pm$ 0.007)	0.930 ( $\pm$ 0.010)	26.792 ( $\pm$ 0.818)	0.249 ( $\pm$ 0.013)	-4.661 ( $\pm$ 0.588)	20.541 ( $\pm$ 0.175)
	VidToMe [ 111 ]	5.294 ( $\pm$ 0.179)	0.597 ( $\pm$ 0.045)	0.956 ( $\pm$ 0.006)	0.950 ( $\pm$ 0.007)	27.550 ( $\pm$ 0.922)	0.252 ( $\pm$ 0.012)	-2.463 ( $\pm$ 0.529)	20.811 ( $\pm$ 0.202)
	VideoComposer [ 201 ] ‡	4.539 ( $\pm$ 0.191)	0.379 ( $\pm$ 0.068)	0.919 ( $\pm$ 0.013)	0.913 ( $\pm$ 0.016)	28.399 ( $\pm$ 1.919)	0.265 ( $\pm$ 0.032)	-7.765 ( $\pm$ 0.589)	20.267 ( $\pm$ 0.382)
	MotionDirector [ 239 ] † ‡	5.128 ( $\pm$ 0.159)	0.556 ( $\pm$ 0.063)	0.952 ( $\pm$ 0.010)	0.954 ( $\pm$ 0.009)	26.365 ( $\pm$ 1.398)	0.264 ( $\pm$ 0.020)	-3.485 ( $\pm$ 0.819)	20.846 ( $\pm$ 0.294)
Attention Injection	Video-P2P [ 120 ] †	4.900 ( $\pm$ 0.133)	0.382 ( $\pm$ 0.069)	0.944 ( $\pm$ 0.007)	0.927 ( $\pm$ 0.010)	23.661 ( $\pm$ 0.827)	0.190 ( $\pm$ 0.011)	-5.769 ( $\pm$ 0.684)	19.735 ( $\pm$ 0.148)

Table 1: V2VBench evaluation results. This table compares 16 diffusion-based text-guided video editing methods across eight performance metrics; higher scores indicate superior performance. Symbols denote specific requirements:

\dagger

for test-time tuning on the source video.

\ddagger

for video large-scaled pretraining. Methods are categorized by their main technological dependencies. The best results within each category are underlined , and the globally best metrics are highlighted in bold font . Values in parentheses represent the standard deviation across five independent runs with different random seeds.

Panel of Figure 12 — Figure 12: Statistics of V2VBench videos and prompts.

Figure 13: Memory usage and running time comparison of 16 diffusion model-based video editing methods. The color represents the category, and the radius indicates peak memory usage.

Figure 14: Further comparisons on V2VBench. Each color represents the metrics for different editing types. Please refer to the appendix for additional details and normalization methods.

Method	Pretrained	Frames Quality	Video Quality	Semantic Consistency	Object Consistency	Frames Text Alignment	Video Text Alignment	Motion Alignment	Frames Pick Score
Tune-A-Video [ 210 ]	SD v1-5	5.016 ( $\pm$ 0.110)	0.449 ( $\pm$ 0.071)	0.934 ( $\pm$ 0.009)	0.916 ( $\pm$ 0.013)	27.774 ( $\pm$ 0.824)	0.257 ( $\pm$ 0.012)	-5.463 ( $\pm$ 0.939)	20.755 ( $\pm$ 0.157)
Tune-A-Video [ 210 ]	SD v2-1	5.020 ( $\pm$ 0.098)	0.486 ( $\pm$ 0.051)	0.935 ( $\pm$ 0.009)	0.920 ( $\pm$ 0.011)	28.186 ( $\pm$ 0.705)	0.261 ( $\pm$ 0.010)	-5.429 ( $\pm$ 0.803)	20.739 ( $\pm$ 0.161)
Vid2Vid-Zero [ 199 ]	SD v1-5	5.089 ( $\pm$ 0.048)	0.473 ( $\pm$ 0.069)	0.919 ( $\pm$ 0.006)	0.912 ( $\pm$ 0.007)	29.012 ( $\pm$ 0.454)	0.274 ( $\pm$ 0.006)	-4.047 ( $\pm$ 0.361)	20.998 ( $\pm$ 0.101)
Vid2Vid-Zero [ 199 ]	SD v2-1	5.107 ( $\pm$ 0.012)	0.510 ( $\pm$ 0.008)	0.920 ( $\pm$ 0.004)	0.910 ( $\pm$ 0.002)	29.008 ( $\pm$ 0.084)	0.277 ( $\pm$ 0.001)	-4.368 ( $\pm$ 0.137)	21.006 ( $\pm$ 0.011)
Fate-Zero [ 151 ]	SD v1-5	5.033 ( $\pm$ 0.021)	0.541 ( $\pm$ 0.066)	0.951 ( $\pm$ 0.003)	0.952 ( $\pm$ 0.002)	25.073 ( $\pm$ 0.154)	0.225 ( $\pm$ 0.002)	-1.425 ( $\pm$ 0.062)	20.704 ( $\pm$ 0.023)
Fate-Zero [ 151 ]	SD v2-1	5.030 ( $\pm$ 0.016)	0.574 ( $\pm$ 0.008)	0.951 ( $\pm$ 0.002)	0.951 ( $\pm$ 0.002)	25.115 ( $\pm$ 0.122)	0.225 ( $\pm$ 0.001)	-1.411 ( $\pm$ 0.060)	20.718 ( $\pm$ 0.016)

Table 2: Quantitative evaluation on V2VBench using Stable Diffusion (SD) [ 159 ] v1-5 and v2-1 backbones. While adopting SD v2-1 generally yields marginal performance gains across most scenarios, these improvements are largely statistically insignificant. Overall, method performance remains robust and consistent across both backbone architectures.

Figure 15: Chronological overview of recent foundational video diffusion models and tools.

Figure 16: Editing spillover failure case : The prompt intends to change the car’s color from white to orange, but the surrounding reed is unexpectedly modified, resulting in artifacts.

Explore similar work

OSVE: One Step Video Editing with One Step Diffusion Models
Jul 22, 2026Habin Lim, Gyeong-Moon ParkText-To-Video Diffusion Models Video Editing
Korea University, Seoul, Republic of Korea
Vera: A Layered Diffusion Model for Content-Preserving Video Editing
Jun 22, 2026Hongkai Zheng, Ta-Ying Cheng, Benjamin Klein +2Text-To-Video Diffusion Models Video Editing
California Institute of Technology · Netflix, Inc.
LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing
Apr 18, 2026Weicheng Wang, Zhicheng Zhang, Zhongqi Zhang +6Video Editing Video Dataset
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1

Jul 22, 2026, cs.CVJ/K move · Enter open · S save

OSVE: One Step Video Editing with One Step Diffusion Models

Habin Lim, Gyeong-Moon Park

Korea University, Seoul, Republic of Korea

Text-To-Video Diffusion Models Video Editing Diffusion-Based Image Editing Diffusion Models

Open paper AskPDF arXiv

Text-guided video editing with diffusion models is impractically slow, hindered by costly multi-step sampling and inversion. We present OSVE, the first framework to successfully adapt one-step Text-to-Image (T2I) models for high-quality video editing, addressing the core challenges of inversion, editability, and temporal consistency. To bypass slow iterative inversion, we train a learnable encoder that predicts the initial noise for each frame in a single forward pass. This encoder is trained with a novel Structure-Aware Editing (SAE) loss on a curated dataset of structurally-aligned image pairs, teaching it to preserve the source video's geometry during edits. For temporal coherence, we introduce Unified-Frame Editing (UFE), a technique that concatenates frame latents to facilitate cross-frame attention in a single generation step. Furthermore, for long videos, a sliding-window strategy with an anchor frame maintains global consistency. Our extensive experiments demonstrate that OSVE achieves editing quality comparable or superior to state-of-the-art multi-step methods, while operating approximately 155--171 times faster. This breakthrough paves the way for practical, real-time video editing applications. Code is available at https://github.com/KU-VGI/OSVE.

Abstract

Explore similar work

OSVE: One Step Video Editing with One Step Diffusion Models

Vera: A Layered Diffusion Model for Content-Preserving Video Editing

LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing