The rapid development of diffusion models (DMs) has significantly advanced image and video applications, making "what you want is what you see" a reality. Among these, video editing has gained substantial attention and seen a swift rise in research activity, necessitating a comprehensive and systematic review of the existing literature. This paper reviews diffusion model-based video editing techniques, including theoretical foundations and practical applications. We begin by overviewing the mathematical formulation and image domain's key methods. Subsequently, we categorize video editing approaches by the inherent connections of their core technologies, depicting evolutionary trajectory. This paper also dives into novel applications, including point-based editing and pose-guided human video editing. Additionally, we present a comprehensive comparison using our newly introduced V2VBench. Building on the progress achieved to date, the paper concludes with ongoing challenges and potential directions for future research.
Figures & tables
Figure 1: This survey focuses on video editing tasks. The edited results are obtained using [ 40 , 57 , 18 , 239 , 63 , 221 ] . We will discuss these methods in Sec. 3 and present more experimental results in Sec. 4 .
Figure 2: The architecture of LDM [ 159 ] . It consists of an encoder E , a decoder D , and a UNet noise predictor ϵθ with cross-attention blocks to incorporate conditioning from domain encoders τθ . Figure adapted from LDM [ 159 ] .
Figure 3: Three representative feature injection methods for image editing. P2P [ 69 ] injects cross-attention maps from the reconstruction branch into the editing branch. PnP [ 188 ] injects self-attention maps. While MasaCtrl [ 16 ] injects self-attention query and key features.
Figure 4: The temporal extension from image to video models involves incorporating 1D temporal convolutions after each 2D spatial convolution (left). Additionally, each 2D spatial attention block is followed by a 1D temporal attention block (right). Figure adapted from Gen-1 [ 47 ] .
Figure 5: Self-attention variations. The query tokens are in red, and the key and value tokens are in blue. W , H , and F denote the input video’s width, height, and frame numbers.
Figure 6: Overview of diffusion-based video editing components.
Figure 7: Taxonomy of diffusion-based video editing methods.
Figure 8: The architecture of VACE [ 88 ] . The VACE [ 88 ] training strategies: (a) Fully Fine-tuning and (b) Context Adapter Tuning. Figure adapted from VACE [ 88 ] .
Figure 9: The t-SNE visualization illustrates the distribution of zT . The samples are obtained from latent inversion on real video frames using a pre-trained T2I diffusion model. The green dots in the background represent samples drawn from i.i.d. Gaussian distribution for reference. Blue dots represent successive frames from the same video, clustered closely together. While frames from different videos (yellow and red dots) are scattered and exhibit no correlation. The plot is adapted from PYoCo [ 55 ] .
Figure 10: Layered Neural Atlases (LNA) [ 94 ] . Original video frames are mapped to flattened 2D atlases by warping models. Edits on the 2D atlases can be propagated to all frames of the video. The plot is adapted from LNA [ 94 ] .
Figure 11: Some source video examples of V2VBench.
Method
Frames Quality
Video Quality
Semantic Consistency
Object Consistency
Frames Text Alignment
Video Text Alignment
Motion Alignment
Frames Pick Score
Network and Training
Tune-A-Video [ 210 ] †
5.016 ( ± 0.110)
0.449 ( ± 0.071)
0.934 ( ± 0.009)
0.916 ( ± 0.013)
27.774 ( ± 0.824)
0.257 ( ± 0.012)
-5.463 ( ± 0.939)
20.755 ( ± 0.157)
SimDA [ 217 ] †
4.987 ( ± 0.104)
0.480 ( ± 0.080)
0.940 ( ± 0.007)
0.930 ( ± 0.010)
26.792 ( ± 0.818)
0.249 ( ± 0.013)
-4.661 ( ± 0.588)
20.541 ( ± 0.175)
VidToMe [ 111 ]
5.294 ( ± 0.179)
0.597 ( ± 0.045)
0.956 ( ± 0.006)
0.950 ( ± 0.007)
27.550 ( ± 0.922)
0.252 ( ± 0.012)
-2.463 ( ± 0.529)
20.811 ( ± 0.202)
VideoComposer [ 201 ] ‡
4.539 ( ± 0.191)
0.379 ( ± 0.068)
0.919 ( ± 0.013)
0.913 ( ± 0.016)
28.399 ( ± 1.919)
0.265 ( ± 0.032)
-7.765 ( ± 0.589)
20.267 ( ± 0.382)
MotionDirector [ 239 ] † ‡
5.128 ( ± 0.159)
0.556 ( ± 0.063)
0.952 ( ± 0.010)
0.954 ( ± 0.009)
26.365 ( ± 1.398)
0.264 ( ± 0.020)
-3.485 ( ± 0.819)
20.846 ( ± 0.294)
Attention Injection
Video-P2P [ 120 ] †
4.900 ( ± 0.133)
0.382 ( ± 0.069)
0.944 ( ± 0.007)
0.927 ( ± 0.010)
23.661 ( ± 0.827)
0.190 ( ± 0.011)
-5.769 ( ± 0.684)
19.735 ( ± 0.148)
Table 1: V2VBench evaluation results. This table compares 16 diffusion-based text-guided video editing methods across eight performance metrics; higher scores indicate superior performance. Symbols denote specific requirements: † for test-time tuning on the source video. ‡ for video large-scaled pretraining. Methods are categorized by their main technological dependencies. The best results within each category are underlined , and the globally best metrics are highlighted in bold font . Values in parentheses represent the standard deviation across five independent runs with different random seeds.
Figure 12: Statistics of V2VBench videos and prompts.
Figure 13: Memory usage and running time comparison of 16 diffusion model-based video editing methods. The color represents the category, and the radius indicates peak memory usage.
Figure 14: Further comparisons on V2VBench. Each color represents the metrics for different editing types. Please refer to the appendix for additional details and normalization methods.
Method
Pretrained
Frames Quality
Video Quality
Semantic Consistency
Object Consistency
Frames Text Alignment
Video Text Alignment
Motion Alignment
Frames Pick Score
Tune-A-Video [ 210 ]
SD v1-5
5.016 ( ± 0.110)
0.449 ( ± 0.071)
0.934 ( ± 0.009)
0.916 ( ± 0.013)
27.774 ( ± 0.824)
0.257 ( ± 0.012)
-5.463 ( ± 0.939)
20.755 ( ± 0.157)
SD v2-1
5.020 ( ± 0.098)
0.486 ( ± 0.051)
0.935 ( ± 0.009)
0.920 ( ± 0.011)
28.186 ( ± 0.705)
0.261 ( ± 0.010)
-5.429 ( ± 0.803)
20.739 ( ± 0.161)
Vid2Vid-Zero [ 199 ]
SD v1-5
5.089 ( ± 0.048)
0.473 ( ± 0.069)
0.919 ( ± 0.006)
0.912 ( ± 0.007)
29.012 ( ± 0.454)
0.274 ( ± 0.006)
-4.047 ( ± 0.361)
20.998 ( ± 0.101)
SD v2-1
5.107 ( ± 0.012)
0.510 ( ± 0.008)
0.920 ( ± 0.004)
0.910 ( ± 0.002)
29.008 ( ± 0.084)
0.277 ( ± 0.001)
-4.368 ( ± 0.137)
21.006 ( ± 0.011)
Fate-Zero [ 151 ]
SD v1-5
5.033 ( ± 0.021)
0.541 ( ± 0.066)
0.951 ( ± 0.003)
0.952 ( ± 0.002)
25.073 ( ± 0.154)
0.225 ( ± 0.002)
-1.425 ( ± 0.062)
20.704 ( ± 0.023)
SD v2-1
5.030 ( ± 0.016)
0.574 ( ± 0.008)
0.951 ( ± 0.002)
0.951 ( ± 0.002)
25.115 ( ± 0.122)
0.225 ( ± 0.001)
-1.411 ( ± 0.060)
20.718 ( ± 0.016)
Table 2: Quantitative evaluation on V2VBench using Stable Diffusion (SD) [ 159 ] v1-5 and v2-1 backbones. While adopting SD v2-1 generally yields marginal performance gains across most scenarios, these improvements are largely statistically insignificant. Overall, method performance remains robust and consistent across both backbone architectures.
Figure 15: Chronological overview of recent foundational video diffusion models and tools.
Figure 16: Editing spillover failure case : The prompt intends to change the car’s color from white to orange, but the surrounding reed is unexpectedly modified, resulting in artifacts.
Text-guided video editing with diffusion models is impractically slow, hindered by costly multi-step sampling and inversion. We present OSVE, the first framework to successfully adapt one-step Text-to-Image (T2I) models for high-quality video editing, addressing the core challenges of inversion, editability, and temporal consistency. To bypass slow iterative inversion, we train a learnable encoder that predicts the initial noise for each frame in a single forward pass. This encoder is trained with a novel Structure-Aware Editing (SAE) loss on a curated dataset of structurally-aligned image pairs, teaching it to preserve the source video's geometry during edits. For temporal coherence, we introduce Unified-Frame Editing (UFE), a technique that concatenates frame latents to facilitate cross-frame attention in a single generation step. Furthermore, for long videos, a sliding-window strategy with an anchor frame maintains global consistency. Our extensive experiments demonstrate that OSVE achieves editing quality comparable or superior to state-of-the-art multi-step methods, while operating approximately 155--171 times faster. This breakthrough paves the way for practical, real-time video editing applications. Code is available at https://github.com/KU-VGI/OSVE.
Video diffusion models have enabled remarkable progress in video generation and editing. However, content preservation remains a core challenge: existing methods regenerate every pixel and often alter elements that should remain unchanged, such as characters or background scenes. We introduce Vera, a layered diffusion framework for content-preserving video editing. Instead of regenerating the entire video, Vera generates an edit layer along with an alpha matte for compositing with the source video, separating creative editing from content preservation by design. To encourage coherent composition with the source video, we extend the text-to-video DiT into a Mixture-of-Transformers (MoT) architecture, with separate DiTs for each layer that interact through joint self-attention. To support the training of Vera, we further construct a high-quality layered dataset with accurate alpha mattes, diverse scenes and dynamics, and visual effects. Across our quantitative benchmark and human preference study, Vera outperforms leading open-source video editing models in content preservation while remaining competitive in edit quality, using 486K frames of layered training data.
Hongkai Zheng, Ta-Ying Cheng, Benjamin Klein +2
California Institute of Technology · Netflix, Inc.
Video editing aims to modify input videos according to user intent. Recently, end-to-end training methods have garnered widespread attention, constructing paired video editing data through video generation or editing models. However, compared to image editing, the high annotation costs of video data severely constrain the scale, quality, and task diversity of video editing datasets when relying on video generative models or manual annotation. To bridge this gap, we propose LIVE, a joint training framework that leverages large-scale, high-quality image editing data alongside video datasets to bolster editing capabilities. To mitigate the domain discrepancy between static images and dynamic videos, we introduce a frame-wise token noise strategy, which treats the latents of specific frames as reasoning tokens, leveraging large pretrained video generative models to create plausible temporal transformations. Moreover, through cleaning public datasets and constructing an automated data pipeline, we adopt a two-stage training strategy to anneal video editing capabilities. Furthermore, we curate a comprehensive evaluation benchmark encompassing over 60 challenging tasks that are prevalent in image editing but scarce in existing video datasets. Extensive comparative and ablation experiments demonstrate that our method achieves state-of-the-art performance. The source code will be publicly available.
Weicheng Wang, Zhicheng Zhang, Zhongqi Zhang +6
Nankai University · Kuaishou Technology · Pengcheng Laboratory +1