FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing
Organizations: The University of Hong Kong · International Digital Economy Academy (IDEA)
Abstract
Image-driven video editing aims to propagate edit contents from the modified first frame to the remaining frames. Existing methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame. Generally, a popular choice for maintaining motion and layout from the source video is intervening in the denoising process by injecting attention during reconstruction. However, such injection often leads to unsatisfactory results, where excessive injection leads to conflicting semantics with the source video while insufficient injection brings limited source representation. Recognizing this, we propose an Editing-awaRE (REE) injection method to modulate the injection intensity of each token. Specifically, we first compute the pixel difference between the source and edited first frame to form a corresponding editing mask. Next, we track the editing area throughout the entire video by using optical flow to warp the first-frame mask. Then, editing-aware feature injection intensity for each token is generated accordingly, where injection is not conducted in editing areas. Building upon REE injection, we further propose a zero-shot image-driven video editing framework with recent-emerging rectified-Flow models, dubbed FREE-Edit. Without fine-tuning or training, our FREE-Edit demonstrates effectiveness in various image-driven video editing scenarios, showing its capability to produce higher-quality outputs compared with existing techniques. Project page: https://free-edit.github.io/page/.
Figures & tables
| Settings | CLIP score | Warp Error |
| 0.920 | 2.106 | |
| 0.937 | 0.595 | |
| 0.877 | 0.764 |
| Settings | Whole-Image | Non-editing Area | Speed (min) | |||||
| CLIP score | Warp Error | CLIP score + | Warp Error + | SSIM + | PSNR + | LPIPS + | ||
| AnyV2V [ 14 ] | 0.908 | 2.190 | 0.911 | 1.872 | 0.684 | 22.08 | 0.218 | 3.61 |
| I2VEdit [ 21 ] | 0.909 | 2.886 | 0.914 | 1.794 | 0.745 | 22.57 | 0.255 | 127 |
| VideoShop [ 6 ] | 0.822 | 7.415 | 0.854 | 5.298 | 0.568 | 18.08 | 0.417 | 2.45 |
| Go-with-the-Flow [ 4 ] | 0.930 | 2.152 | 0.935 | 1.765 | 0.737 | 22.66 | 0.306 | 3.42 |
| FREE-Edit w/ REE injection | 0.937 | 0.595 | 0.941 | 0.471 | 0.811 | 27.05 | 0.161 | 1.53 |
| Ours vs | Editing Consistency | Overall Quality |
| AnyV2V | 79.6% | 81.9% |
| I2VEdit | 78.5% | 74.6% |
| VideoShop | 85.2% | 81.4% |
| Go-with-the-Flow | 72.5% | 68.4% |
| FREE-Edit w/o injection | 82.2% | 71.3% |
| FREE-Edit w/ vanilla injection | 77.8% | 69.8% |
| Method | CLIP-Image | Warp Error |
| TokenFlow [ 7 ] | - | 1.777 |
| VidToMe [ 18 ] | - | 1.570 |
| RF-Edit [ 33 ] | - | 1.729 |
| Ours | 0.937 | 0.595 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Video Editing
Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.
See all 178 papers