Text-to-image diffusion models have advanced video editing. However, existing video editing methods are limited by their high computational overhead and memory consumption. Furthermore, these approaches often sacrifice visual fidelity, leading to temporal inconsistencies and artifacts such as blurring and mosaic-like patterns. To address these dual challenges and strike a balance between computational efficiency and visual fidelity, we propose Edit-Your-Interest, a lightweight, text-driven, zero-shot video editing method. Edit-Your-Interest introduces a Spatio-Temporal Feature Memory bank (SFM) to cache spatial-attention features from previous frames under a fixed memory budget. The SFM maintains a representative candidate pool, reducing the computational overhead of full-sequence spatio-temporal modeling. To query this memory, we propose Feature Most-Similar Propagation (FMP), a similarity-based retrieval rule that propagates relevant cached tokens to the current frame. Together, SFM and FMP form a bounded-memory retrieval-based propagation design that improves temporal consistency while reducing blurring and mosaic-like artifacts. Finally, we introduce a feature-diversity-based SFM update algorithm to keep the memory representative and non-redundant throughout the video sequence. Furthermore, to enable precise object editing, we leverage cross-attention maps to automatically extract masks for the instances of interest. These masks are seamlessly integrated into the diffusion denoising process, enabling fine-grained control over target objects and allowing Edit-Your-Interest to perform highly accurate edits while robustly preserving the background integrity. Extensive experiments demonstrate that Edit-Your-Interest outperforms state-of-the-art methods in both efficiency and visual fidelity. Project page: https://edit-your-interest.github.io/.
Figures & tables
Figure 1: We propose Edit-Your-Interest, a zero-shot video editing method that supports both low-cost global editing (left) and precise instance local editing (right), while effectively preserving the entire background.
Figure 2: The pipeline of Edit-Your-Interest. (Top) We employ DDIM inversion to obtain the initial latent noise and then denoise the sequence via DDIM sampling. (Bottom left) We construct a SFM to cache frame feature tokens. Subsequently, FMP propagates them to the current frame. (Bottom right) We introduce an Automatic Mask Extraction and Injection Strategy. This in-diffusion injection effectively suppresses boundary artifacts between foreground and background regions.
Figure 3: Examples of our method’s results in instance object editing and global editing. As shown, our approach enables not only precise local object editing but also global editing.
Figure 4: Comparison results between our method and direct replacement using semantic masks. It can be seen that our method mitigates boundary artifacts between foreground and background.
Figure 5: Qualitative Comparisons. The results of our proposed method and other state-of-the-art video editing methods are shown. It can be observed that our method not only successfully edits the target but also effectively preserves the background regions.
Method
Text Alignment
Foreground
Background preservation
Global
Fidelity
CLIP-T
CLIP-F
Warp-err
SSIM
PSNR
CLIP-F
Warp-err
FID
FateZero [ 29 ]
31.05
95.98
5.77
81.91
21.49
94.76
6.80
289.92
DMT [ 47 ]
30.78
98.50
1.21
52.35
16.67
98.39
1.15
214.99
GAV [ 15 ]
27.82
96.45
2.11
68.96
17.51
96.44
4.91
243.28
TokenFlow [ 7 ]
31.32
97.03
1.24
77.65
21.31
98.51
1.38
162.59
STEM [ 21 ]
29.89
97.89
1.48
71.79
17.52
98.48
3.47
170.46
Table 1: Quantitative comparison of automatic metrics. Bold indicates the best result. Background preservation denotes background preservation, Foreground denotes foreground coherence, and Global denotes overall video quality.
Method
8 frames
16 frames
32 frames
GPU
RAM
Runtime
GPU
RAM
Runtime
GPU
RAM
Runtime
FateZero [ 29 ]
18.57
71.13
154
27.34 ∗
144.21 ∗
517 ∗
-
-
-
GAV [ 15 ]
17.87
6.87
93
25.40
6.99
242
29.41
7.27
721
DMT [ 47 ]
19.96
3.06
316
30.99
3.07
521
51.97
3.08
935
TokenFlow [ 7 ]
9.64
2.59
102
11.33
2.78
195
11.42
2.82
403
STEM [ 21 ]
9.78
2.84
52
11.46
2.90
97
11.57
2.93
198
Table 2: Comparison of Runtime and Memory Consumption. GPU denotes the GPU memory usage (in GB), RAM denotes the system memory usage (in GB), and Runtime indicates the time required to edite a video (in seconds). Values marked with an asterisk ( ∗ ) are adapted from [ 46 ]
Figure 6: The example of the scoring interface for the user study. Participants are required to select a specific score from the provided table for each video.
Method
LV (mean)
LV (std)
HFR
PSD
GE
DCT-E
Source video
413.50
32.30
4.54×10−3
6.45
3.74
0.88
TokenFlow
596.53
132.99
7.23×10−3
9.73
3.56
0.89
τ=1.0
377.05
57.95
5.24×10−3
6.75
3.61
0.84
τ=0.5
367.94
57.08
5.15×10−3
6.74
3.58
0.82
τ=0.1
378.89
57.28
5.33×10−3
6.75
3.60
0.85
τ=0.01
379.58
58.06
5.45×10−3
6.72
3.61
0.86
Table 3: Representation-level analysis with temperature sweep. For all metrics except LV mean and GE, values closer to source indicate less artifact-induced distortion.
Figure 7: Comparison between FMP and interpolation-like weighted propagation methods. It can be observed that FMP mitigates blurring and mosaic-like artifacts, thereby enhancing the fidelity of edited videos.
Method
Text
Foreground
Fidelity
Override
CLIP-T ↑
CLIP-F ↑
Warp-err ↓
FID ↓
Keyframe candidates + weighted
31.32
97.03
1.24
162.59
N/A
Keyframe candidates + hard
31.46
97.15
1.22
144.75
0.6832
SFM candidates + softmax ( τ=0.5 )
32.20
98.14
1.18
124.03
0.8050
SFM candidates + hard
32.29
97.92
1.19
120.12
0.8031
Table 4: Quantitative comparison of candidate-pool construction and propagation strategies. Override denotes the fraction of tokens propagated from the candidate pool after passing the confidence threshold.
Figure 8: Visualization of temporal mask overlap strategy. This demonstrates that the temporal mask overlap strategy can effectively generate robust instance masks.
λ
Warp-err
τ
PSNR b
SFM-L
Warp-err
0.7
3.92
0.2
28.17
1
1.57
0.8
2.42
0.3
29.26
3
1.33
0.9
1.19
0.4
29.03
5
1.19
0.95
1.44
0.5
27.04
7
1.21
Table 5: Hyperparameters Analysis: the similarity threshold λ , the mask threshold τ , and the length of SFM (SFM-L).
Method
TC
TA
Quality
FateZero
67.50
73.5
71.75
DMT
72.75
69.75
79.75
GAV
71.75
81.50
69.50
TokenFlow
88.75
88.00
80.95
STEM
87.25
84.00
84.00
OURS
90.25
95.5
90.75
Table 6: The results of the user study. Bold indicates the best result.
Figure 9: The computational overhead of our proposed Edit-Your-Interest when processing videos with varying numbers of frames.
Method
Warp-err ↓
PSNR b ↑
Baseline
10.87
21.79
w/o AMEIS
3.12
21.89
w/o FMP
8.53
26.71
OURS
1.19
29.26
Table 7: Ablation Study Results. PSNR b denotes PSNR computed on the background region.
Figure 10: Our proposed Edit-Your-Interest achieves precise editing in multi-instance scenarios by integrating video instance segmentation model. As demonstrated, this method accurately edites specified target objects while preserving the background in complex scenes containing multiple instances of the same class.