High-fidelity generative video editing has seen significant quality improvements by leveraging pre-trained video foundation models. However, their computational cost is a major bottleneck, as they are often designed to inefficiently process the full video context regardless of the inpainting mask's size, even for sparse, localized edits. In this paper, we introduce EditCtrl, an efficient video inpainting control framework that focuses computation only where it is needed. Our approach features a novel local video context module that operates solely on masked tokens, yielding a computational cost proportional to the edit size. This local-first generation is then guided by a lightweight temporal global context embedder that ensures video-wide context consistency with minimal overhead. Not only is EditCtrl 10 times more compute efficient than state-of-the-art generative editing methods, it even improves editing quality compared to methods designed with full-attention. Finally, we showcase how EditCtrl unlocks new capabilities, including multi-region editing with text prompts and autoregressive content propagation.
Figures & tables
Figure 1: EditCtrl: A Real-time Generative Video Editing Pipeline. EditCtrl supports complex, prompt-guided edits on 4K videos, simultaneously handling an arbitrary number of user-defined masks (Top). To maintain real-time performance, our inference pipeline dynamically allocates compute proportional to the edit mask size (Middle). EditCtrl also intelligently propagates object edits from initial frames into the future (after the orange line), ensuring high temporal and object consistency in the resulting edit (Bottom).
Figure 2: EditCtrl Video Diffusion Framework Overview. EditCtrl edits a source video given a target edit mask. Foreground content is masked out, giving the background video that is also down-sampled to a constant resolution regardless of the original resolution. The compact global context of the down-sampled background video and the local context at the mask edit region are then encoded. These are given to trainable local and global adapters inside a pretrained text-to-video diffusion model that denoises tokens zt only in the masked edit region given a text prompt. After diffusion, the tokens are scattered into the masked edit region in the encoded source video latents. Our method shows a proportional speedup with respect to the target mask area ratio.
Figure 3: EditCtrl: Local and Global Control Modules. Given the source video Vsrc and target edit masks Vm , we extract the background content Vb and encode it with a video VAE encoder E . This is then concatenated channel-wise with the down-sampled masks to give the control context C . Tokens in C outside the down-sampled edit mask region are then masked out, giving the local context tokens Clocal which go to the local encoder module cϕ , whose outputs are added to selected transformer layers. The global embedder Gψ receives the query feature tokens and global context tokens produced from the down-sampled background content Vb↓ and modulates the noisy cross-attended features.
Figure 4: “Change the flower color to blue.”
Method
#Par.
PFLOPS ↓
Masked Region Preservation
Text Alignment
Temp. Coherence
Throughput
PSNR ↑
SSIM ↑
LPIPS ×102↓
MSE ×102↓
MAE ×102↓
CLIP ↑
CLIP (M) ↑
CLIP Sim ↓
FPS ↑
ReVideo ( Mou et al., 2024 )
1.5B
193.39
15.52
0.49
27.68
3.49
11.14
9.34
20.01
0.42
0.11
VideoPainter ( Bian et al., 2025 )
5B
817.81
22.63
0.91
7.65
1.02
2.90
8.67
20.20
0.18
0.12
VACE ( Jiang et al., 2025 )
1.3B
76.31
23.84
0.91
5.44
0.92
2.78
9.76
21.51
0.13
0.66
VACE ( Jiang et al., 2025 )
14B
589.19
24.02
0.92
5.13
0.84
2.68
9.85
21.54
0.13
0.10
EditCtrl
1.5B
17.42
24.16
0.92
5.54
0.99
3.01
9.58
21.70
0.15
4.67
Table 1: Video Editing Comparison on VPBench-Edit. EditCtrl outperforms editing baselines as well as the full attention base model in edited video quality, background preservation, and content alignment with the text prompt, all at a much higher throughput and lower computational cost (PFLOPS). In each column, the best , second best , and third best results are marked.
Figure 7: Video Inpainting Comparison. Even with full-attention, baseline methods struggle to inpaint content that is coherent and visually appealing, while our method successfully generates high fidelity content that aligns with the scene using much less compute.
Masked Region Preservation
Text Alignment
Temp. Coherence
Throughput
Method
#Par.
PSNR ↑
SSIM ↑
LPIPS ×102↓
MSE ×102↓
MAE ×102↓
CLIP ↑
CLIP (M) ↑
CLIP Sim ↓
FPS ↑
ProPainter ( Zhou et al., 2023 )
50M
20.97
0.87
9.89
1.24
3.56
7.31
17.18
0.44
5.34
VideoPainter ( Bian et al., 2025 )
5B
23.32
0.89
6.85
0.82
2.62
8.66
21.49
0.15
0.12
VACE ( Jiang et al., 2025 )
1.3B
22.62
0.85
9.30
1.25
4.43
9.77
21.55
0.17
0.66
VACE ( Jiang et al., 2025 )
14B
23.03
0.88
7.65
0.98
3.40
9.27
22.18
0.14
0.10
EditCtrl
1.3B
23.17
0.86
8.52
1.29
3.91
9.81
21.86
0.17
5.24
Table 2: Video Inpainting Comparison. EditCtrl matches or outperforms inpainting baselines for the VPBench-Inp (top) and DAVIS (bottom) test sets while significantly increasing inpainting efficiency.
Figure 8: Local and Global Adapter Ablation. Removing adapter components harms video editing quality, but together they let EditCtrl perform comparably to a method operating with full-attention.
Masked Region Pres.
Text Alignment
Temp. Coherence
Throughput
Method
PSNR ↑
SSIM ↑
LPIPS ×102↓
CLIP ↑
CLIP (M) ↑
CLIP Sim ↓
FPS ↑
VACE
23.84
0.91
5.44
9.76
21.51
0.13
0.10
Ours (Naive)
23.24
0.86
6.96
8.83
20.49
0.19
4.90
Ours (No Gψ )
23.80
0.90
5.74
9.41
21.28
0.15
4.90
Ours
24.16
0.92
5.54
9.58
21.70
0.15
4.67
Table 3: Quantitative Local and Global Adapter Ablation Comparison. We observe a clear improvement in quality at a minimal reduction in efficiency with the local and global control modules. With both adapters, EditCtrl greatly increases throughput and even exceeds the full-attention model in edit quality.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Editing complex timelapse 21:9 ultrawide footage. EditCtrl can also be used with additional conditioning LoRAs ( DiffSynth-Studio, 2025 ) .
Figure 10: Failure Modes for EditCtrl on High Motion (a) & Inaccurate Masks (b).
Figure 11: Content Propagation for Augmented Reality. EditCtrl is particularly suitable for deployment in augmented reality applications given its low latency and ability to propagate content to match the user’s movement.
Building a capable video editor remains significantly harder than a video generator: editing requires (source, instruction, edited) triplets that are prohibitively expensive to annotate and difficult to synthesize at scale, whereas image editing has already reached maturity with millions of such pairs readily available. In this work, we introduce VINCIE-NExT, a unified framework that transfers editing capability from images to videos through in-context visual demonstrations, alleviating the need for large-scale paired video editing data. VINCIE-NExT decomposes video editing into a structured chain of composable sub-tasks (Video -> Image -> Image -> Video), routing editing intent through the image domain and enabling scalable joint training from heterogeneous image and video corpora under a unified diffusion objective. An image editing pair, synthesized by the model or supplied by the user, is prepended as an in-context visual demonstration that serves as a spatial appearance blueprint for every output frame. To ground appearance edits across the interleaved context, we introduce a novel position encoding that links image demonstrations and video frames in a shared spatial coordinate system, enabling pixel-faithful propagation of appearance changes to every output frame. Chain-of-Editing further provides principled test-time scaling: by executing the sub-task chain as progressive diffusion stages, editing quality can be improved by investing additional compute without retraining. Comprehensive experiments on OpenVE-Bench demonstrate the state-of-the-art performance across diverse editing categories, with ablations confirming the effectiveness of each component.
Leigang Qu, Feng Cheng, Ziyan Yang +7
National University of Singapore · ByteDance Seed · University of Science and Technology of China
Current video editing models often rely on expensive paired video data, which limits their practical scalability. In essence, most video editing tasks can be formulated as a decoupled spatiotemporal process, where the temporal dynamics of the pretrained model are preserved while spatial content is selectively and precisely modified. Based on this insight, we propose ImVideoEdit, an efficient framework that learns video editing capabilities entirely from image pairs. By freezing the pre-trained 3D attention modules and treating images as single-frame videos, we decouple the 2D spatial learning process to help preserve the original temporal dynamics. The core of our approach is a Predict-Update Spatial Difference Attention module that progressively extracts and injects spatial differences. Rather than relying on rigid external masks, we incorporate a Text-Guided Dynamic Semantic Gating mechanism for adaptive and implicit text-driven modifications. Despite training on only 13K image pairs for 5 epochs with exceptionally low computational overhead, ImVideoEdit achieves editing fidelity and temporal consistency comparable to larger models trained on extensive video datasets.
Real-time video editing requires low-latency causal generation with bounded computational resources while preserving source fidelity and long-term temporal consistency. We present JoyAI-Video-Edit, a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing without access to future frames or a predefined video duration. Our method combines chunk-wise autoregressive adaptation, Source-Anchored Distribution Matching Distillation (SA-DMD), and Long-Horizon Autoregressive Distillation to reduce train--inference mismatch, preserve source fidelity during two-step generation, and mitigate accumulated temporal drift. Extensive automatic and human evaluations show that JoyAI-Video-Edit substantially outperforms existing streaming editors and remains competitive with strong offline systems on both short and long videos. The complete system achieves end-to-end 720p video editing at approximately 30 FPS on a single Nvidia B200 GPU. Code is available at https://github.com/jd-opensource/JoyAI-Video-Edit.