cs.CVApr 9, 2026
SaveInsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptation
Organizations: The Hong Kong University of Science and Technology · Celia Research HK · City University of Hong Kong
Abstract
Instruction-based video editing is a natural way to control video content with text, but adapting a video generation model into an editor usually appears data-hungry. At the same time, high-quality video editing data remains scarce. In this paper, we show that a video generation backbone can become a strong video editor without large scale video editing data. We present InsEdit, an instruction-based editing model built on HunyuanVideo-1.5. InsEdit combines a visual editing architecture with a video data pipeline based on Mutual Context Attention (MCA), which creates aligned video pairs where edits can begin in the middle of a clip rather than only from the first frame. With only O(100)K video editing data, InsEdit achieves state-of-the-art results among open-source methods on our video instruction editing benchmarks. In addition, because our training recipe also includes image editing data, the final model supports image editing without any modification.
Figures & tables
Figure 1. Representative visual editing results of InsEdit. A teaser figure for InsEdit showing representative visual editing results.
Figure 2. Training efficiency of adapting video generation models into editors. InsEdit reaches strong video editing performance with only K video editing data. A single-column figure showing that InsEdit attains strong video editing performance with about one hundred thousand video editing samples.
Figure 3. Overview of the InsEdit architecture. The semantic module converts the source input and editing instruction into edit-aware tokens, while the vision module denoises the target conditioned on both source features and semantic tokens. A single-column pipeline figure for InsEdit. Built on HunyuanVideo-1.5, the framework contains a reference-aware semantic module that combines instruction features with source-aware cues to produce edit-aware conditioning tokens, and a vision module that combines source-conditioned visual features with noisy target latents inside shared MMDiT blocks to generate the edited target image or video.
Figure 4. Automatic pipeline for video editing data construction. Starting from keyword prompts, the pipeline synthesizes aligned source-target video pairs, converts them into multiple instruction forms, and filters them with VQA-based alignment checking. A pipeline diagram for automatic video editing data construction. The process starts from keyword prompts, expands them into source and target prompts, synthesizes aligned source-target video pairs with Mutual Context Attention, converts each pair into multiple instruction formats, and filters the results with VQA-based alignment verification.
Figure 5. Mutual Context Attention (MCA) for paired video generation. Two denoising branches interact within a shared DiT backbone and exchange attention context so that the source and target videos remain aligned while still supporting diverse editing operations. A diagram of the Mutual Context Attention mechanism for paired video generation. Two denoising branches for the source and target videos run inside a shared DiT backbone, and cross-branch interactions let them exchange attention context to keep the generated pair aligned while enabling different editing operations.
Figure 6. Examples from the MCA-based data construction pipeline. Representative video pairs illustrate temporally localized edits synthesized by our automatic pipeline, where the edited content can emerge only in intermediate video segments instead of being propagated from the first frame. Representative source-target pairs produced by the MCA-based data construction pipeline, highlighting temporally localized edits that begin from intermediate segments rather than the first frame.
| Method | OpenVE-Bench | InsEdit-Bench | Latency (min) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Overall | Local Add | Local Remove | Local Change | Subtitle Edit | Creative Edit | Overall | IC | TVQ | URP | ||
| VACE-14B ( Jiang et al., 2025 ) | 3.01 | 1.76 | 3.99 | 2.47 | 4.41 | 2.17 | 3.08 | 2.39 | 2.64 | 4.21 | 9.08 |
| OmniVideo ( Tan et al., 2025 ) | 3.66 | 2.80 | 4.52 | 3.75 | 4.95 | 1.13 | 4.13 | 4.04 | 4.04 | 4.32 | 20.58 |
| InsViE ( Wu et al., 2025c ) | 3.25 | 2.25 | 3.56 | 2.82 | 4.77 | 3.36 | 2.76 | 2.19 | 2.29 | 3.80 | 1.06 |
| Lucy-Edit ( Team, 2025a ) | 3.77 | 3.92 | 3.95 | 3.93 | 4.23 | 4.19 | 3.64 | 3.24 | 3.38 | 4.30 | 0.6 |
| ICVE ( Liao et al., 2025 ) | 3.76 | 3.77 | 4.50 | 3.87 | 4.68 | 3.54 | 3.54 | 2.91 | 2.94 | 4.79 | 20.60 |
Table 1. Quantitative comparison on video instruction editing. For OpenVE-Bench ( He et al., 2025 ) , we report a subset of the original benchmark metrics due to space limitations; the complete table is provided in the supplementary material. For InsEdit-Bench, we report Overall, Instruction Compliance (IC), Temporal Visual Quality (TVQ), and Unedited Region Preservation (URP), together with inference latency.
Figure 7. Qualitative results on video instruction editing. Representative examples show that InsEdit follows diverse editing instructions while preserving scene fidelity and temporal coherence. A qualitative figure showing representative video instruction editing results produced by InsEdit across multiple scenarios, highlighting instruction following, temporal coherence, and preservation of unedited content.
Figure 8. Qualitative comparison with some baselines on video instruction editing. Compared with UniVideo and VINO, InsEdit better follows editing instructions while preserving scene fidelity and temporal coherence. A single-column qualitative comparison figure for video instruction editing, showing results from UniVideo, VINO, and InsEdit on the same editing cases.
| Variant | Overall | IC | TVQ | URP |
|---|---|---|---|---|
| Edit-only Baseline | 4.41 | 4.24 | 4.30 | 4.70 |
| + Gen Data w/ SigLIP-1 | 4.46 | 4.32 | 4.33 | 4.72 |
| + Gen Data w/ SigLIP-3 | 4.30 | 4.09 | 4.17 | 4.63 |
| + Gen Data w/ SigLIP-3 Avg | 4.32 | 4.11 | 4.21 | 4.65 |
| + Gen Data w/o VLM Vision | 4.10 | 3.83 | 3.80 | 4.52 |
Table 2. Ablation of Stage-2 modeling variants on InsEdit-Bench, including added generation data, VLM-vision input, and SigLIP temporal modeling.
| Group | Variant | Overall | IC | TVQ | URP |
| Baseline | Edit-only Baseline | 4.41 | 4.24 | 4.30 | 4.70 |
| Initialization | w/o Stage 1 Init | 4.15 | 3.94 | 3.95 | 4.57 |
| w/o Stage 1 Consistency Init | 4.26 | 3.99 | 4.04 | 4.74 | |
| w/o Stage 1 VLM-Recon Init | 4.29 | 4.09 | 4.15 | 4.63 | |
| Data Ratio | Image:Video = 1:1 | 4.34 | 4.15 | 4.16 | 4.71 |
| Image:Video = 1:4 | 4.27 | 4.07 | 4.12 | 4.62 |
Table 3. Ablation of the edit-only Stage-2 recipe on InsEdit-Bench, including Stage-1 initialization, image-video training ratio, and prompt-format mixture.
| Method | G_SC | G_PQ | G_O |
|---|---|---|---|
| Gemini2.5 | 7.48 | 8.30 | 7.17 |
| GPT4o | 8.06 | 7.80 | 7.48 |
| Seedream4 | 8.33 | 8.00 | 7.72 |
| UniWorld-V1 ( Lin et al., 2025 ) | 5.04 | 7.56 | 4.98 |
| OmniGen2 ( Wu et al., 2025b ) | 6.79 | 6.68 | 6.18 |
| Flux-Kontext-Dev ( Batifol et al., 2025 ) | 7.23 | 7.28 | 6.53 |
Table 4. Quantitative comparison on GEdit ( Liu et al., 2025 ) . We report the three average metrics used by the benchmark.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Overall | Local Add | Local Remove | Local Change | Background Change | Global Style | Subtitle Edit | Creative Edit | Camera Edit |
|---|---|---|---|---|---|---|---|---|---|
| VACE-14B ( Jiang et al., 2025 ) | 3.01 | 1.76 | 3.99 | 2.47 | 2.81 | 3.46 | 4.41 | 2.17 | 3.09 |
| OmniVideo ( Tan et al., 2025 ) | 3.66 | 2.80 | 4.52 | 3.75 | 4.11 | 3.41 | 4.95 | 1.13 | 3.62 |
| InsViE ( Wu et al., 2025c ) | 3.25 | 2.25 | 3.56 | 2.82 | 2.68 | 3.63 | 4.77 | 3.36 | 3.61 |
| Lucy-Edit ( Team, 2025a ) | 3.77 | 3.92 | 3.95 | 3.93 | 3.25 | 3.64 | 4.23 | 4.19 | 3.54 |
| ICVE ( Liao et al., 2025 ) | 3.76 | 3.77 | 4.50 | 3.87 | 3.51 | 3.87 | 4.68 | 3.54 | 2.84 |
| Ditto ( Bai et al., 2025 ) | 3.44 | 2.48 | 3.53 | 2.89 | 3.52 | 4.48 | 3.69 | 4.14 | 3.33 |
Table 5. More comprehensive category-level results on OpenVE-Bench ( He et al., 2025 ) .
Figure 9. Additional qualitative results on GEdit. Supplementary examples illustrate the image editing performance of InsEdit on the GEdit benchmark. A supplementary qualitative figure showing additional image editing results of InsEdit on the GEdit benchmark.
Figure 10. Additional qualitative video editing results. More supplementary examples illustrate the editing performance of InsEdit across diverse instructions and scenes. A supplementary qualitative figure showing additional video editing results for InsEdit across multiple editing scenarios.
More in this topic
Instruction-Based Video Editing
Instruction-based video editing changes video content from natural-language directions while preserving everything the instruction does not target.
See all 63 papers