VicEdit: Learning to Edit Videos from Visual In-Context Examples

Yuji Wang; Teng Hu; Yuheng Chen; Ran Yi; Han Feng; Weijian Cao; Chengjie Wang; Lizhuang Ma

cs.CVAug 17, 2026

VicEdit: Learning to Edit Videos from Visual In-Context Examples

Authors: Yuji Wang, Teng Hu, Yuheng Chen, Ran Yi, Han Feng, Weijian Cao, Chengjie Wang, Lizhuang Ma,

Organizations: Shanghai Jiao Tong University · Tencent Youtu Lab · Zhejiang University

Abstract

Despite progress in instruction-based video editing, unimodal textual instructions inherently struggle to convey fine-grained textures and complex dynamics. To bridge this perceptual gap, we propose Visual In-context Editing, a new paradigm elevating video editing from textual instructions to multi-modal visual guidance encompassing single image, image pair, and video pair. To facilitate this paradigm, we curate VicEdit-400K, the first large-scale dataset for visual in-context video editing. We develop an automated pipeline to generate 400K high-quality samples across ten task types, ensuring superior visual fidelity and semantic consistency through multi-dimensional filtering. Leveraging this foundation, we introduce VicEdit, a unified framework to bridge visual and textual contexts. To adaptively extract editing semantics from heterogeneous references, we design Modality-Adaptive Semantic Distillation, which produces modality-specific semantic tokens from visual references. These tokens are then synergistically integrated with textual instructions through Dual-Context Injection, enabling the generation process to benefit from both visual and textual signals. Extensive evaluations on VicEditBench demonstrate that VicEdit achieves state-of-the-art performance across both basic instruction editing and visual in-context editing tasks, establishing visual in-context learning as a powerful and controllable paradigm for video editing.

Figures & tables

Figure 1: Top: Qualitative comparison between VicEdit and text-only baselines. VicEdit precisely resolves stylistic drift (via Single Image), spatial drift (via Image Pair), and static dynamics (via Video Pair), demonstrating the necessity of visual exemplars. Bottom: Architectural evolution: (a) text-only methods rely on linguistic semantics; (b) ref-image methods employ auxiliary injection branches; (c) VicEdit extends MLLM contextual reasoning by distilling multi-modal priors into the video DiT for precise, reference-driven editing. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] .

Figure 2: Data construction pipeline that comprises four stages (Sec. 3.2 ), yielding a final dataset of 400K samples that covers basic editing and visual in-context editing tasks. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] .

Figure 4: Overview of VicEdit. It consists of 1) MASD modulates base queries $Q_{base}$ with modality embeddings $E_{m}$ to distill visual tokens $T_{vis}$ ; 2) DCI applies a semantic shift to calibrate instruction extraction, yielding dual-context tokens. These unified embeddings are injected into a Video DiT to steer the generation process. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] .

		Scene & Style Adaptation			Object-Level Manipulation				Spatiotemporal Synthesis
Methods	Overall	Global Style	Local Style	Background Change	Local Change	Local Remove	Local Add	Subtitle Edit	Creative Edit	Inpainting	Outpainting
VACE [ 11 ]	1.68	1.49	1.39	1.55	2.07	1.46	1.26	1.48	1.47	2.31	2.35
Lucy-Edit [ 27 ]	2.35	2.27	2.02	1.57	3.20	1.75	2.30	1.61	2.86	3.01	2.92
DITTO [ 1 ]	2.31	4.01	3.67	1.68	2.03	1.53	1.41	2.81	1.23	2.05	2.67
NovaEdit [ 22 ]	2.76	2.24	1.98	2.40	3.62	3.43	2.64	2.93	2.59	2.78	3.01
VideoCoF [ 38 ]	2.79	2.57	2.24	2.32	3.37	3.61	2.75	3.10	2.20	2.87	2.88

Table 1: Quantitative Comparison on OpenVE-Ext with MLLM.

Figure 5: Comparison with baselines on base editing tasks.

		Scene & Style Adaptation			Object-Level Manipulation				Spatiotemporal Synthesis
Methods	Overall	Global Style	Local Style	Background Change	Local Change	Local Remove	Local Add	Subtitle Edit	Creative Edit	Inpainting	Outpainting
VACE [ 11 ]	1.85	1.45	2.03	1.50	2.25	1.49	1.33	1.43	2.17	2.23	2.60
Lucy-Edit [ 27 ]	2.25	1.94	1.67	1.89	3.30	2.05	2.15	1.43	2.45	2.63	2.97
DITTO [ 1 ]	2.33	3.03	3.60	2.29	2.45	1.64	1.43	2.61	1.46	2.48	2.32
NovaEdit [ 22 ]	2.61	2.08	2.23	2.62	3.07	3.65	2.05	2.46	2.13	2.65	3.18
VideoCoF [ 38 ]	2.90	2.66	2.43	2.37	3.67	3.60	2.44	3.65	1.83	3.06	3.33

Table 2: Quantitative comparison on VicEdit-Bench with MLLM.

Figure 6: Comparison with baselines on visual in-context editing tasks. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] and Senorita [ 45 ] .

			VLM Evaluation				VBench Metrics
Base	MASD	DCI	Overall $\uparrow$	Instr. Comp. $\uparrow$	Detail Fid. $\uparrow$	Visual Qual. $\uparrow$	BC $\uparrow$	MS $\uparrow$	TF $\uparrow$	IQ $\uparrow$	AQ $\uparrow$
✓			3.02	3.12	2.98	2.96	0.968	0.995	0.986	0.711	0.564
✓	✓		3.25	3.40	3.20	3.14	0.969	0.994	0.986	0.712	0.566
✓		✓	3.34	3.56	3.22	3.24	0.971	0.995	0.986	0.715	0.567
✓	✓	✓	3.45	3.65	3.34	3.35	0.975	0.994	0.987	0.718	0.568

Table 3: Quantitative Ablation Study on VicEdit-Bench. We evaluate the impact of MASD and DCI modules across VLM-based assessments and VBench [ 10 ] dimensions.

	Base Editing				Visual In-Context Editing
Training Stage	Overall $\uparrow$	Instr. Comp. $\uparrow$	Detail Fid. $\uparrow$	Visual Qual. $\uparrow$	Overall $\uparrow$	Instr. Comp. $\uparrow$	Detail Fid. $\uparrow$	Visual Qual. $\uparrow$
Base (Kiwi-Edit)	3.08	3.24	3.01	2.98	2.97	3.10	2.98	2.76
Stage 1	3.23	3.36	3.11	3.20	3.10	3.25	3.05	3.01
Stage 2	3.10	3.22	3.02	3.07	3.40	3.62	3.30	3.27
Stage 3	3.28	3.47	3.27	3.10	3.45	3.65	3.34	3.35

Table 4: Quantitative Ablation on Training Stages. We evaluate the performance evolution across different training phases on the OpenVE-Ext and VicEdit-Bench benchmark.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Taxonomy of Editing Tasks and Modalities.

Category	Task Type	SI	IP	VP
Scene & Style Adaptation	Global Style	✓	✓	✓
	Local Style	✓	✓	✓
	Background Change	✓	✓	✓
Object-Level Manipulation	Local Add	$-$	✓	✓
	Local Change	$-$	✓	✓
	Local Remove	$-$	✓	✓

Appendix

Table 5: Supported reference modalities for each task type in VicEdit-400K. SI = Single Image; IP = Image Pair; VP = Video Pair.

Dataset	Scale	Reference Modality				Task Types	Frames	Resolution
Dataset	Scale	Text	SI	IP	VP	Task Types	Frames	Resolution
InsViE [ 35 ]	1.0M	✓	–	–	–	-	25	1024 $\times$ 576
Ditto [ 1 ]	1.0M	✓	–	–	–	3	101	1280 $\times$ 720
Señorita [ 45 ]	2.0M	✓	–	–	–	9	33–64	336 $\times$ 592–1120 $\times$ 1984
OpenVE [ 9 ]	3.0M	✓	–	–	–	8	65–129	1280 $\times$ 720
ReCo [ 42 ]	40K	✓	–	–	–	5	81	480 $\times$ 832

Appendix

Table 6: Comparison of video editing datasets. VicEdit-400K provides the most comprehensive support for multi-modal visual in-context references, including Single Image (SI), Image Pair (IP), and Video Pair (VP).

Figure 9: Examples of VicEdit-Bench. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] or Senorita [ 45 ] .

	VLM Evaluation				VBench Metrics
Method	Overall $\uparrow$	Instr. Comp. $\uparrow$	Detail Fid. $\uparrow$	Visual Qual. $\uparrow$	BC $\uparrow$	MS $\uparrow$	TF $\uparrow$	IQ $\uparrow$	AQ $\uparrow$
VACE [ 11 ]	1.85	1.98	1.79	1.80	0.954	0.994	0.983	0.692	0.543
Lucy-Edit [ 27 ]	2.25	2.40	2.20	2.15	0.962	0.993	0.983	0.701	0.544
DITTO [ 1 ]	2.33	2.44	2.32	2.20	0.966	0.995	0.984	0.703	0.560
NovaEdit [ 22 ]	2.61	2.78	2.49	2.54	0.969	0.994	0.985	0.707	0.562
VideoCoF [ 38 ]	2.90	3.02	2.85	2.85	0.972	0.995	0.986	0.705	0.561

Appendix

Table 7: Quantitative comparison on VicEdit-Bench across VLM-based assessments and VBench [ 10 ] dimensions.

		Scene & Style Adaptation			Object-Level Manipulation				Spatiotemporal Synthesis
Methods	Overall	Global Style	Local Style	Background Change	Local Change	Local Remove	Local Add	Subtitle Edit	Creative Edit	Inpainting	Outpainting
VACE [ 11 ]	2.45	2.11	1.94	2.38	2.87	2.41	2.13	2.24	2.06	2.72	3.09
Lucy-Edit [ 27 ]	3.56	3.04	2.82	4.59	4.52	4.44	4.00	2.41	3.16	3.48	3.15
DITTO [ 1 ]	3.83	4.64	3.87	4.66	4.31	4.65	3.99	3.61	1.94	2.92	3.75
NovaEdit [ 22 ]	3.23	1.70	2.90	4.50	3.70	4.80	1.95	3.74	2.48	3.25	3.23
VideoCoF [ 38 ]	3.41	1.75	3.20	4.55	4.60	4.45	2.90	3.82	2.31	3.15	3.35

Appendix

Table 8: Quantitative Comparison on OpenVE-Ext with Qwen-VL-32B [ 2 ] .

		Scene & Style Adaptation			Object-Level Manipulation				Spatiotemporal Synthesis
Methods	Overall	Global Style	Local Style	Background Change	Local Change	Local Remove	Local Add	Subtitle Edit	Creative Edit	Inpainting	Outpainting
VACE [ 11 ]	2.62	2.15	2.44	2.30	3.10	2.45	2.18	2.32	2.85	3.05	3.40
Lucy-Edit [ 27 ]	3.15	2.80	2.35	2.92	4.25	3.12	3.30	2.28	3.20	3.55	3.75
DITTO [ 1 ]	3.32	3.72	4.21	3.35	3.25	2.68	2.35	3.42	2.10	3.30	3.12
NovaEdit [ 22 ]	3.48	2.95	3.08	3.84	3.92	4.58	3.12	3.36	2.95	3.48	3.52
VideoCoF [ 38 ]	3.81	3.52	3.31	3.75	4.45	4.38	3.61	4.38	2.64	3.92	4.10

Appendix

Table 9: Quantitative comparison on VicEdit-Bench with Qwen-VL-32B [ 2 ] .

Method	#Ref. Modality	#Params	MLLM Time	DiT Time
VACE [ 11 ]	Text-Only	14B	—	$\sim$ 35 min
Lucy-Edit [ 27 ]	Text-Only	5B	—	$\sim$ 4 min
DITTO [ 1 ]	Text-Only	14B	—	$\sim$ 35 min
NovaEdit [ 22 ]	Text-Only	1.3B	—	$\sim$ 15 min
VideoCoF [ 38 ]	Text-Only	14B	—	$\sim$ 35 min
Kiwi-Edit [ 20 ]	Text-Only	5B	$\sim$ 0.4s	$\sim$ 4 min

Appendix

Table 10: Quantitative Comparison of Computational Efficiency. All metrics are measured on a single GPU with a target video length of 81 frames (

480\times 832

). MLLM Time denotes the reference feature extraction time; DiT Time denotes the denoising inference time.

Figure 10: Qualitative ablation of MASD and DCI on (left) a global style editing task and (right) a local object replacement task. MASD preserves texture and shape details, while DCI ensures semantic alignment. Only their combination produces results faithful to both the reference style and the editing instruction. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] or Senorita [ 45 ] .

Figure 11: More comparisons with baselines on base editing tasks. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] or Senorita [ 45 ] .

Figure 12: More comparisons with baselines on visual in-context editing tasks. All examples presented in the figure are collected from the open-source OpenVE [ 9 ] or Senorita [ 45 ] .

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 179 papers

Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance
Mar 2, 2026Yiqi Lin, Guoqiang Liang, Ziyun Zeng +3Video Editing Instruction-Based Video Editing
Show Lab, National University of Singapore
VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling
Oct 8, 2026Leigang Qu, Feng Cheng, Ziyan Yang +7Video Editing Instruction-Based Video Editing
National University of Singapore · ByteDance Seed · University of Science and Technology of China
InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptation
Apr 9, 2026Zhefan Rao, Bin Zou, Haoxuan Che +5Instruction-Based Video Editing Diffusion-Based Video Editing
The Hong Kong University of Science and Technology · Celia Research HK · City University of Hong Kong

Mar 2, 2026J/K move · Enter open · S save

Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

Yiqi Lin, Guoqiang Liang, Ziyun Zeng, Zechen Bai+2

Show Lab, National University of Singapore

Video Editing Instruction-Based Video Editing Video Editing Benchmarks Controllable Video Generation Synthetic Data Generation

Open paper AskPDF arXiv

Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.

Abstract

Video Editing

Explore similar work

Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling

InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptation