RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing
Authors: Bojia Zi, Xiaoyan Yang, Yu Zhou, Ruijie Sun, Lihan Zhang, Bin Liang, Kam-Fai Wong, Haibin Huang, Chi Zhang, Xuelong Li
Organizations: Institute of Artificial Intelligence, China Telecom (TeleAI) · The Chinese University of Hong Kong (CUHK) · Sun Yat-sen University · Fudan University · Tsinghua University
Recent advances in video editing have been largely driven by large-scale instruction-based datasets. However, existing datasets still suffer from two critical limitations. First, target videos are commonly produced by automatic editing models, which may introduce visible artifacts and unreliable supervision signals. Second, most public datasets rely primarily on textual instructions, while lacking visual references that are crucial for precise, identity-preserving, and controllable editing. To address these limitations, we introduce RefVideo-6M, a large-scale reference-guided editing dataset containing 5 million video editing samples and 1 million image editing samples. To ensure reliable supervision, our dataset uses a construction pipeline that treats artifact-free real videos as editing targets and generates quality-filtered input conditions with multiple editing experts. In addition, it provides approximately 6 million visual references, covering diverse reference types and editing scenarios, thereby enabling models to learn fine-grained visual correspondence beyond text-only instructions. Based on RefVideo-6M, we further train a reference-guided video editing model, Ref-MoT, to evaluate the effectiveness and scalability of the proposed dataset. Extensive experiments demonstrate that RefVideo-6M provides substantially more reliable supervision than existing datasets and enables the training of powerful editing models with improved visual quality, controllability, and reference consistency. The open-source dataset is available at https://huggingface.co/datasets/RefVideo6M/RefVideo6M.
Figures & tables
Figure 1: Visualization of RefVideo-6M. Our dataset is a reliable reference-based video editing dataset. It provides a diverse set of reference types, including circles, bounding boxes, light directions, outpainting masks, background images, styles, textures, objects, humans, and clothing references. More visualization results can be found in the supplementary file.
Figure 2: Construction pipeline of RefVideo-6M. We train four experts in addition to eight open-source experts. In our dataset, most tasks reverse the roles of the original and edited videos to construct reliable video pairs.
Datasets
Experts
Tasks
Video Length
Resolution
Ref-Img for Video
Video Pairs
Ref-Img for Image
Image Pairs
InsV2V Cheng et al. (2024)
✗
Free-Form
16
256 2
✗
0.06M
✗
✗
InsViE Wu et al. (2025c)
✗
Free-Form
25
576P
✗
1M
✗
✗
Señorita Zi et al. (2025b)
8
16 Tasks
33-64
336P/1120P
✗
2M
✗
✗
Ditto Bai et al. (2025b)
3
11 Tasks
101
720P
✗
2M
✗
✗
FFP Huang et al. (2026)
2
5 Tasks
81
720P
✗
0.3M
✗
✗
Reco Zhang et al. (2025)
1
4 Tasks
33-64
720P
✗
0.5M
✗
✗
Table 1: Comparison among OpenVE-3M, Reco-500K, Ditto-1M, Señorita-2M, InsV2V and InsViE-1M. For the number of experts, we count only those based on deep learning and exclude traditional operators. For OpenVE-3M, only 2M samples are available in the released open-source dataset. For all datasets, the numbers of experts and tasks are recalculated according to our unified criteria.
Figure 3: Statistics of RefVideo-6M. The dataset includes 26 editing tasks and 6 image editing tasks.
Figure 4: Overview of RefMoT. RefMoT adopts a Mixture-of-Tokens design to efficiently incorporate reference information, reducing training cost while improving reference-guided editing quality and preserving strong generalization.
Method
GPT-5.5 Eval
Gemini-3-Pro Eval
Quantitative Results
User Study
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
CLIPScore
Temp-Cons
Attr. Align.
Halluc. Supp.
Overall Instr.
Attr. Align.
Halluc. Supp.
Overall Instr.
InsViE Wu et al. (2025c)
3.39
2.17
3.94
2.05
2.76
2.86
3.94
1.88
4.30
1.85
3.01
3.00
0.340334
0.984086
0.3%
InsV2V Cheng et al. (2024)
3.48
2.21
4.25
2.33
3.22
3.10
4.09
1.38
4.40
1.75
3.31
2.99
0.345979
0.978932
0.4%
VideoCoF Yang et al. (2025a)
4.02
2.54
4.28
2.23
3.47
3.31
4.33
2.50
4.21
1.93
3.49
3.29
0.348055
0.973684
0.6%
ICVE Liao et al. (2025)
4.02
2.92
4.56
2.77
3.60
3.57
4.52
2.46
4.55
2.54
3.71
3.56
0.349055
0.983105
0.3%
Table 2: Evaluation results on instruction-based editing. The best results are marked in red, and the second-best results are marked in blue. Bg. Pres. denotes Background Preservation, Attr. Align. denotes Attribute Alignment, Halluc. Supp. denotes Hallucination Suppression, and Overall Instr. denotes Overall Instruction Alignment.
Figure 5: Visualization of our model and prior methods for instruction-based and reference-based editing.
Method
GPT-5.5 Eval
Gemini-3-Pro Eval
Quantitative Results
User Study
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
CLIPScore
Temp-Cons
Attr. Align.
Halluc. Supp.
Overall Instr.
Attr. Align.
Halluc. Supp.
Overall Instr.
Kiwi Lin et al. (2026)
3.88
2.96
4.09
3.53
3.29
3.55
4.07
2.35
3.88
2.99
2.98
3.25
0.346991
0.968535
15.1%
UniVideo Wei et al. (2025a)
3.81
3.51
4.18
3.68
3.44
3.73
4.46
3.93
4.22
3.67
3.60
3.98
0.351239
0.971174
14.5%
Bernini Team et al. (2026)
3.80
3.48
4.21
3.76
3.27
3.70
4.15
3.69
3.98
3.73
3.84
3.88
0.350255
0.971026
17.9%
Ours
4.22
3.62
4.59
4.13
3.60
4.03
4.70
3.86
4.51
4.02
3.68
4.16
0.354365
0.973043
52.5%
Table 3: Evaluation results on reference-based editing. The abbreviations remain consistent with those in the previous table.
Type
Method
GPT-5.5 Eval
Gemini-3-Pro Eval
Quantitative Results
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
CLIPScore
Temp-Cons
Attr. Align.
Halluc. Supp.
Overall Instr.
Attr. Align.
Halluc. Supp.
Overall Instr.
Arch
w/o RefMoT
3.95
3.38
4.15
3.79
3.05
3.66
4.10
3.58
3.74
3.51
2.73
3.53
0.357992
0.963980
1/4 RefMoT
4.09
2.47
4.44
3.20
3.36
3.51
4.54
2.22
4.29
2.68
3.35
3.42
0.346781
0.975295
1/3 RefMoT
4.06
2.96
4.58
3.48
3.45
3.71
4.42
2.81
4.31
3.08
3.23
3.57
0.346675
0.974725
1/2 RefMoT
4.06
3.09
4.47
3.65
3.45
3.75
4.37
2.99
4.34
3.32
3.23
3.65
0.349798
0.974311
Table 4: Ablation results on different model architectures and datasets. The abbreviations remain consistent with those in the previous table.
Figure 6: Visualization of baseline datasets and our RefVideo-6M.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Details
Stage 1
Stage 2
Training
Base model
HunyuanVideo1.5
Stage-1 model
Learning rate
1×10−5
1×10−5
Optimizer
AdamW
AdamW
AdamW β1
0.9
0.9
AdamW β2
0.99
0.99
Weight decay
1×10−4
1×10−4
Appendix
Table 5: Training and inference details across different stages. The Stage-2 inference time is measured with the KV-cache version using torch.compile .
Positive only if perfect or nearly perfect; partial completion is negative
Appendix
Table 6: Automatic evaluation configuration. Source and output frames use matched relative timestamps.
Metric
GPT-5.5
Gemini-3-Pro
Accuracy
94.44%
92.22%
Positive recall
91.43%
97.14%
Negative specificity
96.36%
89.09%
Positive precision
94.12%
85.00%
F1
92.75%
90.67%
Balanced accuracy
93.90%
93.12%
Appendix
Table 7: Binary-label agreement with human judgments over 90 samples. Bold indicates the better result between evaluators.
Evaluator
TP
FN
FP
TN
Pos. rec.
Neg. spec.
GPT-5.5
32
3
2
53
91.43%
96.36%
Gemini-3-Pro
34
1
6
49
97.14%
89.09%
Appendix
Table 8: Binary confusion matrices against human labels.
Figure 7: User-study interface for instruction-based video editing without visual references.
Figure 8: User-study interface for reference-based video editing with visual references.
Figure 9: Visualization of style transfer, relighting, weather/season editing, and camera motion instructions. This figure summarizes prompts that modify the global appearance, illumination, environmental condition, or viewpoint trajectory of a video.
Figure 10: Visualization of VFX editing, background swap, human swap, and object addition instructions. These examples demonstrate how the dataset covers local visual effects, scene replacement, identity or person transformation, and insertion of new objects.
Figure 11: Visualization of object removal, object swap, subtitle removal, and object retexture instructions. The figure illustrates editing tasks that remove distracting elements, replace target objects, erase text overlays, or alter material appearance.
Figure 12: Visualization of virtual try-on, watermark removal, and recoloring instructions. These categories focus on changing clothing attributes, removing visible watermarks or branding, and modifying the color of selected regions or objects.
Figure 13: Visualization of video restoration, detection, and outpainting instructions. The examples include upscaling, colorizing grayscale videos, deblurring, grounding or detecting target entities, and expanding the visible scene beyond the original frame.
Figure 14: Visualization of inpainting and structure-conditioned video generation instructions. This figure covers masked region completion, depth-to-video generation, scribble-to-video generation, and conversion from videos to structural controls.
Figure 15: Visualization of Canny-to-video, HED-to-video, and video-to-control instructions. These examples highlight control-based generation, where edge maps or structural guides are used to synthesize realistic videos or extract control representations from videos.
Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.
While instruction-based video editing has seen significant progress, joint audio-visual editing remains constrained by the absence of dedicated datasets and benchmarks. To bridge this gap, we present JAVEdit-100k, the first large-scale, high-quality dataset tailored for instruction-guided joint audio-visual editing. Focusing on human-centric videos, JAVEdit-100k comprises approximately 100K editing triplets spanning five distinct categories, including subject editing and speech editing. This dataset is rigorously constructed via four meticulously designed generation pipelines, seamlessly paired with an agent-in-the-loop quality control mechanism. Furthermore, to address the lack of standardized evaluation within the field, we introduce JAVEditBench, a comprehensive benchmark featuring curated source videos and human-aligned instructions across all editing categories. Finally, we propose JAVEdit, a pioneering baseline model for instruction-guided joint audio-visual editing. Experiments show that \model\ outperforms all baselines on five of six evaluation metrics.
Yinan Chen, Chuming Lin, Zhennan Chen +12
1Zhejiang University · 2Tencent Youtu Lab · 3Nanjing University +3
Instruction-based video editing is a natural way to control video content with text, but adapting a video generation model into an editor usually appears data-hungry. At the same time, high-quality video editing data remains scarce. In this paper, we show that a video generation backbone can become a strong video editor without large scale video editing data. We present InsEdit, an instruction-based editing model built on HunyuanVideo-1.5. InsEdit combines a visual editing architecture with a video data pipeline based on Mutual Context Attention (MCA), which creates aligned video pairs where edits can begin in the middle of a clip rather than only from the first frame. With only O(100)K video editing data, InsEdit achieves state-of-the-art results among open-source methods on our video instruction editing benchmarks. In addition, because our training recipe also includes image editing data, the final model supports image editing without any modification.
Zhefan Rao, Bin Zou, Haoxuan Che +5
The Hong Kong University of Science and Technology · Celia Research HK · City University of Hong Kong