RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing
Authors: Bojia Zi, Xiaoyan Yang, Yu Zhou, Ruijie Sun, Lihan Zhang, Bin Liang, Kam-Fai Wong, Haibin Huang, Chi Zhang, Xuelong Li
Organizations: Institute of Artificial Intelligence, China Telecom (TeleAI) · The Chinese University of Hong Kong (CUHK) · Sun Yat-sen University · Fudan University · Tsinghua University
Recent advances in video editing have been largely driven by large-scale instruction-based datasets. However, existing datasets still suffer from two critical limitations. First, target videos are commonly produced by automatic editing models, which may introduce visible artifacts and unreliable supervision signals. Second, most public datasets rely primarily on textual instructions, while lacking visual references that are crucial for precise, identity-preserving, and controllable editing. To address these limitations, we introduce RefVideo-6M, a large-scale reference-guided editing dataset containing 5 million video editing samples and 1 million image editing samples. To ensure reliable supervision, our dataset uses a construction pipeline that treats artifact-free real videos as editing targets and generates quality-filtered input conditions with multiple editing experts. In addition, it provides approximately 6 million visual references, covering diverse reference types and editing scenarios, thereby enabling models to learn fine-grained visual correspondence beyond text-only instructions. Based on RefVideo-6M, we further train a reference-guided video editing model, Ref-MoT, to evaluate the effectiveness and scalability of the proposed dataset. Extensive experiments demonstrate that RefVideo-6M provides substantially more reliable supervision than existing datasets and enables the training of powerful editing models with improved visual quality, controllability, and reference consistency. The open-source dataset is available at https://huggingface.co/datasets/RefVideo6M/RefVideo6M.
Figures & tables
Figure 1: Visualization of RefVideo-6M. Our dataset is a reliable reference-based video editing dataset. It provides a diverse set of reference types, including circles, bounding boxes, light directions, outpainting masks, background images, styles, textures, objects, humans, and clothing references. More visualization results can be found in the supplementary file.
Figure 2: Construction pipeline of RefVideo-6M. We train four experts in addition to eight open-source experts. In our dataset, most tasks reverse the roles of the original and edited videos to construct reliable video pairs.
Datasets
Experts
Tasks
Video Length
Resolution
Ref-Img for Video
Video Pairs
Ref-Img for Image
Image Pairs
InsV2V Cheng et al. (2024)
✗
Free-Form
16
256 2
✗
0.06M
✗
✗
InsViE Wu et al. (2025c)
✗
Free-Form
25
576P
✗
1M
✗
✗
Señorita Zi et al. (2025b)
8
16 Tasks
33-64
336P/1120P
✗
2M
✗
✗
Ditto Bai et al. (2025b)
3
11 Tasks
101
720P
✗
2M
✗
✗
FFP Huang et al. (2026)
2
5 Tasks
81
720P
✗
0.3M
✗
✗
Reco Zhang et al. (2025)
1
4 Tasks
33-64
720P
✗
0.5M
✗
✗
Table 1: Comparison among OpenVE-3M, Reco-500K, Ditto-1M, Señorita-2M, InsV2V and InsViE-1M. For the number of experts, we count only those based on deep learning and exclude traditional operators. For OpenVE-3M, only 2M samples are available in the released open-source dataset. For all datasets, the numbers of experts and tasks are recalculated according to our unified criteria.
Figure 3: Statistics of RefVideo-6M. The dataset includes 26 editing tasks and 6 image editing tasks.
Figure 4: Overview of RefMoT. RefMoT adopts a Mixture-of-Tokens design to efficiently incorporate reference information, reducing training cost while improving reference-guided editing quality and preserving strong generalization.
Method
GPT-5.5 Eval
Gemini-3-Pro Eval
Quantitative Results
User Study
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
CLIPScore
Temp-Cons
Attr. Align.
Halluc. Supp.
Overall Instr.
Attr. Align.
Halluc. Supp.
Overall Instr.
InsViE Wu et al. (2025c)
3.39
2.17
3.94
2.05
2.76
2.86
3.94
1.88
4.30
1.85
3.01
3.00
0.340334
0.984086
0.3%
InsV2V Cheng et al. (2024)
3.48
2.21
4.25
2.33
3.22
3.10
4.09
1.38
4.40
1.75
3.31
2.99
0.345979
0.978932
0.4%
VideoCoF Yang et al. (2025a)
4.02
2.54
4.28
2.23
3.47
3.31
4.33
2.50
4.21
1.93
3.49
3.29
0.348055
0.973684
0.6%
ICVE Liao et al. (2025)
4.02
2.92
4.56
2.77
3.60
3.57
4.52
2.46
4.55
2.54
3.71
3.56
0.349055
0.983105
0.3%
Table 2: Evaluation results on instruction-based editing. The best results are marked in red, and the second-best results are marked in blue. Bg. Pres. denotes Background Preservation, Attr. Align. denotes Attribute Alignment, Halluc. Supp. denotes Hallucination Suppression, and Overall Instr. denotes Overall Instruction Alignment.
Figure 5: Visualization of our model and prior methods for instruction-based and reference-based editing.
Method
GPT-5.5 Eval
Gemini-3-Pro Eval
Quantitative Results
User Study
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
CLIPScore
Temp-Cons
Attr. Align.
Halluc. Supp.
Overall Instr.
Attr. Align.
Halluc. Supp.
Overall Instr.
Kiwi Lin et al. (2026)
3.88
2.96
4.09
3.53
3.29
3.55
4.07
2.35
3.88
2.99
2.98
3.25
0.346991
0.968535
15.1%
UniVideo Wei et al. (2025a)
3.81
3.51
4.18
3.68
3.44
3.73
4.46
3.93
4.22
3.67
3.60
3.98
0.351239
0.971174
14.5%
Bernini Team et al. (2026)
3.80
3.48
4.21
3.76
3.27
3.70
4.15
3.69
3.98
3.73
3.84
3.88
0.350255
0.971026
17.9%
Ours
4.22
3.62
4.59
4.13
3.60
4.03
4.70
3.86
4.51
4.02
3.68
4.16
0.354365
0.973043
52.5%
Table 3: Evaluation results on reference-based editing. The abbreviations remain consistent with those in the previous table.
Type
Method
GPT-5.5 Eval
Gemini-3-Pro Eval
Quantitative Results
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
Bg. Pres.
Instruction Alignment
Visual Quality
Overall
CLIPScore
Temp-Cons
Attr. Align.
Halluc. Supp.
Overall Instr.
Attr. Align.
Halluc. Supp.
Overall Instr.
Arch
w/o RefMoT
3.95
3.38
4.15
3.79
3.05
3.66
4.10
3.58
3.74
3.51
2.73
3.53
0.357992
0.963980
1/4 RefMoT
4.09
2.47
4.44
3.20
3.36
3.51
4.54
2.22
4.29
2.68
3.35
3.42
0.346781
0.975295
1/3 RefMoT
4.06
2.96
4.58
3.48
3.45
3.71
4.42
2.81
4.31
3.08
3.23
3.57
0.346675
0.974725
1/2 RefMoT
4.06
3.09
4.47
3.65
3.45
3.75
4.37
2.99
4.34
3.32
3.23
3.65
0.349798
0.974311
Table 4: Ablation results on different model architectures and datasets. The abbreviations remain consistent with those in the previous table.
Figure 6: Visualization of baseline datasets and our RefVideo-6M.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Details
Stage 1
Stage 2
Training
Base model
HunyuanVideo1.5
Stage-1 model
Learning rate
1×10−5
1×10−5
Optimizer
AdamW
AdamW
AdamW β1
0.9
0.9
AdamW β2
0.99
0.99
Weight decay
1×10−4
1×10−4
Appendix
Table 5: Training and inference details across different stages. The Stage-2 inference time is measured with the KV-cache version using torch.compile .
Positive only if perfect or nearly perfect; partial completion is negative
Appendix
Table 6: Automatic evaluation configuration. Source and output frames use matched relative timestamps.
Metric
GPT-5.5
Gemini-3-Pro
Accuracy
94.44%
92.22%
Positive recall
91.43%
97.14%
Negative specificity
96.36%
89.09%
Positive precision
94.12%
85.00%
F1
92.75%
90.67%
Balanced accuracy
93.90%
93.12%
Appendix
Table 7: Binary-label agreement with human judgments over 90 samples. Bold indicates the better result between evaluators.
Evaluator
TP
FN
FP
TN
Pos. rec.
Neg. spec.
GPT-5.5
32
3
2
53
91.43%
96.36%
Gemini-3-Pro
34
1
6
49
97.14%
89.09%
Appendix
Table 8: Binary confusion matrices against human labels.
Figure 7: User-study interface for instruction-based video editing without visual references.
Figure 8: User-study interface for reference-based video editing with visual references.
Figure 9: Visualization of style transfer, relighting, weather/season editing, and camera motion instructions. This figure summarizes prompts that modify the global appearance, illumination, environmental condition, or viewpoint trajectory of a video.
Figure 10: Visualization of VFX editing, background swap, human swap, and object addition instructions. These examples demonstrate how the dataset covers local visual effects, scene replacement, identity or person transformation, and insertion of new objects.
Figure 11: Visualization of object removal, object swap, subtitle removal, and object retexture instructions. The figure illustrates editing tasks that remove distracting elements, replace target objects, erase text overlays, or alter material appearance.
Figure 12: Visualization of virtual try-on, watermark removal, and recoloring instructions. These categories focus on changing clothing attributes, removing visible watermarks or branding, and modifying the color of selected regions or objects.
Figure 13: Visualization of video restoration, detection, and outpainting instructions. The examples include upscaling, colorizing grayscale videos, deblurring, grounding or detecting target entities, and expanding the visible scene beyond the original frame.
Figure 14: Visualization of inpainting and structure-conditioned video generation instructions. This figure covers masked region completion, depth-to-video generation, scribble-to-video generation, and conversion from videos to structural controls.
Figure 15: Visualization of Canny-to-video, HED-to-video, and video-to-control instructions. These examples highlight control-based generation, where edge maps or structural guides are used to synthesize realistic videos or extract control representations from videos.