Diffusion Transformers (DiTs) have demonstrated remarkable scalability and quality in image and video generation, prompting growing interest in extending them to controllable generation and editing tasks. However, compared to the image counterparts, progress in video control and editing remains limited, mainly due to the scarcity of paired video data and the high computational cost of training video diffusion models. To address this issue, in this paper, we propose a video-free tuning framework termed ViFeEdit for video diffusion transformers. Without requiring any forms of video training data, ViFeEdit achieves versatile video generation and editing, adapted solely with 2D images. At the core of our approach is an architectural reparameterization that decouples spatial independence from the full 3D attention in modern video diffusion transformers, which enables visually faithful editing while maintaining temporal consistency with only minimal additional parameters. Moreover, this design operates in a dual-path pipeline with separate timestep embeddings for noise scheduling, exhibiting strong adaptability to diverse conditioning signals. Extensive experiments demonstrate that our method delivers promising results of controllable video generation and editing with only minimal training on 2D image data. Codes are available https://github.com/Lexie-YU/ViFeEdit.
Figures & tables
Figure 1: The visualization results of our proposed method ViFeEdit. Our proposed method can adapt text-to-video DiTs to various video editing tasks without any video data. Here, we demonstrate our proposed method on six fine-grained video editing tasks, including style transfer, rigid replacement, non-rigid replacement, color alternation, object addition and object removal.
Figure 2: The architecture of DiT blocks of (a) the original text-to-video Wan2.1 model and (b) our proposed video-free tuner ViFeEdit for video editing and control tasks. Here, we enable text-to-video DiTs to handle diverse video editing and control tasks without any video data. Specifically, the source video CV is jointly fed into the model and interacts with the noisy video latent Z in the 2D spatial attention branch, providing explicit reference guidance.
Figure 3: The visualization results of baselines and our proposed method on consistent style transfer tasks. ∗ means that the pretrained model VACE is further finetuned on the paired image data for style transfer learning.
Method/Setting
VBench
VLM Score
Subject
Background
Temporal
Motion
Color
Structural
Motion
Stylization
Consistency
Consistency
Flickering
Smoothness
Consistency †
Consistency †
3D Chibi Style
OmniConsistency
0.9711
0.9712
0.9948
0.9811
0.9113
90.86
89.67
93.06
VACE ∗
0.9751
0.9758
0.9945
0.9805
0.8737
84.68
86.29
91.13
Ours
0.9811
0.9785
0.9980
0.9872
0.9259
91.13
90.16
93.06
Table 1: The evaluation results of baselines and our proposed method on style transfer task. Here, the experiments are conducted on three target styles, 3D Chibi style, Ghibli Studio style, and American Cartoon style. ∗ means that the pretrained VACE is further finetuned on the paired image data for style transfer learning. † means that the evaluation of consistency is conducted between the source and target video.
Figure 4: The visualization results of baselines and our proposed method on rigid and non-rigid replacement tasks. ∗ means that the pretrained VACE is further finetuned on the paired image data for editing tasks.
Figure 5: The visualization results of baselines and our proposed method on color alternation, object addition and object removal tasks. ∗ means that the pretrained VACE is further finetuned on the paired image data for editing tasks.
Method/Setting
Object Replacement (Rigid & Non-Rigid)
Color Alteration
YN-Acc
MC-Acc
∪ -Acc
∩ -Acc
FiVE-Acc
YN-Acc
MC-Acc
∪ -Acc
∩ -Acc
FiVE-Acc
SDEdit
21.00
32.50
33.50
20.00
26.75
13.00
22.00
23.00
12.00
17.50
VidToMe
36.50
51.00
51.50
36.00
43.75
47.00
51.00
54.00
44.00
49.00
Pyramid-Edit
48.00
67.50
68.00
47.50
57.75
52.00
49.00
59.00
42.00
50.50
Wan-Edit
39.50
52.50
53.00
39.00
46.00
37.00
41.00
44.00
34.00
39.00
VACE ∗
20.50
34.00
34.00
20.50
27.25
82.00
87.00
89.00
80.00
84.50
Table 2: The evaluation results of baselines and our proposed method on FiVE-Benchmark with object replacement (rigid and non-rigid), color alternation, object addition, and object removal tasks. ∗ means that the pretrained VACE is further finetuned on the paired image data for editing tasks.
Task/Setting
Single-Task LoRA
Multi-Task LoRA
YN-Acc
MC-Acc
∪ -Acc
∩ -Acc
FiVE-Acc
YN-Acc
MC-Acc
∪ -Acc
∩ -Acc
FiVE-Acc
Color Alternation
87.00
96.00
98.00
85.00
91.50
87.00
95.00
97.00
85.00
91.00
Object Addition
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Object Removal
80.00
80.00
80.00
80.00
80.00
80.00
90.00
90.00
80.00
85.00
Table 3: The performance of multi-task LoRA and single-task LoRA on FiVE-Bench. Multi-task LoRA is trained with objective addition, removal and color alternation tasks at the same time.
Figure 6: The visualization results of ablation studies. (a) The effectiveness of each proposed component. From left to right, each key component cumulatively builds upon the previous configuration. (b) Comparison with other strategies, including Direct Tuning and Zero-Conv.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Optimizer
AdamW
Learning Rate
1e-4
Weight Decay
0.01
LoRA Rank
32
Scheduler
ConstantLR
LoRA Target Modules
SpaPos.q, k, v, o
Appendix
Table 4: The hyper-parameters of training for all tasks. Here, the training epoch from left to right is for consistent style transfer, object rigid replacement, object non-rigid replacement, color alternation, object addition and object removal tasks.
Figure 7: The visualization results of Ditto and our proposed method on consistent style transfer tasks. Here, our proposed method achieves high consistency and style fidelity with only paired image data.
Method/Setting
Model Size
Training Sets
VBench
Subject
Background
Temporal
Motion
Color
Consistency
Consistency
Flickering
Smoothness
3D Chibi Style
Ditto
14B
Ditto-1M (Video)
0.9749
0.9790
0.9965
0.9759
0.7948
Ours
1.3B
OmniConsistency (Image)
0.9855
0.9806
0.9981
0.9907
0.9357
Ghibli Studio Style
Appendix
Table 5: The evaluation results of Ditto and our proposed method on VBench with consistent style transfer task. Here, Ditto has 14B model size and pre-trained with large-scale paired video dataset, while our proposed method is 1.3B and trained on 100-200 paired image data.
Figure 8: The visualization results of our proposed method on mixed style transfer task. Here, we fine-tune Wan2.1-T2V-1.3B model with mixed style image data. Our proposed method can handle multiple styles transfer at one time.
Figure 9: The visualization results of Wan2.1-VACE-1.3B and our proposed method on depth-to-video tasks. Here, the source videos are generated by Wan2.2-T2V-A14B, and the depth maps are generated by Video Depth Anything.
Figure 10: Visualization results of our method on more challenging editing cases. The left three columns are the source videos, and the right columns are edited videos via our proposed method.