Single-view reference-to-video methods often struggle to preserve identity consistency under large facial-angle variations. This limitation naturally motivates the incorporation of multi-view facial references. However, simply introducing additional reference images exacerbates the copy-paste problem, particularly the view-dependent copy-paste artifact, which reduces facial motion naturalness. Although cross-paired data can alleviate this issue, collecting such data is costly. To balance the consistency and naturalness, we propose Mv2ID, a multi-view conditioned framework under in-paired supervision. We introduce a region-masking training strategy to prevent shortcut learning and extract essential identity features by encouraging the model to aggregate complementary identity cues across views. In addition, we design a reference decoupled-RoPE mechanism that assigns distinct positional encoding to video and conditioning tokens for better modeling of their heterogeneous properties. Furthermore, we construct a large-scale dataset with diverse facial-angle variations and propose dedicated evaluation metrics for identity consistency and motion naturalness. Extensive experiments demonstrate that our method significantly improves identity consistency while maintaining motion naturalness, outperforming existing approaches trained with cross-paired data.
Figures & tables
Figure 1: In this work, we leverage multi-view references to improve identity consistency under large facial angle variations, while maintaining motion naturalness (see enlarged view for trajectory details).
Figure 2: Copy-paste : Within certain time intervals, the generated video collapses to a specific reference image, resulting in reduced motion diversity(more detailed analysis can be found in the Observation in Sec. 3 ).
Figure 3: The overview of the training pipeline. We design the region masking training(Sec. 4.3) and reference-decoupled RoPE(Sec. 4.4). For inference, we use the clean condition images.
Figure 4: Qualitative comparison under large facial-angle changes. Our method,Mv 2 ID, better balances the consistency and facial motion naturalness.
Figure 5: Facial trajectory comparison. Each dot represents the normalized 3D facial direction vector of a frame, and the black dot denotes the origin. Colors indicate the temporal progression of frames, normalized to the range [0,1] across 81 frames.
Figure 6: Single-view vs. Multi-view and qualitative comparison of proposed components. R: RD-RoPE. M : Region-Masking Training.
Figure 7: Facial trajectory for ablation.
Figure 8: User-study results.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Overview of our dataset construction pipeline. We progressively filter raw videos through coarse filtering, clip-level processing, and pose-level processing to obtain human-centric video clips with large facial-angle variation.
Dataset
Human-centric
Large-Angle
Pose
OpenS2V-5M [ 51 ]
×
×
×
Phantom-Data [ 3 ]
×
×
×
HuMoSet [ 2 ]
✓
×
×
Ours
✓
✓
✓
Appendix
Table 3: Comparison with representative open-source subject-to-video related datasets. Our dataset explicitly focuses on human-centric samples with large facial-angle variations.
Figure 10: The high-level understanding of VM and RM.
Figure 11: The visualization results of VM and RM.
Figure 12: Qualitative comparison of different masking ratios in Region Masking. A higher masking ratio helps mitigate the copy-paste issue and produces more natural results.
Masking Ratio
NaturalScore
40%
4.13
60%
4.30
Appendix
Table 4: NaturalScore under different masking ratios. Higher is better.
Figure 13: The h/w offset can accelerate the convergence of training. The model with the h/w offset learns to utilize reference information within only 450 training steps.
Figure 14: More visual results of our method.
Figure 15: The facial trajectory results of our method and compared methods.