Lip synchronization aims to generate realistic talking videos that match given audio, which is essential for high-quality video dubbing. However, current methods have fundamental drawbacks: mask-based approaches suffer from local color discrepancies, while mask-free methods struggle with global background texture misalignment. Furthermore, most methods struggle with diverse real-world scenarios such as stylized avatars, face occlusion, and extreme lighting conditions. In this paper, we propose UniSync, a unified framework designed for achieving high-fidelity lip synchronization in diverse scenarios. Specifically, UniSync uses a mask-free pose-anchored training strategy to keep head motion and eliminate synthesis color artifacts, while employing mask-based blending consistent inference to ensure structural precision and smooth blending. Notably, fine-tuning on compact but diverse videos empowers our model with exceptional domain adaptability, handling complex corner cases effectively. We also introduce the RealWorld-LipSync benchmark to evaluate models under real-world demands, which covers diverse application scenarios including both human faces and stylized avatars. Extensive experiments demonstrate that UniSync significantly outperforms state-of-the-art methods, advancing the field towards truly generalizable and production-ready lip synchronization.
Figures & tables
Figure 1: Comparison with OmniSync on challenging real-world scenarios. While OmniSync shows degraded quality under low-light conditions and occlusions, it completely fails on stylized cartoons, producing no lip modifications. Our method robustly handles all cases with accurate synchronization and high visual fidelity.
Figure 2: Training framework of the proposed method. The Pose-Anchored Fidelity Strategy (PAFS) enforces a direct mapping between pose variation and facial motion. LoRA-based fine-tuning makes the model better adapt to pose-anchored, mask-free training efficiently.
Characteristic
HDTF
AIGC-LipSync
Ours
Video Source
Real-world
AI-generated
Real-world
Resolution
720-1080p
1080p
1080p-4K
Scene Diversity
Low
Medium
High
Camera Angles
Frontal
Varied
Diverse
Lighting
Uniform
Ideal
Extreme
Camera Motion
Minimal
Limited
Dynamic
Table 1: Comparison of lip synchronization benchmarks. Our benchmark emphasizes challenging real-world conditions including extreme lighting, diverse angles, and stylized content.
HDTF Dataset
RealWorld-LipSync Benchmark
Methods
FID ↓
FVD ↓
CSIM ↑
NIQE ↓
BRISQUE ↓
HyperIQA ↑
LSE-C ↑
FID ↓
FVD ↓
CSIM ↑
NIQE ↓
BRISQUE ↓
HyperIQA ↑
LSE-C ↑
GSR ↑
TalkLip
16.680
691.518
0.843
6.377
52.109
44.393
5.880
15.613
447.192
0.629
8.938
56.808
41.227
4.123
71.9%
IP-LAP
9.512
325.691
0.809
6.533
54.402
50.086
7.260
19.458
280.014
0.848
9.202
59.115
43.140
4.686
68.9%
Diff2Lip
12.079
461.341
0.869
6.261
49.361
48.869
7.140
17.631
326.519
0.790
9.359
59.479
42.398
4.318
78.0%
MuseTalk
8.759
231.418
0.862
5.824
46.003
55.397
6.890
16.894
174.946
0.832
8.407
59.142
44.852
4.582
86.5%
LatentSync
8.518
216.899
0.859
6.270
50.861
53.208
8.050
14.039
145.060
0.927
8.106
57.215
44.061
4.943
77.8%
Table 2: Quantitative comparison with previous methods on HDTF and RealWorld-LipSync benchmarks. The best and second-best results are highlighted in Red and Purple .
Figure 4: Quantitative evaluation results compared with recent SOTA methods MuseTalk, LatentSync, and OmniSync. Our method achieves the most accurate lip synchronization and demonstrates superior robustness across diverse real-world production scenarios.
Methods
FID ↓
FVD ↓
CSIM ↑
NIQE ↓
BRISQUE ↓
HyperIQA ↑
LSE-C ↑
w/o PAFS
13.521
177.128
0.824
7.813
52.277
46.897
4.812
w/o TALI
13.742
167.277
0.830
8.438
57.123
41.258
4.855
w/o Gaussian
14.841
191.128
0.805
8.852
58.441
40.019
4.390
UniSync
12.976
141.573
0.869
8.023
55.988
45.818
5.091
Table 3: Ablation study for the proposed UniSync.
τinj
FID ↓
FVD ↓
CSIM ↑
NIQE ↓
BRISQUE ↓
HyperIQA ↑
LSE-C ↑
0.0
13.742
167.277
0.830
8.438
57.123
41.258
4.855
0.2
14.219
179.145
0.915
8.154
53.505
44.782
4.652
0.4
13.006
154.879
0.694
7.975
53.198
44.423
4.678
0.6
14.316
153.091
0.772
8.124
56.504
42.455
4.992
0.8
12.976
141.573
0.869
8.023
55.988
45.818
5.091
1.0
13.538
160.894
0.921
8.465
56.773
45.349
5.264
Table 4: Sensitivity analysis of the injection ratio τinj in TALI. The optimal ratio is 0.8, achieving the best trade-off between temporal consistency (FVD) and generation quality.
Figure 5: Ablation Study for PAFS, TALI and Gaussian smooth.
Metrics
MuseTalk
LatentSync
OmniSync
Ours
Lip Sync Accuracy
2.44
3.52
3.37
3.49
Identity Preservation
2.30
3.12
3.62
3.83
Timing Stability
2.28
3.14
3.64
3.68
Image Quality
2.08
3.07
3.68
4.12
Video Realism
2.14
2.99
3.46
3.86
Table 5: User study results comparing perceptual quality across different methods. Participants rate each metric on a scale of 1-5 (higher is better).