Lip synchronization, the task of generating realistic lip movements synchronized with audio signals, is fundamental to applications such as film dubbing, virtual avatars, and digital content creation. Recent GAN-based methods offer computational efficiency but often lack visual fidelity, while diffusion-based approaches achieve high-quality results at the cost of substantial computational overhead. Moreover, most existing methods are limited to photo-realistic human faces, restricting their applicability to stylized characters and non-human entities. In this paper, we propose an efficient universal lip-sync video generation framework that achieves high-quality results without compromising computational efficiency. We build on LivePortrait, a warp-based portrait animation model, and introduce a training strategy that aligns audio signals to an implicit motion space by distilling knowledge from a pre-trained video-to-motion model. This enables real-time inference while maintaining high visual fidelity, and naturally generalizes to diverse stylistic characters beyond photo-realistic humans. Furthermore, we introduce an expressive control gate that enables explicit manipulation of emotional expressions for content enrichment. Extensive experiments demonstrate that our method achieves state-of-the-art performance in both visual quality and lip-sync accuracy while ensuring real-time inference.
Figures & tables
Figure 1: We present an universal model that generates lip-synchronized video on stylistic humans and characters corresponding to driving audio sequences. Our method ensures computational efficiency without sacrificing high visual fidelity.
Figure 6: Flow diagram of our proposed expressive control gate for injection and transition of emotion states.
HDTF
VoxCeleb2
Gen. Quality
Recon. Quality
Lip-Sync
Gen. Quality
Recon. Quality
Lip-Sync
Methods
FID ↓
FVD ↓
PSNR ↑
LPIPS ↓
CSIM ↑
LMD ↓
AVS ↑
FID ↓
FVD ↓
PSNR ↑
LPIPS ↓
CSIM ↑
LMD ↓
AVS ↑
Wav2Lip [ Prajwal et al.(2020)Prajwal, Mukhopadhyay, Namboodiri, and Jawahar ]
12.50
304.35
30.65
0.135
0.818
7.88
0.3742
10.80
257.85
30.48
0.146
0.791
7.91
0.3658
Diff2Lip [ Mukhopadhyay et al.(2024)Mukhopadhyay, Suri, Gadde, and Shrivastava ]
10.30
260.45
33.04
0.122
0.905
5.32
0.3886
9.80
210.45
34.82
0.057
0.932
5.80
0.3950
MuseTalk [ Zhang et al.(2024)Zhang, Liu, Chen, Wu, He, Zhan, and Zhou ]
9.35
246.75
35.06
0.062
0.932
4.98
0.4103
7.10
203.43
34.57
0.064
0.905
4.98
0.4021
LatentSync [ Li et al.(2024)Li, Zhang, Xu, Lin, Xie, Feng, Peng, Chen, and Xing ]
7.22
151.00
35.16
0.039
0.958
3.86
0.5174
5.70
85.75
34.82
0.042
0.958
4.61
0.5081
Table 2: Quantitative comparison with existing lip-sync video generation methods on HDTF and VoxCeleb2 datasets. Best: bolded , and second: underlined .
Table 7Figure 8Figure 9Figure 10
Figure 13: The visual effectiveness on the proposed expressive control gate.
Figure 14: Perceptual ablation on the expressive control gate.
Lip synchronization aims to generate realistic talking videos that match given audio, which is essential for high-quality video dubbing. However, current methods have fundamental drawbacks: mask-based approaches suffer from local color discrepancies, while mask-free methods struggle with global background texture misalignment. Furthermore, most methods struggle with diverse real-world scenarios such as stylized avatars, face occlusion, and extreme lighting conditions. In this paper, we propose UniSync, a unified framework designed for achieving high-fidelity lip synchronization in diverse scenarios. Specifically, UniSync uses a mask-free pose-anchored training strategy to keep head motion and eliminate synthesis color artifacts, while employing mask-based blending consistent inference to ensure structural precision and smooth blending. Notably, fine-tuning on compact but diverse videos empowers our model with exceptional domain adaptability, handling complex corner cases effectively. We also introduce the RealWorld-LipSync benchmark to evaluate models under real-world demands, which covers diverse application scenarios including both human faces and stylized avatars. Extensive experiments demonstrate that UniSync significantly outperforms state-of-the-art methods, advancing the field towards truly generalizable and production-ready lip synchronization.
Lip synchronization aims to generate visual lip dynamics that align precisely with speech audio. Despite the high generation quality of diffusion models, they often struggle in complex scenarios and suffer from prohibitive inference latency, limiting real-world deployment. We present ComplexSync, a unified diffusion-based framework that enables real-time, high-fidelity lip sync under complex conditions. First, we introduce a dual-stream joint training strategy to mitigate information leakage from reference frames while preserving natural dynamics. Second, we develop a distillation-based acceleration scheme for single-step denoising, achieving a throughput of over 70 FPS. Third, we propose a relational alignment loss that leverages structural priors from Vision Foundation Models (VFMs) to enhance robustness against complex scene factors. Furthermore, we present the first benchmark specifically designed for complex lip synchronization, comprising over 200 challenging video sequences and specialized metrics. Extensive experiments demonstrate that ComplexSync achieves state-of-the-art performance across both standard and complex scenarios while enabling real-time inference.
Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.
Fa-Ting Hong, Runzhen Liu, Luchuan Song +2
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · South China University of Technology