Organizations: Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · South China University of Technology
Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.
Figures & tables
Fig. 1: Motivation of the Dynamic Texture Mixer. Rather than reconstructing the mouth from scratch, we treat each reference as a track on a mixing console and adjust its volume, that is, its aggregation score, according to the driving audio and the source identity.
Fig. 2: Existing mask strategies. (a) Square mask used in DINet [ 2 ] . (b) Fixed mask used in LatentSync [ 3 ] . (c) Adaptive mask used in IPTalker [ 13 ] .
Fig. 3: The framework or our EfficientSync. (a) STAR Sampling first selects, from the source video, a reference pool that is both sharp and topologically diverse, supplying high-quality material for deformation at no inference cost. (b) The Dynamic Texture Mixer then aggregates the spatially aligned references into a high-fidelity mouth feature through a context-aware, channel-wise selection mechanism. (c) The Spatio-Temporal Shifted Adaptive Masking strategy finally composites the synthesized mouth back into the source frame, blending it seamlessly with the preserved background.
Fig. 4: Foundational metrics for STAR Sampling. Visualization of the key geometric and textural measurements extracted from facial landmarks.
Fig. 5: Illustration of STAR Sampling. (1) Each frame’s sharpness is normalized within its k -nearest neighbors in Ftopo -space to obtain a relative clarity D^sharp . (2) The first anchor maximizes clarity over centrality, Dmean,i+ϵD^sharp,i . (3) Each subsequent anchor maximizes the clarity-weighted farthest-point distance until N references are selected.
Fig. 6: Comparison of feature fusion mechanisms. (a) Convolutional Fusion aggregates concatenated reference features using standard convolutional layers. (b) Spatial Self-Attention computes attention weights based on flattened spatial patches. (c) Our proposed Context-Aware Feature Synthesis (CAFS) leverages global context to predict channel-wise routing scores, aggregating diverse candidate features through an efficient channel-wise weighted summation without spatial or channel entanglement.
Fig. 7: Architecture of the Dynamic Texture Mixer. The cropped reference features are encoded and fused with the global context through a Self-Attention block. Linear mapping and Softmax operations translate the updated tokens into channel-specific routing scores, under which the reference features undergo channel-wise weighted aggregation to synthesize the target mouth feature.
Fig. 8: Qualitative comparisons with state-of-the-art works under the cross-audio setting. EfficientSync preserves sharp, identity-consistent intra-oral textures, whereas competing methods either blur the mouth region or hallucinate teeth that deviate from the source identity.
Method
FPS ↑
Mem (MB) ↓
GFLOPs/f ↓
IPLAP [ 23 ]
0.98
8848
2356
DINet [ 2 ]
10.73
349
180
IPTalker [ 13 ]
27.21
565
366
LatentSync [ 3 ]
4.39
11704
39840
KeySync [ 26 ]
1.07
38363
16358
EfficientSync (Ours)
166.01
718
245
TABLE I: Comparison of computational efficiency on a single NVIDIA A100 GPU.
Method
HDTF
VFHQ
SSIM ↑
LPIPS ↓
VL ↑
FID ↓
FVD ↓
Syncconf↑
SSIM ↑
LPIPS ↓
VL ↑
FID ↓
FVD ↓
Syncconf↑
IPLAP [ 23 ]
0.958
0.032
37.615
3.895
95.104
8.039
0.942
0.057
61.359
9.451
117.370
5.751
DINet [ 2 ]
0.952
0.030
47.354
3.690
94.625
8.851
0.925
0.071
59.505
20.399
260.732
5.140
IPTalker [ 13 ]
0.971
0.018
51.750
4.433
61.798
8.885
0.945
0.040
85.791
9.365
167.655
5.530
LatentSync [ 3 ]
0.963
0.028
47.906
2.342
50.820
9.183
0.952
0.042
68.544
5.782
54.241
7.765
KeySync [ 26 ]
0.906
0.064
42.868
11.321
121.321
8.765
0.878
0.097
74.002
23.421
149.354
7.747
TABLE II: Quantitative comparison of reconstruction performance on the HDTF and VFHQ datasets.
Method
HDTF
VFHQ
VL ↑
FID ↓
FVD ↓
Syncconf↑
VL ↑
FID ↓
FVD ↓
Syncconf↑
IPLAP [ 23 ]
37.595
4.052
110.420
5.862
61.534
18.925
159.798
4.476
DINet [ 2 ]
46.039
3.660
118.889
7.355
59.229
11.112
286.577
4.369
IPTalker [ 13 ]
51.733
4.606
87.501
4.622
90.774
10.260
201.464
3.231
LatentSync [ 3 ]
47.714
2.470
78.510
9.037
70.346
7.745
149.811
6.550
KeySync [ 26 ]
42.964
11.258
165.393
8.547
79.739
25.876
258.167
6.989
TABLE III: Quantitative comparison of cross-audio performance on the HDTF and VFHQ datasets.
Variant
SSIM ↑
LPIPS ↓
VL ↑
FID ↓
FVD ↓
Syncconf↑
Reconstruction
w/ Conv Fusion
0.9779
0.0132
57.281
2.055
55.071
8.947
w/ Attn Fusion
0.9786
0.0132
58.680
2.284
54.225
9.043
w/o STAM
0.9761
0.0149
56.900
1.945
53.541
8.670
Ours
0.9791
0.0127
58.691
1.824
49.144
9.107
Cross-Audio
TABLE IV: Ablation study on HDTF.
Fig. 9: Visual comparison for the STAM ablation. As highlighted by the red boxes, removing STAM (w/o STAM) causes spatial discontinuities and blurred background around the lower face, whereas our full model (w STAM) preserves the background and blends seamlessly.
Ref. Num.
Random Sampling
FPS
STAR Sampling
Syncconf↑
VL ↑
Syncconf↑
VL ↑
Syncconf↑
VL ↑
Reconstruction
1
8.391
58.240
8.505
58.206
8.552
58.398
2
8.771
58.538
8.833
58.579
8.910
58.703
4
9.004
58.585
9.065
58.702
9.113
58.720
8
9.076
58.651
9.114
58.669
9.177
58.680
TABLE V: Ablation on the number of reference frames and STAR Sampling on HDTF.
Fig. 10: User Preference Study. We evaluated EfficientSync against random baselines through a user study focusing on Visual Quality and Lip Sync. The results display the preference rates as a percentage of total votes, indicating whether participants preferred our model (Ours), the competing model (Other), or viewed them as comparable (Same).
Fig. 11: Causal knockout experiments guided by Temporal Routing Divergence ( DTR ). DTR -guided channel ablations reveal how DTM adaptively disentangles rigid high-frequency details from flexible low-frequency transitions.
Fig. 12: Temporal trajectories of routing weights on a single video. Visualization of the channel-averaged attention weights allocated by DTM to individual references under STAR and random sampling across varying reference counts.
We present HighSync, an end-to-end diffusion-based framework for high-fidelity lip synchronization that generates photorealistic talking-face videos aligned with arbitrary input audio. Existing approaches consistently struggle to reconcile image quality with synchronization accuracy, producing either visually degraded outputs or temporally inconsistent lip movements. HighSync addresses both challenges simultaneously and, to our knowledge, is the first lip sync model to operate natively at 512*512 resolution, positioning it as a viable solution for professional production environments such as the film and broadcast industries. Central to our approach is the identification and systematic elimination of a data leakage phenomenon that has silently undermined temporal modeling in prior work, preventing models from developing a genuine dependence on the audio signal. Comprehensive evaluations across both perceptual quality and synchronization accuracy metrics confirm that HighSync achieves state-of-the-art performance on both fronts. Source code, pre-trained models, and supplementary video results are publicly available at: https://github.com/saeed5959/high_sync
We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.
Audio-driven lip synchronization aims to match target speech through localized mouth and jaw articulation while preserving subject identity, head pose, and surrounding facial context. Naturally recorded videos provide synchronized reconstruction targets but rarely offer pairs that are matched in identity and pose yet differ primarily in speech-related articulation. Mask-based training enables reconstruction from these videos, but random-reference inputs differ from the aligned self-reference used at inference. We present LiveSync, an iterative self-training framework in which a single mask-based model generates pseudo pairs and is refined for deployment. The model is first trained on a curated real-video corpus augmented with externally generated clips covering under-represented visual conditions. A frozen checkpoint then uses independently sampled audio to generate mouth-edited references from real-video frames, copying pixels outside the edit mask from each source frame. After confidence filtering, these references are paired with aligned masked inputs and original real-frame targets. The retained pseudo pairs are mixed with real training data to refine the same model from its current weights, updating training supervision without adding inference stages. Although we instantiate the framework with a single-pass GAN-based model, the self-training strategy could in principle also be adapted to diffusion-based models. Experimental results show that LiveSync achieves leading synchronization performance on HDTF and competitive performance compared with the baselines on the AIGC-LipSync subset, while maintaining low inference latency.