EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixing

Fa-Ting Hong; Runzhen Liu; Luchuan Song; Hongmin Cai; Chuhua Xian

cs.CVAug 19, 2026

EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixing

Authors: Fa-Ting Hong, Runzhen Liu, Luchuan Song, Hongmin Cai, Chuhua Xian

Organizations: Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · South China University of Technology

Abstract

Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.

Figures & tables

Fig. 1: Motivation of the Dynamic Texture Mixer. Rather than reconstructing the mouth from scratch, we treat each reference as a track on a mixing console and adjust its volume, that is, its aggregation score, according to the driving audio and the source identity.

Fig. 2: Existing mask strategies. (a) Square mask used in DINet [ 2 ] . (b) Fixed mask used in LatentSync [ 3 ] . (c) Adaptive mask used in IPTalker [ 13 ] .

Fig. 3: The framework or our EfficientSync. (a) STAR Sampling first selects, from the source video, a reference pool that is both sharp and topologically diverse, supplying high-quality material for deformation at no inference cost. (b) The Dynamic Texture Mixer then aggregates the spatially aligned references into a high-fidelity mouth feature through a context-aware, channel-wise selection mechanism. (c) The Spatio-Temporal Shifted Adaptive Masking strategy finally composites the synthesized mouth back into the source frame, blending it seamlessly with the preserved background.

Fig. 4: Foundational metrics for STAR Sampling. Visualization of the key geometric and textural measurements extracted from facial landmarks.

Fig. 5: Illustration of STAR Sampling. (1) Each frame’s sharpness is normalized within its

k

-nearest neighbors in

\mathbf{F}_{\text{topo}}

-space to obtain a relative clarity

\hat{D}_{\text{sharp}}

. (2) The first anchor maximizes clarity over centrality,

\frac{\hat{D}_{\text{sharp},i}}{D_{\text{mean},i}+\epsilon}

. (3) Each subsequent anchor maximizes the clarity-weighted farthest-point distance until

N

references are selected.

Fig. 6: Comparison of feature fusion mechanisms. (a) Convolutional Fusion aggregates concatenated reference features using standard convolutional layers. (b) Spatial Self-Attention computes attention weights based on flattened spatial patches. (c) Our proposed Context-Aware Feature Synthesis (CAFS) leverages global context to predict channel-wise routing scores, aggregating diverse candidate features through an efficient channel-wise weighted summation without spatial or channel entanglement.

Fig. 7: Architecture of the Dynamic Texture Mixer. The cropped reference features are encoded and fused with the global context through a Self-Attention block. Linear mapping and Softmax operations translate the updated tokens into channel-specific routing scores, under which the reference features undergo channel-wise weighted aggregation to synthesize the target mouth feature.

Fig. 8: Qualitative comparisons with state-of-the-art works under the cross-audio setting. EfficientSync preserves sharp, identity-consistent intra-oral textures, whereas competing methods either blur the mouth region or hallucinate teeth that deviate from the source identity.

Method	FPS $\uparrow$	Mem (MB) $\downarrow$	GFLOPs/f $\downarrow$
IPLAP [ 23 ]	0.98	8848	2356
DINet [ 2 ]	10.73	349	180
IPTalker [ 13 ]	27.21	565	366
LatentSync [ 3 ]	4.39	11704	39840
KeySync [ 26 ]	1.07	38363	16358
EfficientSync (Ours)	166.01	718	245

TABLE I: Comparison of computational efficiency on a single NVIDIA A100 GPU.

Method	HDTF						VFHQ
Method	SSIM $\uparrow$	LPIPS $\downarrow$	VL $\uparrow$	FID $\downarrow$	FVD $\downarrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VL $\uparrow$	FID $\downarrow$	FVD $\downarrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$
IPLAP [ 23 ]	0.958	0.032	37.615	3.895	95.104	8.039	0.942	0.057	61.359	9.451	117.370	5.751
DINet [ 2 ]	0.952	0.030	47.354	3.690	94.625	8.851	0.925	0.071	59.505	20.399	260.732	5.140
IPTalker [ 13 ]	0.971	0.018	51.750	4.433	61.798	8.885	0.945	0.040	85.791	9.365	167.655	5.530
LatentSync [ 3 ]	0.963	0.028	47.906	2.342	50.820	9.183	0.952	0.042	68.544	5.782	54.241	7.765
KeySync [ 26 ]	0.906	0.064	42.868	11.321	121.321	8.765	0.878	0.097	74.002	23.421	149.354	7.747

TABLE II: Quantitative comparison of reconstruction performance on the HDTF and VFHQ datasets.

Method	HDTF				VFHQ
Method	VL $\uparrow$	FID $\downarrow$	FVD $\downarrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$	VL $\uparrow$	FID $\downarrow$	FVD $\downarrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$
IPLAP [ 23 ]	37.595	4.052	110.420	5.862	61.534	18.925	159.798	4.476
DINet [ 2 ]	46.039	3.660	118.889	7.355	59.229	11.112	286.577	4.369
IPTalker [ 13 ]	51.733	4.606	87.501	4.622	90.774	10.260	201.464	3.231
LatentSync [ 3 ]	47.714	2.470	78.510	9.037	70.346	7.745	149.811	6.550
KeySync [ 26 ]	42.964	11.258	165.393	8.547	79.739	25.876	258.167	6.989

TABLE III: Quantitative comparison of cross-audio performance on the HDTF and VFHQ datasets.

Variant	SSIM $\uparrow$	LPIPS $\downarrow$	VL $\uparrow$	FID $\downarrow$	FVD $\downarrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$
Reconstruction
w/ Conv Fusion	0.9779	0.0132	57.281	2.055	55.071	8.947
w/ Attn Fusion	0.9786	0.0132	58.680	2.284	54.225	9.043
w/o STAM	0.9761	0.0149	56.900	1.945	53.541	8.670
Ours	0.9791	0.0127	58.691	1.824	49.144	9.107
Cross-Audio

TABLE IV: Ablation study on HDTF.

Fig. 9: Visual comparison for the STAM ablation. As highlighted by the red boxes, removing STAM (w/o STAM) causes spatial discontinuities and blurred background around the lower face, whereas our full model (w STAM) preserves the background and blends seamlessly.

Ref. Num.	Random Sampling		FPS		STAR Sampling
Ref. Num.	$\text{Sync}_{\text{conf}}$ $\uparrow$	VL $\uparrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$	VL $\uparrow$	$\text{Sync}_{\text{conf}}$ $\uparrow$	VL $\uparrow$
Reconstruction
1	8.391	58.240	8.505	58.206	8.552	58.398
2	8.771	58.538	8.833	58.579	8.910	58.703
4	9.004	58.585	9.065	58.702	9.113	58.720
8	9.076	58.651	9.114	58.669	9.177	58.680

TABLE V: Ablation on the number of reference frames and STAR Sampling on HDTF.

Fig. 10: User Preference Study. We evaluated EfficientSync against random baselines through a user study focusing on Visual Quality and Lip Sync. The results display the preference rates as a percentage of total votes, indicating whether participants preferred our model (Ours), the competing model (Other), or viewed them as comparable (Same).

Fig. 11: Causal knockout experiments guided by Temporal Routing Divergence ( $\mathcal{D}_{TR}$ ). $\mathcal{D}_{TR}$ -guided channel ablations reveal how DTM adaptively disentangles rigid high-frequency details from flexible low-frequency transitions.

Fig. 12: Temporal trajectories of routing weights on a single video. Visualization of the channel-averaged attention weights allocated by DTM to individual references under STAR and random sampling across varying reference counts.

Lip Synchronization

See all 19 papers

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models
May 16, 2026Saeid Firouzi Daghigh, Majid Iranpour Mobarakeh, Mostafa Alavi +1Lip Synchronization Audio-Visual Synchronization
Department of Computer Engineering and Information Technology, Payam Noor University, Tehran, Iran · independent researcher
Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation
Sep 29, 2026Bangxun TangAudio-Visual Synchronization Audio-Driven Facial Animation
University of California, Irvine
LiveSync: Iterative Self-Training for Lip-Sync Generation
Oct 16, 2026Huaishao Luo, Huan Zhang, Guanbo Bao +3Lip Synchronization Audio-Driven Facial Animation
JD Technology

May 16, 2026J/K move · Enter open · S save

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models

Saeid Firouzi Daghigh, Majid Iranpour Mobarakeh, Mostafa Alavi, Mahdi Bagheri

Department of Computer Engineering and Information Technology, Payam Noor University, Tehran, Iran · independent researcher

Lip Synchronization Audio-Visual Synchronization Audio-Driven Facial Animation

Open paper AskPDF arXiv

We present HighSync, an end-to-end diffusion-based framework for high-fidelity lip synchronization that generates photorealistic talking-face videos aligned with arbitrary input audio. Existing approaches consistently struggle to reconcile image quality with synchronization accuracy, producing either visually degraded outputs or temporally inconsistent lip movements. HighSync addresses both challenges simultaneously and, to our knowledge, is the first lip sync model to operate natively at 512*512 resolution, positioning it as a viable solution for professional production environments such as the film and broadcast industries. Central to our approach is the identification and systematic elimination of a data leakage phenomenon that has silently undermined temporal modeling in prior work, preventing models from developing a genuine dependence on the audio signal. Comprehensive evaluations across both perceptual quality and synchronization accuracy metrics confirm that HighSync achieves state-of-the-art performance on both fronts. Source code, pre-trained models, and supplementary video results are publicly available at: https://github.com/saeed5959/high_sync

Abstract

Lip Synchronization

Explore similar work

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

LiveSync: Iterative Self-Training for Lip-Sync Generation