cs.CVOct 16, 2026

LiveSync: Iterative Self-Training for Lip-Sync Generation

Authors: Huaishao Luo, Huan Zhang, Guanbo Bao, Yinglu Liu, Youzheng Wu, Xiaodong He

Organizations: JD Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Audio-driven lip synchronization aims to match target speech through localized mouth and jaw articulation while preserving subject identity, head pose, and surrounding facial context. Naturally recorded videos provide synchronized reconstruction targets but rarely offer pairs that are matched in identity and pose yet differ primarily in speech-related articulation. Mask-based training enables reconstruction from these videos, but random-reference inputs differ from the aligned self-reference used at inference. We present LiveSync, an iterative self-training framework in which a single mask-based model generates pseudo pairs and is refined for deployment. The model is first trained on a curated real-video corpus augmented with externally generated clips covering under-represented visual conditions. A frozen checkpoint then uses independently sampled audio to generate mouth-edited references from real-video frames, copying pixels outside the edit mask from each source frame. After confidence filtering, these references are paired with aligned masked inputs and original real-frame targets. The retained pseudo pairs are mixed with real training data to refine the same model from its current weights, updating training supervision without adding inference stages. Although we instantiate the framework with a single-pass GAN-based model, the self-training strategy could in principle also be adapted to diffusion-based models. Experimental results show that LiveSync achieves leading synchronization performance on HDTF and competitive performance compared with the baselines on the AIGC-LipSync subset, while maintaining low inference latency.

Figures & tables

Figure 1: Comparison of lip-sync training paradigms. (a) Single-stage mask-based methods reconstruct masked facial regions from audio and reference frames. (b) Two-stage methods use a mask-based generator to construct pseudo pairs for a separate mask-free model. (c) LiveSync uses the same mask-based model to generate spatially aligned pseudo pairs, filter and mix them with real training data, and continue training from the previous round’s weights, forming a closed-loop self-training pipeline. Gray-filled rectangles indicate masked regions.
Figure 2: Overview of LiveSync’s closed-loop self-training. The figure highlights the mismatch between inference, which uses a frame and its aligned masked self, and conventional training, which uses a randomly sampled reference. A frozen checkpoint driven by sampled audio produces a self-generated reference that preserves the target frame outside the edit mask. Confidence-filtered aligned pairs are then mixed with real pairs to refine the same model.
Figure 3: Comparison of representative face masking strategies. LiveSync adopts (c), the mouth-centered rectangular mask, which removes the target lip motion while retaining surrounding facial context. We further use boundary-aware padding when the face crop partially extends beyond the video frame.
MethodLip SyncVisual QualityIdentityEfficiency
 Sync-C ↑\uparrowSync-D ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowFID ↓\downarrowFVD ↓\downarrowLPIPS ↓\downarrowCSIM ↑\uparrowLatency ( ×\times ) ↓\downarrow
HDTF
Wav2Lip [ 19 ]8.4197.15827.6090.886031.727318.1650.18480.47531.0
VideoReTalking [ 3 ]8.1996.93025.9440.850633.741310.5980.20110.43799.2
IP-LAP [ 31 ]4.5449.99229.0350.895220.623243.3970.15650.4936143.9
DINet [ 29 ]6.7878.11826.5070.881914.447151.2890.15070.479718.7
Table 1: Quantitative comparison of LiveSync and all baselines on HDTF and a shared 100-video subset of AIGC-LipSync [ 18 ] . All entries are measured under our evaluation protocol. PSNR is reported in dB. Latency is normalized to Wav2Lip (1.0 ×\times ) and repeated in both dataset blocks. Bold and underlining indicate the best and second-best values, respectively, for each metric within a dataset.
Figure 4: Qualitative comparison of lip-sync outputs. The first column shows the original input frame, followed by outputs from LiveSync and all baselines for the same target audio. Driving words and phonemes are listed at left, and “N/A” indicates that a method failed to generate an output. The examples include profile views, occluded faces, and stylized subjects.
SettingSync-C ↑\uparrowSync-D ↓\downarrowPSNR ↑\uparrowSSIM ↑\uparrowFID ↓\downarrowFVD ↓\downarrowLPIPS ↓\downarrowCSIM ↑\uparrow
HDTF
LiveSync9.1386.39528.9210.914416.589142.5840.11130.5759
w/o Pair8.9386.56627.4610.870428.458194.8230.16560.4672
w/o Pair & GAN8.7186.55928.4960.892432.175251.5200.16060.4998
AIGC-LipSync Benchmark
LiveSync5.7468.25130.8630.931715.370181.5940.14230.5406
Table 2: Cumulative ablation on HDTF and the shared 100-video AIGC-LipSync subset, using the eight metrics of the main comparison. The w/o Pair variant removes self-generated pseudo pairs, and w/o Pair & GAN additionally removes adversarial training. All entries are measured results. Bold marks the best value among the three variants for each metric within each dataset.
Figure 5: Qualitative comparison of the cumulative ablation variants. From top to bottom: full LiveSync, w/o Pair, and w/o Pair & GAN, matching the configurations in Tab. 2 . Each column shows the same example across the three variants. The examples cover diverse facial appearances, stylized characters, and head poses, allowing comparison of mouth shape, lip boundaries, and inner-mouth appearance.
More in this topic

Lip Synchronization

See all 19 papers

Explore similar work

CardsList
Mar 10, 2026

SyncEdit: Rethinking Lip Synchronization as Editing with Audio-Driven Diffusion Models

Lip synchronization refers to the task of modifying facial lip movements such that they are temporally aligned with a given audio signal. It is a fundamental problem in audio-visual synthesis for generating realistic talking-head videos. Recent progress in diffusion-based generative models has led to remarkable advances in lip synchronization. Nevertheless, existing methods typically achieve audio-visual alignment by fine-tuning pre-trained diffusion models on large scale datasets, resulting in substantial computational costs and significant data requirements. Inspired by FlowEdit, we reformulate lip synchronization as a video editing problem. Building upon a pre-trained audio-driven diffusion model, our approach achieves lip synchronization in a training-free manner, without additional fine-tuning or paired data. In this paper, we present SyncEdit, a training-free framework designed for lip synchronization. We reformulate the editing paradigm by substituting the edit sequence in FlowEdit with the target sequence, yielding an unbiased estimation of the desired output. Moreover, we propose annealed noise alignment, which progressively alignes the sampled Gaussian noise with diffusion-model-estimated noise during iterative editing, producing a smooth and stable editing trajectory. Extensive experimental results validate the effectiveness and robustness of the proposed framework. Code is available at \href[]{https://github.com/l1346792580123/SyncEdit}{here}.

Lixiang Lin, Siyuan Jin, Jinshan Zhang

HiThink Research · University of Science and Technology of China · Zhejiang University

Lip SynchronizationVideo EditingAudio-Visual SynchronizationAudio-Driven Facial Animation
AskPDF
Aug 19, 2026

EfficientSync: Real-Time Lip Synchronization via Deformation-Based Reference Texture Mixing

Audio-driven lip synchronization manipulates the mouth region of a talking-face video to match the driving audio while preserving head pose, identity, and background. Although the task is inherently local editing, prevailing approaches reconstruct the entire lower face with heavy GAN- or diffusion-based decoders, incurring substantial latency and, more critically, hallucinating intra-oral details such as teeth and lip wrinkles instead of preserving authentic textures. We contend that the bottleneck in identity preservation is not the scarcity of reference frames, but the lack of a mechanism that faithfully transfers the genuine textures they already contain. We therefore present EfficientSync, a real-time deformation-based framework that retains reference textures rather than resynthesizing them. First, the Dynamic Texture Mixer reformulates multi-reference fusion as channel-wise selection, evaluating each spatially aligned reference in a global context and aggregating them by channel-wise weighted summation, preserving textural integrity at low cost. Second, Spatio-Temporal Shifted Adaptive Masking decomposes the source frame into lip-generation conditions and an independent background prior, suppressing lower-face leakage while blending the synthesized mouth seamlessly into the background. Third, STAR Sampling, a zero-overhead pre-processing step, retrieves the sharpest and most topologically diverse reference frames. Experiments on HDTF and VFHQ show state-of-the-art visual quality and identity preservation at 166 FPS on a single GPU. Video demos: https://alunaticat.github.io/EfficientSync/index.html.

Fa-Ting Hong, Runzhen Liu, Luchuan Song +2

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR · South China University of Technology

Lip SynchronizationAudio-Visual SynchronizationAudio-Driven Facial Animation
AskPDF
Sep 29, 2026

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.

Bangxun Tang

University of California, Irvine

Audio-Visual SynchronizationAudio-Driven Facial AnimationLip Synchronization
AskPDF