Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

Bangxun Tang

cs.CVSep 29, 2026

Beyond Lip Sync: Reference-Grounded Oral Refinement for Audio-Driven Portrait Animation

Authors: Bangxun Tang

Organizations: University of California, Irvine

Abstract

We present RGOR (Reference-Grounded Oral Refinement), an audio-driven lip-sync framework that renders the mouth of the specific person being dubbed rather than a generic one. Existing lip-sync systems follow the audio closely and keep the face recognizable, yet the mouth they render is an average mouth: the shape and texture of the lips, the arrangement of the teeth, and how much of them shows as the mouth opens are not that person's. The problem persists because nothing in current training or evaluation asks for the person's own mouth: perceptual losses accept any plausible mouth, face identity is carried mostly by the skin around it, and the released inference code of inpainting systems uses the unmasked target frame as the reference, which hides the gap. To address this, RGOR conditions every generated frame on frames from separate enrollment recordings of the same person and on HD patches of the mouth that bypass the VAE, and trains the generator against a paired judge that compares each rendered mouth with the person's reference and learns to reject a realistic mouth of someone else. We further build an evaluation protocol and use it to compare open-source and commercial lip-sync systems on held-out identities. Experiments show that RGOR achieves the best or second-best result on most metrics, and preserves the person's own lip and dental detail while keeping synchronization and the rest of the face intact.

Figures & tables

Panel of Figure 1 — Figure 1: Given a video with the mouth masked out, its speech, and enrollment frames and HD patches of the same person, RGOR renders that person’s own lips and teeth in sync with the audio and leaves the rest of the face untouched. Readers can click and play the video clips in this figure using Adobe Acrobat .

Panel of Figure 2 — Figure 2: Overview of RGOR . Each output frame gets its own scattered enrollment reference, and four HD patches bypass the VAE through gated attention; the masked source, the references, and the audio drive the inpainting UNet, and the composited output is trained with image terms, a shape term, and a paired judge that asks whether the rendered mouth belongs to the referenced person.

Panel of Figure 3 — Figure 3: Data construction from NeRSemble.

Method (condition)	MAE $\downarrow$	MAE-out $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	Tex $\to$ 1	Open $\to$ 1	LSE-C $\uparrow$ /D $\downarrow$
Wav2Lip (xref)	0.0463	0.0285	23.7	0.473	0.340	0.200	0.493	4.28/8.19
LatentSync-1.6 (xref)	0.0328	0.0183	26.1	0.334	0.234	0.399	0.451	5.12/6.54
MuseTalk-1.5 (official) †	0.0440	0.0150	23.7	0.446	0.324	0.338	0.628	3.20/8.69
X-Dub (xvid)	0.0483	0.0460	22.5	0.408	0.251	0.555	1.42	3.90/9.11
sync.so lipsync-2 (xvid)	0.0466	0.0447	23.1	0.366	0.207	0.698	0.860	1.70/11.1
Kling lip-sync (xvid)	0.0502	0.0475	22.4	0.354	0.212	0.704	1.09	3.13/9.33

Table 1: Held-out NeRSemble benchmark. †: run as released, with the target frame as reference.

Panel of Figure 4 — Figure 4: Qualitative comparison; click to play the video clips in Adobe Acrobat .

Variant	MAE $\downarrow$	MAE-out $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	Tex $\to$ 1	Open $\to$ 1	LSE-C $\uparrow$ /D $\downarrow$
w/o scattered references	0.0407	0.00992	24.7	0.311	0.184	0.869	0.777	4.28/6.49
w/o HD patches	0.0364	0.00639	25.2	0.295	0.165	0.815	0.740	4.06/6.52
w/o paired judge	0.0367	0.00763	25.3	0.326	0.185	0.778	0.430	3.86/6.95
w/o shape term	0.0395	0.00781	24.7	0.317	0.181	0.842	0.702	3.77/6.79
RGOR (full)	0.0352	0.00634	25.4	0.291	0.161	0.858	0.858	4.24/6.57

Table 2: Ablation on the held-out benchmark (scored on lossless renders).

Figure 6: Human study: share of votes each system receives as the best result on each question.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: The human-study interface for one held-out query clip.

Metric	Identity	Outside the mouth	Realism	Lip-sync
MAE	$-$ 0.0952	$-$ 0.0714	$-$ 0.0714	0.0476
MAE-out	$-$ 0.238	$-$ 0.190	$-$ 0.190	$-$ 0.119
PSNR	$-$ 0.0714	$-$ 0.0952	$-$ 0.0952	0.0238
LPIPS	0.667	0.690	0.690	0.643
DISTS	0.786 ∗	0.810 ∗	0.810 ∗	0.786 ∗
Tex $\to$ 1	0.881 ∗	0.905 ∗	0.905 ∗	0.833 ∗

Appendix

Table 3: Spearman correlation between the human vote shares of the eight systems and the metrics of Table 1 ; ∗ :

p<0.05

(exact permutation test). The eight systems receive the same vote ranking on Outside the mouth and on Realism, so these two columns coincide.

Set	Method (condition)	sees	MAE $\downarrow$	LPIPS $\downarrow$	DISTS $\downarrow$	Tex $\to$ 1
NeRSemble	Wav2Lip (xref)	–	0.0463	0.473	0.340	0.200
	Wav2Lip (official)	✓	0.0352	0.435	0.315	0.239
	LatentSync-1.6 (xref)	–	0.0328	0.334	0.234	0.399
	LatentSync-1.6 (official)	✓	0.0223	0.262	0.189	0.494
	X-Dub (xvid, 51 clips)	–	0.0490	0.412	0.256	0.532
	X-Dub (same-clip, 51 clips)	✓	0.0392	0.366	0.233	0.586

Appendix

Table 4: Same-frame reference leak. A ✓ marks the rows whose reference or input is the target clip.

Figure 8: Failure cases of RGOR : ground truth (top) and ours (bottom) at the same frames, beside the full frame with its fixed oral box. (a) Fast head turn: our mouth stays frontal and is stretched across the turned face. (b) Low-resolution enrollment (HDTF): the gapped lower teeth become a generic tooth row. (c, d) Opening amplitude: the mouth is held too closed (c) or too open (d).

Method (condition)	MAE $\downarrow$	MAE-out $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	Tex $\to$ 1	Open $\to$ 1	LSE-C $\uparrow$ /D $\downarrow$
Wav2Lip (xref)	0.0712	0.0381	20.0	0.408	0.262	0.511	1.17	8.47/6.93
LatentSync-1.6 (xref)	0.0490	0.0199	22.7	0.280	0.183	0.605	0.711	9.30/5.70
MuseTalk-1.5 (official) †	0.0564	0.0161	21.6	0.348	0.239	0.647	0.918	7.38/7.33
X-Dub (xvid)	0.0991	0.0887	17.2	0.445	0.246	0.814	1.65	6.81/8.22
sync.so lipsync-2 (xvid)	0.0938	0.0859	17.7	0.435	0.235	0.801	1.10	1.73/13.0
Kling lip-sync (xvid)	0.0953	0.0872	17.5	0.425	0.235	0.760	1.29	3.82/10.6

Appendix

Table 5: HDTF (48 query clips, 24 speakers) under the strict protocol; columns and marks as in Table 1 . Official Wav2Lip and LatentSync-1.6 rows are in Table 4 . Ground truth’s own SyncNet score is 7.72 / 7.05.

Explore similar work

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models
May 16, 2026Saeid Firouzi Daghigh, Majid Iranpour Mobarakeh, Mostafa Alavi +1Precise Lip Synchronization Speech-To-Text Alignment
Department of Computer Engineering and Information Technology, Payam Noor University, Tehran, Iran · independent researcher
SyncEdit: Rethinking Lip Synchronization as Editing with Audio-Driven Diffusion Models
Mar 10, 2026Lixiang Lin, Siyuan Jin, Jinshan ZhangPrecise Lip Synchronization Audio-Visual Consistency
HiThink Research · University of Science and Technology of China · Zhejiang University
Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing
Sep 30, 2026Rui Liu, Bhavin Jawade, Haoqi Li +4Precise Lip Synchronization Speech-To-Text Alignment
University of Maryland, College Park · Netflix

May 16, 2026, cs.CVJ/K move · Enter open · S save

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models

Saeid Firouzi Daghigh, Majid Iranpour Mobarakeh, Mostafa Alavi, Mahdi Bagheri

Department of Computer Engineering and Information Technology, Payam Noor University, Tehran, Iran · independent researcher

Precise Lip Synchronization Speech-To-Text Alignment Latent Diffusion Model Diffusion Models

Open paper AskPDF arXiv

We present HighSync, an end-to-end diffusion-based framework for high-fidelity lip synchronization that generates photorealistic talking-face videos aligned with arbitrary input audio. Existing approaches consistently struggle to reconcile image quality with synchronization accuracy, producing either visually degraded outputs or temporally inconsistent lip movements. HighSync addresses both challenges simultaneously and, to our knowledge, is the first lip sync model to operate natively at 512*512 resolution, positioning it as a viable solution for professional production environments such as the film and broadcast industries. Central to our approach is the identification and systematic elimination of a data leakage phenomenon that has silently undermined temporal modeling in prior work, preventing models from developing a genuine dependence on the audio signal. Comprehensive evaluations across both perceptual quality and synchronization accuracy metrics confirm that HighSync achieves state-of-the-art performance on both fronts. Source code, pre-trained models, and supplementary video results are publicly available at: https://github.com/saeed5959/high_sync

Abstract

Explore similar work

HighSync: High-Quality Lip Synchronization via Latent Diffusion Models

SyncEdit: Rethinking Lip Synchronization as Editing with Audio-Driven Diffusion Models

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing