Low-quality facial videos suffer from missing fine details, while AI-generated videos may additionally contain implausible teeth and lip movements that do not match the speech. Restoring the mouth region is particularly challenging: enhancing existing details can preserve structural errors, whereas regenerating them risks changing facial identity and articulation. We propose an audio-guided facial video restoration framework that reconstructs unreliable mouth details and aligns lip movements with the speech while preserving facial identity and temporal consistency. Built on a video diffusion transformer, our framework mixes input video latents with noise to retain useful source information while allowing corrupted details to be regenerated. Audio guides lip movements, reference images constrain facial appearance, and previously generated frames promote temporal consistency. We train the model in two stages, first learning visual restoration from degraded images and videos, then learning audio-guided lip correction from degraded dubbed videos. The resulting model supports both restoration with the original audio and visual dubbing with replacement audio. On talking-head restoration benchmarks, our method achieves the lowest LPIPS and DISTS and the best lip synchronization scores among the evaluated restoration baselines. Qualitative results on AI-generated videos further show improvements in facial detail and mouth-region appearance, including clearer tooth boundaries and lip contours.
Figures & tables
Figure 1: Audio-guided facial restoration from AIGC footage. Small facial regions provide limited evidence for lip and teeth structure. The model restores these regions using reference frames and accompanying audio. The three clips illustrate the source scenes and reconstructed heads; the reference bank depicts training-time conditioning. Quantitative references originate from the GT first frame. “Cat” denotes channel-wise fusion implemented by separate linear projections and summation.
Figure 2: Audio-guided facial video restoration. Stage 1 jointly trains on degraded images and videos with their corresponding targets. Images use only flow-matching MSE ( LMSE ) and RGB losses; videos additionally use temporal supervision. Stage 2 uses degraded dubbed sources and target audio for lip correction. Sparse references and continuous history occupy distinct conditioning slots. The VAE, HuBERT, and SyncNet are frozen; the video DiT and conditioning projections are trainable. AIGC thumbnails illustrate the input and target roles.
Reference-based fidelity
Lip sync.
No-reference quality
Method
PSNR ↑ [ 30 ]
SSIM ↑ [ 31 ]
LPIPS ↓ [ 29 ]
DISTS ↓ [ 32 ]
Sync-C ↑ [ 33 ]
Sync-D ↓ [ 33 ]
MUSIQ ↑ [ 34 ]
CLIP-IQA ↑ [ 35 ]
FasterVQA ↑ [ 36 ]
DOVER ↑ [ 37 ]
GT
–
–
–
–
6.959
7.863
62.70
0.4525
0.7735
0.6148
GPEN
26.77
0.7970
0.2267
0.1825
6.309
8.288
76.47
0.7032
0.8931
0.8505
CodeFormer
27.38
0.8045
0.2195
0.1876
5.856
8.693
74.42
0.6649
0.8854
0.7954
InstantRestore
27.42
0.8283
0.1380
0.1421
6.209
8.403
68.11
0.5166
0.8562
0.7704
SparkVSR
30.62
0.8863
0.1755
0.1586
6.545
8.170
65.66
0.4417
0.8459
0.7467
Table 1: Facial restoration under original audio. Results on the 150-video benchmark, averaged equally over common valid clips. Darker and lighter green mark the best and second-best results, respectively.
Lip sync.
No-reference quality
Method
Sync-C ↑ [ 33 ]
Sync-D ↓ [ 33 ]
MUSIQ ↑ [ 34 ]
CLIP-IQA ↑ [ 35 ]
FasterVQA ↑ [ 36 ]
DOVER ↑ [ 37 ]
FlashVSR Full → dubbing
X-Dub
6.137
9.332
73.25
0.6582
0.8750
0.7845
KeySync
7.347
7.840
68.07
0.4908
0.8736
0.7647
TBDub Teacher
7.736
7.543
70.18
0.6208
0.8650
0.7179
TBDub Student
7.469
8.196
69.72
0.5834
0.8641
0.7309
Table 2: Visual dubbing under replacement audio. Equal-video means on the same 30 clips and replacement-audio pairs. Darker and lighter green mark the best and second-best method results.
Figure 3: Facial detail and articulation in AIGC scenes. Four MiniMax-H3 [ 41 ] clips are shown with matched-time head crops located by orange boxes. Left: restoration under original audio. Right: FlashVSR Full (top) or SparkVSR (bottom) is followed by the arrow-marked dubbing methods; Input and SR-only outputs provide visual controls. Ours + jointly reconstructs detail and articulation with a GPT-Image-enhanced reference, without an external SR frontend. Waveforms and word-level IPA labels identify the audio segments. Displayed predictions are unretouched.
Metric
GT
Ours
w/o Audio
Direct Motion
X-Dub (adapted)
PSNR ↑
–
29.10
29.36
28.24
26.63
SSIM ↑
–
0.8852
0.8889
0.8600
0.8415
LPIPS ↓
–
0.0850
0.0856
0.1341
0.1523
MUSIQ ↑
63.26
68.19
69.09
69.49
70.08
CLIP-IQA ↑
0.4971
0.5591
0.5709
0.5807
0.6145
FasterVQA ↑
0.7880
0.8352
0.8432
0.8595
0.8354
Table 3: Ablation comparison on the same 30 videos: 15 from MEAD [ 38 ] and 15 from HDTF [ 39 ] . All variants use the GT first frame. X-Dub [ 8 ] and DirectMotion provide alternatives to source–noise mixing and structured reference injection, respectively. Values are video-wise means; GT is excluded from rankings. Dark and light green mark the best and second-best model results.
Figure 4: Qualitative ablation comparison. Each row shows two selected frames from one clip: MEAD [ 38 ] (top) and HDTF [ 39 ] (bottom). The same frame pair is displayed for GT and each model variant, with ten images per row. All variants use the GT first frame, and predictions are extracted directly from native RGB outputs.
Face video restoration from degraded observations is challenging, as it requires simultaneously recovering visual fidelity, temporal consistency, and subject identity. Existing approaches are often either reference-free, which can lead to identity loss when person-specific facial details are lost, or subject-specific, which limits generalization to unseen identities. We propose a subject-agnostic, reference-guided framework for identity-preserving face video restoration. Our method introduces bimodal perceptual-descriptive identity conditioning into a pretrained flow-based text-to-video generator and employs a two-stage training strategy to strengthen identity guidance during restoration. Experiments show that our approach improves restoration fidelity, temporal consistency, and identity preservation, achieving superior performance under challenging video degradations, including downsampling, blur, noise, and compression artifacts. The code is available under: https://github.com/batuhanntosun/RG-FVR.
Cem Eteke, Batuhan Tosun, Eckehard Steinbach
Chair of Media Technology Munich Institute of Robotics and Machine Intelligence School of Computation, Information, and Technology Technical University of Munich 80333 Munich, Germany
Video face restoration aims to enhance degraded face videos into high-quality results with realistic facial details, stable identity, and temporal coherence. Recent diffusion-based methods have brought strong generative priors to restoration and enabled more realistic detail synthesis. However, existing approaches for face videos still rely heavily on generic diffusion priors and multi-step sampling, which limit both facial adaptation and inference efficiency. These limitations motivate the use of one-step diffusion for video face restoration, yet achieving faithful facial recovery alongside temporally stable outputs remains challenging. In this paper, we propose, DVFace, a one-step diffusion framework for real-world video face restoration. Specifically, we introduce a spatio-temporal dual-codebook design to extract complementary spatial and temporal facial priors from degraded videos. We further propose an asymmetric spatio-temporal fusion module to inject these priors into the diffusion backbone according to their distinct roles. Evaluation on various benchmarks shows that DVFace delivers superior restoration quality, temporal consistency, and identity preservation compared to recent methods. Code: https://github.com/zhengchen1999/DVFace.
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.