cs.CVNov 2, 2026

TaoMate-FVSR: Audio-Guided Facial Video Restoration

Authors: Xinyang Li, Zeran Xu, Jingna Sun, Qijun Gan, Chenwei Zhang, Meiguang Jin, Junfeng Ma

Organizations: Alibaba Group

Abstract

Low-quality facial videos suffer from missing fine details, while AI-generated videos may additionally contain implausible teeth and lip movements that do not match the speech. Restoring the mouth region is particularly challenging: enhancing existing details can preserve structural errors, whereas regenerating them risks changing facial identity and articulation. We propose an audio-guided facial video restoration framework that reconstructs unreliable mouth details and aligns lip movements with the speech while preserving facial identity and temporal consistency. Built on a video diffusion transformer, our framework mixes input video latents with noise to retain useful source information while allowing corrupted details to be regenerated. Audio guides lip movements, reference images constrain facial appearance, and previously generated frames promote temporal consistency. We train the model in two stages, first learning visual restoration from degraded images and videos, then learning audio-guided lip correction from degraded dubbed videos. The resulting model supports both restoration with the original audio and visual dubbing with replacement audio. On talking-head restoration benchmarks, our method achieves the lowest LPIPS and DISTS and the best lip synchronization scores among the evaluated restoration baselines. Qualitative results on AI-generated videos further show improvements in facial detail and mouth-region appearance, including clearer tooth boundaries and lip contours.

Figures & tables

Explore similar work

CardsList
  1. RGFVR: Reference-Guided Face Video Restoration with Flow Matching

    Jun 15, 2026Cem Eteke, Batuhan Tosun, Eckehard SteinbachVideo Diffusion ModelsVideo Restoration

  2. DVFace: Spatio-Temporal Dual-Prior Diffusion for Video Face Restoration

    Apr 16, 2026Zheng Chen, Bowen Chai, Rongjun Gao +7Video Diffusion ModelsImage Restoration