Overview

Audio-visual alignment studies how to match audio and video in time and meaning, such as linking speech to lip movements, sounds to visible sources, and music to edits. Methods include contrastive learning, cross-attention, optimal transport, and temporal alignment losses for recognition and localization, alongside diffusion and flow-matching models for synchronized dubbing and video-to-audio generation. Researchers evaluate these capabilities on tasks and benchmarks such as LRS3 speech recognition, audio-visual segmentation, and video-to-audio generation. Models are improving on defined tasks, but still struggle with fine-grained synchronization, conflicting cues, and reliable evaluation; stronger visual conditioning can also improve alignment while reducing control over the intended audio.

Latest papers

91 papers4 papers in the last four weeks, down 82% on the four weeks before. 0.0% of all new papers.

Nov 23, 2025

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.