Overview

Audio-visual alignment studies how to match audio and video in time and meaning, such as linking speech to lip movements, sounds to visible sources, and music to edits. Methods include contrastive learning, cross-attention, optimal transport, and temporal alignment losses for recognition and localization, alongside diffusion and flow-matching models for synchronized dubbing and video-to-audio generation. Researchers evaluate these capabilities on tasks and benchmarks such as LRS3 speech recognition, audio-visual segmentation, and video-to-audio generation. Models are improving on defined tasks, but still struggle with fine-grained synchronization, conflicting cues, and reliable evaluation; stronger visual conditioning can also improve alignment while reducing control over the intended audio.

Latest papers

CardsList

91 papers4 papers in the last four weeks, down 82% on the four weeks before. 0.0% of all new papers.

  1. SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

    Nov 23, 2025Kaidi Wang, Yi He, Wenhao Guan +9TTS SynthesisAudio-Visual Synchronization