Audio-Visual Alignment

Audio-visual alignment studies how to match audio and video in time and meaning, such as linking speech to lip movements, sounds to visible sources, and music to edits. Methods include contrastive learning, cross-attention, optimal transport, and temporal alignment losses for recognition and localization, alongside diffusion and flow-matching models for synchronized dubbing and video-to-audio generation. Researchers evaluate these capabilities on tasks and benchmarks such as LRS3 speech recognition, audio-visual segmentation, and video-to-audio generation. Models are improving on defined tasks, but still struggle with fine-grained synchronization, conflicting cues, and reliable evaluation; stronger visual conditioning can also improve alignment while reducing control over the intended audio.