Audio-Visual Synchronization
Overview
The field studies temporal and semantic correspondence between audio and video, from matching speech to lip movements to aligning music with dance or sounds with visible events. Methods include contrastive alignment and offset estimation, phoneme- and event-level timing, and cross-attention or diffusion models for generating synchronized media. Benchmarks and learned evaluators measure timing, semantic consistency, and human-perceived quality, though these measures can disagree. A central challenge is achieving fine-grained, human-aligned synchronization in real-time and long-form settings without sacrificing fidelity, identity, or controllability.
Latest papers
66 papers13 papers in the last four weeks, up 160% on the four weeks before. 0.1% of all new papers.