Audio-Visual Representation Learning

Audio-visual representation learning studies how models can encode sound and video so that information in one modality helps interpret the other. Common approaches include contrastive alignment, masked or predictive self-supervision, cross-modal distillation, and attention-based fusion, with settings ranging from speech recognition and sound-event localization to video retrieval and spatial grounding. Evaluation spans benchmarks such as AudioSet, VGGSound, AVE, AVSBench, and LRS3, as well as tests with noisy, missing, mismatched, or unseen inputs. A central tension is how to learn broad shared semantics without losing the fine-grained timing and spatial detail needed for reliable predictions.