Audio-Visual Understanding

Audio-visual understanding asks how AI can use sound and images or video together to identify speech, people, sound sources, events, spatial relations, and social interactions. Methods range from cross-attention and contrastive fusion to audio-visual segmentation, source localization, and multimodal language models that use temporal reasoning, memory, or tools. Benchmarks such as AVE, AVSBench, and VGGSound test event recognition and localization, while newer long-video and streaming evaluations probe evidence tracking across time. Models have improved on many task-specific benchmarks, but reliable audio use, temporal grounding, and robustness to noise, missing or conflicting modalities, and long contexts remain open challenges.