Audio-Visual QA

QA: Question Answering

Overview

AVQA studies how models use synchronized audio, video, and a question to identify relevant evidence and answer about what happens, what is heard, and how the modalities relate. Common approaches include pretrained audio and video encoders, question-guided attention or cross-modal fusion, and methods that retrieve or temporally ground evidence before reasoning. Benchmarks range from music and sound-event questions to long videos, spatial reasoning, and tests of whether answers truly depend on both modalities. Models are improving, but benchmark shortcuts, cross-modal interference, long-context costs, and reliable spatial and temporal grounding remain open challenges.

Latest papers

31 papers3 papers in the last four weeks, down 62% on the four weeks before. 0.0% of all new papers.

Sep 22, 2025

Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy

Speech and audio encoders developed over years of community effort are routinely excluded from video understanding pipelines, not because they fail, but because benchmarks never required listening. We audit 10 video benchmarks and find items largely solvable from visual cues alone: a single-frame probe answers about 76% of AVQA without audio, suggesting poor measurement of audio-visual reasoning. Building on LLaVA-OneVision, we attach a speech/audio encoder and compare five compressor architectures under 25-fold token reduction (25 Hz to 1 Hz). Across 10 benchmarks, with and without filtering, audio yields clear gains on tasks requiring speech comprehension or cross-modal grounding, while vision-centric suites remain largely unaffected. Our results show that speech encoders play a larger role in video understanding than current benchmarks suggest. We will open-source our work at https://github.com/naver-ai/unimambamia-av.