cs.CVNov 14, 2026

Learning Spatially Aware Audio-Visual Features from Videos with Mono Audio

Authors: Mengyu Yang, You Zhang, Vishnu Raj, Olha Townsend, James Hays, Andrea Fanelli, Gautam Bhattacharya

Organizations: Georgia Institute of Technology · Dolby Laboratories

Abstract

Comprehensive audio-visual perception requires both semantic and spatial reasoning. Existing spatial models suffer from poor generalization due to data scarcity, while semantic-only models fail to capture spatial characteristics completely. We introduce a model capable of jointly reasoning over binaural audio and vision on semantic and spatial tasks with improved generalization. To overcome the shortage in real-world data, we develop an automatic pipeline that localizes the sound source in a video, estimates its relative 3D trajectory, and renders binaural audio. We then use this to spatialize VGGSound, allowing us to train at a much larger scale. We demonstrate improved performance and generalization on spatial reasoning tasks while still remaining competitive on semantic tasks compared to semantic-only models.

Figures & tables

Explore similar work

CardsList
  1. Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

    Oct 15, 2026Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4Audio-Visual UnderstandingMultimodal Large Language Models

  2. SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

    Oct 7, 2026Yu Chen, Ruihang Liu, Yangguang Xu +5Audio-Visual UnderstandingSpatial Reasoning Benchmarks

  3. Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

    Aug 10, 2026Zhi Zeng, Cheng Zhang, Zesheng Yang +9Unified Multimodal ModelsAudio-Visual QA