Comprehensive audio-visual perception requires both semantic and spatial reasoning. Existing spatial models suffer from poor generalization due to data scarcity, while semantic-only models fail to capture spatial characteristics completely. We introduce a model capable of jointly reasoning over binaural audio and vision on semantic and spatial tasks with improved generalization. To overcome the shortage in real-world data, we develop an automatic pipeline that localizes the sound source in a video, estimates its relative 3D trajectory, and renders binaural audio. We then use this to spatialize VGGSound, allowing us to train at a much larger scale. We demonstrate improved performance and generalization on spatial reasoning tasks while still remaining competitive on semantic tasks compared to semantic-only models.
Figures & tables
Figure 1: Visualization of our automatic spatialization pipeline. Starting from a video with mono audio, we automatically detect the sound source (car in red ) using SAMA-AVS [ 19 ] and estimate its relative 3D trajectory with Video Depth Anything [ 3 ] and SAM 2 [ 29 ] . We use that information to render binaural audio from the mono source. As modelled in the output frames, listening to the rendered binaural audio will give the sensation of the car swerving towards and then past the listener.
Figure 2: Comparison of CCStereo [ 4 ] , a binaural generation model’s implicit visual grounding, with our spatialization pipeline’s sound source localization on VGG-Sound Source [ 2 ] . CCStereo is unable to generalize to in-the-wild videos without additional training while our pipeline leverages robust localization models to automatically and accurately spatialize videos at scale. More examples are shown in the Supp.
Figure 3: Qualitative examples of CROC augmentation. By cropping the original frame, we spatially offset objects. This creates challenging negative pairs that teach our model to learn the spatial relationship between binaural audio and vision.
Figure 4: Contrastive learning framework. We mine both semantic and spatial negative pairs.
# Clips
Task
Dataset
Train
Eval
Spatial misalignment detection
VGGSound
181K
4.5K
FAIR-Play
1.5K
115
Visually-guided binaural generation
FAIR-Play
1.5K
115
MUSIC-Stereo
12K
1.5K
Cross-modal retrieval
VGGSound
181K
1.5K
Table 1: Dataset summary. Training and evaluation data. Blue indicates native binaural audio while the rest uses our spatialization pipeline to lift mono audio to binaural.
Figure 5: FAIR-Play benchmark. Example frames with manually annotated segmentation masks of ground truth sound sources.
VGG-SS
FAIR-Play
Model
Train Data
Hori.
Vert.
Hori.
Vert.
Random
–
50.0
50.0
50.0
50.0
CAV-MAE [ 10 ]
AS
52.0
54.2
49.6
52.2
Mono2Binaural [ 7 ]
FP
46.5
45.6
76.5
67.8
CCStereo [ 4 ]
FP,VGGS,FT
70.3
71.0
80.9
77.4
Ours w/o CROC
VGGS
68.7
56.1
42.6
48.7
Table 2: Spatial misalignment detection accuracy (%) on our spatialized VGG-SS and the real-world binaural dataset FAIR-Play. For the training data, AS is AudioSet-2M, FP is FAIR-Play, and VGG is VGGSound. A superscript LP means the dataset was used for linear probing, FT for finetuning, and S refers to being spatialized with our pipeline. Bold and underline indicate the best and 2nd best results, respectively.
FAIR-Play
MUSIC-Stereo
Model
Visual Encoder
STFT
ENV
STFT
ENV
Mono-Mono
–
1.939
0.257
1.319
0.202
Mono2Binaural [ 7 ]
ResNet18
0.864
0.135
0.842
0.126
APNet [ 41 ]
ResNet18
0.818
0.131
0.824
0.124
Mono2Binaural [ 7 ]
Ours (w/o CROC)
0.861
0.133
0.832
0.125
Mono2Binaural [ 7 ]
Ours
0.819
0.130
0.804
0.123
Table 3: Comparison of visually-guided binaural generation performance on FAIR-Play and MUSIC-Stereo. We report STFT and Envelope (ENV) distance ( ↓ ). We take existing frameworks and swap the original visual encoder to our spatially aware one. Bold and underline indicate the best and 2nd best results, respectively.
Audio → Vision
Vision → Audio
Model
Train Data
Recall@1
Recall@5
Recall@10
Recall@1
Recall@5
Recall@10
CAV-MAE [ 10 ]
AS
15.4
33.8
42.8
17.4
35.9
45.8
CAV-MAE [ 10 ]
AS,VGG
20.3
43.9
55.1
20.2
45.8
56.9
ImageBind [ 9 ]
++
18.7
39.8
49.1
19.9
40.5
49.1
LanguageBind [ 42 ]
++
6.2
19.6
29.6
8.8
26.8
36.7
Ours (mono, w/ CROC)
VGG
16.0
38.4
50.1
18.7
43.8
54.8
Table 4: Semantic cross-modal retrieval results on VGGSound, reported as percentages (%). Bold and underline indicate the best and 2nd best results, respectively. ++ indicates Internet-scale training data.
Figure 6: Top six samples from a representative cluster after agglomerative clustering of our model’s FAIR-Play visual embeddings, illustrating both semantic (keyboard instruments) and spatial (sources on the right side of the frame) consistency.
Figure 7: The effect of the % of the height or width cropped from a frame to offset objects, on our model’s accuracy for predicting spatial misalignment. A larger offset makes it easier to detect.
Figure 8: Architecture diagram of our model. We initialize the encoders for each modality from pretrained checkpoints and project the output features to a common 512-dimensional space. We mean-pool over space and time to get a single embedding vector per modality and apply our spatial and semantic losses.
Figure 9: Screenshot of the Labelbox video annotation interface used by our annotators to label FAIR-Play videos with ground truth sound sources. Keypoints are placed on the sound source (guitar) and these points are used as input to SAM 2 to get segmentation masks across all frames. Annotators also answer how many sound sources are depicted in each video.
Figure 10: Additional clustering results for FAIR-Play. a) Our audio-visual representation is spatially aware, leading to examples with consistent locations of the sound source in each cluster. b) The bottom cluster also demonstrates semantic consistency, with all sound sources being keyboard instruments.
Figure 11: Additional comparisons of sound source localization between CCStereo and our spatialization pipeline. Correct localizations are required to generate spatially accurate binaural audio.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.
Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.
Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4
POSTECH · KAIST · Hankuk University of Foreign Studies +1
Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities. Existing benchmarks often rely on simulated scenes, evaluate isolated spatial skills, and provide limited diagnostic insight into failure modes. We introduce SAVU-Bench, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. We further introduce SAVU-Diag, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, we introduce SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit. While SAVU-EA substantially improves spatial grounding and joint matching, high-level spatial reasoning remains challenging. Our findings highlight the urgent need for both robust spatial audio perception and deeper integration of cross-modal spatial relations.
Yu Chen, Ruihang Liu, Yangguang Xu +5
University of Western Australia · University of Science and Technology Beijing · Monash University
Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83% average semantic accuracy across the four levels, compared with 37.28% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.
Zhi Zeng, Cheng Zhang, Zesheng Yang +9
Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2