Comprehensive audio-visual perception requires both semantic and spatial reasoning. Existing spatial models suffer from poor generalization due to data scarcity, while semantic-only models fail to capture spatial characteristics completely. We introduce a model capable of jointly reasoning over binaural audio and vision on semantic and spatial tasks with improved generalization. To overcome the shortage in real-world data, we develop an automatic pipeline that localizes the sound source in a video, estimates its relative 3D trajectory, and renders binaural audio. We then use this to spatialize VGGSound, allowing us to train at a much larger scale. We demonstrate improved performance and generalization on spatial reasoning tasks while still remaining competitive on semantic tasks compared to semantic-only models.
Figures & tables
Figure 1: Visualization of our automatic spatialization pipeline. Starting from a video with mono audio, we automatically detect the sound source (car in red ) using SAMA-AVS [ 19 ] and estimate its relative 3D trajectory with Video Depth Anything [ 3 ] and SAM 2 [ 29 ] . We use that information to render binaural audio from the mono source. As modelled in the output frames, listening to the rendered binaural audio will give the sensation of the car swerving towards and then past the listener.
Figure 2: Comparison of CCStereo [ 4 ] , a binaural generation model’s implicit visual grounding, with our spatialization pipeline’s sound source localization on VGG-Sound Source [ 2 ] . CCStereo is unable to generalize to in-the-wild videos without additional training while our pipeline leverages robust localization models to automatically and accurately spatialize videos at scale. More examples are shown in the Supp.
Figure 3: Qualitative examples of CROC augmentation. By cropping the original frame, we spatially offset objects. This creates challenging negative pairs that teach our model to learn the spatial relationship between binaural audio and vision.
Figure 4: Contrastive learning framework. We mine both semantic and spatial negative pairs.
# Clips
Task
Dataset
Train
Eval
Spatial misalignment detection
VGGSound
181K
4.5K
FAIR-Play
1.5K
115
Visually-guided binaural generation
FAIR-Play
1.5K
115
MUSIC-Stereo
12K
1.5K
Cross-modal retrieval
VGGSound
181K
1.5K
Table 1: Dataset summary. Training and evaluation data. Blue indicates native binaural audio while the rest uses our spatialization pipeline to lift mono audio to binaural.
Figure 5: FAIR-Play benchmark. Example frames with manually annotated segmentation masks of ground truth sound sources.
VGG-SS
FAIR-Play
Model
Train Data
Hori.
Vert.
Hori.
Vert.
Random
–
50.0
50.0
50.0
50.0
CAV-MAE [ 10 ]
AS
52.0
54.2
49.6
52.2
Mono2Binaural [ 7 ]
FP
46.5
45.6
76.5
67.8
CCStereo [ 4 ]
FP,VGGS,FT
70.3
71.0
80.9
77.4
Ours w/o CROC
VGGS
68.7
56.1
42.6
48.7
Table 2: Spatial misalignment detection accuracy (%) on our spatialized VGG-SS and the real-world binaural dataset FAIR-Play. For the training data, AS is AudioSet-2M, FP is FAIR-Play, and VGG is VGGSound. A superscript LP means the dataset was used for linear probing, FT for finetuning, and S refers to being spatialized with our pipeline. Bold and underline indicate the best and 2nd best results, respectively.
FAIR-Play
MUSIC-Stereo
Model
Visual Encoder
STFT
ENV
STFT
ENV
Mono-Mono
–
1.939
0.257
1.319
0.202
Mono2Binaural [ 7 ]
ResNet18
0.864
0.135
0.842
0.126
APNet [ 41 ]
ResNet18
0.818
0.131
0.824
0.124
Mono2Binaural [ 7 ]
Ours (w/o CROC)
0.861
0.133
0.832
0.125
Mono2Binaural [ 7 ]
Ours
0.819
0.130
0.804
0.123
Table 3: Comparison of visually-guided binaural generation performance on FAIR-Play and MUSIC-Stereo. We report STFT and Envelope (ENV) distance ( ↓ ). We take existing frameworks and swap the original visual encoder to our spatially aware one. Bold and underline indicate the best and 2nd best results, respectively.
Audio → Vision
Vision → Audio
Model
Train Data
Recall@1
Recall@5
Recall@10
Recall@1
Recall@5
Recall@10
CAV-MAE [ 10 ]
AS
15.4
33.8
42.8
17.4
35.9
45.8
CAV-MAE [ 10 ]
AS,VGG
20.3
43.9
55.1
20.2
45.8
56.9
ImageBind [ 9 ]
++
18.7
39.8
49.1
19.9
40.5
49.1
LanguageBind [ 42 ]
++
6.2
19.6
29.6
8.8
26.8
36.7
Ours (mono, w/ CROC)
VGG
16.0
38.4
50.1
18.7
43.8
54.8
Table 4: Semantic cross-modal retrieval results on VGGSound, reported as percentages (%). Bold and underline indicate the best and 2nd best results, respectively. ++ indicates Internet-scale training data.
Figure 6: Top six samples from a representative cluster after agglomerative clustering of our model’s FAIR-Play visual embeddings, illustrating both semantic (keyboard instruments) and spatial (sources on the right side of the frame) consistency.
Figure 7: The effect of the % of the height or width cropped from a frame to offset objects, on our model’s accuracy for predicting spatial misalignment. A larger offset makes it easier to detect.
Figure 8: Architecture diagram of our model. We initialize the encoders for each modality from pretrained checkpoints and project the output features to a common 512-dimensional space. We mean-pool over space and time to get a single embedding vector per modality and apply our spatial and semantic losses.
Figure 9: Screenshot of the Labelbox video annotation interface used by our annotators to label FAIR-Play videos with ground truth sound sources. Keypoints are placed on the sound source (guitar) and these points are used as input to SAM 2 to get segmentation masks across all frames. Annotators also answer how many sound sources are depicted in each video.
Figure 10: Additional clustering results for FAIR-Play. a) Our audio-visual representation is spatially aware, leading to examples with consistent locations of the sound source in each cluster. b) The bottom cluster also demonstrates semantic consistency, with all sound sources being keyboard instruments.
Figure 11: Additional comparisons of sound source localization between CCStereo and our spatialization pipeline. Correct localizations are required to generate spatially accurate binaural audio.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.