Bearings: Self-Supervised Soundfield Embeddings from First-Order Ambisonics
Authors: Goksenin Yuksel, Marcel van Gerven, Kiki van der Heijden
Organizations: Dept. of Machine Learning and Neural Computing, Donders Institute, Radboud University, Nijmegen, the Netherlands · Mortimer B. Zuckerman Institute, Columbia University, New York, USA
Recently proposed self-supervised audio encoders learn powerful general-purpose representations of sound scenes, yet they are spatially blind. To supply the missing spatial representation of sound scenes, we introduce Bearings. Bearings is a self-supervised framework that learns soundfield embeddings from unlabeled first-order Ambisonics. We pre-train a masked auto-encoder paired with a decoder conditioned on frozen acoustic embeddings from an off-the-shelf single-channel audio encoder. Our results show that the resulting soundfield embeddings form a reusable stream that can be attached to frozen acoustic encoders with a lightweight trainable fusion head. On sound event localization and detection, concatenating our soundfield embeddings with acoustic representations provides the missing spatial information and enables joint detection and localization, raising the location-dependent F-score from below 4 to 50 on TAU-NIGENS 2021 and 39 on STARSS23. To our knowledge, Bearings is the first self-supervised soundfield encoder whose embeddings plug into frozen acoustic encoders without retraining either model.
Figures & tables
Figure 1: Bearings overview. (A) Self-supervised pre-training: a masked ViT encodes seven-channel FOA features (four log-mel spectrograms, three active-intensity vectors), and a decoder that cross-attends to embeddings from a frozen mono encoder reconstructs direction distributions and two-scale diffuseness at the masked patches. (B) Downstream fusion: the frozen Bearings encoder supplies soundfield embeddings that are concatenated with acoustic embeddings from any frozen single-channel acoustic encoder.
TAU-NIGENS 2021
STARSS23
Backbone
Method
ER20∘↓
F20∘↑
LECD↓
LRCD↑
ER20∘↓
F20∘↑
LECD↓
LRCD↑
SELDNet (7-ch FOA + IVs) [ 12 , 15 ]
0.69
33.9
24.1
43.9
0.57
29.9
22.0
47.7
GRAM-Ambisonics (7-ch FOA + IVs) [ 19 ]
0.62
41.0
23.3
66.3
0.56
37.4
22.8
58.8
GRAM-Clean
Single-channel, acoustic only
0.83 [0.81,0.84]
2.8 [2.3,3.3]
102.1 [99.3,104.9]
34.7 [33.0,36.2]
0.83 [0.81,0.86]
2.5 [1.6,3.4]
147.8 [146.2,149.3]
16.6 [15.0,18.2]
+ Bearings (ours)
0.55 [0.53,0.56]
50.4 [48.2,52.7]
19.3 [18.3,20.3]
66.8 [65.0,68.5]
0.51 [0.46,0.55]
38.9 [34.6,43.8]
22.0 [18.3,25.7]
55.5 [49.3,61.4]
Dasheng-Base
Single-channel, acoustic only
0.83 [0.82,0.85]
3.2 [2.6,3.9]
87.1 [84.1,90.3]
43.2 [41.5,45.0]
0.82 [0.80,0.86]
2.9 [2.3,3.5]
146.1 [144.6,147.5]
18.9 [17.1,20.7]
Table 1: Downstream SELD performance. Bold marks the best value per column.
Pretrained spatial audio encoders are increasingly used as general-purpose representations for perceptual tasks, yet their spatial encoding capabilities remain poorly understood. We introduce the Spatial Audio Representation Learning (SARL) benchmark, a controlled framework for evaluating spatial information in pretrained audio models. SARL probes source-level factors (azimuth, elevation, distance, class) and room-level factors (RT60, volume, shape). Experiments across diverse encoders reveal three patterns: input configuration and training paradigm shape spatial encoding; source factors are consistently easier to decode than room factors; and sensitivity analysis under controlled perturbations shows heterogeneous responses to source and room variation. These results reveal systematic biases in current pretrained audio representations. SARL is released as an open-source benchmark for reproducible evaluation of spatial audio representations.
Chuyang Chen, Sivan Ding, Adrian S. Roman +1
Music and Audio Research Laboratory, New York University, USA
Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.
Sonal Kumar, Sinan Hersek, Artem Dementyev +6
University of Maryland, College Park · Google · Google DeepMind
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.