AudioSphere: Towards Self-Supervised Spatial Audio Representation Models
Organizations: Donders Institute, Radboud University, Nijmegen, the Netherlands · Mortimer B Zuckerman Institute, Columbia University, New York, United States
Abstract
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior self-supervised audio embedding models on clip-level classification tasks spanning environmental sound, speech, and music, and on frame-level sound event detection. All code and data are available at https://github.com/labhamlet/AudioSphere and https://huggingface.co/labhamlet.
Figures & tables
| Name | Motion | Impulse Response | Max Overlap | Noise |
|---|---|---|---|---|
| TUT-2018 Real (ov1, 2, 3) ( Adavanne et al., 2019a ) | Static | Real | 1, 2, 3 | Ambient 30 dB |
| TAU-2019 ( Adavanne et al., 2019b ) | Static | Real | 2 | Ambient 30 dB |
| TAU-2020 ( Politis et al., 2020a ) | Dynamic | Real | 2 | Ambient [30-6 dB] |
| TAU-2021 ( Politis et al., 2021 ) | Dynamic | Real | 3 | Dynamic + Ambient |
| STARSS23 ( Shimada et al., 2023 ) | Dynamic | – | 6 | Dynamic + Ambient |
| Model | Corpus | HEAR |
| BEATs | AS | 76.6 |
| SSAM | AS | 79.6 |
| ATST-Clip | AS | 79.8 |
| MW-MAE | AS | 80.0 |
| ATST-Frame | AS | 81.9 |
| Dasheng | Mix (272k h) | 82.8 |
| Method | Overlap 1 | Overlap 2 | Overlap 3 | |||
| SED | ER / F | ER / F | ER / F | |||
| SELDNet ( Adavanne et al., 2019a ) | 0.40 / | 60.3 | 0.49 / | 53.1 | 0.53 / | 51.1 |
| MSEDnet ( Adavanne et al., 2017 ) | 0.35 / | 66.2 | 0.38 / | 61.6 | 0.41 / | 59.5 |
| SEDnet ( Adavanne et al., 2017 ) | 0.38 / | 64.6 | 0.42 / | 61.5 | 0.43 / | 57.2 |
| AudioSphere † | 0.29 0.01 / | 82.1 0.8 | 0.38 0.00 / | 74.7 0.3 | 0.42 0.01 / | 71.1 0.2 |
| DOA | LE ∘ / LR | LE ∘ / LR | LE ∘ / LR | |||
| SED | DOA | |||
| Method | ER / F | LE ∘ / LR | ||
| Supervised | ||||
| SELDNet | 0.34 / | 79.9 | 28.5 / | 85.4 |
| Self-supervised (frozen) | ||||
| BAT | 0.92 / | 19.3 | 57.5 / | 38.0 |
| w2v-SELD | 0.56 / | 65.2 | 80.6 / | 58.4 |
| TAU-2020 | TAU-2021 | ||||||||
| Method | ER | F | LE | LR | ER | F | LE | LR | |
| Supervised baseline | |||||||||
| SELDNet | 0.72 | 37.4 | 22.8 | 60.7 | 0.73 | 30.7 | 24.5 | 40.5 | |
| Probing (frozen backbone) | |||||||||
| Frozen + MLP | 0.57 | 46.6 | 23.5 | 63.2 | 0.75 | 21.0 | 36.9 | 49.1 | |
| Frozen + TM | 0.41 | 63.6 | 15.0 | 75.0 | 0.63 | 37.2 | 24.3 | 61.3 | |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | TAU2020 | TAU2021 | STARSS23 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ER | F | LE | LR | ER | F | LE | LR | ER | F | LE | LR | |
| Baseline (SELDNet) | ||||||||||||
| Baseline FOA | 0.72 | 37.4 | 22.8 | 60.7 | 0.73 | 30.7 | 24.5 | 40.5 | 0.57 | 29.9 | 22.0 | 47.7 |
| Baseline ASM-FOA | 0.77 | 32.2 | 28.0 | 58.3 | 0.72 | 20.5 | 29.4 | 39.7 | 0.62 | 27.6 | 26.3 | 47.5 |
| AudioSphere (frozen) | ||||||||||||
| Frozen + TM, FOA | 0.41 | 63.6 | 15.0 | 75.0 | 0.63 | 37.2 | 24.3 | 61.3 | 0.56 | 37.4 | 22.8 | 58.8 |
| Pipeline | Median DoA | Folded |
|---|---|---|
| BSM + Spatial-AST (BAT) | 13.1 | 0 4.3 |
| AudioSphere (binaural FOA) | 0 3.4 | 0 2.1 |
| ASM + AudioSphere (Table 6 ) | 0 1.9 | – |
| ESC-50 Acc | ER | F | LE | LR | |
| Channel-axis masking / reconstruction target (mask-infilling regime) | |||||
| Tube masking † | |||||
| Per-channel masking † | |||||
| W IVs XYZ † | |||||
| Mels IVs † | |||||
| Masking pattern (drop regime, mask ratio ) | |||||
| ER | F | LE ∘ | LR | |
|---|---|---|---|---|
| Model | ER |
|---|---|
| Supervised | |
| SELDNet | 0.34 |
| AudioSphere | |
| Multi-label classification | 0.40 |
| ACCDOA | 0.28 |
| Method | ER | F | LE ∘ | LR |
|---|---|---|---|---|
| AudioSphere -Clean | 0.29 | 81.9 | 73.0 | 64.3 |
| AudioSphere -W | 0.29 | 82.2 | 73.2 | 65.2 |
| AudioSphere (W dup.) | 0.33 | 79.9 | 73.5 | 62.5 |
| AudioSphere | 0.24 | 85.7 | 14.9 | 77.0 |
| Abbreviation | Task Name | Description | Type |
|---|---|---|---|
| DCASE | DCASE-2016 Task 2 [ Mesaros et al., 2018 ] | Event detection of overlapping office sounds in synthetic mixtures | Scene Analysis |
| FSD50K | FSD50k [ Fonseca et al., 2022 ] | Multilabel, large scale audio tagging | Scene Analysis |
| LC | LibriCount [ Stöter et al., 2018 ] | Speaker Count Identification, Simulated Cocktail Party | Scene Analysis |
| ESC-50 | ESC-50 [ Piczak, ] | Environmental Sound Classification | Environmental Sound Classification |
| CD | Crema-D [ Cao et al., 2014 ] | Emotion Recognition | Speech Analysis |
| VL | VoxLingua107 Top10 [ Valk and Alumäe, 2021 ] | Spoken language identification | Speech Analysis |
| Acoustic Events and Scene Analysis | Speech | Music | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Corpus | DCASE | FSD50K | LC | ESC-50 | CD | VL | SC-5 | NS | BO | Mri-S | Mri-T | mean |
| Wav2Vec 2.0 | LS | 23.5 | 29.4 | 85.3 | 17.4 | 55.7 | |||||||
| HuBERT | LS | 78.3 | 32.8 | 94.0 | 19.8 | 68.7 | |||||||
| WavLM | LS | 27.0 | 25.7 | 93.8 | 18.2 | 59.1 | |||||||
| MAE | AS | – | 33.4 | 66.6 | 63.6 | 65.5 | |||||||
| SSAST | AS + LS | – | 21.4 | 22.0 | 64.2 | 56.2 | |||||||
| Model | Objective / supervision | Pre-training corpus (scale) | Native input | Eval-time conversion | Downstream protocol | Results |
|---|---|---|---|---|---|---|
| Wav2Vec 2.0, HuBERT, WavLM (Base) | SSL, masked prediction | LibriSpeech (960 h) | mono waveform | none | HEAR MLP probe | reproduced (HEAR API) |
| MAE, SSAST, BEATs, MW-MAE, SSAM, ATST | SSL, masked autoencoding / latent prediction | AudioSet ( 4.8–5.8 k h; SSAST also LibriSpeech) | mono spectrogram | none | HEAR MLP probe | reproduced (HEAR API) |
| Dasheng | SSL, masked autoencoding | Mixed ( 272 k h) | mono spectrogram | none | HEAR MLP probe | reproduced (HEAR API) |
| PaSST | supervised (AudioSet labels) | AudioSet | mono spectrogram | none | HEAR MLP probe | reproduced (HEAR API) |
| Spatial-AST (BAT) | supervised (event, DoA, distance labels) | AudioSet + simulated binaural RIRs | binaural | BSM (TAU-2019, RSL2019-dev) | HEAR MLP probe (HEAR, RealSELD , RSL2019-dev) | reproduced |
| ELSA | contrastive (audio–text); supervised spatial-branch pre-training | Spatially augmented Clotho + AudioCaps + Freesound ( 4.7 k h); Spatial LibriSpeech | FOA | – (model not public) | – | cited (TUT-2018 ov. 1 LE) † |
| Dataset | ER | F | LE ∘ | LR |
|---|---|---|---|---|
| Static scenes (ER, F, LE, LR) | ||||
| TUT-2018 Real, ov. 1 | ||||
| TUT-2018 Real, ov. 2 | ||||
| TUT-2018 Real, ov. 3 | ||||
| TAU-2019 | ||||
| Dynamic scenes (ER , F , LE , LR ) | ||||
| Class | ||
|---|---|---|
| Fem. Speech | 0.52 | 25.27 |
| Male Speech | 0.63 | 15.17 |
| Clap | 0.20 | 18.19 |
| Phone | 0.01 | 36.21 |
| Laugh | 0.36 | 17.63 |
| Domestic sounds | 0.30 | 20.54 |
| Class | (MLP temporal) | (MLP temporal) |
|---|---|---|
| Door | 0.00 0.30 | 180.0 11.4 |
| Bell | 0.00 0.17 | 180.0 31.7 |
| Footsteps | 0.01 0.12 | 0 37.5 27.1 |
| Phone | 0.01 0.16 | 0 36.2 26.8 |
| Parameter | Value | Parameter | Value | |
|---|---|---|---|---|
| directSHOrder | 3 | indirectSHOrder | 3 | |
| sampleRate | 32000 | frequencyBands | 8 | |
| maxDiffractionOrder | 10 | transmission | True | |
| indirect | True | indirectRayCount | 15000 | |
| indirectRayDepth | 400 | sourceRayCount | 200 | |
| sourceRayDepth | 20 | threadCount | 16 |
| Parameter | Value |
|---|---|
| Encoder | ViT-Base (12 layers, 768 dim, 12 heads) |
| Decoder | 8 layers, local-global attention |
| Patch size | (channels time freq) |
| Embedding dim | 768 |
| Mask ratio | 80% (tube masking) |
| Segment length | 2 s (200 frames) |
Audio Representation Learning
Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.
See all 222 papers