SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection
Authors: Runbang Wang, Zining Liang, Yin Cao, Qiuqiang Kong
Organizations: Nanjing University, China · The Chinese University of Hong Kong, Hong Kong SAR, China · Shun Hing Institute of Advanced Engineering (SHIAE), Hong Kong SAR, China · Institute of Acoustics, Chinese Academy of Sciences, China
In daily life, people hear speech, footsteps, and music around them. We can often recognize these sounds and judge where they come from. Each sound source can be shown on a separate acoustic map, a rectangular image covering 360∘ horizontally and 180∘ vertically. The map shows the directions occupied by the source as a region and the sound energy within that region. A class label identifies the sound. Predicting these labeled acoustic maps from audio is called semantic acoustic imaging. Such maps could help robots perceive their surroundings and allow augmented reality displays to show sound regions and classes over the real world. Existing models can recognize sound classes and estimate a direction for each source. However, a direction alone does not describe the source region or its energy. Acoustic imaging must also distinguish sound sources in nearby directions, while the number of active sources and the regions they occupy can change over time. We therefore propose the Semantic Acoustic Imaging Detector (SAID), which predicts a separate labeled acoustic map for each active source from audio. First, we pretrain Audio2Sph, SAID's audio encoder, through sound energy estimation across directions without class labels. Then, we train the complete SAID model to predict source regions, energy, and classes together. We also develop a pipeline that generates simulated recordings for pretraining and supports fine-tuning on real recordings. On the official DCASE2026 Task 3 Track A evaluation set, our submitted system ranks first with 0.1080 macro-averaged mean average precision (Macro mAP) and 0.3962 Macro Pearson r. Demos and code are provided at https://github.com/IN03X/SAID.
Figures & tables
Fig. 2: Audio2Sph maps time–frequency features onto a directional grid through learned queries. The panoramic decoder predicts a class-agnostic map only during pretraining.
Fig. 3: Sph2Imaging updates slot queries using multi-scale spatial features and PaSST features. The Map Head predicts a map for each slot; the Active Head and Class Head estimate activity and class, respectively. The Refine Block produces higher-resolution maps for the final output.
Prediction
Macro mAP
Macro Pearson r
Max. JSON
Avg. JSON
Original
0.1202
0.4268
941.50
369.58
Compressed
0.1177
0.4487
18.70
8.08
Table 1: Prediction compression on the full-recording development test set (78 recordings). JSON sizes are in decimal MB per recording.
Rank
Team
Macro mAP
Macro Pearson r
1
CUHK (SAID)
0.1080
0.3962
2
Medisensing
0.0662
0.2765
3
Wuhan University
0.0091
0.3185
4
Samsung Electronics
0.0215
0.1394
5
Deka
0.0001
0.0274
Table 2: Official DCASE2026 Task 3 Track A results, using each team’s best-ranked submission [ 1 , 35 ] . Both metrics are macro-averaged; higher scores are better.
Variant
Macro mAP
Macro Pearson r
Mask AP
Macro Class-F1
SAID (PaSST)
0.1202
0.4268
0.2380
0.3885
SAID (AudioMAE)
0.1134
0.4307
0.2287
0.3951
SAID (None)
0.0260
0.4224
0.2293
0.1347
w/o Audio2Sph pretraining
0.0000
0.0510
0.0000
0.3785
Table 3: Ablations on the development test set. Mask AP is class-agnostic; Macro Class-F1 uses only spatial matches within 20∘ . Higher scores are better.
Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.
Sound events have semantic identities, locations, and trajectories, yet existing audio-language models typically encode clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To bridge this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and QA about what is sounding, where it is, how it moves, and how sources relate. We further introduce ST-Audio Encoder, which jointly learns event semantics and time-resolved source trajectories, and ST-AudioLM, which exposes these audio tokens to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the balance between semantic preservation and localization and yields stronger reasoning than static spatial and localization-oriented baselines. Beyond the controlled benchmark, ST-AudioLM shows zero-shot transfer to real recordings, with further gains after light adaptation. Project page: https://st-audiolm.github.io/.
Spatial audio large language models (LLMs) enable embodied agents, wearable assistants, and immersive systems to recognize sound events, localize sources, and reason about their spatial relationships. However, existing spatial audio LLMs often rely on early fusion of acoustic and spatial features and source-agnostic token representations. These designs make it difficult to preserve the correspondence between individual sound events and their spatial attributes, particularly in multi-source scenes. To address this limitation, we propose SAIL, a Spatial Audio Intelligence framework with LLMs that preserves acoustic-spatial structure and source-level correspondence from audio encoding to LLM alignment. SAIL introduces a Disentangled Spatial Audio Transformer that represents Mel-spectrogram and interaural phase difference features as separate acoustic and spatial streams. Source-discriminative task queries further learn event, direction, and distance information for each source. A Dual-Stream Q-Former then aligns the two streams with the LLM using acoustic and spatial queries organized by source slots. Compared with the early-fusion baseline, SAIL achieves consistent improvements in dual-source sound event detection, direction and distance estimation, and spatial reasoning. These results demonstrate the importance of structured, source-discriminative audio representations for multi-source spatial understanding and reasoning.
Zhengding Luo, Jinyang Wu, Haozhe Ma +3
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Singapore Management University · Tencent Hy Frontier Lab, Singapore +2