Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.
Figures & tables
Figure 1: Overview of the proposed Generation-Analysis-Refinement Sound Source Localization (GAR-SSL) framework. Given an image-audio pair, the model performs three meta-reasoning steps: Generation produces an initial bounding box and audio label, Analysis evaluates Audio-Visual Consistency through role-based reasoning, and Refinement adjusts the localization to obtain a fine-grained final bounding box. This process enables explainable and training-free audio-visual localization.
Figure 2: The proposed training-free framework consists of three stages: ( i ) Generation produces initial bounding boxes and audio classifications from image-audio pairs; ( ii ) Analysis evaluates consistency through role tagging, anchor voting, and scoring, repeated N times for consensus; ( iii ) Refinement applies adaptive gating and geometric operations to adjust localization. All operations are performed via MLLMs prompt engineering without training.
Table 1: Comparison of multi-source sound localization methods on VGGSound-Duet and MUSIC-Duet test sets. We evaluate three types of approaches: ( i ) existing vision-based SSL methods trained with task-specific objectives, ( ii ) off-the-shelf MLLMs baselines (Qwen2.5-Omni, MiniCPM-o, InteractiveOmni) without structured reasoning, and ( iii ) our proposed training-free Generation-Analysis-Refinement framework with N iterations in Stage 2 (Analysis). Bold / underlined fonts denote best/second-best performance.
Table 2: Comparison of single-source sound localization methods on VGGSound-Single and MUSIC-Solo test sets. We evaluate three types of approaches: ( i ) existing vision-based SSL methods trained with task-specific objectives, ( ii ) MLLMs baselines (Qwen2.5-Omni, MiniCPM-o, InteractiveOmni) without structured reasoning, and ( iii ) our proposed training-free Generation-Analysis-Refinement framework with N iterations in Stage 2 (Analysis). Bold / underlined fonts denote best/second-best performance.
Figure 3: Visualization results for (a) MUSIC-Duet and (b) VGGSound-Duet test set. We compare our method with OA-SSL [ 40 ] . More comparisons are in the supplementary document.
Figure 4: Visualization results for VGGSound-Single test set. We compare our method with OA-SSL [ 40 ] . More comparisons are in the supplementary document.
Figure S.5 : Visualization of sound source localization results in VGGSound-Single [ P16 ] dataset. Each row represents a different example, and each column shows the original image, Ground Truth (actual sound source location), prediction results of the OA-SSL [ P38 ] method, and prediction results of the proposed method (Ours).
Figure S.6 : Visualization of sound source localization results on the MUSIC-Solo [ P13 ] dataset. Each row shows a different instrument example. From left to right, we present the Ground Truth bounding box, the prediction produced by OA-SSL [ P38 ], and the prediction generated by our proposed method (Ours).
Figure S.7 : Visualization of sound source localization results in VGGSound-Duet [ P16 ] and MUSIC-Duet [ P13 ] datasets. (a) Results of simultaneously localizing two instrumental sound sources in VGGSound-Duet [ P16 ]. (b) Results in more complex multi-sound source environments in MUSIC-Duet [ P13 ]. Each example includes the original image, Ground Truth, OA-SSL [ P38 ] prediction, and the proposed method prediction (Ours).
Prompt:
You are an assistant for audio-visual sound source localization (SSL).
TASK (Stage A):
Given an IMAGE and an AUDIO clip from the same scene:
1) Locate exactly one main sound-emitting object in the image and output its bounding box as [x1,y1,x2,y2] .
2) Provide a concise visual description of the sound-emitting object.
Large-scale pre-trained image-text models exhibit robust multimodal representations, yet applying the Contrastive Language-Image Pre-training (CLIP) model to audio-visual localization remains challenging. Replacing the classification token ([CLS]) with an audio-embedded token ([V_A]) struggles to capture semantic cues, and the prompt "a photo of a [V_A]" fails to establish meaningful connections between audio embeddings and context tokens. To address these issues, we propose Sound-aware Prompt Learning (SOUPLE), which replaces fixed prompts with learnable context tokens. These tokens incorporate visual features to generate conditional context for a mask decoder, effectively bridging semantic correspondence between audio and visual inputs. Experiments on VGGSound, SoundNet, and AVSBench demonstrate that SOUPLE improves localization and segmentation performance.
Khanh Binh Nguyen, Chae Jung Park
Deakin University, Australia · National Cancer Center, South Korea
Sound source localization (SSL) is a critical technology for determining the position of sound sources in complex environments. However, existing methods face challenges such as high computational costs and precise calibration requirements, limiting their deployment in dynamic or resource-constrained environments. This paper introduces a novel 3D SSL framework, which uses sparse cross-attention, pretraining, and adaptive signal coherence metrics, to achieve accurate and computationally efficient localization with fewer input microphones. The framework also supports operational microphones at unknown positions: their recordings remain available and are used to estimate both source and microphone positions. Preliminary experiments demonstrate its scalability for multi-source localization without requiring additional hardware. This work advances SSL by balancing the model's performance and efficiency and improving its robustness for real-world scenarios.
Yiyuan Yang, Shitong Xu, Niki Trigoni +1
Department of Computer Science, University of Oxford, United Kingdom
Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.
Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4
POSTECH · KAIST · Hankuk University of Foreign Studies +1