cs.CVApr 8, 2026
SaveGenerate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning
Organizations: Kyung Hee University
Abstract
Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.
Figures & tables
Figure 1: Overview of the proposed Generation-Analysis-Refinement Sound Source Localization (GAR-SSL) framework. Given an image-audio pair, the model performs three meta-reasoning steps: Generation produces an initial bounding box and audio label, Analysis evaluates Audio-Visual Consistency through role-based reasoning, and Refinement adjusts the localization to obtain a fine-grained final bounding box. This process enables explainable and training-free audio-visual localization.
Figure 2: The proposed training-free framework consists of three stages: ( i ) Generation produces initial bounding boxes and audio classifications from image-audio pairs; ( ii ) Analysis evaluates consistency through role tagging, anchor voting, and scoring, repeated times for consensus; ( iii ) Refinement applies adaptive gating and geometric operations to adjust localization. All operations are performed via MLLMs prompt engineering without training.
| VGGSound-Duet [ 6 ] | MUSIC-Duet [ 51 ] | |||||
| Method | CAP(%) | [email protected](%) | AUC(%) | CAP(%) | [email protected](%) | AUC(%) |
| Vision Model | ||||||
| Attention 10k (CVPR’18) [ 32 ] | – | 11.5 | 15.2 | – | 21.6 | 19.6 |
| OTS (ECCV’18) [ 3 ] | 10.5 | 12.2 | 15.8 | 11.6 | 13.3 | 18.5 |
| DMC (CVPR’19) [ 13 ] | – | 13.8 | 17.1 | – | 17.5 | 21.1 |
| CoarseToFIne (ECCV’20) [ 29 ] | – | 14.7 | 18.5 | – | 17.6 | 20.6 |
Table 1: Comparison of multi-source sound localization methods on VGGSound-Duet and MUSIC-Duet test sets. We evaluate three types of approaches: ( i ) existing vision-based SSL methods trained with task-specific objectives, ( ii ) off-the-shelf MLLMs baselines (Qwen2.5-Omni, MiniCPM-o, InteractiveOmni) without structured reasoning, and ( iii ) our proposed training-free Generation-Analysis-Refinement framework with iterations in Stage 2 (Analysis). Bold / underlined fonts denote best/second-best performance.
| VGGSound-Single [ 6 ] | MUSIC-Solo [ 51 ] | |||||
| Method | AP(%) | [email protected](%) | AUC(%) | AP(%) | [email protected](%) | AUC(%) |
| Vision Model | ||||||
| Attention 10k (CVPR’18) [ 32 ] | – | 19.2 | 30.6 | – | 37.2 | 38.7 |
| OTS (ECCV’18) [ 3 ] | 29.8 | 32.8 | 35.7 | 69.3 | 26.1 | 35.8 |
| DMC (CVPR’19) [ 13 ] | – | 23.9 | 27.6 | – | 29.1 | 38.0 |
| CoarseToFIne (ECCV’20) [ 29 ] | 28.2 | 29.1 | 34.8 | 70.7 | 33.6 | 39.8 |
Table 2: Comparison of single-source sound localization methods on VGGSound-Single and MUSIC-Solo test sets. We evaluate three types of approaches: ( i ) existing vision-based SSL methods trained with task-specific objectives, ( ii ) MLLMs baselines (Qwen2.5-Omni, MiniCPM-o, InteractiveOmni) without structured reasoning, and ( iii ) our proposed training-free Generation-Analysis-Refinement framework with iterations in Stage 2 (Analysis). Bold / underlined fonts denote best/second-best performance.
Figure 3: Visualization results for (a) MUSIC-Duet and (b) VGGSound-Duet test set. We compare our method with OA-SSL [ 40 ] . More comparisons are in the supplementary document.
Figure 4: Visualization results for VGGSound-Single test set. We compare our method with OA-SSL [ 40 ] . More comparisons are in the supplementary document.
Figure S.5 : Visualization of sound source localization results in VGGSound-Single [ P16 ] dataset. Each row represents a different example, and each column shows the original image, Ground Truth (actual sound source location), prediction results of the OA-SSL [ P38 ] method, and prediction results of the proposed method (Ours).
Figure S.6 : Visualization of sound source localization results on the MUSIC-Solo [ P13 ] dataset. Each row shows a different instrument example. From left to right, we present the Ground Truth bounding box, the prediction produced by OA-SSL [ P38 ], and the prediction generated by our proposed method (Ours).
Figure S.7 : Visualization of sound source localization results in VGGSound-Duet [ P16 ] and MUSIC-Duet [ P13 ] datasets. (a) Results of simultaneously localizing two instrumental sound sources in VGGSound-Duet [ P16 ]. (b) Results in more complex multi-sound source environments in MUSIC-Duet [ P13 ]. Each example includes the original image, Ground Truth, OA-SSL [ P38 ] prediction, and the proposed method prediction (Ours).
| Prompt: |
| You are an assistant for audio-visual sound source localization (SSL). |
| TASK (Stage A): |
| Given an IMAGE and an AUDIO clip from the same scene: |
| 1) Locate exactly one main sound-emitting object in the image and output its bounding box as . |
| 2) Provide a concise visual description of the sound-emitting object. |
Table S.10 : Stage 1 (Generation) Prompt: Audio-Visual Localization
| Prompt: |
| You are an audio classification expert. |
| TASK (Stage B): |
| Listen to the AUDIO and classify the dominant audio event using a short, lowercase class name |
| (e.g., “violin”, “piano”, “dog barking”, “engine”, “drum set”). |
| You must also provide a confidence score in the range . |
Table S.11 : Stage 1 (Generation) Prompt: Audio Classification
| Prompt: |
| You must verify whether the sound suggested by the AUDIO is actually visibly supported within the IMAGE. |
| You must rely only on the given image–audio pair and must not hallucinate unseen content. |
| Context: |
| - previous_bbox |
| - audio_class |
Table S.12 : Stage 2 (Analysis) Prompt
| Prompt: |
| You refine the bounding box of the main sound-emitting object by integrating IMAGE, AUDIO, and Stage 2 analysis results. |
| Context: |
| - previous_bbox |
| - audio_class |
| - image size |
Table S.13 : Stage 3 (Refinement) Prompt
More in this topic
Audio-Visual Understanding
Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.
See all 155 papers