SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts

Khanh Binh Nguyen; Chae Jung Park

cs.CVMar 24, 2026

SOUPLE: Enhancing Audio-Visual Localization and Segmentation with Learnable Prompt Contexts

Authors: Khanh Binh Nguyen, Chae Jung Park

Organizations: Deakin University, Australia · National Cancer Center, South Korea

Abstract

Large-scale pre-trained image-text models exhibit robust multimodal representations, yet applying the Contrastive Language-Image Pre-training (CLIP) model to audio-visual localization remains challenging. Replacing the classification token ([CLS]) with an audio-embedded token ([V_A]) struggles to capture semantic cues, and the prompt "a photo of a [V_A]" fails to establish meaningful connections between audio embeddings and context tokens. To address these issues, we propose Sound-aware Prompt Learning (SOUPLE), which replaces fixed prompts with learnable context tokens. These tokens incorporate visual features to generate conditional context for a mask decoder, effectively bridging semantic correspondence between audio and visual inputs. Experiments on VGGSound, SoundNet, and AVSBench demonstrate that SOUPLE improves localization and segmentation performance.

Figures & tables

Figure 1: Comparison of attention masks produced by ACL-SSL and SouPLe in representative cases. Compared with ACL-SSL, SouPLe yields more refined localization of the sounding objects.

Figure 2: Pipeline of our SouPLe framework. SouPLe takes an audio-visual pair as input, converting the audio signal into a CLIP-compatible token via the Audio Tokenizer and the image into learnable context tokens via the Meta-net. The learnable context tokens are concatenated with the audio-embedded token and used by the text encoder to produce audio-text features for mask decoding. By replacing fixed handcrafted prompts with learnable context tokens, SouPLe improves generalization to the given audio-visual input.

Figure 3: Visual-Audio-Text Alignment (VAT) module

Method	VGG-SS		SoundNet-Flickr
Method	$\mathbf{cIoU}\uparrow$	$\mathbf{AUC}\uparrow$	$\mathbf{cIoU}\uparrow$	$\mathbf{AUC}\uparrow$
Attention	18.50	30.20	66.00	55.80
CoarseToFine	29.10	34.80	-	-
LCBM	32.20	36.60	-	-
LVS	34.40	38.20	71.90	58.20
HardPos	34.60	38.00	76.80	59.20

Table 1: Quantitative results on the VGG-SS and SoundNet-Flickr test sets. All models were trained on 144K samples from VGGSound. SLAVC does not report AUC scores. The last row indicates improvement over ACL-SSL.

{\dagger}

and

{\ddagger}

denote without and with OGL, respectively.

Test Class	Method	cIoU $\uparrow$	AUC $\uparrow$
Heard 110	LVS	28.90	36.20
	EZ-VSL†	31.86	36.19
	EZ-VSL‡	37.25	38.97
	SLAVC†	35.84	-
	SLAVC‡	38.22
	FNAC‡	39.54	39.83

Table 2: Open-set audio-visual localization results on the train/test splits of [ 3 , 21 , 25 ] . The last row indicates improvements over ACL-SSL.

{\dagger}

and

{\ddagger}

denote without and with OGL, respectively. ”50ep” indicates training for 50 epochs.

Method	S4		MS3
(w/o OGL)	mIoU $\uparrow$	F-Score $\uparrow$	mIoU $\uparrow$	F-Score $\uparrow$
SLAVC	28.10	34.60	24.37	25.56
MarginNCE	33.27	45.33	27.31	31.56
FNAC	27.15	31.40	21.98	22.50
Alignment	29.60	35.90	-	-
Baselines:

Table 3: Quantitative results on the AVSBench test sets. The last row indicates the improvement over ACL-SSL.

Method	Extended VGG-SS			Extended Flickr
Method	AP $\uparrow$	max-F1 $\uparrow$	LocAcc $\uparrow$	AP $\uparrow$	max-F1 $\uparrow$	LocAcc $\uparrow$
SLAVC†	32.95	40.00	37.79	51.63	59.10	83.60
MarginNCE†	30.58	36.80	38.25	57.99	61.80	83.94
FNAC†	23.48	33.70	39.50	50.40	62.30	84.73
Alignment†	34.73	40.70	39.94	64.43	66.90	79.60
WAV2CLIP	26.67	33.00	37.71	20.99	24.80	29.60

Table 4: Quantitative results on the Extended VGG-SS and Extended SoundNet-Flickr benchmarks. All models were trained on

144K

samples from VGGSound. Results for prior approaches are from Mo and Morgado [20] . The last row indicates the improvement over ACL-SSL.

{\dagger}

and

{\ddagger}

denote without and with OGL, respectively.

# ctx	$\mathbf{cIoU}\uparrow$	$\mathbf{AUC}\uparrow$
ctx=4	53.21	48.15
ctx=8	52.01	47.32
ctx=16	51.08	46.93

Table 5: Ablation study on context length on the VGG-SS dataset.

# ctx	$\mathbf{V_{A}}$ index	Token Order	$\mathbf{cIoU}\uparrow$	$\mathbf{AUC}\uparrow$
ctx=4	pos=1	$\mathbf{[V_{A}]}[V_{1}][V_{2}][V_{3}][V_{4}]$	49.91	46.21
ctx=4	pos=2	$[V_{1}]\mathbf{[V_{A}]}[V_{2}][V_{3}][V_{4}]$	50.71	46.42
ctx=4	pos=3	$[V_{1}][V_{2}]\mathbf{[V_{A}]}[V_{3}][V_{4}]$	51.08	46.93
ctx=4	pos=4	$[V_{1}][V_{2}][V_{3}]\mathbf{[V_{A}]}[V_{4}]$	52.41	47.72
ctx=4	pos=5	$[V_{1}][V_{2}][V_{3}][V_{4}]\mathbf{[V_{A}]}$	53.21	48.15

Table 6: Ablation study on the position of

[V_{A}]

# ctx	# epochs	$\mathbf{cIoU}\uparrow$	$\mathbf{AUC}\uparrow$
ctx=4	20	53.21	48.15
ctx=4	40	53.66	48.49
ctx=4	50	54.76	49.40

Table 7: Ablation study on training duration.

Figure 4: Sound localization results on VGG-SS, SoundNet-Flickr, and AVSBench, along with a comparison with ACL-SSL.

Method	Fusion?	Ensemble?	$\mathbf{cIoU}\uparrow$	$\mathbf{AUC}\uparrow$
SouPLe			53.21	48.15
	✓		49.93	45.98
		✓	33.63	34.49

Table 8: Ablation study on audio-visual feature fusion.

Figure 5: Failure cases visualization of SouPLe on MS3.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization
Dec 2, 2024Hugo Malard, Michel Olvera, Stephane Lathuiliere +1Image Segmentation Audio-Visual Representation Learning
LTCI, Télécom Paris, Institut Polytechnique de Paris
Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning
Apr 8, 2026Subin Park, Jung Uk KimAudio-Visual Understanding
Kyung Hee University
Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs
Oct 15, 2026Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4Audio-Visual Understanding Multimodal Large Language Models
POSTECH · KAIST · Hankuk University of Foreign Studies +1

Dec 2, 2024J/K move · Enter open · S save

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Hugo Malard, Michel Olvera, Stephane Lathuiliere, Slim Essid

LTCI, Télécom Paris, Institut Polytechnique de Paris

Image Segmentation Audio-Visual Representation Learning

Open paper AskPDF arXiv

Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.

Abstract

Audio-Visual Understanding

Explore similar work

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs