TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Hugo Malard; Michel Olvera; Stephane Lathuiliere; Slim Essid

eess.ASDec 2, 2024

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Authors: Hugo Malard, Michel Olvera, Stephane Lathuiliere, Slim Essid

Organizations: LTCI, Télécom Paris, Institut Polytechnique de Paris

Abstract

Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.

Figures & tables

Figure 1: Our method takes a representation of an image and its associated audio as input, decomposing them into a product of ’semantic’ factors and (spatial or temporal) activations. This decomposition enables locating parts of the original input corresponding to the concept present in both the image and the audio.

Figure 2: Complete pipeline: both the audio and the image are encoded using their respective encoder and their representations are used to perform the co-NMF. FC-CLIP is prompted using the image factors ( $V_{I}$ ) and the segmentation corresponding to the sounding image factor ( $V_{I}^{k\star}$ ) is kept as the final segmentation.

Figure 3: As CLIP and CLAP encode audio and images in different spaces, our method employs semantic anchors to project semantic components (representations soft-masked by $U_{I}^{k}$ and $U_{A}^{k}$ ) in an audio-visual semantic space where standard distances can be considered to compute the penalty function.

Method				S4		MS3
Method	Training-free	Segmenter	#Sources	mask-IoU $\uparrow$	F-Score $\uparrow$	mask-IoU $\uparrow$	F-Score $\uparrow$
SLAVC ( Mo and Morgado, 2022a )	✗	✗	✗	28.10	34.60	24.37	25.56
MarginNCE ( Park et al., 2023 )	✗	✗	✗	33.27	45.33	27.31	31.56
FNAC ( Senocak et al., 2023 )	✗	✗	✗	27.15	31.40	21.98	22.50
Alignment ( Sun et al., 2023 )	✗	✗	✗	29.60	35.90	-	-
TACO w/o segmenter ( $U_{I}^{k^{\star}}$ )	✓	✗	✗	29.68 $\pm$ 0.16	41.91 $\pm$ 0.09	25.88 $\pm$ 0.91	30.72 $\pm$ 0.95

Table 1: Quantitative results of sound-prompted segmentation on the AVS-Bench test sets. Best is in bold and second best underlined . The column #Sources indicates whether the number of sources is known. The * indicates a p-value lower than 0.01 when comparing to the best performing model.

	Segmenter	Training-free	m-IOU $\uparrow$	mAP $\uparrow$
DAVENet ( Hsu et al., 2019 )	-	✗	17.0	16.8
CAVMAE ( Gong et al., 2023 )	-	✗	20.5	26.0
ImageBind ( Girdhar et al., 2023 )	-	✗	18.3	18.1
DenseAV ( Hamilton et al., 2024 )	-	✗	25.5	32.4
TACO w/o segmenter ( $U_{I}^{k^{\star}}$ )	-	✓	27.74 $\pm$ 0.48	35.75 $\pm$ 0.33
A-FC-CLIP	FC-CLIP	✓	45.31	50.36

Table 2: Quantitative results of sound-prompted segmentation on the ADE SP dataset. The * indicates a p-value lower than 0.01 when comparing to the best performing model.

Method	AVSS		ADE SP Semantic
Method	mIoU $\uparrow$	F-Score $\uparrow$	mIoU $\uparrow$	F-Score $\uparrow$
A-FC-CLIP	11.95	13.56	20.91	24.95
TACO	20.50 $\pm 0.10^{*}$	23.02 $\pm 0.09^{*}$	37.70 $\pm 1.61^{*}$	44.39 $\pm 1.92^{*}$

Table 3: Quantitative results of sound-prompted semantic segmentation on the AVSS and ADE SP Semantic datasets. The * indicates a p-value lower than 0.01 when comparing to the best performing model.

Figure 8: mIoU of TACO on S4, with varying number of tags in the word bank.

Word Bank	mask-IoU $\uparrow$	F-Score $\uparrow$
AudioSet tags	64.05 $\pm$ 0.25	71.50 $\pm$ 0.20
GPT5.2 generated	63.83 $\pm$ 0.29	71.18 $\pm$ 0.29

Table 4: Ablation on S4 dataset: choice of general word bank.

Figure 9: Proportion of additional computing time induced by the decomposition. The decomposition allows an improvement of

\sim 12

points with only

\sim 20\%

of additional compute time compared to the baseline (A-FC-CLIP).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	ADE SP	S4	MS3	AVSS	ADE SP S.
A-FC-CLIP	45.31	51.52	33.67	11.95	24.95
A + -FC-CLIP	41.89	51.63	31.67	11.82	23.73

Appendix

Table 5: Impact of clamping on the representation before feeding FC-CLIP. We report mIoU on five datasets. ADE SP S. corresponds to the ADE SP Semantic segmentation dataset. mask-IoU is reported for S4 and S3 and mIoU for all the other datasets.

Table 13

	Word Bank	S4	MS3	ADE SP
$U_{i}^{k^{\star}}$	General	27.78 $\pm$ 0.06	25.88 $\pm$ 0.91	32.42 $\pm$ 0.74
$U_{i}^{k^{\star}}$	Specific	28.76 $\pm$ 0.06	26.57 $\pm$ 0.17	27.74 $\pm$ 0.48
TACO	Specific	64.73 $\pm$ 0.16	45.15 $\pm$ 0.80	54.05 $\pm$ 1.40
TACO	General	64.04 $\pm$ 0.25	43.15 $\pm$ 0.91	51.57 $\pm$ 0.86

Appendix

Table 8: Comparison of the two word banks on sound-prompted segmentation datasets. mask-IoU is reported for S4 and MS3, while mIoU is reported for ADE SP.

Method	mask-IoU $\uparrow$	F-Score $\uparrow$
TACO	43.15 $\pm$ 0.91	47.5 $\pm$ 0.95
TACO w/o penalty	23.52 $\pm$ 0.54	24.23 $\pm$ 0.65
TACO bottom-2	41.12 $\pm$ 0.90	45.17 $\pm$ 1.14

Appendix

Table 9: Ablation on MS3 dataset: impact of penalty function.

	$\displaystyle\mathcal{L}={}$	$\displaystyle\\|X_{A}-U_{A}V_{A}\\|_{2}^{2}+\\|X_{I}-U_{I}V_{I}\\|_{2}^{2}$
		$\displaystyle+\beta_{p}\min_{k}\,\text{CE}(\mathcal{D}_{I^{k}},\mathcal{D}_{A^{k}})$

Appendix

Algorithm 1 TACO algorithm

Figure 10: Illustration of the decomposition

Method	Image Encoder	Audio Encoder	Total
Hamilton et al. (2024)	$1.69\times 10^{10}$	$6.72\times 10^{10}$	$8.41\times 10^{10}$
Park et al. (2024)	$1.69\times 10^{10}$	$4.48\times 10^{10}$	$6.17\times 10^{10}$
Ours	$3.44\times 10^{10}$	$6.31\times 10^{9}$	$4.07\times 10^{10}$

Appendix

Table 10: Comparison of model MACs for image and audio encoders.

Panel of Figure 11 — Figure 11: Semantic segmentation examples from AVSS.

Panel of Figure 12 — Figure 12: Sound-prompted segmentation examples from ADE SP.

Panel of Figure 13 — Figure 13: Sound prompted-segmentation examples from MS3.

Panel of Figure 14 — Figure 14: Examples of typical failure cases on MS3.

Explore similar work

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Neural Audio
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research
Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration
Jun 8, 2026Silas Kwabla Gah, Ebenezer OwusuFew-Shot Segmentation Open-Vocabulary
Department of Computer Science University of Ghana
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
May 29, 2026Yuhan Song, Linhao Zhang, Aiwei Liu +6Audio Tokenizers Acoustic Latent Space
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Basic Model Technology Center, WeChat AI, Tencent Inc.

Feb 5, 2026, cs.CLJ/K move · Enter open · S save

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

Jinchuan Tian, Haoran Wang, Bo-Hao Su, Chien-yu Huang+13

1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research

Audio Understanding Neural Audio Audio Understanding Captions Prior Knowledge Integration

Open paper AskPDF arXiv

Current audio foundation models typically rely on rigid, task-specific supervision, addressing isolated factors of audio rather than the whole. In contrast, human intelligence processes audio holistically, seamlessly bridging physical signals with abstract cognitive concepts to execute complex tasks. Grounded in this philosophy, we introduce Bagpiper, an 8B audio foundation model that interprets physical audio via rich captions, i.e., comprehensive natural language descriptions that encapsulate the critical cognitive concepts inherent in the signal (e.g., transcription, audio events). By pre-training on a massive corpus of 600B tokens, the model establishes a robust bidirectional mapping between raw audio and this high-level conceptual space. During fine-tuning, Bagpiper adopts a caption-then-process workflow, simulating an intermediate cognitive reasoning step to solve diverse tasks without task-specific priors. Experimentally, Bagpiper outperforms Qwen-2.5-Omni on MMAU and AIRBench for audio understanding and surpasses CosyVoice3 and TangoFlux in generation quality, capable of synthesizing arbitrary compositions of speech, music, and sound effects. To the best of our knowledge, Bagpiper is among the first works that achieve unified understanding generation for general audio. Model, data, and code are available at Bagpiper Home Page.

Abstract

Explore similar work

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration

UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception