Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

Sripathi Sridhar; Gordon Wichern; Yoshiki Masuyama; Mark Cartwright; Jonathan Le Roux

eess.ASOct 2, 2026

Evaluating Inference-time Algorithms for Semantic Sound Scene Segmentation

Authors: Sripathi Sridhar, Gordon Wichern, Yoshiki Masuyama, Mark Cartwright, Jonathan Le Roux

Organizations: New Jersey Institute of Technology, Newark NJ, USA · Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA

Abstract

Spatial semantic segmentation of sound scenes (S5) involves the detection and extraction of target sound events in audio files. However, in typical detection-extraction pipelines, errors in the detection module propagate to the extraction module, degrading overall performance. In this work, we develop a multichannel detection-extraction model and evaluate inference-time algorithms to improve the class detection and target sound extraction in a multi-stage setup. We evaluate two fitness functions: binary cross entropy and mixture consistency; two relabeling strategies: naive and conditional confusion matrix relabeling; and three source estimate evaluation models: a multi-label classifier, a fine-tuned single-source classifier, and an off-the-shelf audio judge. We thoroughly evaluate the impact of these design choices on the DCASE 2025 Task 4 dataset, and validate entropy-based gating of inference-time updates on the held-out evaluation set, laying the groundwork for inference-time updates of S5 systems in real-world scenarios.

Figures & tables

Fig. 1: Delta CA-SDRi vs. baseline CA-SDRi on the full test set for M2D with conditional CM relabeler and BCE+MC fitness function.

Relabeler	Non-Target?	Fitness	Evaluator	$\Delta$	CA-SDRi	$\Delta_{\text{oracle}}$	$\Delta$	CA-SDRi	$\Delta_{\text{oracle}}$	$\Delta$	CA-SDRi	$\Delta_{\text{oracle}}$
				All (1500 ex.)			$\leq$ 10 dB (563 ex.)			Oracle Gap (51 ex.)
				Baseline: 12.24 dB			Baseline: 4.79 dB			Baseline: $-$ 0.21 dB
CM	✗	BCE	M2D	-4.10	8.15	1.24	-0.43	4.37	3.08	2.61	2.40	10.04
CM	✓	BCE+MC	M2D	-4.02	8.22	1.42	-0.28	4.52	3.49	2.06	1.85	10.35
CM	✗	BCE	M2D-SC	-2.68	9.56	1.24	0.47	5.26	3.08	5.34	5.13	10.04
CM	✓	BCE+MC	M2D-SC	-2.48	9.77	1.42	0.84	5.64	3.49	5.22	5.01	10.35

Table 1: Performance of various prompt relabeling strategies, fitness functions, and evaluation models on the DCASE 2025 Task 4 test set. CM: Naïve Confusion matrix, CCM: Conditional Confusion matrix, BCE: Binary cross entropy, MC: Mixture consistency, M2D: Multilabel classifier, M2D-SC: Single-class classifier, SAJ: SAM Audio Judge.

\Delta

refers to improvement over the baseline CA-SDRi, while Oracle is the theoretical maximum improvement from the hypotheses evaluated given an ideal fitness function, i.e., computing CA-SDRi against ground truth. Bold indicates best performance; underline indicates second best.

Feature	$\tau$	$N$	CA-SDRi	$\Delta\uparrow$	$\Delta_{\text{g}}\uparrow$	${\text{BL}}_{\text{g}}\downarrow$	${\text{BL}}_{\text{r}}\uparrow$
SAJ R	0.97	5	12.25	+0.003	+0.90	12.21	12.24
SAJ P	1.77	57	12.28	+0.031	+0.82	9.68	12.35
SAJ O	0.99	17	12.25	+0.004	+0.35	11.62	12.25
SAJ F	1.01	55	12.27	+0.024	+0.67	12.04	12.25
H ${}_{\text{sum}}$	3.20	24	12.28	+0.031	+1.96	2.85	12.40
H ${}_{\text{max}}$	0.69	89	12.27	+0.023	+0.39	8.54	12.48

Table 2: Gating threshold sweep results on the test set. SAM Audio Judge scores (R=Recall, P=Precision, O=Overall quality, F=Faithfulness) gate clips with score

\leq\tau

; M2D entropy features gate clips with score

>\tau

. Thresholds

\tau

are oracle-selected by the sweep.

N

: number of gated clips.

\Delta

: change in overall CA-SDRi vs. baseline (12.24 dB).

\Delta_{\text{g}}

: mean CA-SDRi change for gated clips.

\text{BL}_{\text{g}}

\text{BL}_{\text{r}}

: mean baseline CA-SDRi for gated and remaining clips.

Feature	$\tau$	$N$	CA-SDRi	$\Delta\uparrow$	$\Delta_{\text{g}}\uparrow$	${\text{BL}}_{\text{g}}\downarrow$	${\text{BL}}_{\text{r}}\uparrow$
SAJ R	0.97	4	8.61	0.000	$-$ 0.10	11.59	8.60
SAJ P	1.77	81	8.50	$-$ 0.106	$-$ 2.13	10.52	8.51
SAJ O	0.99	35	8.58	$-$ 0.030	$-$ 1.41	9.56	8.59
SAJ F	1.01	73	8.55	$-$ 0.057	$-$ 1.27	10.69	8.51
H ${}_{\text{sum}}$	3.20	37	8.60	$-$ 0.005	$-$ 0.24	4.62	8.70
H ${}_{\text{max}}$	0.69	95	8.54	$-$ 0.070	$-$ 1.20	7.58	8.67

Table 3: Gating on the held-out DCASE 2025 Task 4 evaluation set (1620 clips) with thresholds

\tau

frozen from Table 2 .

\Delta

: change in overall CA-SDRi vs. baseline (8.61 dB); other columns as in Table 2 .

Sound Event Detection

Sound Event Detection identifies which sounds occur in an audio recording and when they begin and end.

See all 60 papers

Lightweight Systems for Spatial Semantic Segmentation of Sound Scenes
Oct 8, 2026Mayank Mishra, Paul Magron, Romain SerizelSpatial Audio Audio Source Separation
Universit´e de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France
Overview and Meta-Analysis of DCASE 2026 Challenge Task 6: Audio Moment Retrieval from Long Audio
Sep 11, 2026Hokuto Munakata, Tatsuya Komatsu, Keisuke Imoto +3Audio-Text Retrieval
LY Corporation, Japan · Kyoto University, Japan · Sony Interactive Entertainment, Japan +2
Training-Free Multi-Step Inference for Target Speaker Extraction
Mar 11, 2026Zhenghai You, Ying Shi, Lantian Li +1Target Speaker Extraction
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China

Oct 8, 2026J/K move · Enter open · S save

Lightweight Systems for Spatial Semantic Segmentation of Sound Scenes

Mayank Mishra, Paul Magron, Romain Serizel

Universit´e de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

Spatial Audio Audio Source Separation Audio Classification

Open paper AskPDF arXiv

Spatial semantic segmentation of sound scenes (S5) jointly performs audio source separation and tagging on a multichannel audio mixture. Therefore, an S5 system can be decomposed into an audio tagging model and a separation model. The baseline S5 system in task~4 of the DCASE 2025 challenge uses M2D and ResUNet models for tagging and separation, respectively. These are large in overall size and therefore not appropriate for use in embedded devices. The main aim of this paper is to assess the performance loss in an S5 system when using lightweight tagging and separation models instead. To this end, we first compare the tagging performance of lightweight audio tagging models, namely PANNs and DyMNs, in terms of label prediction accuracy against the baseline M2D model. Secondly, we assess separation performance in terms of the scale-invariant signal-to-distortion ratio of the baseline ResUNet separator relative to that of a convolutional-based separator using SuDoRMRF. Finally, we combine the tagging and separation models across various configurations and assess the overall performance of the S5 system using the class-aware signal-to-distortion ratio.

Abstract

Sound Event Detection

Explore similar work

Lightweight Systems for Spatial Semantic Segmentation of Sound Scenes

Overview and Meta-Analysis of DCASE 2026 Challenge Task 6: Audio Moment Retrieval from Long Audio

Training-Free Multi-Step Inference for Target Speaker Extraction