Organizations: SHINE research unit - Center for Information and Communication Technology (ICT) - Fondazione Bruno Kessler (FBK), Via Sommarive 18, 38123 Trento, Italy.
In Gaussian model-based multichannel audio source separation, the likelihood of observed mixtures of source signals is parametrized by source spectral variances and by associated spatial covariance matrices. These parameters are estimated by maximizing the likelihood through an Expectation-Maximization algorithm and used to separate the signals by means of multichannel Wiener filtering. We propose to estimate these parameters by applying nonnegative factorization based on prior information on source variances. In the nonnegative factorization, spectral basis matrices can be defined as the prior information. The matrices can be either extracted or indirectly made available through a redundant library that is trained in advance. In a separate step, applying nonnegative tensor factorization, two algorithms are proposed in order to either extract or detect the basis matrices that best represent the power spectra of the source signals in the observed mixtures. The factorization is achieved by minimizing the β-divergence through multiplicative update rules. The sparsity of factorization can be controlled by tuning the value of β. Experiments show that sparsity, rather than the value assigned to β in the training, is crucial in order to increase the separation performance. The proposed method was evaluated in several mixing conditions. It provides better separation quality with respect to other comparable algorithms.
Figures & tables
Fig. 1: Distance measured by different definitions of divergence, with a=1 .
Fig. 2: Examples of controlling the sparsity of Wn of the corrupted power spectrum by selecting the value of β , from the left to the right, respectively, original Wn (training) with β=0.9 , estimated with β=0.1 , estimated with β=0.3 , estimated with β=0.6 , and estimated with β=0.9 .
Fig. 3: Normalized power spectra of true, corrupted, and reconstructed signals, from the left to the right respectively.
Fig. 4: Flowchart of the proposed method
Fig. 5: Normalized Likelihoods (d) of each spectral basis vector in the library Ulib as a function of the separation iterations. First column on the left for the observed mixtures, then the five columns for iterations (1, 10, 25, 40 and 65 respectively) and rows for 3 speech source signals.
Fig. 6: Normalized Likelihoods of each spectral basis matrix Uz as a function of the separation iterations. First column on the left for the observed mixtures, then the five columns for iterations (1, 10, 25, 40 and 65 respectively) and rows for 3 speech source signals.
T60(ms)
130
250
380
Average
K
15
25
40
15
25
40
15
25
40
15
25
40
βs= 0.9
8.53
9.18
9.04
5.50
5.61
5.32
3.58
3.46
3.23
5.87
6.08
5.86
βs= 0.6
9.03
9.71
9.64
6.57
6.51
5.92
4.73
4.48
4.08
6.78
6.90
6.55
βs= 0.3
8.64
9.26
9.66
6.60
6.85
6.84
5.02
4.95
4.44
6.75
7.02
6.98
βs= 0.1
7.88
8.48
9.09
6.26
6.52
6.96
4.63
4.67
4.81
6.26
6.56
6.95
TABLE I: The average SDR of the simulated data as a function of K , T60 and βs , βt=0.9 .
Fig. 7: Iterative improvement of SDR for four different detection cases.
K
10
20
30
40
βs= 0.9
4.52
4.82
5.10
5.43
βs= 0.6
5.60
5.70
5.79
5.88
βs= 0.3
5.72
6.62
6.78
6.62
βs= 0.1
5.67
6.46
6.42
6.43
TABLE II: The average SDR of the live-recorded data as a function of K and βs , βt=0.9 .
Un
Informed
Blind
Div. Factors
βt=0.9,βs=0.3
βe=0.6,βs=0.6
Source images
c~1
c~2
c~3
c~1
c~2
c~3
SDR
11.27
10.42
10.22
9.63
8.67
9.17
ISR
15.13
14.56
17.76
12.90
12.82
16.50
SIR
17.26
18.02
13.12
15.28
13.97
11.95
SAR
14.99
12.99
13.68
14.32
12.33
12.76
TABLE III: The performance of informed and blind source separation of 3-Females of live-recorded mixtures from the SISEC dataset.
T60
130ms
250ms
Average
Source
Three females
Three males
Three females
Three males
K
20
35
50
20
35
50
20
35
50
20
35
50
20
35
50
βs= 0.9
9.45
9.88
9.90
8.43
8.83
8.95
6.34
4.21
4.14
7.70
7.87
7.53
7.98
7.70
7.63
βs= 0.6
10.08
10.20
10.07
8.57
8.90
9.10
6.72
6.11
5.93
7.77
7.97
8.25
8.29
8.30
8.34
βs= 0.3
9.80
10.70
10.56
8.07
8.60
8.94
9.05
9.40
9.01
7.39
7.82
8.11
8.58
9.13
9.16
βs= 0.1
8.95
9.74
10.18
7.63
8.15
8.50
7.88
8.16
8.86
6.76
7.33
7.78
7.81
8.35
8.83
TABLE IV: The SDR of informed separation of live-recorded stereo mixtures from SISEC, βt=0.9 .
Env.
Synthetic convolutive mixtures
Live-recorded mixtures
K
20
35
50
20
35
50
βe
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
βs= 0.9
5.53
5.97
6.00
5.70
5.79
5.63
5.78
5.80
5.68
7.46
7.54
5.83
7.74
7.58
7.95
7.55
7.62
7.80
βs= 0.6
5.92
6.32
6.49
5.71
6.22
5.97
6.18
6.17
5.76
7.49
7.27
6.98
7.66
7.48
7.76
7.61
7.90
8.16
βs= 0.3
5.51
6.79
6.75
5.66
6.01
6.81
5.71
6.90
6.08
6.07
7.06
6.76
6.31
7.20
7.31
7.23
7.62
7.98
βs= 0.1
5.07
5.55
5.75
5.44
6.50
7.07
5.34
6.36
6.20
5.26
5.59
5.41
5.48
6.76
6.74
6.26
7.10
7.11
TABLE V: The average SDR of blind separation of stereo mixtures from SISEC, T60=130ms .
Source
Three females
Three males
K
20
35
50
20
35
50
βe
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
βs= 0.9
8.10
8.27
5.98
8.70
8.35
8.83
8.48
8.80
9.04
6.82
6.81
5.68
6.78
6.81
7.07
6.62
6.44
6.56
βs= 0.6
7.75
8.35
8.12
8.52
8.70
8.72
8.72
8.99
9.20
7.23
6.19
5.84
6.80
6.26
6.80
6.50
6.81
7.13
βs= 0.3
7.29
7.51
7.72
7.40
8.14
8.38
8.43
8.38
9.11
4.85
6.61
5.80
6.00
6.26
6.24
6.03
6.86
6.85
βs= 0.1
5.66
6.97
6.60
5.80
7.26
8.30
7.12
7.99
8.85
4.86
4.21
4.22
5.16
6.26
5.17
5.40
6.21
5.37
TABLE VI: The SDR of blind separation of live-recorded stereo mixtures from SISEC, T60=130 ms.
Source
Three females
Three males
K
20
35
50
20
35
50
βe
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
1.2
0.9
0.6
βs= 0.9
3.65
3.42
3.71
3.55
3.63
3.24
3.56
3.49
3.36
3.90
3.32
3.66
4.71
3.17
3.44
4.03
3.96
3.46
βs= 0.6
3.81
4.52
4.01
4.54
4.65
3.45
5.60
3.54
3.54
3.47
3.48
3.52
5.04
4.01
3.69
4.09
4.22
4.10
βs= 0.3
3.59
6.44
6.29
4.55
4.95
6.50
3.67
4.66
4.36
3.59
3.74
4.20
4.48
5.17
4.34
4.87
5.07
4.47
βs= 0.1
3.63
5.74
6.19
4.61
5.86
6.50
4.67
5.42
4.88
3.37
3.40
3.58
4.85
4.33
4.31
5.11
5.00
4.80
TABLE VII: The SDR of blind separation of live-recorded stereo mixtures from SISEC, T60=250 ms.
Source
Three females
Three males
Method
Proposed
Nesta
Cho
Proposed
Nesta
Cho
Un
Inform
Blind
Initialize
Inform
Blind
Initialize
K
50
50
50
50
βt / βe
0.9
0.6
[ 37 ]
[ 38 ]
0.9
0.6
[ 37 ]
[ 38 ]
βs
0.6
0.6
0.6
0.6
SDR
10.07
9.20
3.92
7.70
8.40
9.10
7.13
2.58
6.50
6.50
TABLE VIII: Performance comparison of live-recorded stereo mixtures from SISEC, T60=130 ms.
Source
Three females
Three males
Method
Proposed
Nesta
Cho
Proposed
Nesta
Cho
Un
Inform
Blind
Initialize
Inform
Blind
Initialize
K
35
35
35
35
βt / βe
0.9
0.6
[ 37 ]
[ 38 ]
0.9
0.6
[ 37 ]
[ 38 ]
βs
0.3
0.3
0.3
0.3
SDR
9.40
6.50
2.32
6.00
6.10
7.82
4.34
1.95
5.20
6.00
TABLE IX: Performance comparison of live-recorded stereo mixtures from SISEC, T60=250 ms.
Fig. 8: The average separation performance of the informed case of the SISEC dataset, as a function of K and βs ( βt=0.9 ). The horizontal axis indicates the values of the tested βs=[0.10.30.60.9] .
Mahmoud Fakhry received the B.Sc. and M.Sc. degrees in Electrical Engineering from the Department of Electrical Engineering, Aswan University, Egypt and the PhD degree in Information and Communication Technology from the University of Trento, Italy, in 2016. From October 2012 to November 2016, he was with the Center for Information and Communication Technology, Fondazione Bruno Kessler, Trento, Italy, as PhD student. Since December 2016, he is a postdoc researcher at the Audio Analysis Lab, Aalborg University, Denmark. His research interests include source modeling using nonnegative matrix and tensor factorization, analysis of dynamic systems, speech separation and enhancement, and voice analysis for early diagnosis of diseases.
Table 18
Piergiorgio Svaizer received the degree in electronic engineering (communications) from the University of Padua, Padua, Italy, in 1989. He is a senior researcher at the Center for Information and Communication Technology, Fondazione Bruno Kessler, Trento, Italy. His main research interests include speech analysis, enhancement and separation, front-end processing and noise reduction algorithms for speech recognition in adverse conditions, and microphone arrays for acoustic scene analysis and acoustic surveillance.
Table 19
Maurizio Omologo (M’88) received the “Laurea” degree (with honors) in electrical engineering from the University of Padua in 1984. He is the head of the SHINE (Speech-Acoustic Scene Analysis and Interpretation) research unit of the Center for Information and Communication Technology, Fondazione Bruno Kessler, Trento, Italy. His current research interests include audio and speech processing, acoustic scene analysis, and automatic speech recognition, in particular for distant-talking scenarios. He is author of more than 100 papers in major international conferences and journals in the field.
Audio source separation models must preserve fine spectral structure while keeping the cost of the separator tractable. Band-split architectures reduce this cost by manually partitioning the spectrum into predefined frequency bands and encoding each band as a token, but the same boundaries are used for every mixture. Because informative spectral structure varies with source content, fixed boundaries can merge discriminative components or spend tokens where little resolution is needed. We introduce Dynamic Band Splitting (DBS), an end-to-end framework that adapts boundary locations and band widths to each mixture under a fixed band-token budget. DBS predicts content-conditioned boundary scores within frequency blocks, selects a prescribed number of boundaries using hard Top-K, and applies content-dependent pooling to obtain compact band tokens. After separation, it smooths and expands these tokens to the original STFT grid, refines them along frequency, and decodes complex masks with a frequency-shared head. DBS improved separation over the corresponding static models on DnR v2, MUSDB18-HQ, and EchoSet, with DBS-BSRNN achieving relative gain of 29.0% in average SI-SDR on DnR v2. These results and boundary analyses on DnR v2 show that content-adaptive frequency partitioning benefits multiple separator backbones under a fixed band-token budget. Code and audio demos are available at https://cslikai.cn/Dynamic-Band-Splitting
Bingzhe Zhang, Kai Li, Xiaolin Hu
Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China · IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China · Chinese Institute for Brain Research (CIBR), Beijing 100010, China
This paper proposes a general framework for stable and effective iterative audio separation with mixture consistency by extending source separation models to a multi-input multi-output (MIMO) configuration. In the field of audio separation, mixture consistency is an essential property for many applications that require accurate phase and timbral information of target sources. While iterative approaches such as diffusion models achieve perceptually superior results in speech enhancement or user-guided target source separation tasks, most existing methods focus on single-step separation with a single-input single-output (SISO) or single-input multi-output (SIMO) configuration through architectural improvements, since mixture-consistent audio separation is generally regarded as a regression problem that admits a unique solution. By extending these architectures to a MIMO configuration, we introduce iterative prediction without compromising the architectural advantages or the characteristics of mixture consistency. We conduct a comprehensive ablation study of combining the framework with discriminators and extending it to a generative model. Experimental results demonstrate significant performance improvements when applying the proposed framework to state-of-the-art separation models.
This paper presents StrADiff, a Structured Source-Wise Adaptive Diffusion Framework for unsupervised blind source separation under linear and nonlinear mixing. The framework treats each latent dimension as a source branch and assigns to it an individual adaptive reverse diffusion mechanism, so that latent sources are recovered directly from observed mixtures through a single end-to-end objective, without supervised source labels or separate post-processing. Source-wise generation, structural regularization, and observation-space reconstruction are optimized jointly during training. In this instantiation, a Gaussian process (GP) prior is used as one example of a source-wise structured prior to impose temporal organization on each recovered trajectory; the framework itself is not restricted to GP priors and can in principle incorporate other structured priors. Theoretical components clarify the induced pushforward source law, the sample-level role of the structured prior, the coupling between source recovery and prior adaptation, and a conditional weak recovery statement in an idealized linear low-noise regime. Experiments on linear and nonlinear mixtures show that StrADiff can recover meaningful latent source trajectories in an unsupervised manner, with particularly stable performance in the linear case and moderate degradation under nonlinear mixing. Beyond classical signal separation, a source branch may also be interpreted as an independent, disentangled, or otherwise interpretable explanatory factor under suitable structural assumptions, suggesting a broader route toward structured latent modeling and future identifiable nonlinear representation learning.