Online Algorithms for Independent Low-Rank Matrix Analysis and Rank-Constrained Spatial Covariance Matrix Estimation Based on Maximum Weighted Likelihood Estimation
Organizations: Graduate School of Information Science and Technology, The University of Tokyo, Tokyo 113-8656, Japan · National Institute of Technology, Kagawa College, Takamatsu, Kagawa 761-8058, Japan · Yamaha Corporation, Hamamatsu-shi, Shizuoka 430-8650, Japan
Real-time multichannel speech extraction (MSE) under diffuse noise conditions is an important task with a wide range of applications, such as speech recognition and hearing aids. In this paper, we propose online algorithms for independent low-rank matrix analysis (ILRMA) and rank-constrained spatial covariance matrix estimation (RCSCME). Previously, we proposed a real-time extension of the RCSCME-based method: an MSE method based on ILRMA and RCSCME using the blockwise batch algorithm. However, it assumes that the spatial characteristics are stationary within a single batch, and thus, in dynamic situations where the target speaker moves, its performance may degrade. To address this problem, we derive the online algorithms for ILRMA and RCSCME in the following three steps. First, we formulate framewise cost functions for ILRMA and RCSCME on the basis of maximum weighted likelihood estimation. Second, we derive the update rules for the framewise cost functions on the basis of auxiliary-function techniques. These naive update rules are computationally costly for real-time execution on a practical machine. Thus, we finally derive the online algorithms by approximating some intermediate parameters with their estimates. Furthermore, we propose stabilization and further acceleration techniques for these online algorithms. In experiments, we simulate situations where a target speaker is stationary or moves and show that the proposed method achieves superior speech extraction performance compared with conventional methods. In addition, using real-world recorded signals, we demonstrate the effectiveness of the proposed method in practical scenarios.
Figures & tables
Figure 1: Flow of RCSCME-based speech extraction method under diffuse noise condition.
Figure 2: Schematic of parallel processing of B-RCSCME. This figure is cited from [ 19 ] .
Figure 3: Schematic of online processing for RCSCME-based speech extraction method.
Table 1: Position of proposed methods among conventional offline BSS and MSE methods and their corresponding online extensions
Figure 4: Room layout for simulating impulse responses. Noise sources were placed at green ‘ × ’ positions. Position of each noise source was perturbed by adding positional noise generated from three-dimensional standard normal distribution and scaled by 0.01 . Directivity of each noise source was hyper cardioid and was set to point outwards from center of room. Circular microphone array consisting of four omnidirectional microphones with radius of 3 cm was placed at red ‘ ◯ ’ position. In experiment described in Section 4 - 4.1 , target speaker is stationary at blue ‘ + ’ position. In experiment described in Section 4 - 4.2 , target speaker moves along blue arrow trajectory at constant speed from 10 to 20 s and from 30 to 40 s after start. At all other times, target speaker is stationary at blue ‘ + ’ position.
Figure 5: Average SDR and SIR improvements for each method in experiment with simulated stationary speaker. For each noise condition, upper and lower panels represent SDR and SIR improvements, respectively. The noise conditions are (a) station, (b) cafe, (c) traffic, (d) cafeteria, (e) restaurant, and (f) public square.
Figure 6: Maximum processing time per frame for all methods. Black dashed line indicates real-time threshold (i.e., 32 ms STFT shift length).
Figure 7: Average SDR and SIR improvements for each method in experiment with simulated moving speaker. For each noise condition, upper and lower panels represent SDR and SIR improvements, respectively. The noise conditions are (a) station, (b) cafe, (c) traffic, (d) cafeteria, (e) restaurant, and (f) public square. Gray shaded areas represent time interval in which target speaker moved around microphone array.
Figure 8: Average SDR (top panel) and SIR (bottom panel) improvements for each method in experiment evaluating online NSR-ILRMA alone with simulated stationary speaker.
Figure 9: Average SDR (top panel) and SIR (bottom panel) improvements for each method in experiment evaluating online NSR-ILRMA alone with simulated moving speaker. Gray shaded areas represent time interval in which target speaker moved around microphone array.
RCSCME
Blockwise batch
Online
NSR-ILRMA
Blockwise batch
Batch-Batch
Batch-Online
Online
Online-Batch
Online-Online
Table 2: Labels for combinations of real-time extensions of NSR-ILRMA and RCSCME based on blockwise batch and online algorithms
Figure 10: Average SDR (top panel) and SIR (bottom panel) improvements for each method in experiment evaluating combinations of blockwise batch and online algorithms with simulated stationary speaker.
Figure 11: Average SDR (top panel) and SIR (bottom panel) improvements for each method in experiment evaluating combinations of blockwise batch and online algorithms with simulated moving speaker. Gray shaded areas represent time interval in which target speaker moved around microphone array.
Figure 12: Maximum processing time per frame for all combinations of real-time extensions of RCSCME-based methods based on blockwise batch and online algorithms. Black dashed line indicates real-time threshold (i.e., 32 ms STFT shift length).
Figure 13: Room layout for real-world recording at the Ito International Research Center, The University of Tokyo. Green shaded area represents cloud of interfereing speakers located 2–4 m aroung microphone array. Two loudspeakers for playing music were embedded in ceiling. Circular microphone array consisting of four omnidirectional microphones with a radius of 3.25 cm was placed at red ‘ ◯ ’ position. Target speaker is stationary at blue ‘ + ’ position.
Figure 14: Average SDR (top panel) and SIR (bottom panel) improvements for each method in experiment using real-world recording.
The Relative Transfer Matrix (ReTM), recently introduced as a generalization of the relative transfer function for multiple receivers and sources, shows promising performance when applied to speech enhancement in noisy environments. Estimating the ReTM of sound sources by exploiting the covariance matrices of multichannel recordings is highly beneficial for practical applications and, to date, remains the only proposed approach. This paper investigates deep learning-based ReTM estimation. We propose three novel supervised learning frameworks using time and short-time frequency transform domain convolutional networks, and a Long Short-Term Memory-based recurrent neural network. Experimental results demonstrate that the proposed models achieve more accurate estimation of the ReTM using five objective metrics compared to the covariance-based method. We also show the effectiveness of the proposed frameworks for speech enhancement, achieving performance on par with the baseline method.
Oshan A. B. Yalegama, Wageesha N. Manamperi
Department of Electronic and Telecommunication Engineering, University of Moratuwa, Sri Lanka · School of Engineering, The Australian National University, Australia
Audio spotforming is a technique for extracting target speech from noisy mixtures by utilizing multiple microphone arrays. Conventional methods estimate a shared target speech component from linearly separated signals obtained by each array using low-rank approximations and apply post filtering (PF) based on this estimated low-rank representation. However, owing to the mismatch between low-rank models and the complex structure of speech signals, directly relying on low-rank approximations for PF can degrade the speech extraction performance. In this study, we leverage the observation that non-target components located in the target speech direction from the perspective of one array can be spatially separated when viewed from other arrays. This insight motivates a new spotforming method for efficient post-filter estimation using non-target estimates across arrays instead of relying on low-rank approximations. Experiments demonstrate that the proposed method outperforms conventional spotforming methods.
Yuto Ishikawa, Li Li, Shogo Seki +1
CyberAgent, Inc., Japan · The University of Tokyo, Japan
The minimum variance distortionless response (MVDR) beamformer is widely used for multichannel speech enhancement due to strong noise suppression while preserving target signals. In practice, its performance is sensitive to microphone self-noise and array mismatches. Existing approaches typically rely on fixed, manually tuned WNG thresholds or diagonal loading, leading to suboptimal performance under unknown or time-varying acoustic conditions. This paper proposes a data-driven MVDR framework that adaptively estimates the WNG constraint using a deep neural network. The network jointly predicts a time-frequency noise mask for covariance estimation and a frequency-dependent WNG threshold, enabling dynamic robustness-directivity control. A differentiable robust MVDR layer is integrated into the framework, allowing end-to-end optimization. Experiments demonstrate consistent improvements in speech quality and intelligibility over conventional fixed-WNG MVDR methods.
Yongyi Deng, Hanchen Pei, Jianbo Ma +3
School of Electronic Information, Wuhan University, Wuhan, Hubei, China · Dolby Laboratories · CIAIC, Northwestern Polytechnical University, Xi’an, Shaanxi, China +1