Lend me an Ear: Speech Enhancement Using a Robotic Arm with a Microphone Array

Speech enhancement performance degrades significantly in noisy environments, limiting the deployment of speech-controlled technologies in industrial settings, such as manufacturing plants. Existing speech enhancement solutions primarily rely on advanced digital signal processing techniques, deep learning methods, or complex software optimization approaches. This paper introduces a novel speech enhancement robotic platform that can reconfigure the geometry of a microphone array and adapt to changing acoustic conditions. A sixteen-microphone array is mounted on a robotic arm manipulator with seven degrees of freedom. The microphones are divided into four groups of four, including one group positioned near the end-effector. The system reconfigures the array by adjusting the manipulator joint angles to place the end-effector microphones closer to the target speaker, thereby improving the reference signal quality. This proposed system is a multimodal sensing, reconfigurable audio capture device that integrates sound source localization techniques, computer vision, inverse kinematics, minimum variance distortionless response beamformer, and time-frequency masking using a deep neural network. Experimental results suggest that this approach outperforms other traditional recording configurations, achieving a higher average scale-invariant signal-to-distortion ratio and lower average word error rate across multiple input signal-to-noise ratio conditions.