RDSE: Real-Time Directional Speech Enhancement Framework for Dual-Mic Commodity Smartphones

Chinmay Deo; Danish Mahajan; Soudeep Ghoshal; Himanshu Buckchash

cs.SDAug 14, 2026

RDSE: Real-Time Directional Speech Enhancement Framework for Dual-Mic Commodity Smartphones

Authors: Chinmay Deo, Danish Mahajan, Soudeep Ghoshal, Himanshu Buckchash

Organizations: National Institute of Technology Jalandhar, Jalandhar, India · Kalinga Institute of Industrial Technology, Bhubaneswar, India · IMC University of Applied Sciences, Krems, Austria

Abstract

Audio-visual zooming on mobile devices is fundamentally constrained by limited microphone count, compute resource, unknown array geometry, and weak assumptions on real-world acoustic conditions. We present RDSE: Real-Time Directional Speech Enhancement Framework, a hybrid pipeline that combines traditional signal processing and modern machine learning methods for dual-microphone low-end smartphones. Unlike tightly coupled AV systems that continuously use visual feeds or embeddings, RDSE decouples visual interaction from feature inference, using camera input solely to parameterize the steering vector for spatial filtering. The proposed system uses four modules: dereverberation network, a beamformer with a temporal convolutional network, and a MetricGAN based denoiser. Extensive evaluation on benchmark data and real-world recordings demonstrates effective zooming without precise geometry calibration. Comparative experiments show competitive performance to existing methods. Proposed system is rigorously tested for statistical validity, ablation study, latency profiling, and practicality through real-time live-streaming.

Figures & tables

Figure 2: Block diagram of the proposed RDSE processing pipeline. Each microphone channel undergoes independent time-frequency transformation and U-Net based dereverberation prior to MVDR beamforming. The beamformer output is further refined using Conv-TasNet based target-speaker extraction and MetricGAN based speech enhancement.

Figure 3: Effect of dereverberation on speech signals in the time-frequency and time domains [ 5 ] .

Figure 4: Comparison of room impulse responses (RIR) under two acoustic conditions. (a) Anechoic condition (

\text{RT60}=0.0\,\text{s}

) showing only the direct path with no reflections. (b) Reverberant condition (

\text{RT60}=0.5\,\text{s}

) showing the direct path followed by decaying reflections.

Figure 5: Distribution of (a) mixture signal energy (RMS amplitude) and (b) GCC-PHAT spatial sharpness (peak-to-mean ratio) across anechoic and reverberant conditions.

Figure 6: Distribution of (a) mixture spectral centroid and (b) inter-microphone level difference (ILD) across anechoic and reverberant conditions.

Condition	STOI	ViSQOL	PESQ
Anechoic (RT 60 = 0.0 s)	0.7754	2.0053	2.5755
Reverberant (RT 60 = 0.5 s)	0.5539	1.871	1.8185

Table 1: Perceptual quality and intelligibility metrics for RDSE

Panel of Figure 7 — Figure 7: Representative waveforms under anechoic and reverberant conditions. Each panel shows the clean target speech, the mixtures recorded at Mic 1 and Mic 2, and the final RDSE output.

Method	STOI	ViSQOL	PESQ	Time (s)	Real-time Feasible
Anechoic (RT 60 = 0.0 s)
Conv-TasNet [ 6 ]	0.6169	—	1.1999	6.05	✓
SepFormer [ 30 ]	0.5015	1.5779	1.6239	22.79	$\times$
FasNet-TAC [ 31 ]	0.4494	1.5670	1.0780	3.50	✓
Reverberant (RT 60 = 0.5 s)
Conv-TasNet [ 6 ]	0.4612	—	1.0544	5.62	✓

Table 2: Neural backbone selection on SpARZ100

Method	STOI	ViSQOL	PESQ	Time (s)	Real-time Feasible	Mic Setup	Test Dataset
Anechoic (RT 60 = 0.0 s)
RDSE (Dereverb-off)	0.7754	2.0053	2.5755	3.95	✓	2-mic	SpARZ100
RDSE (Denoising-off)	0.7356	2.0482	0.8789	10.11	✓	2-mic	SpARZ100
Nair et al. [ 1 ]	0.7100	—	2.0400	—	$\times$	6-mic	AVSpeech
Li et al. [ 23 ]	0.8400	—	2.0300	—	$\times$	2-mic	VoxCeleb2
Ephrat et al. [ 4 ]	—	—	2.4200	—	$\times$	1-mic	TCD-TIMIT

Table 3: comparison with prior work

Configuration	STOI	ViSQOL	PESQ	Time (s)
Anechoic (RT 60 = 0.0 s)
RDSE (Dereverberation-on)	0.7356	2.0482	0.8789	10.11
RDSE (Dereverberation-off)	0.7754	2.0053	2.5755	3.95
RDSE (Denoising-off)	0.7416	2.1047	1.9645	14.85
RDSE (MVDR+Conv-TasNet)	0.7890	2.0713	1.9106	1.38
Reverberant (RT 60 = 0.5 s)

Table 4: Ablation study results under anechoic and reverberant conditions. Inference time is reported as mean seconds per sample.

Figure 8: Time-domain waveforms from a real-world indoor recording (Samsung F24 smartphone). Top to bottom: mixture at Mic 1, mixture at Mic 2, and final processed output.

Condition	Platform	Input (s)	Latency (s)
Live smartphone	MATLAB Mobile	20.0	8.86

Table 5: End-to-end processing latency and real-time factor (RTF).

Figure 9: Bootstrap 95% confidence intervals for (a) STOI, (b) ViSQOL, and (c) PESQ. Anechoic (

\mathrm{RT}_{60}=0.0\,\mathrm{s}

) results are shown on the left and reverberant (

\mathrm{RT}_{60}=0.5\,\mathrm{s}

) on the right. Points denote mean scores and horizontal bars denote confidence bounds. Non-overlapping intervals indicate statistically significant separation. Abbreviations: FT = FasNetTAC, SF = SepFormer, FP = RDSE (Dereverberation-on), MC = RDSE (MVDR+Conv-TasNet), DR = RDSE (Dereverberation-off), ES = RDSE (Denoising-off). Exact values are given in the supplementary material.

Metric	Condition	$\chi^{2}(5)$	$p$ -value	Kendall’s $W$
STOI	Anechoic	85.97	$4.72{\times}10^{-17}$	0.040
STOI	Reverberant	70.83	$6.87{\times}10^{-14}$	0.033
ViSQOL	Anechoic	83.55	$1.51{\times}10^{-16}$	0.039
ViSQOL	Reverberant	45.38	$1.21{\times}10^{-8}$	0.021
PESQ	Anechoic	41.31	$8.12{\times}10^{-8}$	0.091
PESQ	Reverberant	68.26	$2.35{\times}10^{-13}$	0.118

Table 6: Friedman test results for all metric-condition combinations. All tests are highly significant (

p\ll 0.001

), confirming that the six methods are not statistically equivalent. Kendall’s

W

reports the effect size; PESQ yields the highest concordance, indicating the most consistent ranking differences across samples.

STOI — Anechoic
	FT	SF	DR	ES	FP	MC
FT	—	$0.0056$	$1.70{\times}10^{-10}$	$4.28{\times}10^{-6}$	$0.0011$	$2.22{\times}10^{-16}$
SF	$0.0056$	—	$0.014$	$0.072$	$0.108$	$9.87{\times}10^{-6}$
DR	$1.70{\times}10^{-10}$	$0.014$	—	$0.642$	$0.481$	$0.871$
ES	$4.28{\times}10^{-6}$	$0.072$	$0.642$	—	$0.992$	$0.0099$
FP	$0.0011$	$0.108$	$0.481$	$0.992$	—	$0.054$

Table 7: Nemenyi post-hoc

p

-values for STOI pairwise comparisons. Values below 0.05 indicate a statistically significant difference. Green :

p<0.05

; grey : not significant. FT = FasNetTAC, SF = Sepformer, DR = RDSE (Dereverberation-off), ES = RDSE (Denoising-off), FP = RDSE (Dereverberation-on), MC = RDSE (MVDR + Conv-TasNet).

FT vs DR

FT vs ES

FT vs FP

FT vs MC

SF vs DR

FP vs DR

FP vs MC

Metric

Anec.

Rev.

Anec.

Rev.

Anec.

Rev.

Anec.

Rev.

Anec.

Rev.

Anec.

Rev.

Anec.

Rev.

ViSQOL

<0.001

—

<0.001

0.012

<0.001

0.031

<0.001

—

PESQ

<0.001

<0.001

0.003

—

0.004

0.018

—

<0.001

<0.001

—

0.023

Table 8: Summary of significant Nemenyi pairwise comparisons for ViSQOL and PESQ (

p<0.05

). Dashes (—) indicate non-significant differences for that specific pair and condition. Abbreviations: FT = FasNetTAC, SF = Sepformer, DR = RDSE (Dereverberation-off), ES = RDSE (Denoising-off), FP = RDSE (Dereverberation-on), MC = RDSE (MVDR + Conv-TasNet).

Speech Enhancement

Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.

See all 129 papers

Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search
Oct 8, 2026Khubaib Ahmed, Ahsan Adeel, Nasir Saleem +3Speech Enhancement Neural Architecture Search
University of Stirling, UK · Edinburgh Napier University, UK · University of Technology and Applied Sciences, Oman
DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation
Aug 10, 2026Wei Zhou, Wanyi Ning, Yinshang Guo +3Speech Enhancement Noise Robustness
1Yijiahe AI · 2Tianjin University · 3Nanjing University
RT-Tango: Real-Time Distributed Binaural Speech Enhancement for Low-Power Hearing Aid Devices
Jul 2, 2026Z. Benslimane, P. Chouteau, M. Poreba +4Speech Processing Speech Enhancement
Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Universit´e de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France

Oct 8, 2026J/K move · Enter open · S save

Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search

Khubaib Ahmed, Ahsan Adeel, Nasir Saleem, Kia Dashtipour+2

University of Stirling, UK · Edinburgh Napier University, UK · University of Technology and Applied Sciences, Oman

Speech Enhancement Neural Architecture Search Hardware-Aware NAS

Open paper AskPDF arXiv

Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.

Abstract

Speech Enhancement

Explore similar work

Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search

DAVE: A Decoupled Audio-Visual Enhancement Framework for Real-World Speech Separation

RT-Tango: Real-Time Distributed Binaural Speech Enhancement for Low-Power Hearing Aid Devices