Organizations: National Institute of Technology Jalandhar, Jalandhar, India · Kalinga Institute of Industrial Technology, Bhubaneswar, India · IMC University of Applied Sciences, Krems, Austria
Audio-visual zooming on mobile devices is fundamentally constrained by limited microphone count, compute resource, unknown array geometry, and weak assumptions on real-world acoustic conditions. We present RDSE: Real-Time Directional Speech Enhancement Framework, a hybrid pipeline that combines traditional signal processing and modern machine learning methods for dual-microphone low-end smartphones. Unlike tightly coupled AV systems that continuously use visual feeds or embeddings, RDSE decouples visual interaction from feature inference, using camera input solely to parameterize the steering vector for spatial filtering. The proposed system uses four modules: dereverberation network, a beamformer with a temporal convolutional network, and a MetricGAN based denoiser. Extensive evaluation on benchmark data and real-world recordings demonstrates effective zooming without precise geometry calibration. Comparative experiments show competitive performance to existing methods. Proposed system is rigorously tested for statistical validity, ablation study, latency profiling, and practicality through real-time live-streaming.
Figures & tables
Figure 1: Two modes of RDSE
Figure 2: Block diagram of the proposed RDSE processing pipeline. Each microphone channel undergoes independent time-frequency transformation and U-Net based dereverberation prior to MVDR beamforming. The beamformer output is further refined using Conv-TasNet based target-speaker extraction and MetricGAN based speech enhancement.
Figure 3: Effect of dereverberation on speech signals in the time-frequency and time domains [ 5 ] .
Figure 4: Comparison of room impulse responses (RIR) under two acoustic conditions. (a) Anechoic condition ( RT60=0.0s ) showing only the direct path with no reflections. (b) Reverberant condition ( RT60=0.5s ) showing the direct path followed by decaying reflections.
Figure 5: Distribution of (a) mixture signal energy (RMS amplitude) and (b) GCC-PHAT spatial sharpness (peak-to-mean ratio) across anechoic and reverberant conditions.
Figure 6: Distribution of (a) mixture spectral centroid and (b) inter-microphone level difference (ILD) across anechoic and reverberant conditions.
Condition
STOI
ViSQOL
PESQ
Anechoic (RT 60 = 0.0 s)
0.7754
2.0053
2.5755
Reverberant (RT 60 = 0.5 s)
0.5539
1.871
1.8185
Table 1: Perceptual quality and intelligibility metrics for RDSE
Figure 7: Representative waveforms under anechoic and reverberant conditions. Each panel shows the clean target speech, the mixtures recorded at Mic 1 and Mic 2, and the final RDSE output.
Method
STOI
ViSQOL
PESQ
Time (s)
Real-time Feasible
Anechoic (RT 60 = 0.0 s)
Conv-TasNet [ 6 ]
0.6169
—
1.1999
6.05
✓
SepFormer [ 30 ]
0.5015
1.5779
1.6239
22.79
×
FasNet-TAC [ 31 ]
0.4494
1.5670
1.0780
3.50
✓
Reverberant (RT 60 = 0.5 s)
Conv-TasNet [ 6 ]
0.4612
—
1.0544
5.62
✓
Table 2: Neural backbone selection on SpARZ100
Method
STOI
ViSQOL
PESQ
Time (s)
Real-time Feasible
Mic Setup
Test Dataset
Anechoic (RT 60 = 0.0 s)
RDSE (Dereverb-off)
0.7754
2.0053
2.5755
3.95
✓
2-mic
SpARZ100
RDSE (Denoising-off)
0.7356
2.0482
0.8789
10.11
✓
2-mic
SpARZ100
Nair et al. [ 1 ]
0.7100
—
2.0400
—
×
6-mic
AVSpeech
Li et al. [ 23 ]
0.8400
—
2.0300
—
×
2-mic
VoxCeleb2
Ephrat et al. [ 4 ]
—
—
2.4200
—
×
1-mic
TCD-TIMIT
Table 3: comparison with prior work
Configuration
STOI
ViSQOL
PESQ
Time (s)
Anechoic (RT 60 = 0.0 s)
RDSE (Dereverberation-on)
0.7356
2.0482
0.8789
10.11
RDSE (Dereverberation-off)
0.7754
2.0053
2.5755
3.95
RDSE (Denoising-off)
0.7416
2.1047
1.9645
14.85
RDSE (MVDR+Conv-TasNet)
0.7890
2.0713
1.9106
1.38
Reverberant (RT 60 = 0.5 s)
Table 4: Ablation study results under anechoic and reverberant conditions. Inference time is reported as mean seconds per sample.
Figure 8: Time-domain waveforms from a real-world indoor recording (Samsung F24 smartphone). Top to bottom: mixture at Mic 1, mixture at Mic 2, and final processed output.
Condition
Platform
Input (s)
Latency (s)
Live smartphone
MATLAB Mobile
20.0
8.86
Table 5: End-to-end processing latency and real-time factor (RTF).
Figure 9: Bootstrap 95% confidence intervals for (a) STOI, (b) ViSQOL, and (c) PESQ. Anechoic ( RT60=0.0s ) results are shown on the left and reverberant ( RT60=0.5s ) on the right. Points denote mean scores and horizontal bars denote confidence bounds. Non-overlapping intervals indicate statistically significant separation. Abbreviations: FT = FasNetTAC, SF = SepFormer, FP = RDSE (Dereverberation-on), MC = RDSE (MVDR+Conv-TasNet), DR = RDSE (Dereverberation-off), ES = RDSE (Denoising-off). Exact values are given in the supplementary material.
Metric
Condition
χ2(5)
p -value
Kendall’s W
STOI
Anechoic
85.97
4.72×10−17
0.040
Reverberant
70.83
6.87×10−14
0.033
ViSQOL
Anechoic
83.55
1.51×10−16
0.039
Reverberant
45.38
1.21×10−8
0.021
PESQ
Anechoic
41.31
8.12×10−8
0.091
Reverberant
68.26
2.35×10−13
0.118
Table 6: Friedman test results for all metric-condition combinations. All tests are highly significant ( p≪0.001 ), confirming that the six methods are not statistically equivalent. Kendall’s W reports the effect size; PESQ yields the highest concordance, indicating the most consistent ranking differences across samples.
STOI — Anechoic
FT
SF
DR
ES
FP
MC
FT
—
0.0056
1.70×10−10
4.28×10−6
0.0011
2.22×10−16
SF
0.0056
—
0.014
0.072
0.108
9.87×10−6
DR
1.70×10−10
0.014
—
0.642
0.481
0.871
ES
4.28×10−6
0.072
0.642
—
0.992
0.0099
FP
0.0011
0.108
0.481
0.992
—
0.054
Table 7: Nemenyi post-hoc p -values for STOI pairwise comparisons. Values below 0.05 indicate a statistically significant difference. Green : p<0.05 ; grey : not significant. FT = FasNetTAC, SF = Sepformer, DR = RDSE (Dereverberation-off), ES = RDSE (Denoising-off), FP = RDSE (Dereverberation-on), MC = RDSE (MVDR + Conv-TasNet).
FT vs DR
FT vs ES
FT vs FP
FT vs MC
SF vs DR
FP vs DR
FP vs MC
Metric
Anec.
Rev.
Anec.
Rev.
Anec.
Rev.
Anec.
Rev.
Anec.
Rev.
Anec.
Rev.
Anec.
Rev.
ViSQOL
<0.001
—
<0.001
0.012
<0.001
0.031
<0.001
—
—
—
—
—
—
—
PESQ
<0.001
<0.001
0.003
—
—
—
—
0.004
0.018
—
<0.001
<0.001
—
0.023
Table 8: Summary of significant Nemenyi pairwise comparisons for ViSQOL and PESQ ( p<0.05 ). Dashes (—) indicate non-significant differences for that specific pair and condition. Abbreviations: FT = FasNetTAC, SF = Sepformer, DR = RDSE (Dereverberation-off), ES = RDSE (Denoising-off), FP = RDSE (Dereverberation-on), MC = RDSE (MVDR + Conv-TasNet).
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.
Khubaib Ahmed, Ahsan Adeel, Nasir Saleem +3
University of Stirling, UK · Edinburgh Napier University, UK · University of Technology and Applied Sciences, Oman
Audio-visual speech enhancement under real-world conditions remains challenging due to unreliable visual inputs and the lack of large-scale training data with realistic acoustic conditions. Existing approaches usually fuse visual features directly into the separation network, making them vulnerable to degraded visual signals. In this paper, we present DAVE, a decoupled audio-visual enhancement framework for real-world speech separation. Firstly, to address the data scarcity issue, we construct DAVE-Corpus, a large-scale training corpus with 219,411 mixtures generated from public meeting corpora through combinatorial acoustic augmentation. Then, we introduce a progressive multi-objective optimization strategy to jointly improve speech separation, intelligibility, speaker identity preservation, and perceptual quality. We further develop a certified selective enhancement chain that applies scene routing, GAN-based denoising, and loudness normalization only within the no-reference partition, guaranteeing non-degradation of reference-based metrics. Experimental results on the Real-World Audio-Visual Speech Enhancement Challenge demonstrate the robustness of DAVE under both real-world mixed scenarios and visual degradation conditions.
Wei Zhou, Wanyi Ning, Yinshang Guo +3
1Yijiahe AI · 2Tianjin University · 3Nanjing University
Real-time binaural speech enhancement is constrained by latency, computational cost, and inter-device communication, yet existing efficient solutions predominantly address single-channel settings. In this paper, we introduce RT-Tango, a real-time distributed binaural speech enhancement framework designed for streaming on resource-constrained platforms and specifically for hearing aids. RT-Tango relies on a two-stage distributed architecture combining perceptually motivated ERB feature compression, lightweight grouped recurrent mask estimation, and temporal sparsification to reduce computational cost. Stringent latency constraints are addressed by decoupling spectral resolution from algorithmic delay using an asymmetric STFT, together with causal recurrent inference and online estimation of spatial statistics. Experimental results show that RT-Tango achieves competitive speech enhancement while significantly reducing MACs operations and functioning at ultra-low latencies as low as 8 ms.
Z. Benslimane, P. Chouteau, M. Poreba +4
Universit´e Paris-Saclay, CEA, List, F-91120 Palaiseau, France · Universit´e de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France