Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.
Figures & tables
Figure 1: Policy network architecture: LSTM processor with specialized heads for action selection and value estimation. Action masking enforces architectural constraints.
Component
Parameters
Options
Constraint
Encoder
Channels
{128, 192, 256}
E≥128
Bottleneck
Channels
{128, 192, 256}
B≤E
TCN
Blocks per repeat
{4,6,8}
–
Repeats
{2,3,4}
R×B≤24
Expansion ratio
{1.5,2.0,2.5}
–
Visual
Output channels
{128,256}
V≥128
Table 1: Constrained Search Space Configuration
Step
Params (M)
SI-SNRi (dB)
Δ SI-SNR
Reward
Key Changes
0
22.0
15.2
0.00
0.00
Baseline
5
16.2
15.1
-0.1
0.72
E :256 → 192
10
8.4
15.0
-0.2
1.25
B :256 → 192
15
5.1
14.9
-0.3
1.87
R :4 → 3
20
3.8
14.7
-0.5
2.01
X :8 → 6
25
3.1
14.6
-0.6
2.13
H :2.5 → 2.0
Table 2: Architecture Progression Through Progressive Shrinking Search
Model
Params (M)
SI-SNRi (dB)
RTF (CPU)
Audio-only baseline
5.2
10.3
0.15
AVSE baseline
22.0
15.2
0.82
Handcrafted small
6.1
14.8
0.31
Random search best
2.4
13.9
0.22
NAS-optimized (ours)
2.9
14.5
0.19
Table 3: Performance Benchmarking on CEC2 Test Set
Speech enhancement improves intelligibility and listening quality by suppressing noise, reverberation, and competing speakers in recorded or live audio.
In existing Audio-Visual Speech Enhancement (AVSE) methods, objectives such as Scale-Invariant Signal-to-Noise Ratio (SI-SNR) and Mean Squared Error (MSE) are widely used; however, their correlation with perceived speech quality is often suboptimal and provides limited interpretability for optimization. This work proposes a reinforcement learning-based AVSE framework with a Large Language Model (LLM)-based interpretable reward model. An audio LLM generates natural language descriptions of enhanced speech, which are converted by a sentiment analysis model into a 1-5 rating score serving as the PPO reward for fine-tuning a pretrained AVSE model. Compared with scalar metrics, LLM-generated feedback is semantically rich and explicitly describes speech quality improvements. Experiments on the AVSEC-4 dataset show that the proposed method outperforms a supervised baseline and a DNSMOS-based RL baseline in PESQ, STOI, neural quality metrics, and subjective listening tests.
Chih-Ning Chen, Jen-Cheng Hou, Hsin-Min Wang +3
Department of Electrical Engineering, National Taiwan University, Taiwan · Research Center for Information Technology Innovation, Academia Sinica, Taiwan · Center for Hearing Research, University of California Irvine, USA
Lightweight audio-visual speech enhancement (AVSE) models face a critical trade-off between computational efficiency and cross-modal alignment accuracy. While simple concatenation lacks relational expressiveness, dense cross-attention incurs computational overhead and is prone to unreliable cross-modal correspondence under strong acoustic interference. We propose Sparse Graph-Guided Mamba (SG-Mamba), a lightweight AVSE framework that integrates a sparse heterogeneous graph with a linear-complexity Mamba backbone. The graph explicitly models modality-specific relations through content-adaptive attention and cross-frame audio-visual connections, while Mamba captures long-range temporal context. We further introduce an audio skip connection to preserve spectral detail without sacrificing noise suppression. Evaluated on LRS3, SG-Mamba achieves competitive or superior performance against strong lightweight baselines and reaches 13.091 dB SI-SDR under noise-only condition. It also remains robust in cluttered multi-speaker conditions with a competitive cost of 3.45 G MACs (or 6.90 G FLOPs). Results on VoxCeleb2 further suggest that explicit structural priors improve robustness, generalizability, and computational efficiency in lightweight AVSE.
Guo-Ruei Tseng, Hung-Shin Lee, Hsin-Min Wang +1
Dept. of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan · Grad. Inst. of AI Interdisciplinary Applied Technology, National Taiwan Normal University, Taiwan · Inst. of Information Science, Academia Sinica, Taiwan
Audio-visual speech enhancement (AVSE) uses a target speaker's visible articulation to recover that speaker's voice from overlapped speech. Yet most evaluation protocols rely on synthetic mixtures and reliable video. The ISCSLP 2026 Real-World AVSE Challenge addresses both gaps. Track1 combines naturally recorded two-talker mixtures, which lack a clean reference, with reference-available remixes of the same speakers; Track2 additionally degrades the target video in five ways and adds 3-m far-field recordings. Sixteen and twelve teams were ranked on speaker-disjoint test data by rank averaging over waveform fidelity, predicted quality, transcription accuracy, and speaker similarity. On Track1 remixes, the best system reaches 12.7dB SI-SDR and 0.85 STOI, but natural recordings remain harder: even the lowest CER rises from 9.0% to 14.7%. The leading systems use video mainly for speaker attribution rather than signal reconstruction, and the top two lose under 0.5dB SI-SDR on Track2. UTMOS and DNSMOS rank systems differently from the other metrics, so no single metric captures target-speech recovery. We release the baselines, evaluator, and official results.
Kai Li, Wenze Ren, Junjie Li +11
1Tsinghua University · 2National Taiwan University · 3The Hong Kong Polytechnic University +8