Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized Voice Quality Enhancement (AV-PVQE), which approaches these requirements from the other direction. We start from a personalized speech enhancement model that reconstructs a requested voice at high quality but confuses the target in 46% of two-speaker mixtures despite clean enrollment. Adding mouth features at its speaker-conditioning input and jointly fine-tuning the visual and reconstruction networks reduces this rate to 1.6%, with no future frames and 20 ms of algorithmic delay. Compared with an online autoregressive audio-visual extractor, AV-PVQE yields separation gains on two synthetic benchmarks and larger gains on recorded meetings, and keeps its advantage on excerpts with more speakers than the fine-tuning mixtures. In personalized P.835 listening tests on two meeting corpora, it improves overall quality over this extractor by 0.57 and 0.63 MOS, with similar mean rating relative to the starting model. Preservation and rejection tests show that it keeps the target intact when no competing voice is present and suppresses competing speech when the target is absent.
Figures & tables
Figure 1: AV-PVQE: new layers (orange) connect two pretrained networks (blue), and ct enters PVQE’s speaker-conditioning input. All trainable blocks except the enrollment encoder are fine-tuned.
Target recovery
Predicted quality (DNSMOS)
Dataset
Model
SI-SNRi (dB) ↑
Picked (%) ↑
WAcc (%) ↑
Spk-sim ↑
SIG ↑
BAK ↑
OVRL ↑
P.808 ↑
LRS3
AVASE
8.72
96.83
88.89
.890
3.508
2.803
2.564
2.915
PVQE
−2.31
54.17
28.57
.727
3.641
3.354
2.916
3.434
AV-PVQE (ours)
9.03
98.38
90.91
.913
3.563
2.903
2.648
3.106
VoxCeleb2
AVASE
3.81
85.52
75.00
.843
3.683
1.887
2.146
2.807
PVQE
−2.97
54.12
28.57
.725
3.617
3.046
2.754
3.501
Table 1: AV-PVQE gives the best target recovery on every corpus. Higher is better. Bold marks the best target-recovery value per corpus; DNSMOS columns are not ranked because the scorer receives no enrollment and cannot identify the recovered speaker (Sec. 4.1 ).
SI-SNRi (dB) ↑
Picked (%) ↑
Mixture group
AVASE
AV-PVQE
AVASE
AV-PVQE
2-spk-mix
4.27
12.16
84.10
93.33
3-spk-mix
1.92
7.13
72.57
91.15
4-spk-mix
0.65
4.91
56.82
75.00
Table 2: AV-PVQE keeps its lead over AVASE on MTM excerpts with more speakers than the two-speaker fine-tuning mixtures. Groups count participants who speak anywhere in an excerpt.
Target-only audio
Competing-only audio ↑
Model
Long TSOS ↓
Within ±3 dB (%) ↑
Attenuation (dB)
Static video (dB)
Shifted video (dB)
AVASE
5
98.50
.810
2.25
8.62
PVQE
142
97.42
.839
2.46
—
AV-PVQE (ours)
7
96.93
.665
32.83
18.28
Table 3: AV-PVQE nearly eliminates PVQE’s prolonged target suppression and rejects competing speech far more strongly than AVASE (LRS3). Long TSOS counts events over 1 s; clip-mean attenuation should be near zero for target-only audio and large for competing-only audio. Bold marks the best value per column.
Scale
AVASE
PVQE
(Ours) AV-PVQE
Δ vs AVASE [95% CI]
AMI
SIG
2.87
3.58
3.47
+.60 [.31, .90]
BAK
2.67
3.03
3.11
+.44 [.19, .69]
OVRL
2.11
2.74
2.68
+.57 [.32, .81]
MCoRec
SIG
1.65
2.39
2.63
+.98 [.74, 1.23]
Table 4: AV-PVQE receives higher personalized P.835 ratings than AVASE on both meeting corpora. All six AV-PVQE–AVASE difference intervals exclude zero. Ratings use a 1–5 scale; the final column reports the difference and its 95% bootstrap interval. Bold means are highest within each row.
Figure 2: Target preference on LRS3 by target-to-interferer ratio; the equal group spans ±2 dB. PVQE follows the louder voice; AV-PVQE selects the target at least 97% of the time at every level.
The goal of this paper is to provide a new perspective on audio-visual target speaker extraction (AV-TSE) by decoupling separation and target selection. Conventional AV-TSE systems typically integrate audio and visual features deeply to re-learn the entire separation process, which can act as a fidelity ceiling due to the noisy nature of in-the-wild audio-visual datasets. To address this, we propose Plug-and-Steer, which assigns high-fidelity separation to a frozen audio-only backbone and limits the role of the visual modality strictly to target selection. We introduce the Latent Steering Matrix (LSM), a minimalist linear transformation that re-routes latent features within the backbone to anchor the target speaker to a designated channel. Experiments across four representative architectures show that our method effectively preserves the acoustic priors of diverse backbones, achieving perceptual quality comparable to that of the original backbones. Audio samples are available at: https://plugandsteer.github.io
Doyeop Kwak, Suyeon Lee, Joon Son Chung
Korea Advanced Institute of Science and Technology, South Korea
Audio-visual speech enhancement (AVSE) uses a target speaker's visible articulation to recover that speaker's voice from overlapped speech. Yet most evaluation protocols rely on synthetic mixtures and reliable video. The ISCSLP 2026 Real-World AVSE Challenge addresses both gaps. Track1 combines naturally recorded two-talker mixtures, which lack a clean reference, with reference-available remixes of the same speakers; Track2 additionally degrades the target video in five ways and adds 3-m far-field recordings. Sixteen and twelve teams were ranked on speaker-disjoint test data by rank averaging over waveform fidelity, predicted quality, transcription accuracy, and speaker similarity. On Track1 remixes, the best system reaches 12.7dB SI-SDR and 0.85 STOI, but natural recordings remain harder: even the lowest CER rises from 9.0% to 14.7%. The leading systems use video mainly for speaker attribution rather than signal reconstruction, and the top two lose under 0.5dB SI-SDR on Track2. UTMOS and DNSMOS rank systems differently from the other metrics, so no single metric captures target-speech recovery. We release the baselines, evaluator, and official results.
Kai Li, Wenze Ren, Junjie Li +11
1Tsinghua University · 2National Taiwan University · 3The Hong Kong Polytechnic University +8
Target speech extraction (TSE) aims to recover a target speaker's voice from a mixture. While recent text-prompted approaches have shown promise, most approaches assume fully overlapped mixtures, limiting insight into behavior across realistic overlap ratios. We introduce VorTEX (Various overlap ratio for Target speech EXtraction), a text-prompted TSE architecture with a Decoupled Adaptive Multi-branch (DAM) Fusion block that separates primary extraction from auxiliary regularization pathways. To enable controlled analysis, we construct PORTE, a two-speaker dataset spanning overlap ratios from 0% to 100%. We further propose Suppression Ratio on Energy (SuRE), a diagnostic metric that detects suppression behavior not captured by conventional measures. Experiments show that existing models exhibit suppression or residual interference under overlap, whereas VorTEX achieves the highest separation fidelity across 20-100% overlap (e.g., 5.50 dB at 20% and 2.04 dB at 100%) while maintaining zero SuRE, indicating robust extraction without suppression-driven artifacts.
Ro-hoon Oh, Jihwan Seol, Bugeun Kim
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea