Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized Voice Quality Enhancement (AV-PVQE), which approaches these requirements from the other direction. We start from a personalized speech enhancement model that reconstructs a requested voice at high quality but confuses the target in 46% of two-speaker mixtures despite clean enrollment. Adding mouth features at its speaker-conditioning input and jointly fine-tuning the visual and reconstruction networks reduces this rate to 1.6%, with no future frames and 20 ms of algorithmic delay. Compared with an online autoregressive audio-visual extractor, AV-PVQE yields separation gains on two synthetic benchmarks and larger gains on recorded meetings, and keeps its advantage on excerpts with more speakers than the fine-tuning mixtures. In personalized P.835 listening tests on two meeting corpora, it improves overall quality over this extractor by 0.57 and 0.63 MOS, with similar mean rating relative to the starting model. Preservation and rejection tests show that it keeps the target intact when no competing voice is present and suppresses competing speech when the target is absent.
Figures & tables
Figure 1: AV-PVQE: new layers (orange) connect two pretrained networks (blue), and ct enters PVQE’s speaker-conditioning input. All trainable blocks except the enrollment encoder are fine-tuned.
Target recovery
Predicted quality (DNSMOS)
Dataset
Model
SI-SNRi (dB) ↑
Picked (%) ↑
WAcc (%) ↑
Spk-sim ↑
SIG ↑
BAK ↑
OVRL ↑
P.808 ↑
LRS3
AVASE
8.72
96.83
88.89
.890
3.508
2.803
2.564
2.915
PVQE
−2.31
54.17
28.57
.727
3.641
3.354
2.916
3.434
AV-PVQE (ours)
9.03
98.38
90.91
.913
3.563
2.903
2.648
3.106
VoxCeleb2
AVASE
3.81
85.52
75.00
.843
3.683
1.887
2.146
2.807
PVQE
−2.97
54.12
28.57
.725
3.617
3.046
2.754
3.501
Table 1: AV-PVQE gives the best target recovery on every corpus. Higher is better. Bold marks the best target-recovery value per corpus; DNSMOS columns are not ranked because the scorer receives no enrollment and cannot identify the recovered speaker (Sec. 4.1 ).
SI-SNRi (dB) ↑
Picked (%) ↑
Mixture group
AVASE
AV-PVQE
AVASE
AV-PVQE
2-spk-mix
4.27
12.16
84.10
93.33
3-spk-mix
1.92
7.13
72.57
91.15
4-spk-mix
0.65
4.91
56.82
75.00
Table 2: AV-PVQE keeps its lead over AVASE on MTM excerpts with more speakers than the two-speaker fine-tuning mixtures. Groups count participants who speak anywhere in an excerpt.
Target-only audio
Competing-only audio ↑
Model
Long TSOS ↓
Within ±3 dB (%) ↑
Attenuation (dB)
Static video (dB)
Shifted video (dB)
AVASE
5
98.50
.810
2.25
8.62
PVQE
142
97.42
.839
2.46
—
AV-PVQE (ours)
7
96.93
.665
32.83
18.28
Table 3: AV-PVQE nearly eliminates PVQE’s prolonged target suppression and rejects competing speech far more strongly than AVASE (LRS3). Long TSOS counts events over 1 s; clip-mean attenuation should be near zero for target-only audio and large for competing-only audio. Bold marks the best value per column.
Scale
AVASE
PVQE
(Ours) AV-PVQE
Δ vs AVASE [95% CI]
AMI
SIG
2.87
3.58
3.47
+.60 [.31, .90]
BAK
2.67
3.03
3.11
+.44 [.19, .69]
OVRL
2.11
2.74
2.68
+.57 [.32, .81]
MCoRec
SIG
1.65
2.39
2.63
+.98 [.74, 1.23]
Table 4: AV-PVQE receives higher personalized P.835 ratings than AVASE on both meeting corpora. All six AV-PVQE–AVASE difference intervals exclude zero. Ratings use a 1–5 scale; the final column reports the difference and its 95% bootstrap interval. Bold means are highest within each row.
Figure 2: Target preference on LRS3 by target-to-interferer ratio; the equal group spans ±2 dB. PVQE follows the louder voice; AV-PVQE selects the target at least 97% of the time at every level.