WhisperVC-AV: Audio-Visual Content Restoration for Noise-Robust Whisper-to-Normal Voice Conversion

Ziyue Yin; Dong Liu; Ming Li

eess.ASSep 26, 2026

WhisperVC-AV: Audio-Visual Content Restoration for Noise-Robust Whisper-to-Normal Voice Conversion

Authors: Ziyue Yin, Dong Liu, Ming Li

Organizations: Digital Innovation Research Center, Duke Kunshan University, Kunshan, China · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · School of Computer Science, Wuhan University, Wuhan, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China

Abstract

Environmental noise obscures the content cues needed for whisper-to-normal conversion. We propose WhisperVC-AV, which restores acoustic content features while leaving the original WhisperVC conversion module unchanged. Its context-guided restoration module combines temporal acoustic context with synchronized lip features through attention and gated residual correction. Experiments on AISHELL6-Whisper show lower character error rates (CERs) than WhisperVC across three ASR systems, on clean speech and under all six signal-to-noise ratio (SNR) conditions with MUSAN noise. The largest gains occur at 0 dB SNR, where Qwen3-ASR CER falls from 36.29% to 28.88%. WhisperVC-AV also improves predicted speech quality while maintaining speaker similarity. The CER gains extend to unseen background noise without retraining, while visual controls support the use of utterance-specific lip cues. Audio examples are available on our demo page.

Figures & tables

Figure 1: Overview of WhisperVC-AV. Acoustic and visual cues restore content features while the original WhisperVC conversion module remains unchanged. All speech branches share the same frozen ASR encoder. Dashed paths and feature-loss connections are used only during training.

Figure 2: Context-guided content restoration. Acoustic queries attend to projected lip features

z

within

\mathcal{N}(t)

. Acoustic context

u_{t}

and visual context

c_{t}

jointly determine the gated residual, which the identity path adds to content features

H

to obtain restored features

\hat{H}

ASR	System	Clean	0 dB	5 dB	10 dB	15 dB	20 dB	25 dB	Noisy avg.
Qwen3-ASR	WhisperVC [ 11 ]	11.47	36.29	26.23	18.91	15.40	13.43	12.34	20.43
	MP-SENet-DNS + WhisperVC	24.42	43.13	34.31	26.97	22.57	19.72	17.90	27.43
	SEMamba + WhisperVC	25.34	61.63	52.04	43.18	35.67	31.63	28.71	42.14
	Direct AV adaptation	10.40	29.99	21.51	16.18	13.20	11.65	11.05	17.26
	WhisperVC-A	10.11	30.92	21.76	16.08	13.27	11.66	10.87	17.43
	WhisperVC-AV (ours)	9.86	28.88	20.60	15.31	12.77	11.36	10.57	16.58

Table 1: Character error rates (CER, %) on clean speech and under six SNR conditions with MUSAN noise. The cascades enhance waveforms before WhisperVC. WhisperVC-A is the separately trained audio-only ablation. Noisy avg. averages the six noisy conditions; lower is better, with each condition’s best result in bold.

ASR	Compared with	Reduction [95% CI]
Qwen3-ASR	WhisperVC	3.85 [3.57, 4.13]
	Direct AV adaptation	0.68 [0.48, 0.88]
	WhisperVC-A	0.85 [0.73, 0.97]
Finetuned Whisper	WhisperVC	9.07 [7.76, 10.46]
(Large-V3) in [ 10 ]	Direct AV adaptation	3.51 [2.64, 4.47]
	WhisperVC-A	1.10 [0.69, 1.54]

Table 2: Mean CER reductions of WhisperVC-AV over six SNR conditions with MUSAN noise (percentage points), with pointwise 95% confidence intervals (CIs). All nine two-sided paired tests yield raw

p=0.000005

(199,999 label swaps; plus-one correction) and Holm-adjusted

p=0.000045

Table 3: Predicted speech quality, speaker similarity, and reference consistency for clean input and the average over six SNR conditions with MUSAN noise. Higher values are better; bold indicates the best rounded score within each condition.

Voice Conversion

Voice conversion changes a speaker's vocal identity or delivery while preserving the spoken content.

See all 34 papers

WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation
Mar 9, 2026Zihao Fang, Yingda Shen, Zifan Guan +3Zero-Shot Voice Conversion Speech Processing
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Honor Device Co., Ltd, China
FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching
Mar 4, 2026Fabian Ritter-Gutierrez, Md Asif Jalal, Pablo Peso Parada +5Speech Processing Voice Conversion
Samsung Electronics R&D Institute UK (SRUK), London, United Kingdom · Samsung Electronics, Mobile eXperience Business, Suwon, Republic of Korea
MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Jun 8, 2026Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6Zero-Shot Voice Conversion Diffusion Transformer
The University of New South Wales, Australia · WeNet Open Source Community, China

Mar 9, 2026J/K move · Enter open · S save

WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation

Zihao Fang, Yingda Shen, Zifan Guan, Tongtong Song+2

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Honor Device Co., Ltd, China

Zero-Shot Voice Conversion Speech Processing Synthetic Data Augmentation

Open paper AskPDF arXiv

Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.

Abstract

Voice Conversion

Explore similar work

WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation

FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching

MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion