VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing

Kaidi Wang; Wenhao Guan; Peijie Chen; Weijie Wu; Xuetao Lin; Lin Li; Qingyang Hong

cs.SDOct 18, 2026

VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing

Authors: Kaidi Wang, Wenhao Guan, Peijie Chen, Weijie Wu, Xuetao Lin, Lin Li, Qingyang Hong

Organizations: School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China

Abstract

Automatic video dubbing (AVD) requires temporally aligned speech, transcripts, and video, yet existing corpora vary in audio quality, transcription reliability, and speaking-face visibility. We introduce VoxDub, a bilingual lip-synchronized audio-visual dataset containing 766K clips and approximately 1.58K hours of in-the-wild Chinese and English recordings, covering varied speakers, scenes, speaking styles, and linguistic content. Its curation pipeline combines speech-centered segmentation with dual-ASR transcription checking, speech quality filtering, face visibility checks, and lip-speech synchronization verification. Text-to-speech (TTS) experiments show that training on VoxDub yields lower word error rates and higher speaker similarity than training on the compared audio-visual corpora, LRS3 and CN-CVS. AVD experiments show broadly comparable speech quality and subjective synchronization. These results support the utility of VoxDub for TTS and AVD training. More details are available on the project website (https://voxdub-dataset.github.io/VoxDub/).

Figures & tables

Dataset	Language	Scale	Transcript	Visual	Data Source	Seg. Modality	Full Lip-Speech Align.
LibriTTS [ 26 ]	EN	585 h	✓	$\times$	Audiobook	Audio	–
Emilia [ 12 ]	Multi	101K+ h	✓	$\times$	In-the-wild	Audio	–
MEAD [ 25 ]	EN	40+ h	✓	✓	Studio recording	–	–
HDTF [ 27 ]	EN	15.8 h	$\times$	✓	In-the-wild	Visual	Assumed
CMLR [ 28 ]	ZH	61+ h	✓	✓	In-the-wild	Audio	Assumed
LRS3 [ 1 ]	EN	400+ h	✓	✓	In-the-wild	Visual	Explicit

Table 1: Comparison with representative speech and audio-visual datasets.

Figure 1: Overview of the VoxDub dataset construction pipeline.

Figure 2: Corpus composition: (a) language, (b) gender, and (c) clip duration.

Figure 3: Quality-score distributions: (a) inter-ASR WER, (b) DNSMOS P.835 OVRL, and (c) LSE-C.

Train Data	Seed-TTS test-en					DubBench-EN
Train Data	WER $\downarrow$	SIM-o $\uparrow$	UT-MOS $\uparrow$	CMOS $\uparrow$	SMOS $\uparrow$	WER $\downarrow$	SIM-o $\uparrow$	UT-MOS $\uparrow$	CMOS $\uparrow$	SMOS $\uparrow$
Emilia	1.55	0.696	3.85	0.19	4.08	2.40	0.635	3.58	0.14	3.86
LibriTTS	3.92	0.603	3.69	0.07	3.79	8.38	0.511	3.46	0.03	3.51
LRS3	12.54	0.448	3.36	-0.16	3.47	12.34	0.506	3.06	-0.07	3.55
VoxDub-EN	3.01	0.601	3.92	0.14	3.96	4.34	0.562	3.42	0.09	3.67

Table 2: English TTS results on Seed-TTS test-en and DubBench-EN. WER is reported as a percentage.

Train Data	Seed-TTS test-zh					DubBench-ZH
Train Data	WER $\downarrow$	SIM-o $\uparrow$	UT-MOS $\uparrow$	CMOS $\uparrow$	SMOS $\uparrow$	WER $\downarrow$	SIM-o $\uparrow$	UT-MOS $\uparrow$	CMOS $\uparrow$	SMOS $\uparrow$
Emilia	1.42	0.750	3.14	0.06	4.03	1.72	0.740	3.55	-0.03	4.11
CN-CVS	12.52	0.482	2.81	-0.22	3.64	7.17	0.593	2.71	-0.25	3.48
VoxDub-ZH	2.63	0.613	3.22	-0.02	3.88	3.08	0.652	2.70	-0.18	3.97

Table 3: Chinese TTS results on Seed-TTS test-zh and DubBench-ZH. WER is reported as a percentage.

Figure 4: Comparison of DNSMOS OVRL scores across different corpora. Error bars denote standard deviations.

Lang	Train Data	WER $\downarrow$	SIM-o $\uparrow$	UT-MOS $\uparrow$	LSE-C $\uparrow$	CMOS $\uparrow$	SMOS $\uparrow$	Sync-MOS $\uparrow$
EN	LRS3	6.10	0.652	3.51	7.10	-0.06	3.75	4.27
EN	VoxDub-EN	5.44	0.649	3.54	6.29	-0.02	3.83	4.12
ZH	CN-CVS	1.40	0.728	2.78	6.05	0.06	3.65	4.01
ZH	VoxDub-ZH	1.41	0.730	2.74	5.79	0.09	3.71	4.07

Table 4: AVD results on DubBench-EN and DubBench-ZH. WER is reported as a percentage.

Audio-Visual Alignment

Audio-visual alignment links sounds, speech, and music to the people, objects, events, and moments that produce or accompany them.

See all 87 papers

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
Nov 23, 2025Kaidi Wang, Yi He, Wenhao Guan +9TTS Synthesis Audio-Visual Synchronization
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
TBDub: Production-Oriented Visual Dubbing
Sep 5, 2026Bihan Li, Xinyang Li, Zeran Xu +2Video Diffusion Models Audio-Driven Facial Animation
TaoLive AIGC, Taobao & Tmall Group of Alibaba
DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
Mar 15, 2026Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi +3Expressive Speech Synthesis TTS Synthesis
FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. +1

Nov 23, 2025J/K move · Enter open · S save

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

Kaidi Wang, Yi He, Wenhao Guan, Weijie Wu+8

School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China+1

TTS Synthesis Audio-Visual Synchronization Audio-Visual Alignment

Open paper AskPDF arXiv

Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.

Abstract

Audio-Visual Alignment

Explore similar work

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

TBDub: Production-Oriented Visual Dubbing

DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization