TBDub: Production-Oriented Visual Dubbing

Bihan Li; Xinyang Li; Zeran Xu; Meiguang Jin; Junfeng Ma

cs.CVSep 5, 2026

TBDub: Production-Oriented Visual Dubbing

Authors: Bihan Li, Xinyang Li, Zeran Xu, Meiguang Jin, Junfeng Ma

Organizations: TaoLive AIGC, Taobao & Tmall Group of Alibaba

Abstract

Visual dubbing must synchronize mouth motion with replacement speech while preserving identity, appearance, and temporal consistency. Although X-Dub provides a strong mask-free video-editing baseline, its application to livestream and generated-video content reveals limitations in production-domain robustness, temporal and motion stability, identity and oral-detail preservation, and inference efficiency. We present TBDub, a production-oriented extension of X-Dub that combines task-adaptive post-training with task-aware few-step distillation. Post-training adapts the video DiT using production-domain data, production-specific conditioning and filtering, and enhanced audio features to obtain a 30-step Teacher. Distillation adapts DMD/DMD2 to conditional video editing and compresses the Teacher into a two-step Student. On 38 TalkVid clips, the Teacher improves all eight reported reconstruction, perceptual, identity, and synchronization metrics over X-Dub. In the MOS evaluation, it improves lip-sync consistency, identity consistency, and visual quality over X-Dub by 0.14, 0.95, and 0.90 points, while the Student achieves the highest lip-sync and visual-quality scores and remains close to the Teacher in identity consistency. In paired end-to-end generation timing from the first VAE encode through the final VAE decode on a single NVIDIA H20 GPU at $512\times512$ , the Student reaches 7.13 effective FPS and reduces total latency by $13.93\times$ ; the DiT stage alone is accelerated by $42.49\times$ . The Student largely retains the Teacher's generation quality and audiovisual synchronization. The code is available on GitHub at \https://github.com/TaoLiveAIGC/TBDub, and the 30-step Teacher and two-step Student weights are available on Hugging Face at https://huggingface.co/TaoLiveAIGC/TBDub.

Figures & tables

Layer	Inherited or adopted mechanisms	TBDub task adaptation
Base model	Causal video VAE, latent video DiT, reference-conditioned mask-free editing	Full-DiT post-training with motion dropout and spatial/temporal weighted flow matching
Conditions	X-Dub reference-video, audio-conditioning, motion-context, and CFG interfaces	Replacement of the original Whisper/wav2vec 2.0 audio path with an English-pretrained HuBERT-large frontend, concatenating layers 9–12 with a learned projection and segment-level temporal alignment
Data	X-Dub generative bootstrapping, identity/articulation/visual-quality filtering, and relighting	Production-domain data ingestion, audiovisual calibration and production-specific failure rules, $3{:}1$ mixture, asymmetric condition degradation, and an identity-matched clean oral prior
Distillation and sampling	X-Dub three-route dynamic CFG; DMD score difference and online FakeScore; DMD2 two-timescale updates and multi-step generation	Guided Teacher as the DMD real-score, differentiable two-step generation, condition-state handling, temporary region-selective supervision, and causal first-latent handling

Table 1: Technical boundary between inherited mechanisms and TBDub’s task-specific adaptations.

Method	Mouth PSNR $\uparrow$	Mouth SSIM $\uparrow$	Crop LPIPS $\downarrow$	Mouth LPIPS $\downarrow$	FID $\downarrow$	CSIM $\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	Response Mouth PSNR
Ground truth	–	–	–	–	–	–	6.28	8.35	–
X-Dub	22.20	0.76	0.07	0.08	9.96	0.88	5.66	9.28	23.05
LipForcing	24.33	0.82	0.05	0.06	4.34	0.91	6.13	8.37	29.28
KeySync	23.23	0.82	0.05	0.08	16.19	0.86	6.97	7.82	22.61
LatentSync	31.30	0.95	0.01	0.02	3.70	0.97	7.76	7.00	30.98
TBDub Teacher (30 steps)	27.13	0.87	0.06	0.06	9.40	0.91	6.80	7.98	25.67

Table 2: Objective comparison on 38 TalkVid clips. Bold and underline indicate the best and second-best results, respectively, in each metric column with a stated preference direction; rankings are determined from the unrounded values. Response Mouth PSNR measures response magnitude rather than quality and is not ranked. Ground truth is reported only for synchronization.

Method	Ratings	Lip-sync MOS $\uparrow$	Identity MOS $\uparrow$	Visual-quality MOS $\uparrow$
KeySync	114	2.75	2.40	2.33
LatentSync	114	1.91	2.23	1.87
LipForcing	114	2.18	2.88	2.52
X-Dub	114	3.57	2.83	2.88
TBDub Teacher (30 steps)	114	3.71	3.78	3.78
TBDub Student (2 steps, EMA)	114	3.85	3.72	3.80

Table 3: Subjective MOS results on 38 TalkVid clips. Each method contains 114 rating records. All dimensions range from 0 to 5, with higher values being better. Bold and underline indicate the best and second-best results, respectively, in each MOS column.

Method	Model / steps	FPS $\uparrow$
KeySync	Default best configuration	1.05
LatentSync	Default best configuration	1.31
LipForcing	14B open-source model, full Wan VAE	2.83
Original X-Dub	30 steps	0.51
TBDub Teacher	30 steps	0.51
TBDub Student	2 steps	7.13

Table 4: End-to-end generation throughput under the stated VAE-to-VAE boundary on one NVIDIA H20 at

512\times 512

. Higher FPS is better; bold and underline indicate the best and second-best results, respectively.

Method	VAE enc. $\downarrow$	DiT $\downarrow$	Other $\downarrow$	VAE dec. $\downarrow$	Total $\downarrow$	Eff. FPS $\uparrow$
Teacher (30 steps)	9.29 (1.26%)	696.77 (94.83%)	5.15 (0.70%)	23.54 (3.20%)	734.75	0.51
Student (2 steps)	9.27 (17.58%)	16.40 (31.09%)	3.55 (6.73%)	23.54 (44.61%)	52.76	7.13

Table 5: Paired stage-wise profile for one 376-valid-frame sample. Latency columns are in seconds; percentages show each stage’s share of total timed end-to-end generation latency within the VAE-to-VAE boundary. Lower latency and higher effective FPS are better, and bold indicates the best result in each column based on the unrounded values.

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 710 papers

Nov 23, 2025

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.

Kaidi Wang, Yi He, Wenhao Guan +9

School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1

TTS Synthesis Audio-Visual Synchronization Audio-Visual Alignment

Ask PDF

Apr 14, 2026

CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.

Gaoxiang Cong, Liang Li, Jiaxin Ye +4

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China +2

Audio-Visual Alignment Diffusion Transformer

Ask PDF

Oct 18, 2026

VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing

Automatic video dubbing (AVD) requires temporally aligned speech, transcripts, and video, yet existing corpora vary in audio quality, transcription reliability, and speaking-face visibility. We introduce VoxDub, a bilingual lip-synchronized audio-visual dataset containing 766K clips and approximately 1.58K hours of in-the-wild Chinese and English recordings, covering varied speakers, scenes, speaking styles, and linguistic content. Its curation pipeline combines speech-centered segmentation with dual-ASR transcription checking, speech quality filtering, face visibility checks, and lip-speech synchronization verification. Text-to-speech (TTS) experiments show that training on VoxDub yields lower word error rates and higher speaker similarity than training on the compared audio-visual corpora, LRS3 and CN-CVS. AVD experiments show broadly comparable speech quality and subjective synchronization. These results support the utility of VoxDub for TTS and AVD training. More details are available on the project website (https://voxdub-dataset.github.io/VoxDub/).

Kaidi Wang, Wenhao Guan, Peijie Chen +4

School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China

Audio-Visual Alignment

Ask PDF

Abstract

Video Diffusion Models

Explore similar work

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing