DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization

Ngoc-Son Nguyen; Thanh V. T. Tran; Jeongsoo Choi; Hieu-Nghia Huynh-Nguyen; Truong-Son Hy; Van Nguyen

cs.CVMar 15, 2026

DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization

Authors: Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi, Hieu-Nghia Huynh-Nguyen, Truong-Son Hy, Van Nguyen

Organizations: FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. · University of Alabama at Birmingham, USA

Abstract

Video dubbing requires content accuracy, expressive prosody, high-quality acoustics, and precise lip synchronization, yet existing approaches struggle on all four fronts. To address these issues, we propose DiFlowDubber, the first video dubbing framework built upon a discrete flow matching backbone with a novel two-stage training strategy. In the first stage, a zero-shot text-to-speech (TTS) system is pre-trained on large-scale corpora, where a deterministic architecture captures linguistic structures, and the Discrete Flow-based Prosody-Acoustic (DFPA) module models expressive prosody and realistic acoustic characteristics. In the second stage, we propose the Content-Consistent Temporal Adaptation (CCTA) to transfer TTS knowledge to the dubbing domain: its Synchronizer enforces cross-modal alignment for lip-synchronized speech. Complementarily, the Face-to-Prosody Mapper (FaPro) conditions prosody on facial expressions, whose outputs are then fused with those of the Synchronizer to construct rich, fine-grained multimodal embeddings that capture prosody-content correlations, guiding the DFPA to generate expressive prosody and acoustic tokens for content-consistent speech. Experiments on two benchmark datasets demonstrate that DiFlowDubber outperforms prior methods across multiple evaluation metrics.

Figures & tables

Figure 1: Overall Inference Pipeline of DiFlowDubber. Face-to-Prosody Mapper module predicts global prosody priors that capture global prosody and stylistic cues from facial expressions. Content-Consistent Temporal Adaptation module generates discrete content tokens conditioned on lip movements, text, and prosody priors, ensuring consistency with the target text transcription and temporal alignment. Discrete Flow-based Prosody-Acoustic module generates diverse yet globally consistent prosody tokens under the guidance of the prosody prior, together with corresponding acoustic tokens. The speech waveform is synthesized from the predicted tokens and speaker embedding via a Codec Decoder.

Figure 2: Detailed of DiFlowDubber Pipeline. Our framework is a two-stage pipeline. The first stage performs zero-shot TTS pre-training, where a simple deterministic content modeling architecture efficiently captures linguistic structures (orange dashed box). For prosody and acoustic attributes, we adopt (b) Discrete Flow-based Prosody-Acoustic (DFPA) module to model expressive prosodic variations and realistic acoustic diversity from the corpus. In the second stage, the model is adapted to the video dubbing task, where (a) Content-Consistent Temporal Adaptation module transfers consistent content knowledge from the TTS domain and generates temporally aligned content representations. Meanwhile, FaPro extracts a global prosody prior from facial expression cues. The DFPA module then models the joint distribution of prosody and acoustic tokens conditioned on the global prosody prior, latent content representations, and speaker embedding.

Model	Setting 1.0					Setting 2.0
Model	LSE-C $\uparrow$	LSE-D $\downarrow$	WER $\downarrow$	SECS $\uparrow$	UTMOS $\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	WER $\downarrow$	SECS $\uparrow$	MOS-S $\uparrow$	UTMOS $\uparrow$
Ground Truth	8.12	6.59	3.85	100.00	4.18	8.12	6.59	3.85	100.00	4.34 $\pm$ 0.18	4.18
V2C-Net [ 2 ] CVPR’22	1.97	12.17	90.47	51.52	1.81	1.82	12.09	94.59	44.19	-	1.76
HPMDubbing [ 11 ] CVPR’23	7.85	7.19	16.05	85.09	2.16	3.98	9.50	29.82	73.55	-	2.01
StyleDubber [ 13 ] ACL’24	3.87	10.92	13.14	87.72	3.14	3.74	11.00	14.18	82.07	4.02 $\pm$ 0.24	3.04
Speaker2Dubber [ 60 ] ACM MM’24	3.76	10.56	16.98	74.73	3.61	3.45	11.17	18.10	69.28	3.69 $\pm$ 0.25	3.64

Table 1: Performance on the Chem dataset under both Setting 1.0 and Setting 2.0. In the Dub 1.0 Setting, the ground truth audio is used as the reference, while in the Dub 2.0 Setting, a non-ground-truth audio sample from the same speaker within the dataset is used as the reference. The best and second-best results are shown in bold and underlined , respectively.

Model	Setting 1.0					Setting 2.0
Model	LSE-C $\uparrow$	LSE-D $\downarrow$	WER $\downarrow$	SECS $\uparrow$	UTMOS $\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	WER $\downarrow$	SECS $\uparrow$	MOS-S $\uparrow$	UTMOS $\uparrow$
Ground Truth	7.13	6.78	22.41	100.00	3.94	7.13	6.78	22.41	100.00	4.58 $\pm$ 0.13	3.94
V2C-Net [ 2 ] CVPR’22	5.59	9.52	47.82	80.98	2.41	5.34	9.76	49.09	71.51	-	2.40
HPMDubbing [ 11 ] CVPR’23	5.76	9.13	45.51	85.11	2.14	5.82	9.10	44.15	71.99	-	2.11
StyleDubber [ 13 ] ACL’24	6.12	9.03	18.88	93.79	3.73	6.09	9.08	19.58	86.67	3.83 $\pm$ 0.24	3.71
Speaker2Dubber [ 60 ] ACM MM’24	5.27	9.84	17.07	94.50	3.69	5.19	9.93	17.42	85.76	3.57 $\pm$ 0.26	3.73

Table 2: Performance on the GRID dataset is evaluated under the same two Dub settings as in the Chem dataset.

#	Model	AVSync $\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	WER $\downarrow$	SECS $\uparrow$	UTMOS $\uparrow$
1	Ours	0.721	8.31	6.73	9.65	84.59	4.02
Zero-shot TTS Pre-training
2	w/o Zero-shot TTS Pre-training	0.715	8.17	6.79	12.04	83.72	3.53
Synchronizer
3	w/o ( $\mathcal{L}_{\text{VT}}+\mathcal{L}_{\text{ST}}$ )	0.652	8.26	6.77	17.15	84.29	3.90
4	w/o $\mathcal{L}_{\text{VT}}$	0.681	8.36	6.71	16.57	83.94	3.87

Table 3: Ablation study results on the Chem test set.

Model	#Params	NFE	RTF $\downarrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	WER $\downarrow$	SECS $\uparrow$	UTMOS $\uparrow$
ProDubber	200M	5	0.08	2.58	12.54	9.45	72.13	3.85
EmoDubber	116M	10	0.05	8.11	6.92	11.72	90.62	3.82
DiFlowDubber	250M	1	0.03	8.25	6.78	10.72	83.63	2.82
		2	0.04	8.27	6.74	10.99	83.34	2.79
		4	0.04	8.32	6.73	10.79	83.82	3.50
		8	0.05	8.29	6.73	10.28	84.50	3.85

Table 4: Analysis of NFE in inference under the Dub 1.0 Setting. #Params exclude video-related encoders.

Figure 3: Mel-spectrogram visualization compared with Ground-Truth speech.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The detailed architecture of Synchronizer .

Figure 5: Alignment visualization of the Synchronizer . Left: Video-Text alignment matrix. Right: Speech-Text alignment matrix.

Figure 6: Qualitative comparison of mel-spectrograms. Our method produces spectra closest to the ground-truth, with clearer harmonics, cleaner pauses, and better temporal synchronization than baseline methods.

Model	FFE $\downarrow$	GPE $\downarrow$	VDE $\downarrow$	Emo-SIM $\uparrow$
HPMDubbing	0.535	0.473	0.289	0.979
StyleDubber	0.583	0.493	0.360	0.976
Speaker2Dubber	0.639	0.519	0.408	0.979
ProDubber	0.653	0.562	0.436	0.959
EmoDubber	0.426	0.408	0.220	0.977
DiFlowDubber (ours)	0.395	0.361	0.209	0.983

Appendix

Table 5: Evaluation of prosodic expressiveness and emotional consistency on the Chem dataset (Setting 2.0).

Expressive Speech Synthesis

See all 23 papers

CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing
Apr 14, 2026Gaoxiang Cong, Liang Li, Jiaxin Ye +4Audio-Visual Alignment Diffusion Transformer
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China +2
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
Nov 23, 2025Kaidi Wang, Yi He, Wenhao Guan +9TTS Synthesis Audio-Visual Synchronization
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects
Aug 16, 2026Yusheng Dai, Kangdi Wang, Baolong Gao +4Sound Effects Generation
Monash University · University of Chinese Academy of Sciences · Tsinghua University

Apr 14, 2026J/K move · Enter open · S save

CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

Gaoxiang Cong, Liang Li, Jiaxin Ye, Zhedong Zhang+3

Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China+2

Audio-Visual Alignment Diffusion Transformer

Open paper AskPDF arXiv

Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.

Abstract

Expressive Speech Synthesis

Explore similar work

CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects