DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
Authors: Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi, Hieu-Nghia Huynh-Nguyen, Truong-Son Hy, Van Nguyen
Organizations: FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. · University of Alabama at Birmingham, USA
Video dubbing requires content accuracy, expressive prosody, high-quality acoustics, and precise lip synchronization, yet existing approaches struggle on all four fronts. To address these issues, we propose DiFlowDubber, the first video dubbing framework built upon a discrete flow matching backbone with a novel two-stage training strategy. In the first stage, a zero-shot text-to-speech (TTS) system is pre-trained on large-scale corpora, where a deterministic architecture captures linguistic structures, and the Discrete Flow-based Prosody-Acoustic (DFPA) module models expressive prosody and realistic acoustic characteristics. In the second stage, we propose the Content-Consistent Temporal Adaptation (CCTA) to transfer TTS knowledge to the dubbing domain: its Synchronizer enforces cross-modal alignment for lip-synchronized speech. Complementarily, the Face-to-Prosody Mapper (FaPro) conditions prosody on facial expressions, whose outputs are then fused with those of the Synchronizer to construct rich, fine-grained multimodal embeddings that capture prosody-content correlations, guiding the DFPA to generate expressive prosody and acoustic tokens for content-consistent speech. Experiments on two benchmark datasets demonstrate that DiFlowDubber outperforms prior methods across multiple evaluation metrics.
Figures & tables
Figure 1: Overall Inference Pipeline of DiFlowDubber. Face-to-Prosody Mapper module predicts global prosody priors that capture global prosody and stylistic cues from facial expressions. Content-Consistent Temporal Adaptation module generates discrete content tokens conditioned on lip movements, text, and prosody priors, ensuring consistency with the target text transcription and temporal alignment. Discrete Flow-based Prosody-Acoustic module generates diverse yet globally consistent prosody tokens under the guidance of the prosody prior, together with corresponding acoustic tokens. The speech waveform is synthesized from the predicted tokens and speaker embedding via a Codec Decoder.
Figure 2: Detailed of DiFlowDubber Pipeline. Our framework is a two-stage pipeline. The first stage performs zero-shot TTS pre-training, where a simple deterministic content modeling architecture efficiently captures linguistic structures (orange dashed box). For prosody and acoustic attributes, we adopt (b) Discrete Flow-based Prosody-Acoustic (DFPA) module to model expressive prosodic variations and realistic acoustic diversity from the corpus. In the second stage, the model is adapted to the video dubbing task, where (a) Content-Consistent Temporal Adaptation module transfers consistent content knowledge from the TTS domain and generates temporally aligned content representations. Meanwhile, FaPro extracts a global prosody prior from facial expression cues. The DFPA module then models the joint distribution of prosody and acoustic tokens conditioned on the global prosody prior, latent content representations, and speaker embedding.
Model
Setting 1.0
Setting 2.0
LSE-C ↑
LSE-D ↓
WER ↓
SECS ↑
UTMOS ↑
LSE-C ↑
LSE-D ↓
WER ↓
SECS ↑
MOS-S ↑
UTMOS ↑
Ground Truth
8.12
6.59
3.85
100.00
4.18
8.12
6.59
3.85
100.00
4.34 ± 0.18
4.18
V2C-Net [ 2 ] CVPR’22
1.97
12.17
90.47
51.52
1.81
1.82
12.09
94.59
44.19
-
1.76
HPMDubbing [ 11 ] CVPR’23
7.85
7.19
16.05
85.09
2.16
3.98
9.50
29.82
73.55
-
2.01
StyleDubber [ 13 ] ACL’24
3.87
10.92
13.14
87.72
3.14
3.74
11.00
14.18
82.07
4.02 ± 0.24
3.04
Speaker2Dubber [ 60 ] ACM MM’24
3.76
10.56
16.98
74.73
3.61
3.45
11.17
18.10
69.28
3.69 ± 0.25
3.64
Table 1: Performance on the Chem dataset under both Setting 1.0 and Setting 2.0. In the Dub 1.0 Setting, the ground truth audio is used as the reference, while in the Dub 2.0 Setting, a non-ground-truth audio sample from the same speaker within the dataset is used as the reference. The best and second-best results are shown in bold and underlined , respectively.
Model
Setting 1.0
Setting 2.0
LSE-C ↑
LSE-D ↓
WER ↓
SECS ↑
UTMOS ↑
LSE-C ↑
LSE-D ↓
WER ↓
SECS ↑
MOS-S ↑
UTMOS ↑
Ground Truth
7.13
6.78
22.41
100.00
3.94
7.13
6.78
22.41
100.00
4.58 ± 0.13
3.94
V2C-Net [ 2 ] CVPR’22
5.59
9.52
47.82
80.98
2.41
5.34
9.76
49.09
71.51
-
2.40
HPMDubbing [ 11 ] CVPR’23
5.76
9.13
45.51
85.11
2.14
5.82
9.10
44.15
71.99
-
2.11
StyleDubber [ 13 ] ACL’24
6.12
9.03
18.88
93.79
3.73
6.09
9.08
19.58
86.67
3.83 ± 0.24
3.71
Speaker2Dubber [ 60 ] ACM MM’24
5.27
9.84
17.07
94.50
3.69
5.19
9.93
17.42
85.76
3.57 ± 0.26
3.73
Table 2: Performance on the GRID dataset is evaluated under the same two Dub settings as in the Chem dataset.
#
Model
AVSync ↑
LSE-C ↑
LSE-D ↓
WER ↓
SECS ↑
UTMOS ↑
1
Ours
0.721
8.31
6.73
9.65
84.59
4.02
Zero-shot TTS Pre-training
2
w/o Zero-shot TTS Pre-training
0.715
8.17
6.79
12.04
83.72
3.53
Synchronizer
3
w/o ( LVT+LST )
0.652
8.26
6.77
17.15
84.29
3.90
4
w/o LVT
0.681
8.36
6.71
16.57
83.94
3.87
Table 3: Ablation study results on the Chem test set.
Model
#Params
NFE
RTF ↓
LSE-C ↑
LSE-D ↓
WER ↓
SECS ↑
UTMOS ↑
ProDubber
200M
5
0.08
2.58
12.54
9.45
72.13
3.85
EmoDubber
116M
10
0.05
8.11
6.92
11.72
90.62
3.82
DiFlowDubber
250M
1
0.03
8.25
6.78
10.72
83.63
2.82
2
0.04
8.27
6.74
10.99
83.34
2.79
4
0.04
8.32
6.73
10.79
83.82
3.50
8
0.05
8.29
6.73
10.28
84.50
3.85
Table 4: Analysis of NFE in inference under the Dub 1.0 Setting. #Params exclude video-related encoders.
Figure 3: Mel-spectrogram visualization compared with Ground-Truth speech.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The detailed architecture of Synchronizer .
Figure 5: Alignment visualization of the Synchronizer . Left: Video-Text alignment matrix. Right: Speech-Text alignment matrix.
Figure 6: Qualitative comparison of mel-spectrograms. Our method produces spectra closest to the ground-truth, with clearer harmonics, cleaner pauses, and better temporal synchronization than baseline methods.
Model
FFE ↓
GPE ↓
VDE ↓
Emo-SIM ↑
HPMDubbing
0.535
0.473
0.289
0.979
StyleDubber
0.583
0.493
0.360
0.976
Speaker2Dubber
0.639
0.519
0.408
0.979
ProDubber
0.653
0.562
0.436
0.959
EmoDubber
0.426
0.408
0.220
0.977
DiFlowDubber (ours)
0.395
0.361
0.209
0.983
Appendix
Table 5: Evaluation of prosodic expressiveness and emotional consistency on the Chem dataset (Setting 2.0).
Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.
Gaoxiang Cong, Liang Li, Jiaxin Ye +4
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China +2
Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.
Kaidi Wang, Yi He, Wenhao Guan +9
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
Automatic video dubbing in the wild remains fundamentally limited by two competing constraints: hierarchical methods depend on brittle, multi-stage preprocessing pipelines that severely restrict data scalability and practical deployment, while holistic approaches operating on uncropped video suffer from weak temporal alignment and speaker-utterance ambiguity in multi-speaker settings. To overcome these limitations, we propose CineDub, a unified diffusion-based model that achieves precise multi-speaker dialogue dubbing directly from uncropped videos, without face cropping or speaker diarization. Central to our approach is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm, where holistic visual representations and a semantic-bundled transcription format are encoded independently, yet implicitly coupled through cross-modal training to resolve speaker ambiguity and enable precise multi-speaker multi-turn dialogue dubbing. Building on the unified temporal cues captured by holistic visual features, we further extend CineDub to joint speech and audio generation. We introduce an Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation, and a decoupled textual branch control mechanism to resolve cross-prompt interference during simultaneous generation. We also release two in-the-wild benchmarks, CineDub-Multi for multi-speaker dialogue dubbing and CineDub-SA for video-to-speech-and-audio (V2SA) generation, to enable evaluation under realistic conditions. Experiments show that CineDub achieves state-of-the-art results on established single-speaker dubbing and video-to-audio benchmarks while excelling in multi-speaker dialogue dubbing and acoustically coherent joint generation.
Yusheng Dai, Kangdi Wang, Baolong Gao +4
Monash University · University of Chinese Academy of Sciences · Tsinghua University