Automatic video dubbing in the wild remains fundamentally limited by two competing constraints: hierarchical methods depend on brittle, multi-stage preprocessing pipelines that severely restrict data scalability and practical deployment, while holistic approaches operating on uncropped video suffer from weak temporal alignment and speaker-utterance ambiguity in multi-speaker settings. To overcome these limitations, we propose CineDub, a unified diffusion-based model that achieves precise multi-speaker dialogue dubbing directly from uncropped videos, without face cropping or speaker diarization. Central to our approach is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm, where holistic visual representations and a semantic-bundled transcription format are encoded independently, yet implicitly coupled through cross-modal training to resolve speaker ambiguity and enable precise multi-speaker multi-turn dialogue dubbing. Building on the unified temporal cues captured by holistic visual features, we further extend CineDub to joint speech and audio generation. We introduce an Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation, and a decoupled textual branch control mechanism to resolve cross-prompt interference during simultaneous generation. We also release two in-the-wild benchmarks, CineDub-Multi for multi-speaker dialogue dubbing and CineDub-SA for video-to-speech-and-audio (V2SA) generation, to enable evaluation under realistic conditions. Experiments show that CineDub achieves state-of-the-art results on established single-speaker dubbing and video-to-audio benchmarks while excelling in multi-speaker dialogue dubbing and acoustically coherent joint generation.
Figures & tables
Figure 1: (a) Existing dialogue dubbing methods rely on complex preprocessing to provide active cropped speaker regions with corresponding timestamps and transcripts, limiting scalability and practicality. (b) Generating accompanying sound effects requires a separate V2A model, which inevitably introduces ghost speech and acoustic incoherence with the dubbed speech. (c) CineDub jointly generates multi-speaker dialogue with coherent audio through a unified end-to-end model, requiring only an uncropped video and an MLLM-generated semantic-bundled transcription.
Figure 2: Visualization of SynchFormer self-attention maps (magenta: active speaker; white: inactive speaker). (a) Single-speaker setting: attention concentrates on the lip region. (b) Two-speaker setting: attention dynamically shifts to the active speaker across turns. (c) Failure cases: attention drifts to a non-speaking face during overlapping speech or becomes ambiguous at shot boundaries.
Figure 3: Overview of CineDub under the ICHC paradigm. The holistic visual condition cv (left), extracted by SynchFormer (Sec. 3.2.1 ), encodes both event-level audio-visual correspondences and fine-grained lip-sync alignment. To address the speaker assignment ambiguity in cv , the semantic-bundled transcription ct (right) provides per-segment speaker-utterance grounding cues, implicitly coupling with cv via multi-conditional training (Sec. 3.2.2 ). CineDub adopts a unified DiT backbone that supports both joint video-to-speech-and-audio generation and each subtask. ct and ca are routed through decoupled textual branches to prevent cross-prompt interference, with learnable meta-tokens replacing inactive branches during single-task inference (Sec. 3.3.2 ).
GRID
CHEM
Method
WER ↓
SIM ↑
MCD DTW ↓
MCD SL ↓
LSE-D ↓
LSE-C ↑
WER ↓
SIM ↑
MCD DTW ↓
MCD SL ↓
LSE-D ↓
LSE-C ↑
GT
13.59
1.00
0.00
0.00
7.43
6.20
1.44
1.00
0.00
0.00
6.67
7.99
Hierarchical
HPMDubbing [ 10 ]
41.68
0.86
6.91
7.02
9.24
5.71
27.39
0.86
7.03
8.40
9.63
4.15
StyleDubber [ 13 ]
17.34
0.91
6.18
6.57
8.93
6.23
12.10
0.91
6.15
6.24
11.09
3.89
Speak2Dub [ 59 ]
14.87
0.92
6.04
6.59
9.78
5.34
15.41
0.89
7.88
8.29
11.02
3.64
EmoDubber [ 12 ]
16.58
0.93
4.43
4.44
6.83
7.14
12.27
0.92
5.77
5.79
6.96
7.93
Table 1: Single-speaker video dubbing results on GRID and CHEM. Methods are grouped into Hierarchical methods that rely on face or lip crops and Holistic methods that operate on full-frame video. ∗ denotes the more challenging zero-shot voice cloning setting. Gray entries are quoted from original papers without released checkpoints and are listed unranked for reference only.
Method
cpWER (WER) ↓
UTMOS ↑
MCD DTW ↓
Desync ↓
GT
10.47 (8.84)
2.54
0.00
0.219
AlignDiT [ 7 ]
57.49 (32.54)
2.75
8.98
0.567
DeepAudio [ 55 ]
55.53 ( 12.67 )
3.45
8.15
0.666
FunCineForge 4 4 footnotemark: 4 [ 34 ]
43.47 (25.30)
3.45
10.47
0.882
CineDub (ours)
13.93 ( 13.06 )
2.76
8.06
0.255
w/ flat transcript
31.08 (21.71)
2.83
8.81
0.396
Table 2: Multi-speaker dialogue video dubbing results on CineDub-Multi. cpWER extends WER by additionally penalizing speaker assignment errors, providing a more complete measure of correctness in multi-turn generation.
Method
KL ↓
FD VGG ↓
FD PaSST ↓
IS ↑
Desync ↓
IB ↑
ThinkSound [ 33 ]
1.60
1.10
116.08
11.73
0.53
0.26
Hunyuan-Foley [ 42 ]
1.74
2.36
100.53
11.58
0.57
0.32
AudioX [ 47 ]
1.59
1.24
103.37
14.94
1.23
0.26
MMAudio [ 5 ]
1.63
0.91
68.44
13.44
0.49
0.29
Omni2Sound [ 15 ]
1.35
0.53
48.20
15.79
0.49
0.34
DualDub ∗ [ 46 ]
2.91
2.29
–
11.50
–
0.24
Table 3: Video-to-audio results on VGGSound test set. Top: V2A expert models; Bottom: Unified V2SA models. ∗ Evaluation without textual audio prompts. Gray entries (unranked) are cited from original papers without released checkpoints.
Speech Metrics
Audio Metrics
Method
WER Qwen ↓
UTMOS ↑
MCD DTW ↓
LSE-D ↓
LSE-C ↑
KL ↓
FD VGG ↓
FD PaSST ↓
IS ↑
IB ↑
Desync ↓
GT
15.78
1.50
0.02
9.13
2.47
0.00
0.02
0.00
5.00
0.35
0.31
AlignDiT [ 7 ]
38.98
1.72
11.33
10.30
2.10
–
–
–
–
–
–
+MMAudio [ 5 ]
53.33
1.30
10.99
9.92
2.20
1.20
2.95
287.64
3.01
0.27
0.34
DeepAudio [ 55 ]
20.14
2.48
11.20
11.12
1.15
–
–
–
–
–
–
+MMAudio [ 5 ]
51.92
1.36
11.58
9.99
2.14
1.17
2.13
226.55
3.29
0.30
0.31
Table 4: Video to speech and audio joint generation results on CineDub-SA. Cascaded baselines dub speech first, then mix audio generated by MMAudio [ 5 ] .
Speech (GRID)
Audio (VGGSound)
Strategy
WER↓
LSE-D↓
LSE-C↑
FD VGG ↓
IS↑
IB↑
Native Joint
13.48
7.36
6.46
0.60
14.49
0.32
L→A
13.21
7.58
6.32
0.61
14.52
0.32
A→L (ours)
10.36
7.27
6.55
0.53
14.59
0.33
Table 5: Ablation on curriculum learning strategies. Each variant is trained for joint V2SA generation and evaluated on GRID (speech) and VGGSound (audio).
Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.
Kaidi Wang, Yi He, Wenhao Guan +9
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.
Gaoxiang Cong, Liang Li, Jiaxin Ye +4
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China +2
Video dubbing requires content accuracy, expressive prosody, high-quality acoustics, and precise lip synchronization, yet existing approaches struggle on all four fronts. To address these issues, we propose DiFlowDubber, the first video dubbing framework built upon a discrete flow matching backbone with a novel two-stage training strategy. In the first stage, a zero-shot text-to-speech (TTS) system is pre-trained on large-scale corpora, where a deterministic architecture captures linguistic structures, and the Discrete Flow-based Prosody-Acoustic (DFPA) module models expressive prosody and realistic acoustic characteristics. In the second stage, we propose the Content-Consistent Temporal Adaptation (CCTA) to transfer TTS knowledge to the dubbing domain: its Synchronizer enforces cross-modal alignment for lip-synchronized speech. Complementarily, the Face-to-Prosody Mapper (FaPro) conditions prosody on facial expressions, whose outputs are then fused with those of the Synchronizer to construct rich, fine-grained multimodal embeddings that capture prosody-content correlations, guiding the DFPA to generate expressive prosody and acoustic tokens for content-consistent speech. Experiments on two benchmark datasets demonstrate that DiFlowDubber outperforms prior methods across multiple evaluation metrics.
Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi +3
FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. +1