Automatic video dubbing in the wild remains fundamentally limited by two competing constraints: hierarchical methods depend on brittle, multi-stage preprocessing pipelines that severely restrict data scalability and practical deployment, while holistic approaches operating on uncropped video suffer from weak temporal alignment and speaker-utterance ambiguity in multi-speaker settings. To overcome these limitations, we propose CineDub, a unified diffusion-based model that achieves precise multi-speaker dialogue dubbing directly from uncropped videos, without face cropping or speaker diarization. Central to our approach is the Implicitly-Coupled Holistic Conditioning (ICHC) paradigm, where holistic visual representations and a semantic-bundled transcription format are encoded independently, yet implicitly coupled through cross-modal training to resolve speaker ambiguity and enable precise multi-speaker multi-turn dialogue dubbing. Building on the unified temporal cues captured by holistic visual features, we further extend CineDub to joint speech and audio generation. We introduce an Ambient-to-Linguistic Curriculum Learning (ALC) to mitigate sub-task degradation, and a decoupled textual branch control mechanism to resolve cross-prompt interference during simultaneous generation. We also release two in-the-wild benchmarks, CineDub-Multi for multi-speaker dialogue dubbing and CineDub-SA for video-to-speech-and-audio (V2SA) generation, to enable evaluation under realistic conditions. Experiments show that CineDub achieves state-of-the-art results on established single-speaker dubbing and video-to-audio benchmarks while excelling in multi-speaker dialogue dubbing and acoustically coherent joint generation.
Figures & tables
Figure 1: (a) Existing dialogue dubbing methods rely on complex preprocessing to provide active cropped speaker regions with corresponding timestamps and transcripts, limiting scalability and practicality. (b) Generating accompanying sound effects requires a separate V2A model, which inevitably introduces ghost speech and acoustic incoherence with the dubbed speech. (c) CineDub jointly generates multi-speaker dialogue with coherent audio through a unified end-to-end model, requiring only an uncropped video and an MLLM-generated semantic-bundled transcription.
Figure 2: Visualization of SynchFormer self-attention maps (magenta: active speaker; white: inactive speaker). (a) Single-speaker setting: attention concentrates on the lip region. (b) Two-speaker setting: attention dynamically shifts to the active speaker across turns. (c) Failure cases: attention drifts to a non-speaking face during overlapping speech or becomes ambiguous at shot boundaries.
Figure 3: Overview of CineDub under the ICHC paradigm. The holistic visual condition cv (left), extracted by SynchFormer (Sec. 3.2.1 ), encodes both event-level audio-visual correspondences and fine-grained lip-sync alignment. To address the speaker assignment ambiguity in cv , the semantic-bundled transcription ct (right) provides per-segment speaker-utterance grounding cues, implicitly coupling with cv via multi-conditional training (Sec. 3.2.2 ). CineDub adopts a unified DiT backbone that supports both joint video-to-speech-and-audio generation and each subtask. ct and ca are routed through decoupled textual branches to prevent cross-prompt interference, with learnable meta-tokens replacing inactive branches during single-task inference (Sec. 3.3.2 ).
GRID
CHEM
Method
WER ↓
SIM ↑
MCD DTW ↓
MCD SL ↓
LSE-D ↓
LSE-C ↑
WER ↓
SIM ↑
MCD DTW ↓
MCD SL ↓
LSE-D ↓
LSE-C ↑
GT
13.59
1.00
0.00
0.00
7.43
6.20
1.44
1.00
0.00
0.00
6.67
7.99
Hierarchical
HPMDubbing [ 10 ]
41.68
0.86
6.91
7.02
9.24
5.71
27.39
0.86
7.03
8.40
9.63
4.15
StyleDubber [ 13 ]
17.34
0.91
6.18
6.57
8.93
6.23
12.10
0.91
6.15
6.24
11.09
3.89
Speak2Dub [ 59 ]
14.87
0.92
6.04
6.59
9.78
5.34
15.41
0.89
7.88
8.29
11.02
3.64
EmoDubber [ 12 ]
16.58
0.93
4.43
4.44
6.83
7.14
12.27
0.92
5.77
5.79
6.96
7.93
Table 1: Single-speaker video dubbing results on GRID and CHEM. Methods are grouped into Hierarchical methods that rely on face or lip crops and Holistic methods that operate on full-frame video. ∗ denotes the more challenging zero-shot voice cloning setting. Gray entries are quoted from original papers without released checkpoints and are listed unranked for reference only.
Method
cpWER (WER) ↓
UTMOS ↑
MCD DTW ↓
Desync ↓
GT
10.47 (8.84)
2.54
0.00
0.219
AlignDiT [ 7 ]
57.49 (32.54)
2.75
8.98
0.567
DeepAudio [ 55 ]
55.53 ( 12.67 )
3.45
8.15
0.666
FunCineForge 4 4 footnotemark: 4 [ 34 ]
43.47 (25.30)
3.45
10.47
0.882
CineDub (ours)
13.93 ( 13.06 )
2.76
8.06
0.255
w/ flat transcript
31.08 (21.71)
2.83
8.81
0.396
Table 2: Multi-speaker dialogue video dubbing results on CineDub-Multi. cpWER extends WER by additionally penalizing speaker assignment errors, providing a more complete measure of correctness in multi-turn generation.
Method
KL ↓
FD VGG ↓
FD PaSST ↓
IS ↑
Desync ↓
IB ↑
ThinkSound [ 33 ]
1.60
1.10
116.08
11.73
0.53
0.26
Hunyuan-Foley [ 42 ]
1.74
2.36
100.53
11.58
0.57
0.32
AudioX [ 47 ]
1.59
1.24
103.37
14.94
1.23
0.26
MMAudio [ 5 ]
1.63
0.91
68.44
13.44
0.49
0.29
Omni2Sound [ 15 ]
1.35
0.53
48.20
15.79
0.49
0.34
DualDub ∗ [ 46 ]
2.91
2.29
–
11.50
–
0.24
Table 3: Video-to-audio results on VGGSound test set. Top: V2A expert models; Bottom: Unified V2SA models. ∗ Evaluation without textual audio prompts. Gray entries (unranked) are cited from original papers without released checkpoints.
Speech Metrics
Audio Metrics
Method
WER Qwen ↓
UTMOS ↑
MCD DTW ↓
LSE-D ↓
LSE-C ↑
KL ↓
FD VGG ↓
FD PaSST ↓
IS ↑
IB ↑
Desync ↓
GT
15.78
1.50
0.02
9.13
2.47
0.00
0.02
0.00
5.00
0.35
0.31
AlignDiT [ 7 ]
38.98
1.72
11.33
10.30
2.10
–
–
–
–
–
–
+MMAudio [ 5 ]
53.33
1.30
10.99
9.92
2.20
1.20
2.95
287.64
3.01
0.27
0.34
DeepAudio [ 55 ]
20.14
2.48
11.20
11.12
1.15
–
–
–
–
–
–
+MMAudio [ 5 ]
51.92
1.36
11.58
9.99
2.14
1.17
2.13
226.55
3.29
0.30
0.31
Table 4: Video to speech and audio joint generation results on CineDub-SA. Cascaded baselines dub speech first, then mix audio generated by MMAudio [ 5 ] .
Speech (GRID)
Audio (VGGSound)
Strategy
WER↓
LSE-D↓
LSE-C↑
FD VGG ↓
IS↑
IB↑
Native Joint
13.48
7.36
6.46
0.60
14.49
0.32
L→A
13.21
7.58
6.32
0.61
14.52
0.32
A→L (ours)
10.36
7.27
6.55
0.53
14.59
0.33
Table 5: Ablation on curriculum learning strategies. Each variant is trained for joint V2SA generation and evaluated on GRID (speech) and VGGSound (audio).
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China +2
FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. +1