CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

Gaoxiang Cong; Liang Li; Jiaxin Ye; Zhedong Zhang; Hongming Shan; Yuankai Qi; Qingming Huang

cs.SDApr 14, 2026

CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

Authors: Gaoxiang Cong, Liang Li, Jiaxin Ye, Zhedong Zhang, Hongming Shan, Yuankai Qi, Qingming Huang

Organizations: Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China · Hangzhou Dianzi University, Hangzhou, China ·

Abstract

Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.

Figures & tables

Figure 1: (a) Illustration of Visual Voice Cloning (V2C) task. (b) Explicit alignment manner requires an external forced tool to compute the ground-truth duration of each phoneme in advance. (c) Implicit alignment eliminates the dependency on external forced tools. (d) We propose a movie dubbing architecture built upon CoSync-DiT, which is structured around a cognitively inspired listen–watch–articulate paradigm to progressively guide the denoising trajectory in flow matching.

Dataset	Speaker	Main Scenes	Experiment Setting
CelebV-Dub [ 38 ]	Multi-Speakers	Vlogs, Dramas	Setting 1 & 2
Chem [ 17 ]	Single-Speaker	Teaching Video	Setting 1 & 2
CinePile-Dub [ 35 ]	Multi-Speakers	Live Action Movies	Zero-shot Setting

Table 1: Overview of evaluation configurations across three experimental settings in various dubbing benchmarks.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$
GT	100.00	3.85	100.00	0.00	3.86
HPMDubbing [ 11 ] (CVPR’23)	57.05	17.52	78.80	0.477	3.34
StyleDubber [ 13 ] (ACL’24)	61.87	10.62	80.95	0.440	3.39
EmoDubber [ 12 ] (CVPR’25)	75.60	11.86	85.10	0.420	3.70
FlowDubber [ 10 ] (MM’25)	75.37	11.51	83.47	0.417	3.66
HD-Dubber [ 24 ] (TPAMI’25)	64.78	14.23	79.59	0.410	3.60

Table 2: Compared with SOTA methods on the Chem benchmark under Setting 1.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$
GT	73.01	3.85	100.00	0.00	3.86
HPMDubbing [ 11 ] (CVPR’23)	44.67	18.48	77.49	0.458	3.33
StyleDubber [ 13 ] (ACL’24)	50.00	11.98	79.09	0.426	3.34
EmoDubber [ 12 ] (CVPR’25)	67.53	12.01	79.41	0.427	3.71
FlowDubber [ 10 ] (MM’25)	60.61	15.24	78.95	0.411	3.66
HD-Dubber [ 24 ] (TPAMI’25)	52.70	14.80	77.77	0.430	3.59

Table 3: Compared with SOTA methods on the Chem benchmark under Setting 2.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$
GT	100.00	4.15	100.00	0.000	3.38
HPMDubbing [ 11 ] (CVPR’23)	17.26	21.99	73.01	0.441	2.47
StyleDubber [ 13 ] (ACL’24)	26.39	10.79	79.38	0.415	2.65
EmoDubber [ 12 ] (CVPR’25)	22.08	13.44	76.59	0.407	3.26
FlowDubber [ 10 ] (MM’25)	22.24	13.95	76.76	0.423	3.27
Produbber [ 52 ] (CVPR’25)	26.97	5.18	73.49	0.421	3.12

Table 4: Compared with SOTA Dubbing methods on the CelebV-Dub dataset under Setting 1. Unlike single dubbing scenario ( e.g . , Chem), CelebV-Dub includes diverse video content (vlogs and dramas) with highly expressive speech.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$
GT	66.13	4.15	100.00	0.000	3.38
HPMDubbing [ 11 ] (CVPR’23)	12.52	23.64	69.25	0.447	2.47
StyleDubber [ 13 ] (ACL’24)	19.42	7.03	74.87	0.405	2.63
EmoDubber [ 12 ] (CVPR’25)	18.78	16.37	76.30	0.422	3.30
FlowDubber [ 10 ] (MM’25)	19.32	14.94	74.18	0.420	3.31
Produbber [ 52 ] (CVPR’25)	23.13	6.41	71.38	0.432	3.14

Table 5: Compared with SOTA methods on the CelebV-Dub dataset under Setting 2.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$
GT	100.00	2.56	100.00	0.00	3.15
HPMDubbing [ 11 ] (CVPR’23)	42.72	99.20	61.22	0.391	3.23
StyleDubber [ 13 ] (ACL’24)	41.65	74.96	63.88	0.362	2.89
HD-Dubber [ 24 ] ](TPAMI’25)	5.84	42.75	56.32	0.337	2.58
EmoDubber [ 12 ] (CVPR’25)	23.10	49.57	70.08	0.337	3.05
FlowDubber [ 10 ] (MM’25)	22.68	54.41	67.39	0.354	3.04

Table 6: Zero-shot main experiment results on CinePile-Dub dataset (Out-of-Domain). All models are trained on the training set of the basic dubbing dataset CelebV-Dub.

Figure 3: Performance comparison between the proposed method and the SOTA baseline under the different Number of Function Evaluations (NFE) steps.

Methods	SPKSIM (%) $\uparrow$	WER (%) $\downarrow$	EMOSIM (%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$
w/o Style Adapting	19.64	6.84	77.24	0.385	3.38
w/o Visual Calibrating	53.25	6.40	80.17	0.419	3.45
w/o Context Aligning	52.75	7.39	80.04	0.446	3.44
w/o JSAR	51.30	8.72	80.14	0.431	3.39
w/o Semantic Consistency	52.34	8.39	80.19	0.392	3.42
w/o Temporal Consistency	51.37	6.58	80.24	0.425	3.44

Table 7: Results of ablation study on CelebV-Dub dataset under Setting 2.

Figure 4: Visual comparison of the generated mel-spectrograms. The blue arrows highlight specific regions requiring attention for audio-visual synchronization.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$	AVSync(%) $\uparrow$
GT	100.00	4.15	100.00	0.000	3.38	100.00
HPMDubbing [ 11 ] (CVPR’23)	17.26	21.99	73.01	0.441	2.47	47.49
StyleDubber [ 13 ] (ACL’24)	26.39	10.79	79.38	0.415	2.65	21.38
EmoDubber [ 12 ] (CVPR’25)	22.08	13.44	76.59	0.407	3.26	32.94
FlowDubber [ 10 ] (MM’25)	22.24	13.95	76.76	0.423	3.27	33.40
Produbber [ 52 ] (CVPR’25)	26.97	5.18	73.49	0.421	3.12	15.96

Table 8: Compared with SOTA Dubbing methods on the CelebV-Dub dataset under Setting 1. Unlike single dubbing scenario ( e.g . , Chem), CelebV-Dub includes diverse video content (vlogs and dramas) with highly expressive speech.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$	AVSync(%) $\uparrow$
GT	66.13	4.15	100.00	0.000	3.38	100.00
HPMDubbing [ 11 ] (CVPR’23)	12.52	23.64	69.25	0.447	2.47	42.67
StyleDubber [ 13 ] (ACL’24)	19.42	7.03	74.87	0.405	2.63	20.75
EmoDubber [ 12 ] (CVPR’25)	18.78	16.37	76.30	0.422	3.30	32.14
FlowDubber [ 10 ] (MM’25)	19.32	14.94	74.18	0.420	3.31	32.37
Produbber [ 52 ] (CVPR’25)	23.13	6.41	71.38	0.432	3.14	14.92

Table 9: Compared with SOTA methods on the CelebV-Dub dataset under Setting 2.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$	AVSync(%) $\uparrow$
GT	100.00	2.56	100.00	0.00	3.15	100.00
HPMDubbing [ 11 ] (CVPR’23)	42.72	99.20	61.22	0.391	3.23	17.00
StyleDubber [ 13 ] (ACL’24)	41.65	74.96	63.88	0.362	2.89	22.56
HD-Dubber [ 24 ] ](TPAMI’25)	5.84	42.75	56.32	0.337	2.58	19.33
EmoDubber [ 12 ] (CVPR’25)	23.10	49.57	70.08	0.337	3.05	13.75
FlowDubber [ 10 ] (MM’25)	22.68	54.41	67.39	0.354	3.04	13.58

Table 10: Zero-shot results on CinePile-Dub dataset (Out-of-Domain Movie). All models are trained on the training set of the basic dubbing dataset CelebV-Dub.

Table	Reference audio Setting	Zero-shot Setting (OOD Movie)
Tab. 8	Setting 1	$\times$
Tab. 9	Setting 2	$\times$
Tab. 10	Setting 1	✓
Tab. 12	Setting 2	✓
Tab. 13	Setting 1	✓
Tab. 14	Setting 2	✓

Table 11: Summary of the experiment configurations in the supplementary materials. Note that Setting 1 and Setting 2 define the reference audio selection strategy. Setting 1 uses the target speech as the reference audio, while Setting 2 uses an unaligned utterance from the same speaker. The Zero-shot condition indicates whether the target domain data is excluded during the training phase.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$	AVSync(%) $\uparrow$
GT	60.58	2.56	100.00	0.00	3.15	100.00
HPMDubbing [ 11 ] (CVPR’23)	30.75	93.86	52.93	0.382	3.24	13.87
StyleDubber [ 13 ] (ACL’24)	28.67	77.61	57.15	0.372	2.88	16.48
HD-Dubber [ 24 ] ](TPAMI’25)	7.04	64.12	47.66	0.330	2.58	18.86
EmoDubber [ 12 ] (CVPR’25)	19.43	57.73	62.46	0.347	3.05	13.00
FlowDubber [ 10 ] (MM’25)	19.86	55.93	63.86	0.346	3.05	13.05

Table 12: Zero-shot results on the CinePile-Dub dataset (Out-of-Domain Movie) under Setting 2. All models are trained on the training set of the basic dubbing dataset CelebV-Dub. The reference audio is from different segments of the same speaker.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$	AVSync(%) $\uparrow$
GT	100.00	4.15	100.00	0.000	3.38	100.00
AlignDiT* [ 6 ]	61.51	18.35	76.33	0.338	3.25	24.03
AlignDiT [ 6 ]	58.90	20.98	77.39	0.342	3.36	31.77
Ours	60.04	5.59	77.41	0.332	3.40	45.24

Table 13: Zero-shot evaluation results on the CinePile-Dub dataset (Out-of-domain Movie) under Setting 1. The asterisk (*) denotes the official AlignDiT checkpoint pre-trained on the large-scale LRS3 dataset. All evaluated models face completely unseen target domains.

Methods	SPKSIM(%) $\uparrow$	WER(%) $\downarrow$	EMOSIM(%) $\uparrow$	Sync-KL $\downarrow$	DNSMOS $\uparrow$	AVSync(%) $\uparrow$
GT	100.00	4.15	100.00	0.000	3.38	100.00
AlignDiT* [ 6 ]	44.01	35.90	68.80	0.359	3.23	18.72
AlignDiT [ 6 ]	38.58	34.52	66.74	0.358	3.37	22.50
Ours	47.24	7.53	70.12	0.319	3.39	31.79

Table 14: Zero-shot evaluation results on the CinePile-Dub dataset (Out-of-domain Movie) under Setting 2. All evaluated models face completely unseen target domains and non-target reference audio. The asterisk (*) denotes the official AlignDiT checkpoint pre-trained on the large-scale LRS3 dataset.

Figure 5: Visual comparison of the generated mel-spectrograms with ground truth. The blue and white bounding boxes highlight regions where different models exhibit significant differences in duration and spectrogram details. Furthermore, the blue arrows pinpoint specific temporal regions that require critical attention for evaluating audio-visual synchronization.

Figure 6: Visual comparison of the generated mel-spectrograms with ground truth under zero-shot setting (out-of-domain movie scenario). The blue and white bounding boxes highlight regions where different models exhibit significant differences in duration and spectrogram details. Furthermore, the blue arrows pinpoint specific temporal regions that require critical attention for evaluating audio-visual synchronization.

Audio-Visual Alignment

Audio-visual alignment links sounds, speech, and music to the people, objects, events, and moments that produce or accompany them.

See all 87 papers

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
Nov 23, 2025Kaidi Wang, Yi He, Wenhao Guan +9TTS Synthesis Audio-Visual Synchronization
School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China +1
DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization
Mar 15, 2026Ngoc-Son Nguyen, Thanh V. T. Tran, Jeongsoo Choi +3Expressive Speech Synthesis TTS Synthesis
FPT Software AI Center, Vietnam · KAIST, South Korea · Independent researcher, work done in part while author was at the FPT Software AI Center, Vietnam. +1
CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects
Aug 16, 2026Yusheng Dai, Kangdi Wang, Baolong Gao +4Sound Effects Generation
Monash University · University of Chinese Academy of Sciences · Tsinghua University

Nov 23, 2025J/K move · Enter open · S save

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

Kaidi Wang, Yi He, Wenhao Guan, Weijie Wu+8

School of Informatics, Xiamen University, China · MiLM Plus, Xiaomi Inc., China · School of Electronic Science and Engineering, Xiamen University, China+1

TTS Synthesis Audio-Visual Synchronization Audio-Visual Alignment

Open paper AskPDF arXiv

Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.

Abstract

Audio-Visual Alignment

Explore similar work

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization

CineDub: Scaling End-to-End Video Dubbing to Multi-Speaker Dialogues with Coherent Sound Effects