Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis

Akshita Gupta; Tatiana Likhomanenko; Karren Dai Yang; Richard He Bai; Zakaria Aldeneh; Navdeep Jaitly

cs.MMNov 26, 2024

Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis

Authors: Akshita Gupta, Tatiana Likhomanenko, Karren Dai Yang, Richard He Bai, Zakaria Aldeneh, Navdeep Jaitly

Organizations: TU Darmstadt · Apple

Abstract

Unified decoder-only transformers have shown promise for multimodal generation, yet how sequence design affects synchronization across modalities with heterogeneous sampling rates remains underexplored. We investigate this through video-text-to-speech (VTTS) synthesis, a controlled task requiring fine-grained temporal alignment between sparse text, video, and continuous speech. Using Visatronic, a unified decoder-only transformer trained on VoxCeleb2, we examine complementary modality contributions, positional encoding strategies, the effects of modality ordering on in-domain performance and cross-domain transfer, and phoneme-level timing errors. Global sequential indexing, which assigns position IDs in concatenation order, and co-temporal ordered indexing, which assigns identical IDs to temporally corresponding video and speech tokens, achieve comparable in-domain synchronization. Under domain shift, however, synchronization depends on the interaction between token ordering and position-ID design. Text-first co-temporal indexing transfers most robustly among the evaluated variants without requiring explicit timestamp metadata. Text and video provide complementary signals: text supports intelligibility, while video contributes temporal cues and emotional expressiveness. Modality ordering also introduces a trade-off: video-first ordering achieves stronger in-domain performance, whereas text-first ordering generalizes more robustly among the timestamp-informed prefix variants. Training on diverse, in-the-wild data is associated with more transferable behavior, although isolating its contribution requires targeted ablations. We introduce TimeSync, a phoneme-level synchronization metric that reveals temporal misalignments overlooked by frame-level metrics. Together, these findings establish VTTS as a testbed for understanding temporal synchronization in unified multimodal decoders.

Figures & tables

Method	Intelligibility ( $\uparrow$ )	Naturalness ( $\uparrow$ )	Synchronization ( $\uparrow$ )
GT	4.79 ±0.05	4.79 ±0.05	4.73 ±0.06
GT (discrete)	4.32 ±0.11	3.80 ±0.11	4.59 ±0.07
Lip to Speech Synthesis
Lip2Speech ( Kim et al., 2023 )	2.21 ±0.08	2.20 ±0.09	2.69 ±0.08
SVTS ( Mira et al., 2022 )	2.17 ±0.08	2.15 ±0.09	2.71 ±0.09
VCA-GAN ( Kim et al., 2021 )	2.19 ±0.08	2.20 ±0.09	2.71 ±0.08

Table 1: Human evaluation on LRS3. Mean opinion scores (MOS) (1-5) with 95% confidence intervals for our VoxCeleb2-trained models evaluated on LRS3. VTTS (TV-CoTemporal) achieves better performance in Intelligibility (3.62) and Synchronization (3.12), while both methods perform equally well in Naturalness (3.01). GT is an upper bound, while GT (discrete) shows degradation due to speech quantization and vocoding artifacts. These results demonstrate our model maintains good perceptual quality even on out-of-distribution data. For baseline methods, we use publicly available model checkpoints from their respective repositories to generate speech samples, which we then evaluate using our human evaluation protocol. “-” indicates unavailable metrics under the same standardized setup; this applies to dubbing baselines where only naturalness scores were available.

Method	video-speech emotions ( $\uparrow$ )	speech emotions ( $\uparrow$ )	Angry ( $\uparrow$ )	Happy ( $\uparrow$ )	Fearful ( $\uparrow$ )
GT	4.62 ±0.07	4.92 ±0.04	4.36	4.43	4.20
GT (discrete)	4.41 ±0.10	4.37 ±0.12	–	–	–
TTS (dmel Bai et al. (2024) )	3.57 ±0.14	3.20 ±0.15	3.46	3.58	3.17
VTTS
Video-Causal-Streaming	3.66 ±0.16	3.36 ±0.15	–	–	–
TV-CoTemporal	3.79 ±0.15	3.31 ±0.17	–	–	–

Table 2: Human evaluation of emotional expressiveness on VoxCeleb2. Mean Opinion Scores (1-5) with 95% confidence intervals for emotional consistency between video and speech ( video-speech emotions ), perceived emotion quality in speech alone ( speech emotions ), and specific emotion categories ( Angry, Happy, Fearful ). VTTS (TV-CoTemporal) achieves the highest score in video-speech emotion alignment (3.79), while VTTS (VT-Scaled) performs best in overall speech emotion quality (3.39) and in all individual emotion categories. These results demonstrate the advantage of visual conditioning for expressive speech generation. Ground truth (GT) serves as the reference, and GT (discrete) provides the upper bound achievable with our quantization-based approach. Per-category labels were collected only for GT, TTS, and VTTS (VT-Scaled) due annotation-budget constraints.

Method	Training Dataset	WER (%) $\downarrow$
V2SFlow ( Choi et al., 2025 )	LRS3	28.5
LipVoicer ( Yemini et al., 2024 )	LRS3	21.4
Lip2Speech ( Kim et al., 2023 )	LRS3	57.4
SVTS ( Mira et al., 2022 )	LRS3	82.4
VCA-GAN ( Kim et al., 2021 )	LRS3	90.6
DiffV2S ( Choi et al., 2023 )	LRS3	39.2

Table 3: Cross-dataset generalization and validation against prior work on full-set LRS3. Primary comparisons are reported on the full LRS3 test set for fairness. VoiceCraft-Dub* is shown for reference only and is not directly comparable because it adapts a large pretrained TTS backbone with additional AV-specific fine-tuning.

Method	Input Modality	GT WER ( $\downarrow$ )	GT (discrete) WER ( $\downarrow$ )	WER ( $\downarrow$ )	TimeSync (s) ( $\downarrow$ )
TTS (dmel Bai et al. (2024) )	Text	4.0 ±0.1	10.5 ±0.1	19.0 $\mathord{+}8.5$	-
VTTS
VT-Scaled	Video-Text			17.2 $\mathord{+}6.7$	-
TTS	Text	2.6 ±0.1	10.1 ±0.2	14.7 $\mathord{+}4.6$	0.62 ±0.98
VTTS
Video-Causal-Streaming	Text-Video			14.5 $\mathord{+}4.4$	0.49 ±0.63

Table 4: WER and TimeSync on VoxCeleb2. We report three WERs: (i) GT WER from original audio, (ii) GT (discrete) WER from reconstructed audio using GT speech tokens (lower bound), and (iii) WER from generated audio by a model. All values are computed using Whisper-large v2. The first two rows correspond to PL.v1 transcripts; the remaining rows use PL.v2. TimeSync is reported for the PL.v2 section. Among non-global prefix variants, VTTS (VT-Scaled) achieves 12.2% WER, while global-indexing ablations reach 12.1–12.7% WER. TimeSync further shows improved phoneme-level synchronization with video-conditioned models. Rows labeled “global” compare Global Sequential Indexing as an ablation against their non-global counterparts (TV-CoTemporal and VT-Scaled).

Table 5

Method	Iterations	GT WER ( $\downarrow$ )	GT (discrete) WER ( $\downarrow$ )	WER ( $\downarrow$ )
TTS	2M	2.6 ±0.1	10.1 ±0.2	17.3 $\mathord{+}7.2$
VTTS
TV-CoTemporal	2M			17.0 $\mathord{+}6.9$
LTTS (LT-Scaled)	2M			14.9 $\mathord{+}4.8$
VT-Scaled	2M			12.2 $\mathord{+}2.1$
TTS	3M	2.6 ±0.1	10.1 ±0.2	14.7 $\mathord{+}4.6$

Table 7: Convergence analysis. Training-iteration comparison shows faster convergence when video modality is used in addition to text. At 2M iterations, VTTS variants are comparable to or better than TTS at the same budget, and VTTS (VT-Scaled) already reaches 12.2% WER. TTS reaches 14.7% WER at 3M iterations, while VTTS (VT-Scaled) maintains a smaller gap (+2.1%) from GT (discrete) WER, indicating more efficient optimization with multimodal conditioning.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Order	Position IDs	Shared	Causal	Intended setting
TV-CoTemporal	T $\to$ V $\to$ S	timestamp, shared axis	yes	prefix	transfer; dubbing unseen speakers
VT-Scaled	V $\to$ T $\to$ S	timestamp-scaled video	no	prefix	in-domain synthesis
VC-Streaming	T $\to$ V/S (I)	temporal order	yes	video $t^{\prime}\leq t$	streaming, low latency
TV-Global	T $\to$ V $\to$ S	concatenation order	no	prefix	ID ablation
VT-Global	V $\to$ T $\to$ S	concatenation order	no	prefix	ID ablation

Appendix

Table 8: Sequence-design variants. Each variant is fully specified by the three per-token quantities of Figure 4 : token order, timestamp, and how the position ID is computed. A speaker token precedes all modalities in every variant and is omitted from the table. All variants use full-prefix conditioning-every conditioning token precedes the speech tokens-except Video-Causal-Streaming, in which video and speech are interleaved in temporal order so that speech at time

t

attends only to video at

t^{\prime}\leq t

. Order: T, V, S denote text, video and speech tokens; (I) marks interleaving; VC-Streaming abbreviates Video-Causal-Streaming. Shared: whether video and speech tokens share position IDs. Causal: "prefix" means all conditioning tokens precede speech.

Method	Intelligibility ( $\uparrow$ )	Naturalness ( $\uparrow$ )	Synchronization ( $\uparrow$ )
GT	4.55 ±0.09	4.79 ±0.05	4.57 ±0.10
GT (discrete)	3.95 ±0.13	3.77 ±0.15	4.36 ±0.12
TTS (dmel Bai et al. (2024) )	3.17 ±0.19	2.92 ±0.21	1.98 ±0.15
VTTS
Video-Causal-Streaming	3.19 ±0.17	2.99 ±0.16	2.28 ±0.17
TV-CoTemporal	3.35 ±0.17	3.02 ±0.19	2.50 ±0.21

Appendix

Table 9: Human evaluation on VoxCeleb2. Mean Opinion Scores (MOS) (1-5) with 95% confidence intervals for Intelligibility, Naturalness, and Synchronization. VTTS (VT-Scaled) achieves the best performance in Intelligibility (3.48) and Naturalness (3.20), while VTTS (TV-CoTemporal) performs best in Synchronization (2.50). Both outperform the TTS baseline across all metrics. Ground truth (GT) serves as a reference, and GT (discrete) is an upper bound due to speech discretization.

Method	Training Dataset	WER (%) $\downarrow$	TimeSync (s) $\downarrow$
V2SFlow ( Choi et al., 2025 )	LRS3	28.5	-
LipVoicer ( Yemini et al., 2024 )	LRS3	21.4	-
Lip2Speech ( Kim et al., 2023 )	LRS3	57.4	-
SVTS ( Mira et al., 2022 )	LRS3	82.4	-
VCA-GAN ( Kim et al., 2021 )	LRS3	90.6	-
DiffV2S ( Choi et al., 2023 )	LRS3	39.2	-

Appendix

Table 10: LRS3 transfer-focused setting. This setting is introduced in this work for detailed zero-shot transfer analysis. Unlike the main full-set WER results, we additionally report TimeSync in this setting.

Method	WER (%) $\downarrow$
VDTTS-VOXCELEB2 ( Hassid et al., 2022 )	48.0
VDTTS-LSVSR ( Hassid et al., 2022 )	25.0
VTTS (VT-Scaled, VoxCeleb2-trained, ours)	12.2

Appendix

Table 11: Reference-only VoxCeleb2-trained WER values. All rows are trained on VoxCeleb2 and evaluated on VoxCeleb2. Rows from ( Hassid et al., 2022 ) use an external evaluation protocol; our best VoxCeleb2-trained non-global prefix variant is included for context from Table 4 . These rows are not directly comparable.

Figure 11

Figure 12

Figure 13

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 287 papers

Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
Oct 3, 2025Kaisi Guan, Xihua Wang, Zhengfeng Lai +5Audio-Video Generation Video Diffusion Models
1Renmin University of China · 2Apple
OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation
Jul 26, 2026Jun Zhan, Chen Yang, Yitian Gong +23Variational Autoencoders Audio-Video Generation
1Fudan University · 2Shanghai Innovation Institute · 3MOSI Intelligence +1
Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition
Mar 11, 2026Inyong Koo, yeeun Seong, Minseok Son +2Multimodal Emotion Recognition Audio-Visual Alignment
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · Graduate School of Green Growth and Sustainability, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea

Oct 3, 2025J/K move · Enter open · S save

Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

Kaisi Guan, Xihua Wang, Zhengfeng Lai, Xin Cheng+4

1Renmin University of China · 2Apple

Audio-Video Generation Video Diffusion Models Audio-Visual Synchronization Cross-Modal Attention

Open paper AskPDF arXiv

This study focuses on a challenging yet promising task, Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text conditions, meanwhile ensuring both modalities are aligned with text. Despite progress in joint audio-video training, two critical challenges still remain unaddressed: (1) a single, shared text caption where the text for video is equal to the text for audio often creates modal interference, confusing the pretrained backbones, and (2) the optimal mechanism for cross-modal feature interaction remains unclear. To address these challenges, we first propose the Hierarchical Visual-Grounded Captioning (HVGC) framework that generates pairs of disentangled captions, a video caption, and an audio caption, eliminating interference at the conditioning stage. Based on HVGC, we further introduce BridgeDiT, a novel dual-tower diffusion transformer, which employs a Dual CrossAttention (DCA) mechanism that acts as a robust ``bridge" to enable a symmetric, bidirectional exchange of information, achieving both semantic and temporal synchronization. Extensive experiments on three benchmark datasets, supported by human evaluations, demonstrate that our method achieves state-of-the-art results on most metrics. Comprehensive ablation studies further validate the effectiveness of our contributions, offering key insights for the future T2SV task. All the codes and checkpoints will be publicly released.

Abstract

Unified Multimodal Models

Explore similar work

Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction

OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition