Unified decoder-only transformers have shown promise for multimodal generation, yet how sequence design affects synchronization across modalities with heterogeneous sampling rates remains underexplored. We investigate this through video-text-to-speech (VTTS) synthesis, a controlled task requiring fine-grained temporal alignment between sparse text, video, and continuous speech. Using Visatronic, a unified decoder-only transformer trained on VoxCeleb2, we examine complementary modality contributions, positional encoding strategies, the effects of modality ordering on in-domain performance and cross-domain transfer, and phoneme-level timing errors. Global sequential indexing, which assigns position IDs in concatenation order, and co-temporal ordered indexing, which assigns identical IDs to temporally corresponding video and speech tokens, achieve comparable in-domain synchronization. Under domain shift, however, synchronization depends on the interaction between token ordering and position-ID design. Text-first co-temporal indexing transfers most robustly among the evaluated variants without requiring explicit timestamp metadata. Text and video provide complementary signals: text supports intelligibility, while video contributes temporal cues and emotional expressiveness. Modality ordering also introduces a trade-off: video-first ordering achieves stronger in-domain performance, whereas text-first ordering generalizes more robustly among the timestamp-informed prefix variants. Training on diverse, in-the-wild data is associated with more transferable behavior, although isolating its contribution requires targeted ablations. We introduce TimeSync, a phoneme-level synchronization metric that reveals temporal misalignments overlooked by frame-level metrics. Together, these findings establish VTTS as a testbed for understanding temporal synchronization in unified multimodal decoders.
Figures & tables
Method
Intelligibility ( ↑ )
Naturalness ( ↑ )
Synchronization ( ↑ )
GT
4.79 ±0.05
4.79 ±0.05
4.73 ±0.06
GT (discrete)
4.32 ±0.11
3.80 ±0.11
4.59 ±0.07
Lip to Speech Synthesis
Lip2Speech ( Kim et al., 2023 )
2.21 ±0.08
2.20 ±0.09
2.69 ±0.08
SVTS ( Mira et al., 2022 )
2.17 ±0.08
2.15 ±0.09
2.71 ±0.09
VCA-GAN ( Kim et al., 2021 )
2.19 ±0.08
2.20 ±0.09
2.71 ±0.08
Table 1: Human evaluation on LRS3. Mean opinion scores (MOS) (1-5) with 95% confidence intervals for our VoxCeleb2-trained models evaluated on LRS3. VTTS (TV-CoTemporal) achieves better performance in Intelligibility (3.62) and Synchronization (3.12), while both methods perform equally well in Naturalness (3.01). GT is an upper bound, while GT (discrete) shows degradation due to speech quantization and vocoding artifacts. These results demonstrate our model maintains good perceptual quality even on out-of-distribution data. For baseline methods, we use publicly available model checkpoints from their respective repositories to generate speech samples, which we then evaluate using our human evaluation protocol. “-” indicates unavailable metrics under the same standardized setup; this applies to dubbing baselines where only naturalness scores were available.
Method
video-speech emotions ( ↑ )
speech emotions ( ↑ )
Angry ( ↑ )
Happy ( ↑ )
Fearful ( ↑ )
GT
4.62 ±0.07
4.92 ±0.04
4.36
4.43
4.20
GT (discrete)
4.41 ±0.10
4.37 ±0.12
–
–
–
TTS (dmel Bai et al. (2024) )
3.57 ±0.14
3.20 ±0.15
3.46
3.58
3.17
VTTS
Video-Causal-Streaming
3.66 ±0.16
3.36 ±0.15
–
–
–
TV-CoTemporal
3.79 ±0.15
3.31 ±0.17
–
–
–
Table 2: Human evaluation of emotional expressiveness on VoxCeleb2. Mean Opinion Scores (1-5) with 95% confidence intervals for emotional consistency between video and speech ( video-speech emotions ), perceived emotion quality in speech alone ( speech emotions ), and specific emotion categories ( Angry, Happy, Fearful ). VTTS (TV-CoTemporal) achieves the highest score in video-speech emotion alignment (3.79), while VTTS (VT-Scaled) performs best in overall speech emotion quality (3.39) and in all individual emotion categories. These results demonstrate the advantage of visual conditioning for expressive speech generation. Ground truth (GT) serves as the reference, and GT (discrete) provides the upper bound achievable with our quantization-based approach. Per-category labels were collected only for GT, TTS, and VTTS (VT-Scaled) due annotation-budget constraints.
Method
Training Dataset
WER (%) ↓
V2SFlow ( Choi et al., 2025 )
LRS3
28.5
LipVoicer ( Yemini et al., 2024 )
LRS3
21.4
Lip2Speech ( Kim et al., 2023 )
LRS3
57.4
SVTS ( Mira et al., 2022 )
LRS3
82.4
VCA-GAN ( Kim et al., 2021 )
LRS3
90.6
DiffV2S ( Choi et al., 2023 )
LRS3
39.2
Table 3: Cross-dataset generalization and validation against prior work on full-set LRS3. Primary comparisons are reported on the full LRS3 test set for fairness. VoiceCraft-Dub* is shown for reference only and is not directly comparable because it adapts a large pretrained TTS backbone with additional AV-specific fine-tuning.
Method
Input Modality
GT WER ( ↓ )
GT (discrete) WER ( ↓ )
WER ( ↓ )
TimeSync (s) ( ↓ )
TTS (dmel Bai et al. (2024) )
Text
4.0 ±0.1
10.5 ±0.1
19.0 +8.5
-
VTTS
VT-Scaled
Video-Text
17.2 +6.7
-
TTS
Text
2.6 ±0.1
10.1 ±0.2
14.7 +4.6
0.62 ±0.98
VTTS
Video-Causal-Streaming
Text-Video
14.5 +4.4
0.49 ±0.63
Table 4: WER and TimeSync on VoxCeleb2. We report three WERs: (i) GT WER from original audio, (ii) GT (discrete) WER from reconstructed audio using GT speech tokens (lower bound), and (iii) WER from generated audio by a model. All values are computed using Whisper-large v2. The first two rows correspond to PL.v1 transcripts; the remaining rows use PL.v2. TimeSync is reported for the PL.v2 section. Among non-global prefix variants, VTTS (VT-Scaled) achieves 12.2% WER, while global-indexing ablations reach 12.1–12.7% WER. TimeSync further shows improved phoneme-level synchronization with video-conditioned models. Rows labeled “global” compare Global Sequential Indexing as an ablation against their non-global counterparts (TV-CoTemporal and VT-Scaled).
Table 5
Method
Iterations
GT WER ( ↓ )
GT (discrete) WER ( ↓ )
WER ( ↓ )
TTS
2M
2.6 ±0.1
10.1 ±0.2
17.3 +7.2
VTTS
TV-CoTemporal
2M
17.0 +6.9
LTTS (LT-Scaled)
2M
14.9 +4.8
VT-Scaled
2M
12.2 +2.1
TTS
3M
2.6 ±0.1
10.1 ±0.2
14.7 +4.6
Table 7: Convergence analysis. Training-iteration comparison shows faster convergence when video modality is used in addition to text. At 2M iterations, VTTS variants are comparable to or better than TTS at the same budget, and VTTS (VT-Scaled) already reaches 12.2% WER. TTS reaches 14.7% WER at 3M iterations, while VTTS (VT-Scaled) maintains a smaller gap (+2.1%) from GT (discrete) WER, indicating more efficient optimization with multimodal conditioning.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Variant
Order
Position IDs
Shared
Causal
Intended setting
TV-CoTemporal
T → V → S
timestamp, shared axis
yes
prefix
transfer; dubbing unseen speakers
VT-Scaled
V → T → S
timestamp-scaled video
no
prefix
in-domain synthesis
VC-Streaming
T → V/S (I)
temporal order
yes
video t′≤t
streaming, low latency
TV-Global
T → V → S
concatenation order
no
prefix
ID ablation
VT-Global
V → T → S
concatenation order
no
prefix
ID ablation
Appendix
Table 8: Sequence-design variants. Each variant is fully specified by the three per-token quantities of Figure 4 : token order, timestamp, and how the position ID is computed. A speaker token precedes all modalities in every variant and is omitted from the table. All variants use full-prefix conditioning-every conditioning token precedes the speech tokens-except Video-Causal-Streaming, in which video and speech are interleaved in temporal order so that speech at time t attends only to video at t′≤t . Order: T, V, S denote text, video and speech tokens; (I) marks interleaving; VC-Streaming abbreviates Video-Causal-Streaming. Shared: whether video and speech tokens share position IDs. Causal: "prefix" means all conditioning tokens precede speech.
Method
Intelligibility ( ↑ )
Naturalness ( ↑ )
Synchronization ( ↑ )
GT
4.55 ±0.09
4.79 ±0.05
4.57 ±0.10
GT (discrete)
3.95 ±0.13
3.77 ±0.15
4.36 ±0.12
TTS (dmel Bai et al. (2024) )
3.17 ±0.19
2.92 ±0.21
1.98 ±0.15
VTTS
Video-Causal-Streaming
3.19 ±0.17
2.99 ±0.16
2.28 ±0.17
TV-CoTemporal
3.35 ±0.17
3.02 ±0.19
2.50 ±0.21
Appendix
Table 9: Human evaluation on VoxCeleb2. Mean Opinion Scores (MOS) (1-5) with 95% confidence intervals for Intelligibility, Naturalness, and Synchronization. VTTS (VT-Scaled) achieves the best performance in Intelligibility (3.48) and Naturalness (3.20), while VTTS (TV-CoTemporal) performs best in Synchronization (2.50). Both outperform the TTS baseline across all metrics. Ground truth (GT) serves as a reference, and GT (discrete) is an upper bound due to speech discretization.
Method
Training Dataset
WER (%) ↓
TimeSync (s) ↓
V2SFlow ( Choi et al., 2025 )
LRS3
28.5
-
LipVoicer ( Yemini et al., 2024 )
LRS3
21.4
-
Lip2Speech ( Kim et al., 2023 )
LRS3
57.4
-
SVTS ( Mira et al., 2022 )
LRS3
82.4
-
VCA-GAN ( Kim et al., 2021 )
LRS3
90.6
-
DiffV2S ( Choi et al., 2023 )
LRS3
39.2
-
Appendix
Table 10: LRS3 transfer-focused setting. This setting is introduced in this work for detailed zero-shot transfer analysis. Unlike the main full-set WER results, we additionally report TimeSync in this setting.
Method
WER (%) ↓
VDTTS-VOXCELEB2 ( Hassid et al., 2022 )
48.0
VDTTS-LSVSR ( Hassid et al., 2022 )
25.0
VTTS (VT-Scaled, VoxCeleb2-trained, ours)
12.2
Appendix
Table 11: Reference-only VoxCeleb2-trained WER values. All rows are trained on VoxCeleb2 and evaluated on VoxCeleb2. Rows from ( Hassid et al., 2022 ) use an external evaluation protocol; our best VoxCeleb2-trained non-global prefix variant is included for context from Table 4 . These rows are not directly comparable.
This study focuses on a challenging yet promising task, Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text conditions, meanwhile ensuring both modalities are aligned with text. Despite progress in joint audio-video training, two critical challenges still remain unaddressed: (1) a single, shared text caption where the text for video is equal to the text for audio often creates modal interference, confusing the pretrained backbones, and (2) the optimal mechanism for cross-modal feature interaction remains unclear. To address these challenges, we first propose the Hierarchical Visual-Grounded Captioning (HVGC) framework that generates pairs of disentangled captions, a video caption, and an audio caption, eliminating interference at the conditioning stage. Based on HVGC, we further introduce BridgeDiT, a novel dual-tower diffusion transformer, which employs a Dual CrossAttention (DCA) mechanism that acts as a robust ``bridge" to enable a symmetric, bidirectional exchange of information, achieving both semantic and temporal synchronization. Extensive experiments on three benchmark datasets, supported by human evaluations, demonstrate that our method achieves state-of-the-art results on most metrics. Comprehensive ablation studies further validate the effectiveness of our contributions, offering key insights for the future T2SV task. All the codes and checkpoints will be publicly released.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Jun Zhan, Chen Yang, Yitian Gong +23
1Fudan University · 2Shanghai Innovation Institute · 3MOSI Intelligence +1
Audio-visual emotion recognition (AVER) methods typically fuse utterance-level features, and even frame-level attention models seldom address the frame-rate mismatch across modalities. In this paper, we propose a Transformer-based framework focusing on the temporal alignment of multimodal features. Our design employs a multimodal self-attention encoder that simultaneously captures intra- and inter-modal dependencies within a shared feature space. To address heterogeneous sampling rates, we incorporate Temporally-aligned Rotary Position Embeddings (TaRoPE), which implicitly synchronize audio and video tokens. Furthermore, we introduce a Cross-Temporal Matching (CTM) loss that enforces consistency among temporally proximate pairs, guiding the encoder toward better alignment. Experiments on CREMA-D and RAVDESS datasets demonstrate consistent improvements over recent baselines, suggesting that explicitly addressing frame-rate mismatch helps preserve temporal cues and enhances cross-modal fusion.
Inyong Koo, yeeun Seong, Minseok Son +2
School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea · Graduate School of Green Growth and Sustainability, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea