The dual-stream transformer architecture-based joint audio-video generation method has become the dominant paradigm in current research. By incorporating pre-trained video diffusion models and audio diffusion models, along with a cross-modal interaction attention module, high-quality, temporally synchronized audio-video content can be generated with minimal training data. In this paper, we first revisit the dual-stream transformer paradigm and further analyze its limitations, including model manifold variations caused by the gating mechanism controlling cross-modal interactions, biases in multi-modal background regions introduced by cross-modal attention, and the inconsistencies in multi-modal classifier-free guidance (CFG) during training and inference, as well as conflicts between multiple conditions. To alleviate these issues, we propose Cross-Modal Context Learning (CCL), equipped with several carefully designed modules. Temporally Aligned RoPE and Partitioning (TARP) effectively enhances the temporal alignment between audio latent and video latent representations. The Learnable Context Tokens (LCT) and Dynamic Context Routing (DCR) in the Cross-Modal Context Attention (CCA) module provide stable unconditional anchors for cross-modal information, while dynamically routing based on different training tasks, further enhancing the model's convergence speed and generation quality. During inference, Unconditional Context Guidance (UCG) leverages the unconditional support provided by LCT to facilitate different forms of CFG, improving train-inference consistency and further alleviating conflicts. Through comprehensive evaluations, CCL achieves state-of-the-art performance compared with recent academic methods while requiring substantially fewer resources.
Figures & tables
Figure 1: We demonstrate several capabilities of CCL, including multilingual human speech generation, environmental sound synthesis, music generation, background speech generation, storyboard-style scene transitions, and dialogue generation, as well as applicability to real-world scenarios such as beauty tutorial production.
Figure 2
Figure 4: The pipeline of our proposed Cross-Modal Context Learning . CCL follows the conventional dual-stream transformer architecture, equipped with several novel-designed modules, enabling efficient and effective joint audio-video generation with high consistency. The figure illustrates the implementation details of proposed modules. For Dynamic Context Routing, the various colors denote that the corresponding colored paths on the left are in an activated state.
Figure 5: The visualization of the training loss when adopting the gate mechanism compared with leveraging the CCL. We only sampled the loss for the joint audio-video generation task and applied the EMA operation. Notably, due to the instability of the loss during the early stages of training, we begin visualizing from iteration 100.
Figure 6: The visualization of attention maps in CCL. We observe that background tokens in both the audio and video modalities attend more strongly to the LCT.
Method
Data
Audio Quality
Lip Sync-easy
Lip Sync-hard
AV-Alignment-easy
AV-Alignment-hard
CU ↑
PQ ↑
WER ↓
Sync-C ↑
Sync-D ↓
Sync-C ↑
Sync-D ↓
DeSync ↓
IB ↑
DeSync ↓
IB ↑
Ovi [ 33 ]
30.7M
5.88
6.14
17.51%
8.47
7.32
5.96
8.57
1.11
0.20
1.18
0.21
LTX-2 [ 13 ]
–
5.12
5.35
11.04%
7.67
7.90
5.47
8.93
1.18
0.19
1.20
0.21
MOVA [ 50 ]
50M
6.34
7.24
8.09%
7.79
7.70
5.38
8.87
1.15
0.20
1.22
0.19
CCL
4M
5.71
6.42
4.62 %
8.45
7.48
6.12
8.25
1.11
0.23
1.19
0.22
Table 1: We compare CCL with existing methods on the test set, including Ovi [ 33 ] , LTX-2 [ 13 ] , and MOVA [ 50 ] . The evaluation metrics cover audio quality, lip sync, and AV-Alignment. The results show that CCL substantially outperforms prior approaches, further validating the effectiveness of our proposed method.
TARP
LCT/DCR
UCG
SyncCFG [ 20 ]
Audio Quality
Lip Sync
AV-Alignment
WER ↓
Sync-C ↑
Sync-D ↓
DeSync ↓
IB ↑
4.52 %
4.53
10.81
1.20
0.20
✓
4.67%
5.05
10.17
1.20
0.20
✓
✓
4.81%
5.62
9.12
1.18
0.21
✓
✓
✓
4.63%
6.14
8.09
1.16
0.22
✓
✓
✓
✓
5.59%
5.85
8.46
1.18
0.20
Table 2: We conduct extensive ablation studies to validate the effectiveness of each proposed module. We also compare our design choices with existing alternatives featuring similar components. The results indicate that every introduced module contributes to consistent performance improvements.
Multimodal generative models have shown remarkable progress in single-modality video and audio synthesis, yet truly joint audio-video generation remains an open challenge. In this paper, I explore four key contributions to advance this field. First, I release two high-quality, paired audio-video datasets. The datasets consisting on 13 hours of video-game clips and 64 hours of concert performances, each segmented into consistent 34-second samples to facilitate reproducible research. Second, I train the MM-Diffusion architecture from scratch on our datasets, demonstrating its ability to produce semantically coherent audio-video pairs and quantitatively evaluating alignment on rapid actions and musical cues. Third, I investigate joint latent diffusion by leveraging pretrained video and audio encoder-decoders, uncovering challenges and inconsistencies in the multimodal decoding stage. Finally, I propose a sequential two-step text-to-audio-video generation pipeline: first generating video, then conditioning on both the video output and the original prompt to synthesize temporally synchronized audio. My experiments show that this modular approach yields high-fidelity generations of audio video generation.
Recent advances in Diffusion Transformers (DiTs) have enabled high-quality joint audio-video generation, producing videos with synchronized audio within a single model. However, existing controllable generation frameworks are typically restricted to video-only control. This restricts comprehensive controllability and often leads to suboptimal cross-modal alignment. To bridge this gap, we present MMControl, which enables users to perform Multi-Modal Control in joint audio-video generation. MMControl introduces a dual-stream conditional injection mechanism. It incorporates both visual and acoustic control signals, including reference images, reference audio, depth maps, and pose sequences, into a joint generation process. These conditions are injected through bypass branches into a joint audio-video Diffusion Transformer, enabling the model to simultaneously generate identity-consistent video and timbre-consistent audio under structural constraints. Furthermore, we introduce modality-specific guidance scaling, which allows users to independently and dynamically adjust the influence strength of each visual and acoustic condition at inference time. Extensive experiments demonstrate that MMControl achieves fine-grained, composable control over character identity, voice timbre, body pose, and scene layout in joint audio-video generation.
Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1
Jun Zhan, Chen Yang, Yitian Gong +23
1Fudan University · 2Shanghai Innovation Institute · 3MOSI Intelligence +1