Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.
Figures & tables
Figure 1: Long-Video to Audio (LV2A) task overview. The challenge is framed as training models on fixed-length segments while requiring them to generalize to variable-length (long-form) audio outputs during inference.
Figure 2: We analyze the role of positional embeddings in V2A models such as MMAudio [ 4 ] , built on MMDiT [ 22 ] . Without positional embeddings (a), MMAudio fails to capture temporal structure, producing redundant audio dominated by prominent visual objects ( e.g . , car crashing). With adjusted positional embeddings (b), alignment improves but sound quality degrades over long sequences (see scene C). (c) On UnAV100 [ 9 ] , both configurations show performance drops across durations, with MMAudio without positional embeddings performing worst in distribution matching (FD PANN↓ ) and multimodal alignment (IB-Score ↑ ).
Figure 3: Overview of our proposed framework. Left: A comprehensive end-to-end flow-matching model that operates across both multimodal and single-modal blocks, handling inputs in both compressed and original spaces. Middle: A temporal routing mechanism designed to efficiently process tokens in a time-aware manner. Right: A multimodal routing strategy that leverages strong correlations between the two modalities for enhanced integration.
UnAV100
Method
Size
Distribution Matching
Audio Quality
MM Align.
Temporal Align.
FD VGG↓
FD PANNs↓
FD PASST↓
KL PASST↓
ISC PANNs↑
ISC PASST↑
IB-Score ↑
DeSync ↓
MMAudio-S [ 4 ]
157M
4.62
10.69
349.11
1.85
5.92
6.02
28.63
0.906
MMAudio-L [ 4 ]
1.03B
3.86
9.01
296.79
1.79
6.18
6.41
30.71
0.593
MMAudio-L + NTK [ 4 ]
1.03B
4.08
8.43
301.45
1.76
6.05
6.17
30.24
0.599
LoVA [ 5 ]
1.06B
3.36
7.50
223.29
1.56
6.17
8.46
24.62
1.232
Table 1: Comparison of methods across various evaluation metrics on UnAV100 [ 9 ] and LongVale [ 10 ] .
VGGSound
Method
FD VGG↓
ISC PANNs↑
ISC PASST↑
IB-Score ↑
DeSync ↓
MMAudio-S [ 4 ]
1.66
18.02
12.46
32.27
0.444
MMAudio-L [ 4 ]
0.97
17.40
13.33
33.22
0.442
LoVA [ 5 ]
1.70
9.73
9.91
-
-
V-AURA [ 44 ]
2.88
10.08
-
27.64
0.654
MMHNet-S (ours)
1.54
16.73
12.52
32.11
0.460
Table 2: Comparison of methods under a fixed audio length ∼ 10 seconds on VGGSound. Baseline results are based on the reports in [ 4 , 5 ] .
Figure 4: Visualization of audio spectogram from MMHNet and competing methods on UnAV100.
UnAV100
Core Network
FD VGG↓
FD PANNs↓
ISC PANNs↑
IB-Score ↑
DeSync ↓
Transformers
3.36
9.00
6.42
28.41
0.638
Causal Mamba-2
2.28
9.18
5.85
33.32
0.497
Non-Causal Mamba-2
3.35
5.87
7.62
36.82
0.439
LongVale
Core Network
FD VGG↓
FD PANNs↓
ISC PANNs↑
IB-Score ↑
DeSync ↓
Table 3: Ablation of the core networks of MMHNet by comparing among transformers, Causal Mamba-2 and Mamba-2. Evaluation is performed on UnAV100 and LongVale datasets.
UnAV100
Variant
FD PANNs↓
FD PASST↓
ISC PANNs↑
ISC PASST↑
IB-Score ↑
DeSync ↓
Non-Hierarchical
6.31
264.43
6.58
6.72
35.00
0.621
Hierarchical
5.87
217.00
7.62
8.21
36.82
0.439
LongVale
Variant
FD PANNs↓
FD PASST↓
ISC PANNs↑
ISC PASST↑
IB-Score ↑
DeSync ↓
Non-Hierarchical
11.76
442.12
2.59
2.29
26.34
0.669
Table 4: Hierarchical vs. non-hierarchical method comparison.
Threshold
FD VGG↓
FD PANNs↓
ISC PANNs↑
IB-Score ↑
DeSync ↓
0.3
3.24
8.13
7.45
33.44
0.460
0.4
3.52
6.94
8.61
35.55
0.431
0.5
1.80
5.29
8.10
36.27
0.410
0.6
3.64
6.91
8.66
35.58
0.426
0.7
15.03
33.06
2.81
0.02
1.210
Table 5: Comparison of various threshold values on UnAV100.
Figure 5: Comparison with past methods on various duration splits of audio-video data on UnAV100 (FD PANNs↓ and IB-Score ↑ ).
Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.
Shaohua Pan, Junbao Chen, Shengyi He +8
Baidu, China · Beijing Institute of Technology, China
Multimodal generative models have shown remarkable progress in single-modality video and audio synthesis, yet truly joint audio-video generation remains an open challenge. In this paper, I explore four key contributions to advance this field. First, I release two high-quality, paired audio-video datasets. The datasets consisting on 13 hours of video-game clips and 64 hours of concert performances, each segmented into consistent 34-second samples to facilitate reproducible research. Second, I train the MM-Diffusion architecture from scratch on our datasets, demonstrating its ability to produce semantically coherent audio-video pairs and quantitatively evaluating alignment on rapid actions and musical cues. Third, I investigate joint latent diffusion by leveraging pretrained video and audio encoder-decoders, uncovering challenges and inconsistencies in the multimodal decoding stage. Finally, I propose a sequential two-step text-to-audio-video generation pipeline: first generating video, then conditioning on both the video output and the original prompt to synthesize temporally synchronized audio. My experiments show that this modular approach yields high-fidelity generations of audio video generation.
Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose Ripple, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.
Yanbo Ding, Zhizhi Guo, Quanyue Song +4
Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences +2