Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.
Figures & tables
Figure 1. Encore enables two capabilities: (top & bottom-left) long audio-video generation with consistent identity and audio over 500 seconds; (bottom-right) long cross-modal generation, including audio-to-video (above dashed line) and video-to-audio (below dashed line).
Figure 2. Overview of Encore. The model jointly denoises audio (left) and video (right) within a shared DiT architecture. Each stream receives reference signals (reference audio/image), continuation signals (input audio and motion frames, with error injection on the video side), and noisy latents, all packed into a unified sequence with reference-shifted position embeddings. Text prompts encoded by Gemma provide semantic conditioning via cross-attention, while a dedicated audio-video cross-attention enables synchronization between the two modalities. Adaptive Signal Routing (ASR, right panel) introduces per-layer, per-head learnable biases ( αℓ,hanc , αℓ,hct ) within self-attention and learnable scales ( αℓ,hsem , αℓ,hsync ) on cross-attention outputs, allowing the model to dynamically modulate the influence of each conditioning signal.
Methods
Video Quality
Audio Quality
Audio-Video Synchronization
Error Accumulation
MSc ↑
MSm ↑
AS ↑
ID ↑
CS ↑
CE ↑
CU ↑
PC ↓
PQ ↑
LSE-C ↑
LSE-D ↓
AV-A ↑
Δ AS ↓
Δ ID ↓
SVI
1.79
99.46
0.55
0.81
-
-
-
-
-
-
-
-
0.06
0.22
Helios
1.22
99.03
0.59
0.77
-
-
-
-
-
-
-
-
0.09
0.23
OVI
3.41
98.30
0.50
0.72
0.30
5.38
5.87
2.62
6.25
1.04
11.66
0.64
0.08
0.32
LTX2.3
2.61
99.44
0.45
0.72
0.23
6.25
7.24
4.52
7.65
1.40
10.94
0.76
0.06
0.26
Encore (Ours)
2.51
99.47
0.46
0.86
0.34
5.63
6.53
2.97
6.95
2.36
10.34
0.88
0.02
0.07
Table 1. Quantitative comparison on long audio-video generation. Bold indicates the best result and underline indicates the second best. SVI and Helios are video-only methods and thus not evaluated on audio or synchronization metrics.
Methods
Video Quality
Audio Quality
A-V Sync.
MSc ↑
MSm ↑
ID ↑
PC ↓
LSE-C ↑
AV-A ↑
LTX-2.3
3.36
99.29
0.865
3.84
2.36
0.730
OVI
1.37
99.30
0.894
2.30
2.18
0.631
Encore (Ours)
2.52
99.41
0.888
2.68
3.46
0.867
Table 2. Short-clip comparison with the LTX-2.3 backbone and OVI.
Methods
Video Quality
Audio Quality
A-V Sync.
MSc ↑
MSm ↑
ID ↑
CS ↑
PC ↓
LSE-C ↑
AV-A ↑
w/o SVI
2.43
99.41
0.77
0.29
3.20
1.02
0.61
w/o ASR
1.26
99.12
0.80
0.18
4.21
1.11
0.78
w/o αℓ,hanc
2.38
99.42
0.85
0.33
3.03
2.32
0.88
w/o αℓ,hct
2.74
98.81
0.86
0.32
3.38
2.29
0.90
w/o αℓ,hsem
1.16
99.22
0.84
0.19
4.20
1.24
0.78
Table 3. Ablation results on key components.
Figure 3. Qualitative comparison on real-human videos. Baselines exhibit progressive identity drift and scene changes, while Encore maintains consistent appearance and audio throughout 30 seconds.
Figure 4. Qualitative comparison on anime-character and animal videos. Encore maintains stable identity and coherent audio across diverse content types over 30 seconds.
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.
Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.
Recent audio-visual generation models can synthesize synchronized video and sound in a unified diffusion process, but their inference cost remains high because long video token sequences require repeated attention computation across denoising steps. A variety of acceleration techniques have been developed for video generation models, including low-bit quantization, attention sparsification, and feature caching. However, since these methods are originally designed for video generation, directly applying them to audio-visual models overlooks the interactions between the audio and video branches and may therefore disrupt audio-video synchronization. We present a synchronization-aware acceleration framework for efficient audio-visual generation. Our key observation is that bidirectional audio-video cross-attention reveals structured interactions between the two branches, with high responses often concentrated on a few sound-related visual and temporal regions. Guided by this interaction pattern, we introduce a protected sparse attention strategy that preserves high-fidelity computation for synchronization-critical tokens while sparsifying redundant attention interactions. By explicitly accounting for cross-modal dependence during acceleration, our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.
Shengchuan Gao, Teng Hu, Bohao Feng +4
Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · Alibaba Cloud Computing