Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

Shaohua Pan; Junbao Chen; Shengyi He; Jingfeng Xue; Wen Tao; Haocheng Feng; Siming Fan; Dongwei Pan

cs.MMAug 28, 2026

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

Authors: Shaohua Pan, Junbao Chen, Shengyi He, Jingfeng Xue, Wen Tao, Haocheng Feng, Siming Fan, Dongwei Pan,

Organizations: Baidu, China · Beijing Institute of Technology, China

Abstract

Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.

Figures & tables

Figure 1. Encore enables two capabilities: (top & bottom-left) long audio-video generation with consistent identity and audio over 500 seconds; (bottom-right) long cross-modal generation, including audio-to-video (above dashed line) and video-to-audio (below dashed line).

Figure 2. Overview of Encore. The model jointly denoises audio (left) and video (right) within a shared DiT architecture. Each stream receives reference signals (reference audio/image), continuation signals (input audio and motion frames, with error injection on the video side), and noisy latents, all packed into a unified sequence with reference-shifted position embeddings. Text prompts encoded by Gemma provide semantic conditioning via cross-attention, while a dedicated audio-video cross-attention enables synchronization between the two modalities. Adaptive Signal Routing (ASR, right panel) introduces per-layer, per-head learnable biases ( $\alpha_{\ell,h}^{anc}$ , $\alpha_{\ell,h}^{ct}$ ) within self-attention and learnable scales ( $\alpha_{\ell,h}^{sem}$ , $\alpha_{\ell,h}^{sync}$ ) on cross-attention outputs, allowing the model to dynamically modulate the influence of each conditioning signal.

Methods	Video Quality				Audio Quality					Audio-Video Synchronization			Error Accumulation
Methods	MSc $\uparrow$	MSm $\uparrow$	AS $\uparrow$	ID $\uparrow$	CS $\uparrow$	CE $\uparrow$	CU $\uparrow$	PC $\downarrow$	PQ $\uparrow$	LSE-C $\uparrow$	LSE-D $\downarrow$	AV-A $\uparrow$	$\Delta$ AS $\downarrow$	$\Delta$ ID $\downarrow$
SVI	1.79	99.46	0.55	0.81	-	-	-	-	-	-	-	-	0.06	0.22
Helios	1.22	99.03	0.59	0.77	-	-	-	-	-	-	-	-	0.09	0.23
OVI	3.41	98.30	0.50	0.72	0.30	5.38	5.87	2.62	6.25	1.04	11.66	0.64	0.08	0.32
LTX2.3	2.61	99.44	0.45	0.72	0.23	6.25	7.24	4.52	7.65	1.40	10.94	0.76	0.06	0.26
Encore (Ours)	2.51	99.47	0.46	0.86	0.34	5.63	6.53	2.97	6.95	2.36	10.34	0.88	0.02	0.07

Table 1. Quantitative comparison on long audio-video generation. Bold indicates the best result and underline indicates the second best. SVI and Helios are video-only methods and thus not evaluated on audio or synchronization metrics.

Methods	Video Quality			Audio Quality	A-V Sync.
Methods	MSc $\uparrow$	MSm $\uparrow$	ID $\uparrow$	PC $\downarrow$	LSE-C $\uparrow$	AV-A $\uparrow$
LTX-2.3	3.36	99.29	0.865	3.84	2.36	0.730
OVI	1.37	99.30	0.894	2.30	2.18	0.631
Encore (Ours)	2.52	99.41	0.888	2.68	3.46	0.867

Table 2. Short-clip comparison with the LTX-2.3 backbone and OVI.

Methods	Video Quality			Audio Quality		A-V Sync.
Methods	MSc $\uparrow$	MSm $\uparrow$	ID $\uparrow$	CS $\uparrow$	PC $\downarrow$	LSE-C $\uparrow$	AV-A $\uparrow$
w/o SVI	2.43	99.41	0.77	0.29	3.20	1.02	0.61
w/o ASR	1.26	99.12	0.80	0.18	4.21	1.11	0.78
w/o $\alpha_{\ell,h}^{anc}$	2.38	99.42	0.85	0.33	3.03	2.32	0.88
w/o $\alpha_{\ell,h}^{ct}$	2.74	98.81	0.86	0.32	3.38	2.29	0.90
w/o $\alpha_{\ell,h}^{sem}$	1.16	99.22	0.84	0.19	4.20	1.24	0.78

Table 3. Ablation results on key components.

Figure 3. Qualitative comparison on real-human videos. Baselines exhibit progressive identity drift and scene changes, while Encore maintains consistent appearance and audio throughout 30 seconds.

Figure 4. Qualitative comparison on anime-character and animal videos. Encore maintains stable identity and coherent audio across diverse content types over 30 seconds.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Feb 24, 2026Christian Simon, Masato Ishii, Wei-Yao Wang +8Audio-Video Generation Audio Generation
†Sony Group Corporation · ♣Sony AI
Soundwich: Video Generation with Layered and Controllable Audio
Sep 30, 2026Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek +2Audio-Video Generation Audio-Visual Synchronization
Simon Fraser University · University of Cyprus · CYENS Centre of Excellence +1
Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
Aug 16, 2026Shengchuan Gao, Teng Hu, Bohao Feng +4Audio-Video Generation Diffusion Model Inference Acceleration
Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · Alibaba Cloud Computing

Feb 24, 2026J/K move · Enter open · S save

Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

Christian Simon, Masato Ishii, Wei-Yao Wang, Koichi Saito+7

†Sony Group Corporation · ♣Sony AI

Audio-Video Generation Audio Generation Length Generalization

Open paper AskPDF arXiv

Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.

Abstract

Audio-Video Generation

Explore similar work

Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

Soundwich: Video Generation with Layered and Controllable Audio

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention