Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

Christian Simon; Masato Ishii; Wei-Yao Wang; Koichi Saito; Akio Hayakawa; Dongseok Shim; Zhi Zhong; Shuyang Cui

cs.CVFeb 24, 2026

Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

Authors: Christian Simon, Masato Ishii, Wei-Yao Wang, Koichi Saito, Akio Hayakawa, Dongseok Shim, Zhi Zhong, Shuyang Cui,

Organizations: †Sony Group Corporation · ♣Sony AI

Abstract

Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.

Figures & tables

Figure 1: Long-Video to Audio (LV2A) task overview. The challenge is framed as training models on fixed-length segments while requiring them to generalize to variable-length (long-form) audio outputs during inference.

Figure 2: We analyze the role of positional embeddings in V2A models such as MMAudio [ 4 ] , built on MMDiT [ 22 ] . Without positional embeddings (a), MMAudio fails to capture temporal structure, producing redundant audio dominated by prominent visual objects ( e.g . , car crashing). With adjusted positional embeddings (b), alignment improves but sound quality degrades over long sequences (see scene C). (c) On UnAV100 [ 9 ] , both configurations show performance drops across durations, with MMAudio without positional embeddings performing worst in distribution matching (FD ${}_{PANN}\downarrow$ ) and multimodal alignment (IB-Score $\uparrow$ ).

Figure 3: Overview of our proposed framework. Left: A comprehensive end-to-end flow-matching model that operates across both multimodal and single-modal blocks, handling inputs in both compressed and original spaces. Middle: A temporal routing mechanism designed to efficiently process tokens in a time-aware manner. Right: A multimodal routing strategy that leverages strong correlations between the two modalities for enhanced integration.

UnAV100
Method	Size	Distribution Matching				Audio Quality		MM Align.	Temporal Align.
Method	Size	FD ${}_{\textrm{VGG}}$ $\downarrow$	FD ${}_{\textrm{PANNs}}$ $\downarrow$	FD ${}_{\textrm{PASST}}$ $\downarrow$	KL ${}_{\textrm{PASST}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	ISC ${}_{\textrm{PASST}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
MMAudio-S [ 4 ]	157M	4.62	10.69	349.11	1.85	5.92	6.02	28.63	0.906
MMAudio-L [ 4 ]	1.03B	3.86	9.01	296.79	1.79	6.18	6.41	30.71	0.593
MMAudio-L + NTK [ 4 ]	1.03B	4.08	8.43	301.45	1.76	6.05	6.17	30.24	0.599
LoVA [ 5 ]	1.06B	3.36	7.50	223.29	1.56	6.17	8.46	24.62	1.232

Table 1: Comparison of methods across various evaluation metrics on UnAV100 [ 9 ] and LongVale [ 10 ] .

VGGSound
Method	FD ${}_{\textrm{VGG}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	ISC ${}_{\textrm{PASST}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
MMAudio-S [ 4 ]	1.66	18.02	12.46	32.27	0.444
MMAudio-L [ 4 ]	0.97	17.40	13.33	33.22	0.442
LoVA [ 5 ]	1.70	9.73	9.91	-	-
V-AURA [ 44 ]	2.88	10.08	-	27.64	0.654
MMHNet-S (ours)	1.54	16.73	12.52	32.11	0.460

Table 2: Comparison of methods under a fixed audio length

\sim

10 seconds on VGGSound. Baseline results are based on the reports in [ 4 , 5 ] .

Figure 4: Visualization of audio spectogram from MMHNet and competing methods on UnAV100.

UnAV100
Core Network	FD ${}_{\textrm{VGG}}$ $\downarrow$	FD ${}_{\textrm{PANNs}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
Transformers	3.36	9.00	6.42	28.41	0.638
Causal Mamba-2	2.28	9.18	5.85	33.32	0.497
Non-Causal Mamba-2	3.35	5.87	7.62	36.82	0.439
LongVale
Core Network	FD ${}_{\textrm{VGG}}$ $\downarrow$	FD ${}_{\textrm{PANNs}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$

Table 3: Ablation of the core networks of MMHNet by comparing among transformers, Causal Mamba-2 and Mamba-2. Evaluation is performed on UnAV100 and LongVale datasets.

UnAV100
Variant	FD ${}_{\textrm{PANNs}}$ $\downarrow$	FD ${}_{\textrm{PASST}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	ISC ${}_{\textrm{PASST}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
Non-Hierarchical	6.31	264.43	6.58	6.72	35.00	0.621
Hierarchical	5.87	217.00	7.62	8.21	36.82	0.439
LongVale
Variant	FD ${}_{\textrm{PANNs}}$ $\downarrow$	FD ${}_{\textrm{PASST}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	ISC ${}_{\textrm{PASST}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
Non-Hierarchical	11.76	442.12	2.59	2.29	26.34	0.669

Table 4: Hierarchical vs. non-hierarchical method comparison.

Threshold	FD ${}_{\textrm{VGG}}$ $\downarrow$	FD ${}_{\textrm{PANNs}}$ $\downarrow$	ISC ${}_{\textrm{PANNs}}$ $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
0.3	3.24	8.13	7.45	33.44	0.460
0.4	3.52	6.94	8.61	35.55	0.431
0.5	1.80	5.29	8.10	36.27	0.410
0.6	3.64	6.91	8.66	35.58	0.426
0.7	15.03	33.06	2.81	0.02	1.210

Table 5: Comparison of various threshold values on UnAV100.

Panel of Figure 5 — Figure 5: Comparison with past methods on various duration splits of audio-video data on UnAV100 (FD ${}_{\textrm{PANNs}}$ $\downarrow$ and IB-Score $\uparrow$ ).

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing
Aug 28, 2026Shaohua Pan, Junbao Chen, Shengyi He +8Audio-Video Generation Audio-Visual Synchronization
Baidu, China · Beijing Institute of Technology, China
Diffusion Models for Joint Audio-Video Generation
Mar 17, 2026Alejandro Paredes La TorreAudio-Video Generation Multimodal Diffusion Models
Duke University
Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory
Jul 29, 2026Yanbo Ding, Zhizhi Guo, Quanyue Song +4Audio-Video Generation Long Video Generation
Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd., China · School of Artificial Intelligence, University of Chinese Academy of Sciences +2

Aug 28, 2026J/K move · Enter open · S save

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

Shaohua Pan, Junbao Chen, Shengyi He, Jingfeng Xue+7

Baidu, China · Beijing Institute of Technology, China

Audio-Video Generation Audio-Visual Synchronization Long Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.

Abstract

Audio-Video Generation

Explore similar work

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

Diffusion Models for Joint Audio-Video Generation

Ripple: Real-Time Streaming Audio-Video Generation With Cross-Modal Recurrent Memory