Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

Shengchuan Gao; Teng Hu; Bohao Feng; Luchen Li; Wenqiang Wang; Hongqian Deng; Ran Yi

cs.CVAug 16, 2026

Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

Authors: Shengchuan Gao, Teng Hu, Bohao Feng, Luchen Li, Wenqiang Wang, Hongqian Deng, Ran Yi

Organizations: Shanghai Jiao Tong University · Alibaba Token Hub, Alibaba Group · Alibaba Cloud Computing

Abstract

Recent audio-visual generation models can synthesize synchronized video and sound in a unified diffusion process, but their inference cost remains high because long video token sequences require repeated attention computation across denoising steps. A variety of acceleration techniques have been developed for video generation models, including low-bit quantization, attention sparsification, and feature caching. However, since these methods are originally designed for video generation, directly applying them to audio-visual models overlooks the interactions between the audio and video branches and may therefore disrupt audio-video synchronization. We present a synchronization-aware acceleration framework for efficient audio-visual generation. Our key observation is that bidirectional audio-video cross-attention reveals structured interactions between the two branches, with high responses often concentrated on a few sound-related visual and temporal regions. Guided by this interaction pattern, we introduce a protected sparse attention strategy that preserves high-fidelity computation for synchronization-critical tokens while sparsifying redundant attention interactions. By explicitly accounting for cross-modal dependence during acceleration, our method improves inference efficiency while keeping video quality, audio quality, and audio-video synchronization.

Figures & tables

Figure 1: Our method accelerates audio-visual generation while preserving high visual fidelity. On Ovi (left) and LTX-2.3 (right), our synchronization-aware protection better preserves frame quality and remains closer to dense inference , while achieving nearly $2\times$ and $1.4\times$ end-to-end speedup.

Figure 2: Motivation of synchronization-aware protection. From left to right: a generated video frame, the extracted audio-visual saliency and protected regions, feature error of vanilla sparse attention, feature error of our protected sparse attention, and their error difference. Red areas in the difference map indicate regions where our method reduces the approximation error compared with vanilla sparsification.

Figure 3: Overview of the proposed synchronization-aware acceleration framework. Cross-modal interactions between audio and video branches are used to guide protected sparse attention and cache reuse.

Base Model	Method	Video Quality				Audio / Sync.			Efficiency
Base Model	Method	PSNR $\uparrow$	LPIPS $\downarrow$	SSIM $\uparrow$	Q-Align $\uparrow$	Sync-C $\uparrow$	Sync-D $\downarrow$	VISQOL $\uparrow$	Speedup $\uparrow$	Latency (s) $\downarrow$
Ovi	Dense Baseline	$\infty$	0.0000	1.0000	0.7953	4.688	8.172	Ref	1.000x	448.75
	SVG	21.7630	0.2511	0.7706	0.7601	3.658	8.729	3.3670	1.300x	344.83
	BWCache	21.9480	0.2444	0.7672	0.7560	3.822	8.635	3.5192	1.774x	252.71
	TeaCache	24.7775	0.1494	0.8365	0.7546	3.743	8.651	3.4610	1.811x	247.83
	SeaCache	22.1419	0.2389	0.7743	0.7218	3.901	8.311	3.3588	2.471x	181.47

Table 1: Main comparison on Ovi and LTX-2.3.

Figure 4: Saliency-conditioned feature error analysis. (a) Feature error across different cross-modal saliency ranks. (b) Error reduction on top-

5\%

protected queries.

Figure 5: Layer–timestep comparison of feature error.

Figure 6: Qualitative comparison of dense inference, unprotected acceleration, and our protected acceleration. The protected computation path better preserves salient visual structures and reduces local artifacts under acceleration.

Attn	Cache	Audio	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Sync-C $\uparrow$	Speedup $\uparrow$
			21.5338	0.7689	0.2462	3.626	2.230x
✓			24.2551	0.8182	0.1533	3.939	2.152x
	✓		22.9203	0.7821	0.1915	3.852	2.063x
✓	✓	✓	25.6857	0.8385	0.1398	4.606	1.992x

Table 2: Component ablation under sparse computation. Checkmarks indicate whether the corresponding protection strategy is enabled.

Prot. Ratio	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Sync-C $\uparrow$	Speedup $\uparrow$
2%	24.8926	0.8133	0.1672	4.539	2.02x
5%	25.6857	0.8385	0.1398	4.606	1.992x
10%	25.8201	0.8413	0.1369	4.648	1.861x

Table 3: Ablation study on the protection ratio.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Illustration of a typical dual-stream audio-visual diffusion block. The video and audio branches are processed separately and exchange information through bidirectional audio-video attention.

Saliency Strategy	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Sync-C $\uparrow$	Sync-D $\downarrow$
No Protection	21.5338	0.7689	0.2462	3.626	8.653
Random 5%	22.0219	0.7762	0.2376	3.819	8.578
Video Self-Attn Score 5%	23.5128	0.8025	0.2140	3.764	8.602
Cross-Modal Saliency 5%	25.6857	0.8385	0.1398	4.606	8.378

Appendix

Table 4: Ablation on saliency selection strategies.

$K$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	Sync-C $\uparrow$	Sync-D $\downarrow$
2	25.7033	0.8408	0.1378	4.629	8.492
4	25.6857	0.8385	0.1398	4.606	8.378
8	24.3362	0.8166	0.1702	4.325	8.522

Appendix

Table 5: Ablation on the saliency update interval.

Figure 8: Protected-token overlap across denoising-step intervals. The high overlap at small intervals supports lagged saliency reuse, while the decrease at larger intervals motivates periodic saliency updates.

Panel of Figure 9 — Figure 9: Additional qualitative comparisons on LTX-2.3.

Panel of Figure 10 — Figure 10: Additional qualitative comparisons on LTX-2.3.

Panel of Figure 11 — Figure 11: Additional qualitative comparisons on LTX-2.3.

Panel of Figure 12 — Figure 12: Additional qualitative comparisons on LTX-2.3.

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing
Aug 28, 2026Shaohua Pan, Junbao Chen, Shengyi He +8Audio-Video Generation Audio-Visual Synchronization
Baidu, China · Beijing Institute of Technology, China
CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation
Oct 22, 2026Lifan Jiang, Yibo Zhao, Senlin cheng +6Audio-Video Generation Diffusion Model Caching
State Key Lab of CAD&CG, Zhejiang University · Ant Group
Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training
Oct 4, 2026Shuyuan Tu, Qi Tian, Yinming Huang +8Audio-Video Generation High-Resolution Video Generation
Fudan University · Tencent Hunyuan Foundation Model Team · Zhejiang University

Aug 28, 2026J/K move · Enter open · S save

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

Shaohua Pan, Junbao Chen, Shengyi He, Jingfeng Xue+7

Baidu, China · Beijing Institute of Technology, China

Audio-Video Generation Audio-Visual Synchronization Long Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Existing audio-video generation methods produce well-synchronized clips but are limited to short durations, while long-video generation methods extend duration through chunk-based iterative synthesis yet lack audio entirely. Generating long audio-video jointly is fundamentally harder than either task alone: each chunk must simultaneously maintain video temporal coherence, audio temporal coherence, and cross-modal synchronization, whose conditioning signals enter the model through different pathways. In this work, we present Encore for long-form synchronized audio-video generation. Our key insight is to factor this challenge into: (1) local continuity handled by iterative generation with explicit cross-chunk context propagation, and (2) global consistency enforced via reference audio-video signals with shifted position embedding. Building on this design, we propose Adaptive Signal Routing (ASR), which introduces learnable attention biases within self-attention and learnable residual scales on cross-attention outputs, enabling the model to adaptively modulate the influence of each conditioning signal. Trained end-to-end for joint audio-video generation, Encore also supports infinite-length audio-to-video and video-to-audio synthesis at inference by conditioning on the ground-truth modality throughout the denoising process. Experiments on our extended VerseBench for long audio-video evaluation demonstrate that Encore significantly outperforms existing methods in both generation quality and temporal coherence. Code and data for this paper are at https://github.com/shaohua-pan/Encore.

Abstract

Audio-Video Generation

Explore similar work

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model Training