cs.CVMar 9, 2026

Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows

Authors: Shentong Mo, Yibing Song

Organizations: CMU / MBZUAI · DAMO Academy, Alibaba Group · Hupan Laboratory

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment, where both semantics and rhythmics of the generated audio segments shall correspond to those in the video frames. Previous studies leverage a two-stage design where the AV encoders are firstly aligned via contrastive learning, then the encoded video representations guide the audio generation process. We observe that both contrastive learning and global video guidance are effective in aligning overall AV semantics while limiting temporally rhythmic synchronization. In this work, we propose FoleyFlow to first align unimodal AV encoders via masked modeling training, where the masked audio segments are recovered under the guidance of the corresponding video segments. After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency. Then, we develop a dynamic conditional flow for the final audio generation. Built upon the efficient velocity flow generation framework, our dynamic conditional flow utilizes temporally varying video features as the dynamic condition to guide corresponding audio segment generations. To this end, we extract coherent semantic and rhythmic representations during masked AV alignment, and use this representation of video segments to guide audio generation temporally. Our audio results are evaluated on the standard benchmarks and largely surpass existing results under several metrics. The superior performance indicates that FoleyFlow is effective in generating coordinated audios that are both semantically and rhythmically coherent to various video sequences.

Figures & tables

Figure 1: Comparison with state-of-the-art approaches. We achieve the best results regarding semantical (Align Acc) and rhythmical (KLD, FAD) metrics.
Figure 2: Illustration of the proposed Foley-Flow framework.
MethodKLD ↓\downarrowFAD ↓\downarrowAlign Acc ↑\uparrow
SpecVQGAN [ 10 ]3.786.6348.79
Im2Wav [ 46 ]2.546.3274.31
Diff-Foley [ 17 ]3.156.4082.47
FoleyGen [ 19 ]2.892.5973.83
V2A-Mapper [ 52 ]2.780.9974.37
Seeing & Hearing [ 53 ]2.622.6378.95
Table 1: Comparison results on VGGSound test set for video-to-audio generation. The KLD, FAD, and Align Acc values are reported.
VAMAGVAFKLD ↓\downarrowFAD ↓\downarrowAlign Acc ↑\uparrow
✗✗3.156.4082.47
✓✗1.681.5796.29
✗✓1.921.8993.86
✓✓0.970.5298.97
Table 2: Ablation studies on Video-Audio Masking Alignment (VAMA) and Generalized Video-Audio Flow (GVAF) for video-to-audio generation. The KLD, FAD, and Align Acc values are reported.
VideoAudioKLD ↓\downarrowFAD ↓\downarrowAlign Acc ↑\uparrow
ImageBindImageBind1.381.2797.29
MAEImageBind1.171.0698.32
EVA-CLIPImageBind1.010.7398.52
ImageBindAudioMAE1.361.1397.74
MAEAudioMAE1.060.6598.62
EVA-CLIPAudioMAE0.970.5298.97
Table 3: Ablation studies on video-audio encoders for video-to-audio generation. The KLD, FAD, and Align Acc are reported.
Flow TypeKLD ↓\downarrowFAD ↓\downarrowAlign Acc ↑\uparrow
DCF1.280.8597.73
GDCF0.970.5298.97
Table 4: Ablation studies on Dynamic Conditional Flow (DCF) and Generalized Dynamic Conditional Flow (GDCF) for video-to-audio generation. The KLD, FAD, and Align Acc are reported.
Masking RatioKLD ↓\downarrowFAD ↓\downarrowAlign Acc ↑\uparrow
0.31.251.0698.12
0.51.160.8798.37
0.71.090.6598.78
0.80.970.5298.97
0.91.020.5698.96
Table 5: Ablation studies on masking ratio for video-to-audio generation. The KLD, FAD, and Align Acc values are reported.
More in this topic

Audio-Video Generation

Audio-video generation creates or edits synchronized sound and moving images from text, media, or other multimodal inputs.

See all 157 papers

Explore similar work

CardsList
Jun 12, 2026

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics, enabling synchronized, versatile audio synthesis for diverse tasks. Existing VTA methods either have multi-modal control but weak temporal alignment or strong alignment but lack reference audio conditioning and semantic precision. FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy preserving training synchronization, and an adverb-based data augmentation algorithm leveraging signal processing and large language models to enhance textual supervision with nuanced semantics. Experiments on AudioCaps, VGGSound, and Greatest Hits demonstrate its competitive controllable VTA performance against existing methods. Demo samples are available at https://foleygenex.github.io/FoleyGenEx.

Shiyao Wang, Xijuan Zeng, Hui Wang +4

Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China · Kling Team, Kuaishou Technology, Beijing, China

Audio-Video GenerationAudio-Visual SynchronizationSound Effects GenerationAudio-Visual AlignmentAudio Generation
AskPDF
Jul 7, 2026

Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.

Thanh V. T. Tran, Ngoc-Son Nguyen, Luong Tran +4

FPT Software AI Center, Vietnam · NVIDIA Corporation, USA

Audio-Video GenerationAudio-Visual Synchronization
AskPDF
Mar 16, 2026

AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer

Existing video-to-audio (V2A) generation methods predominantly rely on text prompts alongside visual information to synthesize audio. However, two critical bottlenecks persist: semantic granularity gaps in training data, such as conflating acoustically distinct sounds under coarse labels, and textual ambiguity in describing micro-acoustic features. These bottlenecks make it difficult to perform fine-grained sound synthesis using text-controlled modes. To address these limitations, we propose AC-Foley, an audio-conditioned V2A model that directly leverages reference audio to achieve precise and fine-grained control over generated sounds. This approach enables fine-grained sound synthesis, timbre transfer, zero-shot sound generation, and improved audio quality. By directly conditioning on audio signals, our approach bypasses the semantic ambiguities of text descriptions while enabling precise manipulation of acoustic attributes. Empirically, AC-Foley achieves state-of-the-art performance for Foley generation when conditioned on reference audio, while remaining competitive with state-of-the-art video-to-audio methods even without audio conditioning. Code and demo are available at: https://ff2416.github.io/AC-Foley-Page

Pengjun Fang, Yingqing He, Yazhou Xing +3

The Hong Kong University of Science and Technology · University of Central Florida

Audio-Video GenerationAudio GenerationSound Effects Generation
AskPDF