Semantic-Critical Regions for Video-to-Audio Generation

Yun Fang; Wei Guo; Zhuonan Liang; Heng Wang; Weidong Cai

cs.CVOct 16, 2026

Semantic-Critical Regions for Video-to-Audio Generation

Authors: Yun Fang, Wei Guo, Zhuonan Liang, Heng Wang, Weidong Cai

Organizations: The University of Sydney, Sydney, Australia

Abstract

In real-world videos, sound events are often associated with localized and fine-grained visual cues, such as object motion, object-part dynamics, contact events, and interactions between objects. Existing video-to-audio (V2A) methods employ implicit conditioning based on global visual representations or explicit control derived from temporal and spatial cues. However, subtle sound-producing and sound-indicative interactions remain insufficiently captured, leaving room for more precise visual conditioning of sound-event semantics and temporal dynamics. Motivated by this observation, we refer to regions containing such sound-relevant visual evidence as Semantic-Critical Regions (SCRs) and introduce SCR-V2A, a V2A generation framework that explicitly localizes and encodes SCRs as fine-grained visual guidance for audio generation. Specifically, we first identify SCRs through sound-related visual localization. We then transform the localized regions into a control representation through adapter-based dual visual encoding, which jointly captures global visual context and fine-grained sound-relevant evidence. The representation is used to condition a downstream audio generation model. Experiments on both in-domain and out-of-domain benchmarks demonstrate that SCR-V2A achieves strong performance in distribution matching and audio quality while remaining competitive in cross-modal alignment and requiring few trainable parameters. These results highlight the value of prioritizing fine-grained, sound-relevant visual regions for V2A generation.

Figures & tables

Figure 1. SCR regions highlight fine-grained visual evidence associated with sound-producing events. A video sequence of two people speaking is shown as a filmstrip. The corresponding SCR heatmaps highlight the active speaker's mouth in red or purple in each frame, and the aligned audio waveform below uses the same colors to indicate the alternating speakers.

Figure 2. Overview of SCR-V2A. Our method formulates Semantic-Critical Regions (SCRs) as the control for V2A generation. The SCRs are identified through Sound-Related Visual Localization, transformed into a control representation via Adapter-Based Dual Visual Encoding, and injected into the diffusion model for Control-Guided Audio Generation. Pipeline of SCR-V2A from input video to generated audio. A shared video encoder predicts sound-related tags and produces SCR heatmaps through a trainable classifier and heatmap adapter. Original-video and heatmap features are fused, passed through a control adapter into a frozen audio diffusion U-Net, and combined with text features to generate an audio spectrogram. Colored arrows distinguish video, localization, fused-condition, audio-generation, and text paths; flame and lock icons mark trainable and frozen components.

Figure 3. Qualitative Results. Blue boxes highlight representative temporal segments where spectral energy patterns are compared across methods. Yellow marks indicate whether the sound event is present in each generated audio. Qualitative comparison for a video of a musician playing a double bass. Five video frames and their SCR heatmaps emphasize the moving hand, bow, face, and instrument. Below, spectrograms from SCR-V2A, Diff-Foley, Seeing and Hearing, MMAudio, and ground-truth audio are aligned in rows. Cyan boxes mark five temporal segments; yellow check marks indicate a detected musical event and crosses indicate a missing event. SCR-V2A contains the event in all five segments and most closely follows the ground-truth temporal pattern.

Method	#Params	VGGSound						CountixAV
Method	#Params	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
Diff-Foley ( Luo et al., 2023 )	859M	25.60	7.99	3.17	9.85	19.87	0.884	42.11	9.82	3.36	5.36	21.13	0.853
Seeing&Hearing ( Xing et al., 2024 )	415M	27.70	5.40	2.90	7.51	35.40	1.207	35.72	8.98	2.65	3.26	34.95	1.198
V2A-Mapper ( Wang et al., 2024a )	229M	9.69	1.21	2.90	12.36	22.00	1.230	21.43	3.99	3.07	5.46	18.41	1.210
SS2A ( Guo et al., 2026b )	85M	16.21	2.61	2.29	9.47	23.98	1.213	26.49	5.04	2.54	3.11	22.68	1.212
FoleyCrafter ( Zhang et al., 2026 )	1.22B	13.51	2.32	2.55	10.33	27.73	1.220	24.21	3.51	2.79	5.06	27.21	1.272

Table 1. Objective Evaluation. Comparison of SCR-V2A with baseline methods on the VGGSound (in-domain) and CountixAV (out-of-domain) test sets. Note that Seeing&Hearing ( Xing et al., 2024 ) directly optimizes the IB score at test time and therefore achieves the highest IB score. Best results are shown in bold , and second-best results are underlined .

Method	OAQ $\uparrow$	Sem.Align. $\uparrow$	Temporal.Align. $\uparrow$
See & Hear	3.17 $\pm$ 1.21	3.63 $\pm$ 1.27	2.83 $\pm$ 1.21
V2A-Mapper	3.23 $\pm$ 1.13	3.13 $\pm$ 1.28	2.73 $\pm$ 1.23
MMAudio	3.80 $\pm$ 0.85	3.87 $\pm$ 0.90	3.87 $\pm$ 0.94
SCR-V2A (Ours)	3.93 $\pm$ 0.94	4.10 $\pm$ 0.88	3.50 $\pm$ 0.97

Table 2. Subjective Evaluation. The best and the second best results are shown in bold and by underlined .

Figure 4. Temporal Sound Event Composition. Two examples of temporally composed sound events. In the first sequence, SCR heatmaps shift between a person's face and a gun while the generated spectrogram is divided into talking, gun-shooting, and talking intervals. In the second sequence, heatmaps shift among a barking dog, a gun, and a basketball player, with the aligned spectrogram divided into dog-barking, gun-shooting, and basketball-playing intervals.

Visual Prior	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
Object BBox ∗	14.36	3.48	2.97	11.80	23.45	1.168
Object Mask ∗	14.99	3.29	2.92	11.93	23.38	1.157
Optical Flow †	19.52	4.01	3.11	10.64	21.88	0.803
Attention Rollout	9.74	2.35	2.22	12.98	26.51	0.924
SCR	8.13	1.07	1.67	13.81	29.95	0.680

Table 3. Comparison of Different Visual Priors. ∗ : extracted using YOLOv8 ( Jocher et al., 2023 ) ; † : extracted using RAFT ( Teed and Deng, 2020 ) . Best results are shown in bold , and second-best results are underlined .

Condition	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
None	33.20	9.31	4.82	7.14	10.62	1.842
Tag only	25.85	5.31	4.08	10.31	21.06	1.536
SCR only	23.47	5.89	3.71	8.62	19.04	1.183
RGB only	22.74	5.76	3.49	9.27	20.82	1.284
Tag + SCR	17.59	3.62	3.28	10.05	24.02	0.997
Tag + RGB	13.79	1.62	2.97	13.76	27.18	1.174

Table 4. Contribution of Conditioning Components. Best and second-best results are shown in bold and underlined .

Figure 5. Sound Source Awareness. Comparison of rough visual guidance and SCR guidance for a video of a man playing an acoustic guitar. Rough guidance segments the entire person, whereas the SCR heatmap concentrates on the guitar and the player's hands. A reference image marks the guitar body and strumming hand as the actual sound-producing region.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$\lambda$	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
0% (SCR only)	9.75	1.88	2.33	13.01	27.81	0.808
30%	8.82	1.57	2.02	13.68	29.12	0.699
50%	8.13	1.07	1.67	13.81	29.95	0.680
70%	10.70	2.11	2.51	12.11	25.10	0.991
100% (RGB only)	14.43	3.37	3.10	11.90	23.30	1.170

Appendix

Table 5. Effect of Mixing Ratio in Visual Prior Conditioning.

Figure 6. Performance Trends across Different Mixing Ratios. Three line charts plot FD, ImageBind score, and DeSync against SCR mixing ratios of 0, 30, 50, 70, and 100 percent. Performance is best at a 50 percent mixing ratio: FD and DeSync reach their minima while the ImageBind score reaches its maximum. All three metrics worsen when the ratio increases from 50 to 100 percent.

Panel of Figure 7 — Figure 7. Screenshots of Subjective Evaluation Pages. Four screenshots of the SCR-V2A human-evaluation website. The first shows instructions, consent, three evaluation criteria, and a one-to-five rating scale. The other screenshots show sample pages containing four anonymized video-and-audio variants, with separate one-to-five controls for overall quality, semantic alignment, and temporal alignment. Example videos depict food preparation, tennis, and guitar playing.

Model	Distribution Matching			Audio Quality	Cross-modal Alignment
Model	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
SCR-V2A	8.13	1.07	1.67	13.81	29.95	0.680
w. uniformly random heatmaps	18.34	4.02	3.11	10.63	23.07	1.180
w. shuffled heatmaps	22.79	3.74	3.18	11.18	21.36	1.172
w. inverted heatmaps	30.15	4.29	3.33	10.96	17.54	1.285

Appendix

Table 6. Effect of Negative Heatmap Control.

Audio Generation

Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.

See all 133 papers

Aug 5, 2026

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.

Zehua Chen, Junyou Wang, Yuxuan Jiang +5

Tsinghua University, Beijing, China · Hefei University of Technology, Hefei, China · Monash University, Australia +1

Audio-Video Generation Audio Diffusion Models

Ask PDF

Jun 26, 2025

Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

We propose a step-by-step video-to-audio (V2A) generation method that provides finer control over the generation process and more realistic audio synthesis. Inspired by traditional Foley workflows, our approach enables incremental generation of complementary sounds, allowing users to author multiple sound events induced by a video. To avoid the need for costly multi-reference video-audio datasets, each generation step is formulated as a negatively guided V2A process that discourages duplication of sounds already present in previously generated tracks. The guidance model is trained by finetuning a pre-trained V2A model on audio pairs from non-overlapping segments of the same video, encouraging it to leverage acoustic context while remaining visually grounded, and enabling training with standard single-reference audiovisual datasets. Objective and subjective evaluations demonstrate that our method enhances the separability of generated sounds at each step and improves the overall quality of the final composite audio, outperforming existing baselines. Our project page is available at: https://ahykw.github.io/sbsv2a/.

Akio Hayakawa, Masato Ishii, Takashi Shibuya +1

Sony AI, Tokyo, Japan · Sony Group Corporation, Tokyo, Japan

Audio-Video Generation Sound Effects Generation Audio Generation

Ask PDF

Jul 7, 2026

Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space

Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.

Thanh V. T. Tran, Ngoc-Son Nguyen, Luong Tran +4

FPT Software AI Center, Vietnam · NVIDIA Corporation, USA

Audio-Video Generation Audio-Visual Synchronization

Ask PDF

Abstract

Audio Generation

Explore similar work

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space