Semantic-Critical Regions for Video-to-Audio Generation

Yun Fang; Wei Guo; Zhuonan Liang; Heng Wang; Weidong Cai

cs.CVOct 16, 2026

Semantic-Critical Regions for Video-to-Audio Generation

Authors: Yun Fang, Wei Guo, Zhuonan Liang, Heng Wang, Weidong Cai

Organizations: The University of Sydney, Sydney, Australia

Abstract

In real-world videos, sound events are often associated with localized and fine-grained visual cues, such as object motion, object-part dynamics, contact events, and interactions between objects. Existing video-to-audio (V2A) methods employ implicit conditioning based on global visual representations or explicit control derived from temporal and spatial cues. However, subtle sound-producing and sound-indicative interactions remain insufficiently captured, leaving room for more precise visual conditioning of sound-event semantics and temporal dynamics. Motivated by this observation, we refer to regions containing such sound-relevant visual evidence as Semantic-Critical Regions (SCRs) and introduce SCR-V2A, a V2A generation framework that explicitly localizes and encodes SCRs as fine-grained visual guidance for audio generation. Specifically, we first identify SCRs through sound-related visual localization. We then transform the localized regions into a control representation through adapter-based dual visual encoding, which jointly captures global visual context and fine-grained sound-relevant evidence. The representation is used to condition a downstream audio generation model. Experiments on both in-domain and out-of-domain benchmarks demonstrate that SCR-V2A achieves strong performance in distribution matching and audio quality while remaining competitive in cross-modal alignment and requiring few trainable parameters. These results highlight the value of prioritizing fine-grained, sound-relevant visual regions for V2A generation.

Figures & tables

Figure 1. SCR regions highlight fine-grained visual evidence associated with sound-producing events. A video sequence of two people speaking is shown as a filmstrip. The corresponding SCR heatmaps highlight the active speaker's mouth in red or purple in each frame, and the aligned audio waveform below uses the same colors to indicate the alternating speakers.

Figure 2. Overview of SCR-V2A. Our method formulates Semantic-Critical Regions (SCRs) as the control for V2A generation. The SCRs are identified through Sound-Related Visual Localization, transformed into a control representation via Adapter-Based Dual Visual Encoding, and injected into the diffusion model for Control-Guided Audio Generation. Pipeline of SCR-V2A from input video to generated audio. A shared video encoder predicts sound-related tags and produces SCR heatmaps through a trainable classifier and heatmap adapter. Original-video and heatmap features are fused, passed through a control adapter into a frozen audio diffusion U-Net, and combined with text features to generate an audio spectrogram. Colored arrows distinguish video, localization, fused-condition, audio-generation, and text paths; flame and lock icons mark trainable and frozen components.

Figure 3. Qualitative Results. Blue boxes highlight representative temporal segments where spectral energy patterns are compared across methods. Yellow marks indicate whether the sound event is present in each generated audio. Qualitative comparison for a video of a musician playing a double bass. Five video frames and their SCR heatmaps emphasize the moving hand, bow, face, and instrument. Below, spectrograms from SCR-V2A, Diff-Foley, Seeing and Hearing, MMAudio, and ground-truth audio are aligned in rows. Cyan boxes mark five temporal segments; yellow check marks indicate a detected musical event and crosses indicate a missing event. SCR-V2A contains the event in all five segments and most closely follows the ground-truth temporal pattern.

Method	#Params	VGGSound						CountixAV
Method	#Params	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
Diff-Foley ( Luo et al., 2023 )	859M	25.60	7.99	3.17	9.85	19.87	0.884	42.11	9.82	3.36	5.36	21.13	0.853
Seeing&Hearing ( Xing et al., 2024 )	415M	27.70	5.40	2.90	7.51	35.40	1.207	35.72	8.98	2.65	3.26	34.95	1.198
V2A-Mapper ( Wang et al., 2024a )	229M	9.69	1.21	2.90	12.36	22.00	1.230	21.43	3.99	3.07	5.46	18.41	1.210
SS2A ( Guo et al., 2026b )	85M	16.21	2.61	2.29	9.47	23.98	1.213	26.49	5.04	2.54	3.11	22.68	1.212
FoleyCrafter ( Zhang et al., 2026 )	1.22B	13.51	2.32	2.55	10.33	27.73	1.220	24.21	3.51	2.79	5.06	27.21	1.272

Table 1. Objective Evaluation. Comparison of SCR-V2A with baseline methods on the VGGSound (in-domain) and CountixAV (out-of-domain) test sets. Note that Seeing&Hearing ( Xing et al., 2024 ) directly optimizes the IB score at test time and therefore achieves the highest IB score. Best results are shown in bold , and second-best results are underlined .

Method	OAQ $\uparrow$	Sem.Align. $\uparrow$	Temporal.Align. $\uparrow$
See & Hear	3.17 $\pm$ 1.21	3.63 $\pm$ 1.27	2.83 $\pm$ 1.21
V2A-Mapper	3.23 $\pm$ 1.13	3.13 $\pm$ 1.28	2.73 $\pm$ 1.23
MMAudio	3.80 $\pm$ 0.85	3.87 $\pm$ 0.90	3.87 $\pm$ 0.94
SCR-V2A (Ours)	3.93 $\pm$ 0.94	4.10 $\pm$ 0.88	3.50 $\pm$ 0.97

Table 2. Subjective Evaluation. The best and the second best results are shown in bold and by underlined .

Figure 4. Temporal Sound Event Composition. Two examples of temporally composed sound events. In the first sequence, SCR heatmaps shift between a person's face and a gun while the generated spectrogram is divided into talking, gun-shooting, and talking intervals. In the second sequence, heatmaps shift among a barking dog, a gun, and a basketball player, with the aligned spectrogram divided into dog-barking, gun-shooting, and basketball-playing intervals.

Visual Prior	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
Object BBox ∗	14.36	3.48	2.97	11.80	23.45	1.168
Object Mask ∗	14.99	3.29	2.92	11.93	23.38	1.157
Optical Flow †	19.52	4.01	3.11	10.64	21.88	0.803
Attention Rollout	9.74	2.35	2.22	12.98	26.51	0.924
SCR	8.13	1.07	1.67	13.81	29.95	0.680

Table 3. Comparison of Different Visual Priors. ∗ : extracted using YOLOv8 ( Jocher et al., 2023 ) ; † : extracted using RAFT ( Teed and Deng, 2020 ) . Best results are shown in bold , and second-best results are underlined .

Condition	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
None	33.20	9.31	4.82	7.14	10.62	1.842
Tag only	25.85	5.31	4.08	10.31	21.06	1.536
SCR only	23.47	5.89	3.71	8.62	19.04	1.183
RGB only	22.74	5.76	3.49	9.27	20.82	1.284
Tag + SCR	17.59	3.62	3.28	10.05	24.02	0.997
Tag + RGB	13.79	1.62	2.97	13.76	27.18	1.174

Table 4. Contribution of Conditioning Components. Best and second-best results are shown in bold and underlined .

Figure 5. Sound Source Awareness. Comparison of rough visual guidance and SCR guidance for a video of a man playing an acoustic guitar. Rough guidance segments the entire person, whereas the SCR heatmap concentrates on the guitar and the player's hands. A reference image marks the guitar body and strumming hand as the actual sound-producing region.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

$\lambda$	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
0% (SCR only)	9.75	1.88	2.33	13.01	27.81	0.808
30%	8.82	1.57	2.02	13.68	29.12	0.699
50%	8.13	1.07	1.67	13.81	29.95	0.680
70%	10.70	2.11	2.51	12.11	25.10	0.991
100% (RGB only)	14.43	3.37	3.10	11.90	23.30	1.170

Appendix

Table 5. Effect of Mixing Ratio in Visual Prior Conditioning.

Figure 6. Performance Trends across Different Mixing Ratios. Three line charts plot FD, ImageBind score, and DeSync against SCR mixing ratios of 0, 30, 50, 70, and 100 percent. Performance is best at a 50 percent mixing ratio: FD and DeSync reach their minima while the ImageBind score reaches its maximum. All three metrics worsen when the ratio increases from 50 to 100 percent.

Panel of Figure 7 — Figure 7. Screenshots of Subjective Evaluation Pages. Four screenshots of the SCR-V2A human-evaluation website. The first shows instructions, consent, three evaluation criteria, and a one-to-five rating scale. The other screenshots show sample pages containing four anonymized video-and-audio variants, with separate one-to-five controls for overall quality, semantic alignment, and temporal alignment. Example videos depict food preparation, tennis, and guitar playing.

Model	Distribution Matching			Audio Quality	Cross-modal Alignment
Model	FD $\downarrow$	FAD $\downarrow$	KLD $\downarrow$	IS $\uparrow$	IB (%) $\uparrow$	DeSync $\downarrow$
SCR-V2A	8.13	1.07	1.67	13.81	29.95	0.680
w. uniformly random heatmaps	18.34	4.02	3.11	10.63	23.07	1.180
w. shuffled heatmaps	22.79	3.74	3.18	11.18	21.36	1.172
w. inverted heatmaps	30.15	4.29	3.33	10.96	17.54	1.285

Appendix

Table 6. Effect of Negative Heatmap Control.

Audio Generation

Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.

See all 133 papers

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation
Aug 5, 2026Zehua Chen, Junyou Wang, Yuxuan Jiang +5Audio-Video Generation Audio Diffusion Models
Tsinghua University, Beijing, China · Hefei University of Technology, Hefei, China · Monash University, Australia +1
Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance
Jun 26, 2025Akio Hayakawa, Masato Ishii, Takashi Shibuya +1Audio-Video Generation Sound Effects Generation
Sony AI, Tokyo, Japan · Sony Group Corporation, Tokyo, Japan
Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space
Jul 7, 2026Thanh V. T. Tran, Ngoc-Son Nguyen, Luong Tran +4Audio-Video Generation Audio-Visual Synchronization
FPT Software AI Center, Vietnam · NVIDIA Corporation, USA

Aug 5, 2026J/K move · Enter open · S save

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

Zehua Chen, Junyou Wang, Yuxuan Jiang, Zhenying Fang+4

Tsinghua University, Beijing, China · Hefei University of Technology, Hefei, China · Monash University, Australia+1

Audio-Video Generation Audio Diffusion Models

Open paper AskPDF arXiv

Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.

Abstract

Audio Generation

Explore similar work

Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation

Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance

Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space