FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts

You Li; Dewei Zhou; Fan Ma; Fu Li; Dongliang He; Yi Yang

cs.SDMar 20, 2026

FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts

Authors: You Li, Dewei Zhou, Fan Ma, Fu Li, Dongliang He, Yi Yang

Organizations: The State Key Lab of Brain-Machine Intelligence, Zhejiang University · ReLER, CCAl, Zhejiang University, China · Intelligent Creation, ByteDance, China

Abstract

Recent Video-to-Audio (V2A) methods have achieved remarkable progress, enabling the synthesis of realistic, high-quality audio. However, they struggle with fine-grained temporal control in multi-event scenarios or when visual cues are insufficient, such as small regions, off-screen sounds, or occluded or partially visible objects. In this paper, we propose FoleyDirector, a framework that, for the first time, enables precise temporal guidance in DiT-based V2A generation while preserving the base model's audio quality and allowing seamless switching between V2A generation and temporally controlled synthesis. FoleyDirector introduces Structured Temporal Scripts (STS), a set of captions corresponding to short temporal segments, to provide richer temporal information. These features are integrated via the Script-Guided Temporal Fusion Module, which employs Temporal Script Attention to fuse STS features coherently. To handle complex multi-event scenarios, we further propose Bi-Frame Sound Synthesis, enabling parallel in-frame and out-of-frame audio generation and improving controllability. To support training and evaluation, we construct the DirectorSound dataset and introduce VGGSoundDirector and DirectorBench. Experiments demonstrate that FoleyDirector substantially enhances temporal controllability while maintaining high audio fidelity, empowering users to act as Foley directors and advancing V2A toward more expressive and controllable generation.

Figures & tables

Figure 2: Overview of our method. (a) Extraction pipeline of segment-level STS features. (b) Structure of the SG-TFM module, where Temporal Script Attention introduces control signals. (c) Bi-Frame Sound Synthesis Framework, which leverages the controllability of our method in T2A and V2A to enable parallel rendering of in-frame and out-of-frame sounds. Fused block represents the single-modal transformer block in MMAudio.

\Block 2-1 Model	Counterfactual (C)			Temporal (T)			Overall (O)			Quality (Q)
	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	FD ${}_{\text{VGG}}$ $\downarrow$
MMAudio	0.1775	0.2908	0.1825	0.2787	0.4591	0.2972	0.2281	0.3749	0.2378	8.55
ThinkSound	0.1141	0.2062	0.1208	0.2197	0.3042	0.2206	0.1669	0.2552	0.1707	7.17
Video-Foley	0.2009	0.3684	0.1976	0.2265	0.3674	0.2350	0.2137	0.3679	0.2163	8.03
Hunyuan-Foley	0.2241	0.3808	0.2331	0.2309	0.4265	0.2572	0.2275	0.4037	0.2451	7.51
Ours	0.5226	0.6101	0.5284	0.4129	0.5823	0.4354	0.4677	0.5962	0.4819	6.19

Table 1: Evaluation on DirectorBench. C denotes Counterfactual control, T denotes Temporal control, O denotes Overall performance, and FD

{}_{\text{VGG}}

(lower is better) measures visual quality.

\Block 2-1 Model	Distribution matching					Audio quality	Semantic align	Temporal align
	FD ${}_{\text{VGG}}$ $\downarrow$	FD ${}_{\text{PANN}}$ $\downarrow$	FD ${}_{\text{PaSST}}$ $\downarrow$	KL ${}_{\text{PANN}}$ $\downarrow$	KL ${}_{\text{PaSST}}$ $\downarrow$	ISC ${}_{\text{PANN}}$ $\uparrow$	IB $\uparrow$	DeSync $\downarrow$
GT	0.00	0.00	0.00	0.00	0.00	12.73	0.33	0.625
MMAudio	1.45	8.23	92.80	1.67	1.47	14.38	0.32	0.439
Hunyuan-Foley	2.39	14.10	114.09	2.03	1.81	12.74	0.31	0.543
Thinksound	2.75	17.32	174.80	2.11	1.92	8.13	0.22	0.560
Ours(w/o STS)	1.27	8.58	94.15	1.65	1.43	13.81	0.32	0.438

Table 2: Comparison of V2A models on distribution matching, audio quality, semantic alignment, and temporal alignment metrics. Lower values (

\downarrow

) indicate better performance for FD, KL, and DeSync; higher values (

\uparrow

) indicate better performance for ISC and IB-score. We also report inference w/o STS tokens, demonstrates that our method can flexibly switch between V2A and temporally controlled modes.

\Block 2-1 ID	\Block 2-1 Model	Overall (O)
		Precision $\uparrow$	Recall $\uparrow$	F1 $\uparrow$
➀	Base	0.1448	0.1963	0.1311
➁	+ STS	0.4102	0.5432	0.4252
➂	+ Interleaved Rope	0.4209	0.5582	0.4389
➃	+ Bi-Frame	0.4677	0.5962	0.4819
➄	$\text{w/o Bi-Frame}_{\text{(sub)}}$	0.3928	0.5373	0.4178

Table 3: Ablation on DirectorBench. O denotes Overall performance. Green rows report testing on the full benchmark, blue rows report test on the reannotated subset.

Figure 3: Qualitative comparison. We visualize the spectrograms of generated audio (by prior works and our method) and the ground-truth. The curves in the figure represent the recognition confidence for the corresponding sound categories, while the rectangular regions along the curves indicate the time intervals where our fine-grained scripts provide control.

Table 4: User Study and STS duration experiments

Figure 4: Ablation on training iterations. We show the model metrics after training for different numbers of iterations. The metrics are normalized to the range of 0.3–0.8, with higher values indicating better performance.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: System Prompt. The system prompt we used in annotation pipeline

\Block 2-1 Model		Distribution matching					Quality	Semantic	Temporal
	Params	FD ${}_{\text{VGG}}$ $\downarrow$	FD ${}_{\text{PANN}}$ $\downarrow$	FD ${}_{\text{PaSST}}$ $\downarrow$	KL ${}_{\text{PANN}}$ $\downarrow$	KL ${}_{\text{PaSST}}$ $\downarrow$	ISC ${}_{\text{PANN}}$ $\uparrow$	IB $\uparrow$	DeSync $\downarrow$
Medium	621M	1.45	8.23	92.80	1.67	1.47	14.38	0.32	0.439
Large	1.03B	1.47	8.22	93.77	1.66	1.46	14.41	0.32	0.456

Appendix

Table 5: Comparison of MMAudio models of different sizes on VGGSound-Director.All the model are 44KHZ.

Figure 6: Visual Results in DirectorBench. We present several results from DirectorBench.

Figure 7: Different Architecture. We present the architectural diagrams of different design variants of SG-TFM. (a) concatenates the STS token with features from other modalities; (b) first fuses the STS token with video features using a Fuse-Attention layer; (c) combines the previous two approaches; and (d) shows the design we currently adopt.

Figure 8: Visual Results in VGGSound-Director. We present several results from VGGSound-Director, comparing the mel-spectrograms generated by our method with those from other approaches and with the ground-truth audio. We also compute the L1 similarity between each generated mel-spectrogram and the ground truth.

Drop	FD ${}_{\text{VGG}}$ $\downarrow$	KL ${}_{\text{PANN}}$ $\downarrow$	ISC ${}_{\text{PANN}}$ $\uparrow$	IB $\uparrow$	DeSync $\downarrow$
0.0	1.51	1.31	14.63	0.33	0.467
0.1	1.17	1.42	14.84	0.33	0.432
0.3	1.61	1.37	14.63	0.32	0.456
0.5	1.67	1.42	14.81	0.32	0.461

Appendix

Table 6: Comparison of MMAudio models of different sizes on VGGSound-Director. All models are 44kHz.

Figure 9: Visual Results in DirectorBench. We present several results from DirectorBench.

Figure 10: Visual Results in VGGSound-Director. We present several results from VGGSound-Director, comparing the mel-spectrograms generated by our method with those from other approaches and with the ground-truth audio. We also compute the L1 similarity between each generated mel-spectrogram and the ground truth.

Figure 11: Visual Results in VGGSound-Director. We present several results from VGGSound-Director, comparing the mel-spectrograms generated by our method with those from other approaches and with the ground-truth audio. We also compute the L1 similarity between each generated mel-spectrogram and the ground truth.

Arch	FD ${}_{\text{VGG}}$ $\downarrow$	KL ${}_{\text{PANN}}$ $\downarrow$	ISC ${}_{\text{PANN}}$ $\uparrow$	IB $\uparrow$	DeSync $\downarrow$
Concat	1.29	1.26	14.62	0.33	0.476
Attn	1.74	1.54	15.49	0.32	0.535
C+A	1.43	1.30	14.59	0.32	0.476
Ours	1.17	1.42	14.84	0.33	0.432

Appendix

Table 7: Comparison of MMAudio models of different sizes on VGGSound-Director. All models are 44kHz.

Audio Generation

Audio generation creates speech, music, and sound from text, media, symbolic controls, or reference audio, with growing focus on controllability and realism.

See all 133 papers

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision
Jun 12, 2026Shiyao Wang, Xijuan Zeng, Hui Wang +4Audio-Video Generation Audio-Visual Synchronization
Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China · Kling Team, Kuaishou Technology, Beijing, China
AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer
Mar 16, 2026Pengjun Fang, Yingqing He, Yazhou Xing +3Audio-Video Generation Audio Generation
The Hong Kong University of Science and Technology · University of Central Florida
ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
Apr 16, 2026Jianxuan Yang, Xinyue Guo, Zhi Cheng +10Audio-Video Generation Multimodal Generation
MiLM Plus, Xiaomi Inc. · Wuhan University

Jun 12, 2026J/K move · Enter open · S save

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

Shiyao Wang, Xijuan Zeng, Hui Wang, Shiwan Zhao+3

Academy for Advanced Interdisciplinary Studies, Nankai University, Tianjin, China · Kling Team, Kuaishou Technology, Beijing, China

Audio-Video Generation Audio-Visual Synchronization Sound Effects Generation Audio-Visual Alignment Audio Generation

Open paper AskPDF arXiv

We present FoleyGenEx, a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics, enabling synchronized, versatile audio synthesis for diverse tasks. Existing VTA methods either have multi-modal control but weak temporal alignment or strong alignment but lack reference audio conditioning and semantic precision. FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy preserving training synchronization, and an adverb-based data augmentation algorithm leveraging signal processing and large language models to enhance textual supervision with nuanced semantics. Experiments on AudioCaps, VGGSound, and Greatest Hits demonstrate its competitive controllable VTA performance against existing methods. Demo samples are available at https://foleygenex.github.io/FoleyGenEx.

Abstract

Audio Generation

Explore similar work

FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision

AC-Foley: Reference-Audio-Guided Video-to-Audio Synthesis with Acoustic Transfer

ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling