Organizations: Shanghai Film Academy, Shanghai University · Shanghai Engineering Research Center of Motion Picture Special Effects · University of Surrey, UK
Foley art plays a pivotal role in enhancing immersive auditory experiences in film, yet manual creation of spatio-temporally aligned audio remains labor-intensive. We propose FoleyDesigner, a novel framework inspired by professional Foley workflows, integrating film clip analysis, spatio-temporally controllable Foley generation, and professional audio mixing capabilities. FoleyDesigner employs a multi-agent architecture for precise spatio-temporal analysis. It achieves spatio-temporal alignment through latent diffusion models trained on spatio-temporal cues extracted from video frames, combined with large language model (LLM)-driven hybrid mechanisms that emulate post-production practices in film industry. To address the lack of high-quality stereo audio datasets in film, we introduce FilmStereo, the first professional stereo audio dataset containing spatial metadata, precise timestamps, and semantic annotations for eight common Foley categories. For applications, the framework supports interactive user control while maintaining seamless integration with professional pipelines, including 5.1-channel Dolby Atmos systems compliant with ITU-R BS.775 standards, thereby offering extensive creative flexibility. Extensive experiments demonstrate that our method achieves superior spatio-temporal alignment compared to existing baselines, with seamless compatibility with professional film production standards. The project page is available at https://gekiii996.github.io/FoleyDesigner/ .
Figures & tables
Figure 1: FoleyDesigner Overview. The left column detailing the actual steps of a human Foley designer. The right column presents the corresponding simulated functional modules of FoleyDesigner, showcasing outputs at each phase, resulting in a soundtrack suitable for film use.
Figure 2: FoleyDesigner Architecture. Our pipeline for automated Foley generation consists of three stages, (1) Fine-Grained Film Decomposition : analyzes silent video and generates hierarchical Foley scripts; (2) Spatio-Temporal Foley Generation : produces spatially-controlled stereo audio using DiT-based diffusion conditioned on visual cues; (3) Foley Refinement : applies multi-agent processing to refine audio quality and generate 5.1 surround output.
Figure 3: FilmStereo Construction. Our four-step pipeline for creating spatially and temporally annotated audio data. (1) Audio filtering and extension, (2) Spatial simulation with random positioning, (3) Spatial-rich caption generation via chain-of-thought, and (4) Temporal annotation with event detection.
Figure 4: Qualitative Results. Qualitative comparison showing temporal alignment between video events and generated audio across two scenarios. Left: explosion sequence with three distinct events. Right: excavation scene with repetitive shoveling actions. Checkmarks indicate successful synchronization, while crosses mark temporal misalignment.
Method
IS ↑
KL ↓
FAD ↓
CLAP ↑
Stable Audio [ 11 ]
10.50
1.86
2.37
0.594
SpatialSonic [ 32 ]
13.79
1.37
1.93
0.672
Ours
12.36
1.40
1.88
0.679
Table 1: Audio Quality. Metrics include Inception Score, KL Divergence, Fréchet Audio Distance, and CLAP score for audio quality assessment. Best and second-best results are highlighted. ↓ indicates lower is better, ↑ indicates higher is better.
Method
GCC ↓
CRW ↓
FSAD ↓
IoU ↑
Stable Audio [ 11 ]
61.17
51.44
0.343
24.5
See2Sound [ 7 ]
60.03
51.17
0.291
21.3
SpatialSonic [ 32 ]
49.20
36.87
0.163
27.8
Ours
48.79
34.23
0.138
32.2
Table 2: Spatio-Temporal Alignment Results. Metrics include GCC and CRW for spatial accuracy, FSAD for stereo quality, and IoU for temporal alignment. Best and second-best results are highlighted. ↓ indicates lower is better, ↑ indicates higher is better.
Method
IB ↑
SRS ↑
CCS ↑
AV-Sync ↑
Stable Audio [ 11 ]
0.216
5.31
5.8
0.512
See2Sound [ 7 ]
0.105
3.03
3.0
0.601
SpatialSonic [ 32 ]
0.251
5.91
4.5
0.545
Ours
0.402
8.27
6.2
0.726
Table 3: Film Foley Performance. Evaluation on film clips covering audio-visual synchronization (ImageBind Score, AV-Sync) and cinematic quality (Sonic Richness Score, Cinematic Clarity Score). Best and second-best results are highlighted.
Configuration
GCC ↓
CRW ↓
FSAD ↓
FAD ↓
w/o STC
62.02
55.89
0.297
2.14
Full Model
48.79
34.23
0.138
1.88
Table 4: Ablation study results on the FilmStereo dataset. STC refers to spatio-temporal cues including trajectory information and spatial positioning prompts. Best results are highlighted.
Figure 5: Human Evaluation Results. We compared the selection ratio of four methods from five perspectives: (1) Immerse, (2) Emo Align, (3) Tempo Align, (4) Spatial Align, and (5) Timbre.
Figure 1: FilmStereo Dataset Pipeline. The process begins with sourcing data using randomly sampled parameters to define sound event attributes, followed by a simulated sound design scenario in Step 2 to generate film foley annotations. The resulting data undergoes manual verification to ensure quality and accuracy.
Dataset
Duration (hours)
Temporal
Spatial
LAION-Audio
4.3K
✗
✗
AudioCaps
110
✗
✗
VGG-Sound
550
✗
✗
AudioTime
15.3
✓
✗
CompA-order
1.5
✓
✗
Simstereo
116
✗
✓
Table 1: Comparison of existing datasets. FilmStereo provide temporal and spatial annotations simultaneously.
Category
Percentage (%)
Ambience & Environments
4.70
Bio & Organic
12.87
Cultural & Abstract
21.87
Foley & Physical Interactions
19.01
Designed & Abstract
17.95
Dynamic Systems
6.88
Table 2: Percentage distribution. The table shows the proportion of audio clips belonging to each of the eight major sound design categories.
Figure 2: FilmStereo Distribution Analysis. (a) Room size distribution. (b) Motion type distribution. (c) Spatial positioning across azimuth angles and depth zones.
Stage
Time (s)
Component
1. Visual Analysis
2s
VLM
2. Script Decomposition
34s
LLM Agents
3. Audio Generation
8s
DiT Diffusion
4. Foley Refinement
64s
LLM Agents
Total
108s
vs. End-to-End ( ∼ 5s)
Table 3: Inference Time Analysis. Time consumption in seconds (s) for generating a 3s stereo clip on a single A6000 GPU.
Method
FAD ↓
ImgBind ↑
IoU ↑
GCC ↓
CRW ↓
Diff-Foley
1.85
0.383
31.80
-
-
FoleyCrafter
1.69
0.430
32.00
-
-
Tango (Fine-tuned)
2.26
0.328
28.60
54.82
45.36
Ours
1.88
0.402
32.20
48.79
34.23
Table 4: Generation Quality and Spatial Alignment. ↓ indicates lower is better, ↑ indicates higher is better.
Method
ER ↑
LSD ↓
LE ↓
RT60E ↓
Single-Stage
68.5%
34.20
7.63
0.92
Ours
84.2%
18.42
2.86
1.08
Table 5: Ablation Study on Agents Framework. Best results are highlighted. ER: Event Recall (%); LSD: Log-spectral distance (dB); LE: Loudness Error (LU); RT60E: RT60 Error (s).
Figure 3: User study setup . (a) Standard 5.1 surround sound speaker configuration showing Front Left (FL), Center (C), Front Right (FR), Low Frequency Effects (LFE), Left Surround (LS), and Right Surround (RS) positions. (b) Professional mixing studio environment.
Figure 4: Participants conducting the perceptual evaluation in the professional mixing studio environment.
Figure 5: Questionnaire details . This is the survey questionnaire we designed and used in the user study.
Figure 6: Temporal Analysis. Each video frame corresponds to a temporal segment in the spectrogram below. Yellow checkmarks indicate successful audio-visual synchronization. Our method achieves consistent temporal alignment across key events, while baseline methods show varying degrees of synchronization failure regardless of their output channel configuration.
Figure 7: Spatial Analysis. Our method demonstrates proper stereo separation with left channel (L) strengthening and right channel (R) weakening, while SpatialSonic (stereo output) shows limited spatial variation despite having two-channel capability.
Figure 8: Spatial Analysis. Our method demonstrates proper stereo separation with left channel (L) weakening and right channel (R) strengthening, while SpatialSonic (stereo output) shows limited spatial variation despite having two-channel capability.
The State Key Lab of Brain-Machine Intelligence, Zhejiang University · ReLER, CCAl, Zhejiang University, China · Intelligent Creation, ByteDance, China