FoleyForge: Free-form, Controllable, Rectified-flow Editing of Sound Effects
Organizations: Adobe · New York University
Abstract
We present FoleyForge, a generative model for instruction-based editing of sound effects. Given an input clip and a free-form instruction describing the desired change, it applies the edit while preserving the input's identity. A single model handles semantic, stylistic, and DSP-like changes. We introduce a data pipeline that guides an audio language model to author both DSP-based acoustic edits and similarity-preserving semantic edits, thus addressing the lack of publicly available audio editing data. Such data is then used to fine-tune a rectified-flow audio generation model, which we extend to expose attribute controls for loudness change, edit strength, and an input-derived contour envelope, each tunable via classifier-free guidance, yielding a more expressive editing model. FoleyForge achieves state-of-the-art instruction adherence at competitive audio quality and source faithfulness.
Figures & tables
| Subtle Set (ground truth, ) | Creative Set (no ground-truth, ) | |||||||||||||
| Editing Strength | Audio Quality | Editing Strength | Audio Quality | |||||||||||
| Model | alig | faith | dir | Qwen | Gemini | FAD | PQ | alig | faith | dir | Qwen | Gemini | FAD | PQ |
| Ground truth | 0.388 | 0.747 | 0.133 | 3.28 | 4.72 | 0.340 | 6.77 | – | – | – | – | – | – | – |
| SAO-Instruct [ 24 ] | 0.361 | 0.720 | 0.066 | 2.94 | 4.58 | 0.382 | 6.71 | 0.389 | 0.831 | 0.093 | 2.69 | 3.65 | 0.406 | 6.76 |
| SDEdit [ 22 ] | 0.293 | 0.298 | 0.127 | 3.28 | 3.99 | 0.432 | 6.58 | 0.339 | 0.443 | 0.094 | 3.34 | 4.31 | 0.423 | 6.66 |
| EchoEdit+ [ 6 ] | 0.365 | 0.902 | 0.029 | 2.52 | 4.10 | 0.349 | 6.85 | 0.371 | 0.895 | 0.031 | 2.31 | 3.45 | 0.380 | 6.79 |
Audio Editing
Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.
See all 62 papers