Woosh: A Sound Effects Foundation Model

Gaëtan Hadjeres; Marc Ferras; Khaled Koutini; Benno Weck; Alexandre Bittar; Thomas Hummel; Zineb Lahrichi; Hakim Missoum

cs.SDApr 2, 2026

Woosh: A Sound Effects Foundation Model

Authors: Gaëtan Hadjeres, Marc Ferras, Khaled Koutini, Benno Weck, Alexandre Bittar, Thomas Hummel, Zineb Lahrichi, Hakim Missoum,

Organizations: Sony AI · Sony Group Corporation

Abstract

The audio research community depends on open generative models as foundational tools for building novel approaches and establishing baselines. In this report, we present Woosh, Sony AI's publicly released sound effect foundation model, detailing its architecture, training process, and an evaluation against other popular open models. Being optimized for sound effects, we provide (1) a high-quality audio encoder/decoder model and (2) a text-audio alignment model for conditioning, together with (3) text-to-audio and (4) video-to-audio generative models. Distilled text-to-audio and video-to-audio models are also included in the release, allowing for low-resource operation and fast inference. Our evaluation on both public and private data shows competitive or better performance for each module when compared to existing open alternatives like StableAudio-Open and TangoFlux. Inference code and model weights are available at https://github.com/SonyResearch/Woosh. Demo samples can be found at https://sonyresearch.github.io/Woosh/.

Figures & tables

Panel of Figure 1 — Figure 1: Inference-time layout of the Woosh-Flow (left) and Woosh-VFlow (right) models for text-to-audio and video-to-audio generation, respectively.

Figure 2: VOCOS decoder architecture as a cascade on ConvNeXt blocks, used in Woosh-AE.

			AudioCaps-Test			InternalSFX-Test
Model	SampleRate	Params	MelDist	STFTDist	SI-SDR	MelDist	STFTDist	SI-SDR
Descript	44.1 kHz	76 M	0.150	1.44	9.69	0.081	0.376	7.29
Encodec	48 kHz	19 M	0.266	2.00	7.79	0.161	0.784	5.22
SAO-VAE	44.1 kHz	156 M	0.217	1.55	$-$ 0.08	0.121	0.462	$-$ 2.92
Woosh-AE-Public	48 kHz	221 M	0.032	1.18	20.79	0.021	0.272	12.70
Woosh-AE-Private	48 kHz	221 M	0.074	1.25	22.52	0.024	0.288	16.03

Table 1: Woosh-AE evaluation on the public AudioCaps-Test and the InternalSFX-Test datasets.

Figure 3: Woosh-CLAP training block diagram for a positive pair of samples. Only the text encoder is used at generation time as conditioning.

Model	Params	AudioCaps-Test				InternalSFX-Test
		T2A		A2T		T2A		A2T
		R@5	R@10	R@5	R@10	R@5	R@10	R@5	R@10
LAION-CLAP	355 M + 31 M	0.469	0.618	0.467	0.650	0.280	0.188	0.215	0.328
Woosh-CLAP-Public	355 M + 86 M	0.428	0.546	0.462	0.595	0.138	0.198	0.281	0.401
Woosh-CLAP-Private	355 M + 86 M	0.095	0.167	0.153	0.227	0.592	0.655	0.694	0.766

Table 2: Woosh-CLAP evaluation on the public AudioCaps-Test and the InternalSFX-Test datasets. Recall at 5 (R@5) and recall at 10 (R@10) are given for text-to-audio (T2A) and audio-to-text (A2T) retrieval.

Figure 4: Multimodal transformer stack in the Woosh-Flow diffusion model, formed by MultiStream (MS) and SingleStream (SS) blocks.

Panel of Figure 5 — Figure 5: MultiStream transformer block diagram (left). Both self-attention and feed-forward network (FFN) outputs are computed independently for each modality sequence. SingleStream transformer block diagram (right). Self-attention is computed on the concatenation of modality sequences along the time dimension. Instead of an FFN block, non-linear features are in a parallel path and added after the attention project layer.

Approach	Params	NFEs	AudioCaps-Test			InternalSFX-Test
			FD $\downarrow$	KL $\downarrow$	CLAP $\uparrow$	FD $\downarrow$	KL $\downarrow$	CLAP $\uparrow$
SAO	1057 M	2 $\times$ 100	150.1	4.068	0.1457	341.0	2.690	0.2544
TangoFlux	515 M	2 $\times$ 100	131.9	1.700	0.3471	378.1	2.729	0.2249
Woosh-Flow-Public	337 M	$\approx$ 2 $\times$ 70	109.1	1.599	0.3719	282.7	2.248	0.3752
Woosh-DFlow-Public	337 M	4	132.1	1.836	0.3229	322.5	2.404	0.3093
Woosh-Flow-Private	337 M	$\approx$ 2 $\times$ 70	213.8	4.238	0.1824	246.9	2.348	0.3010

Table 3: Woosh-Flow and Woosh-DFlow results on AudioCaps-Test and InternalSFX-Test. We provide the number latent diffusion model parameters (Params), the number of neural function evaluations at inference time (NFEs), and the FD, KL, and CLAP metrics described in Section 4.5 . All models are conditioned on text.

Dataset	Approach	Params	TC	FD $\downarrow$	KL $\downarrow$	IB $\uparrow$	DeSync $\downarrow$	CLAP $\uparrow$
FoleyBench	Ground truth	-	$\checkmark$	0.000	0.000	0.300	0.628	0.231
	MMAudio-M	621M	$\checkmark$	30.480	1.983	0.297	0.400	0.292
	Woosh-VFlow	413M	$\checkmark$	24.136	1.774	0.293	0.575	0.325
	Woosh-DVFlow	413M	$\checkmark$	24.422	1.834	0.278	0.647	0.303
	MMAudio-M	621M	$\times$	38.517	2.466	0.268	0.439	0.195
	Woosh-VFlow	413M	$\times$	34.382	2.401	0.269	0.602	0.176

Table 4: Woosh-VFlow and Woosh-DVFlow results on FoleyBench (top) and OGameData-Test (bottom). We provide the number of latent diffusion model parameters (Params), whether the model is text-conditioned or not (TC), the FD, KL, and CLAP metrics described in Section 4.5 , and the task-specific IB and DeSync metrics described in Section 5.5 .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

		$\displaystyle\mathcal{L}_{\textrm{disc}}\leftarrow\textrm{ReLU}\big[1-\texttt{disc}(x_{r}^{\textrm{true}})\big]+\text{ReLU}\big[1+\texttt{disc}(x_{r}^{\textrm{fake}}.\texttt{detach()})\big]$
		$\displaystyle\mathcal{L}_{\textrm{disc}}.\texttt{backward()}$

Appendix

Algorithm 1 Distillation Training with Adversarial Loss

Approach	Params	TC	FD	KL	IB	DeSync	CLAP
Ground truth	-	$\checkmark$	-0.000	0.000	0.331	0.630	0.198
MMAudio-M	621M	$\checkmark$	30.539	1.413	0.329	0.449	0.224
Woosh-VFlow	413M	$\checkmark$	25.711	1.691	0.282	0.633	0.278
Woosh-DVFlow	413M	$\checkmark$	25.469	1.722	0.278	0.673	0.235
MMAudio-M	621M	Recap	31.445	1.540	0.309	0.448	0.166
Woosh-VFlow	413M	Recap	19.442	1.338	0.270	0.620	0.161

Appendix

Table 5: Woosh-VFlow and Woosh-DVFlow results on VGG-Sound.

Audio-Text Alignment

See all 17 papers

Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens
Feb 18, 2026Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds +3Audio-Language Models Audio Generation
Stanford University · SCB 10X · University of Southern California +2
SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving
Mar 9, 2026Ayush Barik, Sofia Stoica, Nikhil Sarda +4Text-to-Audio Generation Diffusion Model Inference Acceleration
University of Illinois Urbana-Champaign · Assured Intelligence
Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Audio-Language Models
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research

Feb 18, 2026J/K move · Enter open · S save

Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds, Guangzhi Sun+2

Stanford University · SCB 10X · University of Southern California+2

Audio-Language Models Audio Generation

Open paper AskPDF arXiv

Current audio language models are predominantly text-first, either extending pre-trained text LLM backbones or relying on semantic-only audio tokens, limiting general audio modeling. This paper presents a systematic empirical study of native audio foundation models that apply next-token prediction to audio at scale, jointly modeling semantic content, acoustic details, and text to support both general audio generation and cross-modal capabilities. We provide comprehensive empirical insights for building such models: (1) We systematically investigate design choices -- data sources, text mixture ratios, and token composition -- establishing a validated training recipe. (2) We conduct the first scaling law study for discrete audio models via IsoFLOP analysis on 64 models spanning

3{\times}10^{18}

3{\times}10^{20}

FLOPs, finding that optimal data grows 1.6

\times

faster than optimal model size. (3) We apply these lessons to train SODA (Scaling Open Discrete Audio), a suite of models from 135M to 4B parameters on 500B tokens, comparing against our scaling predictions and existing models. SODA serves as a flexible backbone for diverse audio/text tasks -- we demonstrate this by fine-tuning for voice-preserving speech-to-speech translation, using the same unified architecture.

Abstract

Audio-Text Alignment

Explore similar work

Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

SoundWeaver: Compositional Warm-Starting for Text-to-Audio Diffusion Serving

Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions