Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing

Ilpo Viertola; Giulio Cengarle; Gouthaman KV; Daniel Arteaga; Lie Lu

cs.SDSep 24, 2026

Spot, Separate, and Enhance: Fully Generative Approach for Audio Mixing

Authors: Ilpo Viertola, Giulio Cengarle, Gouthaman KV, Daniel Arteaga, Lie Lu

Organizations: Dolby Laboratories · Tampere University

Abstract

We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources, and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments. Project page: https://sse-ai.notion.site

Figures & tables

Figure 1 : Overview of SSE . Given a video with degraded and unbalanced audio, SSE enhances it according to the given textual description. First, all the modalities are encoded with specific encoders, and the visual and audio features are aligned and fused. The fused and textual features are then passed to the DiT . The DiT generates enhanced audio that closely follows the original content and user guidance. Finally, the generated audio is decoded into a waveform representation with the SkipDACVAE decoder.

Model	FD $\downarrow$	KL $\downarrow$	FD $\downarrow$	KL $\downarrow$	IS $\uparrow$	IB $\uparrow$	Sync $\downarrow$	CLAP $\uparrow$
	PANNs		PaSST
Input	3.55	0.35	43.39	0.26	3.16	30.49	51.80	40.56
VisAH* [ 11 ]	2.97	0.31	40.63	0.21	3.04	31.08	51.80	44.06
SSE	1.26	0.14	41.30	0.12	3.13	32.44	48.67	45.06

Table 1 : Proposed metrics calculated for DegradedMix test data. We could not evaluate VisAH-FM [ 22 ] as the model is closed-source. *: retrained with the DegradedMix data for fair comparison.

Variant	Env $\downarrow$	Mag $\downarrow$	Was $\downarrow$	$\text{KL}_{\mathrm{PaSST}}\downarrow$
Input	6.79	23.13	1.95	25.65
VisAH* [ 11 ]	5.16	15.70	1.22	21.08
SSE	3.65	10.87	0.90	11.60

Table 2 : VGAH [ 11 ] metrics calculated for DegradedMix test data. We could not evaluate VisAH-FM [ 22 ] as the model is closed-source. *: retrained with the DegradedMix data for fair comparison.

Model	FD $\downarrow$	KL $\downarrow$	FD $\downarrow$	KL $\downarrow$	IS $\uparrow$	IB $\uparrow$	Sync $\downarrow$	CLAP $\uparrow$
	PANNs		PaSST
Input	2.30	0.29	31.86	0.21	3.11	31.19	51.19	38.30
VisAH [ 11 ]	1.26	0.18	18.06	0.11	3.06	31.81	49.71	39.56
SSE	1.04	0.13	35.78	0.10	3.13	32.74	47.69	46.78

Table 3 : Proposed metrics calculated for MuddyMix [ 11 ] test data. We could not evaluate VisAH-FM [ 22 ] as the model is closed-source and the authors do not provide samples for the MuddyMix test set.

Model	Env $\downarrow$	Mag $\downarrow$	Was $\downarrow$	$\text{KL}_{\mathrm{PaSST}}\downarrow$
Input	6.29	22.69	1.96	20.74
VisAH [ 11 ]	3.38	9.99	0.84	11.37
VisAH-FM [ 22 ]	2.74	8.28	0.63	9.70
SSE	3.46	9.86	0.88	9.61

Table 4 : VGAH [ 11 ] metrics calculated for MuddyMix [ 11 ] test data.

Comparison (A vs. B)	Wins (A–B)	Pref. rate	$p$ -value
SSE -L vs. SSE -S	68–22	75.6%	$<0.001$
SSE -L vs. Original audio	72–18	80.0%	$<0.001$
SSE -L vs. VisAH [ 11 ]	80–10	88.9%	$<0.001$
SSE -S vs. Original audio	47–43	52.2%	$0.752$
SSE -S vs. VisAH [ 11 ]	57–33	63.3%	$0.015$
Original audio vs. VisAH [ 11 ]	62–28	68.9%	$<0.001$

Table 5 : Pairwise preference test results for UGC enhancement. Preference rate is the proportion of trials in which the first condition was preferred over the second, with 95% confidence intervals.

p

-values are from a two-sided binomial test against chance (50%).

Model	FD $\downarrow$	KL $\downarrow$	FD $\downarrow$	KL $\downarrow$	IS $\uparrow$	IB $\uparrow$	Sync $\downarrow$	CLAP $\uparrow$
	PANNs		PaSST
Small	1.12	0.14	37.81	0.11	3.13	32.76	47.82	47.02
Base	1.06	0.13	37.28	0.10	3.12	32.64	48.84	47.10
Large	1.04	0.13	35.78	0.10	3.13	32.74	47.69	46.78

Table 6 : Ablation on different model variants. We use MuddyMix [ 11 ] data. Preferred configuration is highlighted in yellow.

Variant	FD $\downarrow$	KL $\downarrow$	FD $\downarrow$	KL $\downarrow$	IS $\uparrow$	IB $\uparrow$	Sync $\downarrow$	CLAP $\uparrow$
	PANNs		PaSST
DACVAE	3.38	0.21	143.55	0.28	3.03	32.50	48.35	48.13
SkipDACVAE	1.04	0.13	35.78	0.10	3.13	32.74	47.69	46.78

Table 7 : Effect of using SkipDACVAE. We use MuddyMix [ 11 ] data. Preferred configuration is highlighted in yellow.

Type	FD $\downarrow$	KL $\downarrow$	FD $\downarrow$	KL $\downarrow$	IS $\uparrow$	IB $\uparrow$	Sync $\downarrow$	CLAP $\uparrow$
	PANNs		PaSST
Original	1.24	0.16	39.15	0.12	3.13	32.58	49.97	44.82
Summarized	1.22	0.16	38.47	0.12	3.12	32.66	49.11	44.78
Audio-based	1.12	0.14	37.81	0.11	3.13	32.76	47.82	47.02

Table 8 : Ablation on different caption types. We use small-variant and MuddyMix [ 11 ] . Preferred configuration is highlighted in yellow.

Explore similar work

Soundwich: Video Generation with Layered and Controllable Audio
Sep 30, 2026Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek +2Audio-Video Generation Video Generation
Simon Fraser University · University of Cyprus · CYENS Centre of Excellence +1
Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects
Aug 5, 2026Yisu Zong, Jinjie Shi, Joshua ReissAudio Editing Mix
Centre for Digital Music, Queen Mary University of London, UK
Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
May 27, 2026Jiahao Mei, Heinrich Dinkel, Yadong Niu +7Modern Generative Audio Models Acoustic Latent Space
1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

Soundwich: Video Generation with Layered and Controllable Audio

Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek, Daniel Cohen-Or+1

Simon Fraser University · University of Cyprus · CYENS Centre of Excellence+1

Audio-Video Generation Video Generation Audio Understanding

Open paper AskPDF arXiv

Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.

Abstract

Explore similar work

Soundwich: Video Generation with Layered and Controllable Audio

Diff2Mix: Controllable Music Mixing via Diffusion Models and Differentiable Audio Effects

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text