Cinematic Audio Source Separation Using Visual Cues

Kang Zhang; Suyeon Lee; Arda Senocak; Joon Son Chung

cs.MMMar 27, 2026

Cinematic Audio Source Separation Using Visual Cues

Authors: Kang Zhang, Suyeon Lee, Arda Senocak, Joon Son Chung

Organizations: School of Electrical Engineering, KAIST · Graduate School of Artificial Intelligence, UNIST

Abstract

Cinematic Audio Source Separation (CASS) aims to decompose mixed film audio into speech, music, and sound effects, enabling applications like dubbing and remastering. Existing CASS approaches are audio-only, overlooking the inherent audio-visual nature of films, where sounds often align with visual cues. We present the first framework for audio-visual CASS (AV-CASS), leveraging visual context to enhance separation quality. Our method formulates CASS as a conditional generative modeling problem using conditional flow matching, enabling multimodal audio source separation. To address the lack of cinematic datasets with isolated sound tracks, we introduce a training data synthesis pipeline that pairs in-the-wild audio and video streams (e.g., facial videos for speech, scene videos for effects) and design a dedicated visual encoder for this dual-stream setup. Trained entirely on synthetic data, our model generalizes effectively to real-world cinematic content and achieves strong performance on synthetic, real-world, and audio-only CASS benchmarks. Code and demo are available at https://cass-flowmatching.github.io.

Figures & tables

Figure 1: Illustration of the Cinematic Audio Source Separation (CASS) task. The audio stream from a movie is separated into distinct tracks: speech, sound effects, and music.

Figure 2: Architecture of AV-CASS. The fusion module integrates visual features from the facial and scene encoders into $\bm{c}^{V}$ , which serves as a conditioning input along with a mixture audio $\bm{s}^{A}$ for the vector field estimator $\bm{u}_{\theta}$ .

Figure 3: Extraction of dual-stream visual inputs from a real-world cinematic video during inference. Since no architectural changes are required, the AV-CASS model can be used with real-world cinematic videos for inference.

Figure 4: Comparison of MRX, BandIt, and AV-CASS on a real-world movie sample. Input video frames $\bm{v}^{f}$ and $\bm{v}^{s}$ are shown at the top, with the input audio spectrogram $\bm{s}^{A}$ placed for each stem. Yellow boxes highlight the bicycle bell, red boxes indicate cheering, and dotted boxes show elements misplaced in non-target stems. The dotted pink box in BandIt’s FX shows speech artifacts. Best viewed when zoomed in. This sample can also be viewed in the supplementary video.

Method	MRX [ 48 ]	BandIt [ 59 ]	AV-CASS (Ours)
MOS ( $\uparrow$ )	2.55 $\pm$ 0.10	3.78 $\pm$ 0.10	4.13 $\pm$ 0.09

Table 1: MOS results of CASS models on real-world samples. The scores are computed based on 95% confidence intervals.

Method	DX ( $\downarrow$ )	FX ( $\downarrow$ )	MX ( $\downarrow$ )
MRX [ 48 ]	2.63	33.72	6.01
BandIt [ 59 ]	0.60	22.41	0.35
DAVIS-Flow [ 29 ]	5.88	14.58	35.94
AV-CASS (Ours)	0.46	19.81	0.32

Table 2: Wrong Placement Ratio (WPR [%]) on real-world samples. Lower is better.

Predictive Model
Method	A-V	FAD ( $\downarrow$ )	KL ( $\downarrow$ )	SI-SDRi ( $\uparrow$ )	PESQ ( $\uparrow$ )	WPR ( $\downarrow$ )
Hybrid Demucs [ 14 ]	✗	2.05	1.03	13.57	2.16	5.24
HT Demucs [ 52 ]	✗	2.08	1.06	13.41	2.06	9.23
MRX [ 48 ]	✗	3.47	1.67	10.60	1.89	14.91
BandIt [ 59 ]	✗	2.15	1.14	14.40	2.15	4.65
Generative Model

Table 3: Results on AVDnR dataset (objective scores). All models are trained on our training data. A-V indicates audio-visual.

Method	H-Demucs	HT Demucs	MRX	BandIt	MSDM	Ours
MOS ( $\uparrow$ )	3.14 $\pm$ 0.15	3.01 $\pm$ 0.14	1.90 $\pm$ 0.13	3.12 $\pm$ 0.14	2.79 $\pm$ 0.14	3.90 $\pm$ 0.13

Table 4: MOS results on AVDnR dataset.

Figure 5: Comparison of our audio-only model (Ours-AO), DAVIS-Flow [ 29 ] , and our audio-visual model (AV-CASS) on a clip from the AVDnR testset. The input video frames and the GT audio spectrograms are shown at the top. Yellow boxes highlight the bird chirping present in $\bm{v}^{s}$ and the FX tracks. Dotted boxes indicate misplaced segments. Better viewed when zoomed in.

		FAD ( $\downarrow$ )	KL ( $\downarrow$ )	SI-SDRi ( $\uparrow$ )	PESQ ( $\uparrow$ )	WPR ( $\downarrow$ )
DnRv2 [ 48 ]	Hybrid Demucs [ 14 ]	3.66	1.47	9.19	2.03	4.77
	HT Demucs [ 52 ]	3.72	1.37	8.59	1.95	9.14
	MRX [ 48 ]	5.01	1.77	7.48	1.78	16.56
	BandIt [ 59 ]	2.75	1.55	7.68	1.97	3.67
	MSDM [ 44 ]	6.25	1.54	9.09	2.07	5.33
	Ours (Audio-only)	1.95	1.33	8.10	1.93	2.13

Table 5: Audio-only CASS results. Metrics are averaged across three sources, except PESQ, which is evaluated only on the speech source. All models are trained on our dataset.

Method	$\bm{v}^{f}$	$\bm{v}^{s}$	FAD ( $\downarrow$ )	KL ( $\downarrow$ )	SI-SDRi ( $\uparrow$ )	PESQ ( $\uparrow$ )
Audio-only	✗	✗	1.63	1.15	12.23	2.08
+ Facial stream	✓	✗	0.91	1.00	12.13	2.21
+ Scene stream	✗	✓	0.87	1.00	12.27	2.24
+ Both (Ours)	✓	✓	0.84	0.93	12.32	2.26

Table 6: Ablation results on visual streams.

Method	$\bm{v}^{f}$	$\bm{v}^{s}$	DX	FX	MX
Audio-only	✗	✗	0.0265	4.6507	1.3443
+ Facial stream	✓	✗	0.0164	4.3310	1.2028
+ Scene stream	✗	✓	0.0289	4.0939	1.2282
+ Both (Ours)	✓	✓	0.0255	4.2854	1.2077

Table 7: Ablation results on visual streams with Wrong Placement Ratio (WPR [%]) for each stem. Lower is better.

Figure 6: Impact of the number of sampling steps. Dotted lines denote Hybrid Demucs (second-best FAD) on the left and BandIt (highest SI-SDRi) on the right.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Distributions of the synthesized training data for loudness in LKFS, true peak in dBFS, number of segments for each stem, and the duration of each segment in seconds.

Figure 8: Instructions and scoring interface for the MOS test on real-world movie clips.

Figure 9: Instructions and scoring interface for the MOS test on AVDnR dataset.

Method	A-V	P/G	FAD ( $\downarrow$ )				KL ( $\downarrow$ )				SI-SDRi [dB] ( $\uparrow$ )				WPR [%] ( $\downarrow$ )				PESQ ( $\uparrow$ )
Method	A-V	P/G	Avg.	DX	FX	MX	Avg.	DX	FX	MX	Avg.	DX	FX	MX	Avg.	DX	FX	MX	DX
Hybrid Demucs [ 14 ]	✗	P	2.05	1.66	1.34	3.16	1.03	0.99	0.95	1.14	13.57	12.75	13.59	14.38	5.24	0.12	12.67	2.93	2.16
HT Demucs [ 52 ]	✗	P	2.08	1.97	1.62	2.65	1.06	1.09	0.92	1.17	13.41	12.68	13.33	14.21	9.23	0.15	19.97	7.57	2.06
MRX [ 48 ]	✗	P	3.47	2.67	2.02	4.01	1.67	2.00	1.02	1.97	10.60	11.23	11.34	12.31	14.91	2.18	37.60	4.94	1.89
BandIt [ 59 ]	✗	P	2.15	2.38	2.96	1.11	1.14	0.82	1.19	1.41	14.40	13.33	14.39	15.48	4.65	0.04	12.46	1.46	2.15
MSDM [ 44 ]	✗	P	2.90	3.84	3.03	3.56	1.70	2.47	0.97	1.66	11.63	11.48	9.38	10.94	5.65	1.65	11.49	3.82	2.12

Appendix

Table 8: Detailed objective metric results on AVDnR for each stem. All models are trained on our training data. A-V refers to audio-visual, P refers to predictive model, and G refers to generative model.

	DnRv2 [ 48 ]			DnRv3 [ 60 ]
Method	DX	FX	MX	DX	FX	MX
Hybrid Demucs [ 14 ]	0.26	10.94	3.13	0.19	7.75	3.39
HT Demucs [ 52 ]	0.26	21.45	5.71	0.20	19.67	5.61
MSDM [ 44 ]	2.42	10.20	3.38	0.18	8.86	4.41
MRX [ 48 ]	2.00	41.50	6.19	3.25	39.73	6.94
BandIt [ 59 ]	0.24	7.36	3.40	0.09	6.56	3.95

Appendix

Table 9: Wrong Placement Ratio (WPR [%]). Ratio of residual sounds from other tracks within each track type for the DnRv2 and DnRv3 test sets. Lower is better.

Figure 10: Comparison of spectrograms from MRX, BandIt, and AV-CASS (ours) on AVDnR. Facial frames $\bm{v}^{f}$ , scene frames $\bm{v}^{s}$ , and the input audio spectrogram $\bm{s}^{A}$ shown at the top. Ground truth audio spectrogram is on top of the models’ results for each stem. The orange box in the DX stem highlights that our model reconstructs speech most accurately compared to others. Pink solid boxes indicate the presence of vacuum cleaning in $\bm{v}^{s}$ and the spectrograms of the FX stem, while dotted boxes indicate incorrectly placed vacuum cleaning sounds.

Figure 11: Comparison of the separation results of our model and other methods on AVDnR dataset.

Figure 12: Comparison of the separation results of our model and other methods on AVDnR dataset.

Figure 13: Comparison of the separation results of our model and other methods on AVDnR dataset.

Figure 14: Comparison of the separation results of our model and other methods on AVDnR dataset.

Figure 15: Comparison of the separation results of our model and other methods on AVDnR dataset.

Figure 16: Spectrogram comparison of separated outputs from our audio-only model (Ours-AO), audio-visual model (AV-CASS), and ground truth (GT) on AVDnR.

Figure 17: Spectrogram comparison of separated outputs from our audio-only model (Ours-AO), audio-visual model (AV-CASS), and ground truth (GT) on AVDnR.

Audio Source Separation

Audio source separation isolates individual speakers, instruments, or sound events from mixtures for listening, analysis, and audio editing.

See all 65 papers

LiteCASS: A Lightweight End-to-End Network for Real-Time Stereo Cinematic Audio Source Separation
Sep 20, 2026Yuanxin Guo, Qiang Ji, Mengmei Liu +3Spatial Audio Audio Source Separation
Southwestern University of Finance and Economics, Chengdu, China · Xiaomi Automobile Co., Ltd., Beijing, China · Wuhan University, Wuhan, China
SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model
Oct 8, 2026Aviad Dahan, Rajaei Khatib, Yonatan Bitton +3Audio-Video Generation Audio Source Separation
Tel Aviv University · Google
FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation
Aug 22, 2026Yi Yuan, Xubo Liu, Haohe Liu +3Audio Source Separation Flow Matching
School of Computer Science and Electronic Engineering, University of Surrey, Guildford, UK · Department of Informatics, King’s College London, London, UK

Sep 20, 2026J/K move · Enter open · S save

LiteCASS: A Lightweight End-to-End Network for Real-Time Stereo Cinematic Audio Source Separation

Yuanxin Guo, Qiang Ji, Mengmei Liu, Yuhan Lv+2

Southwestern University of Finance and Economics, Chengdu, China · Xiaomi Automobile Co., Ltd., Beijing, China · Wuhan University, Wuhan, China

Spatial Audio Audio Source Separation

Open paper AskPDF arXiv

Cinematic audio source separation (CASS) decomposes a soundtrack into dialogue, music, and sound-effects (SFX) stems. Existing CASS methods, however, suffer from two critical limitations: they rely on heavily parameterized network architectures and GPU-class hardware, limiting their use in real-time and resource-constrained scenarios, and they are overwhelmingly designed for monaural signals, leaving the stereo scenario largely unexplored. We present LiteCASS, to our knowledge the first lightweight end-to-end network for real-time stereo CASS. LiteCASS combines deterministic STFT subband rearrangement with two jointly trained compact U-Nets: the first extracts dialogue, and the second separates music and SFX from the predicted non-speech component. A multi-task waveform-domain L1 loss supervises all stems. On a spatialized stereo extension of DnR v3, LiteCASS-K8 uses only 1.06M parameters and 0.72G MACs per second, while achieving the highest averaged SI-SDR among the compared CASS baselines.

Abstract

Audio Source Separation

Explore similar work

LiteCASS: A Lightweight End-to-End Network for Real-Time Stereo Cinematic Audio Source Separation

SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model

FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation