Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

Kai Peng; Yunzhe Shen; Miao Zhang; Leiye Liu; Yidong Han; Wei Ji; Jingjing Li; Yongri Piao

cs.CVMar 15, 2026

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

Authors: Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Wei Ji, Jingjing Li, Yongri Piao,

Organizations: Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Abstract

The ability to capture and segment sounding objects in dynamic visual scenes is crucial for the development of Audio-Visual Segmentation (AVS) tasks. While significant progress has been made in this area, the interaction between audio and visual modalities still requires further exploration. In this work, we aim to answer the following questions: How can a model effectively suppress audio noise while enhancing relevant audio information? How can we achieve discriminative interaction between the audio and visual modalities? To this end, we propose SDAVS, equipped with the Selective Noise-Resilient Processor (SNRP) module and the Discriminative Audio-Visual Mutual Fusion (DAMF) strategy. The proposed SNRP mitigates audio noise interference by selectively emphasizing relevant auditory cues, while DAMF ensures more consistent audio-visual representations. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on benchmark AVS datasets, especially in multi-source and complex scenes. The code and model are available at https://github.com/happylife-pk/SDAVS.

Figures & tables

Fig. 1: In the first scene (a), the piano is the actual sounding object. However, during the piano performance, the interaction between the cat and the person interferes with the audio signal. In the second scene (b), the guitar is the true sounding object, while the person remains silent. Existing methods struggle to distinguish the correct sounding object.

Fig. 2: The overall pipeline of the proposed SDAVS, which includes the Selective Noise-Resilient Processor (SNRP) and Discriminative Audio-Visual Mutual Fusion (DAMF) modules. The SNRP module first filters noise and enhances audio-relevant features. The refined video features and original audio features are then fed into the DAMF module aligning their perception regions, which performs discriminative fusion using spatial-temporal-channel (STC) enhancement and bidirectional cross-modal attention. In DAMF, different icon shapes represent distinct perception regions per modality. After applying DAMF, the icons become more consistent, indicating improved cross-modal alignment.

Method	Backbone	S4			MS3
Method	Backbone	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
TPAVI [ 1 ]	PVT-v2	78.7	87.9	83.3	54.0	64.5	59.3
AVSC [ 36 ]	Swin-base	81.3	88.6	85.0	59.5	65.8	62.7
CATR [ 7 ]	PVT-v2	81.4	89.6	85.5	59.0	70.0	64.5
SelM [ 37 ]	PVT-v2	83.5	91.2	87.4	60.3	71.3	65.8
AQFormer [ 6 ]	PVT-v2	81.6	89.4	85.5	62.2	72.7	67.5

TABLE I: Comparison with state-of-the-art methods on the S4 and MS3 settings. The best results are highlighted in bold, while the second-best results are underlined.

\mathcal{J}\&\mathcal{F}_{m}

means the average value of

\mathcal{J}

and

\mathcal{F}_{m}

Method	Backbone	AVSS
Method	Backbone	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
TPAVI [ 1 ]	PVT-v2	29.8	35.2	32.5
CATR [ 7 ]	PVT-v2	32.8	38.5	35.7
AVSegFormer [ 4 ]	PVT-v2	37.3	42.8	40.1
BAVS [ 21 ]	Swin-base	33.6	37.5	35.6
SelM [ 37 ]	PVT-v2	41.3	46.9	44.1

TABLE II: Comparison with state-of-the-art methods on the AVSS settings. The best results are highlighted in bold, while the second-best results are underlined.

\mathcal{J}\&\mathcal{F}_{m}

means the average value of

\mathcal{J}

and

\mathcal{F}_{m}

. * denotes the two-stage methods.

Fig. 3: Qualitative comparison of AVSBench [ 1 ] , AVSegFormer [ 4 ] , AVSStone [ 19 ] and Ours on the AVSBench-object dataset. (a) demonstrates the accuracy of the model segmentation, and (b) demonstrates the model’s suppression of the incorrect segmentation, i.e., the segmentation of non-sounding objects.

Fig. 4: Qualitative comparison between AVSBench [ 1 ] , AVSegFormer [ 4 ] and Ours on the AVSS dataset, demonstrating our model’s excellent ability to interpret semantic information.

Module	S4			MS3
Module	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
w/o SNRP	84.1	90.8	87.5	70.7	79.0	74.9
w/o DAMF	83.2	90.5	86.9	69.1	78.2	73.7
SDAVS(Ours)	85.5	92.0	88.8	72.3	80.8	76.6

TABLE III: The impact of the proposed modules on the model’s effectiveness, and the absence of any module leads to performance degradation.

SNRP	S4			MS3
SNRP	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
w/o CFS	84.3	90.9	87.6	70.6	79.1	74.9
w/o SFS	84.6	91.3	88.0	70.5	79.1	74.8
SDAVS(Ours)	85.5	92.0	88.8	72.3	80.8	76.6

TABLE IV: Ablation study on SNRP components (CFS and SFS).

Fig. 5: feature maps visualizations in SNRP presented in sequence from left to right. (a) raw images, (b) ground truth masks, (c) feature maps before processing by SNRP, (d) feature maps after processing CFS, and (e) feature maps after processing SFS.

Position	S4			MS3
Position	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
Pre	85.5	92	88.8	72.3	80.8	76.6
After	84.3	90.6	87.5	71.1	78.7	74.9

TABLE V: Ablation study on the placement of the SNRP module. ’Pre’ indicates placing SNRP before the fusion stage, while ’After’ indicates placing it after the fusion stage.

DAMF	S4			MS3
DAMF	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
only A $\to$ V	83.4	90.9	87.2	69.6	78.8	74.2
only V $\to$ A	83.2	90.7	87.0	69.3	78.6	74.0
w/o STC	83.8	91.0	87.4	70.5	79.3	74.9
SDAVS(Ours)	85.5	92.0	88.8	72.3	80.8	76.6

TABLE VI: Ablation study on DAMF. A

\to

V means audio-prompt method and V

\to

A means video-prompt method.

Processing Stage	CKA $\uparrow$	KL $\downarrow$	JS $\downarrow$
before DAMF	0.33	22.1	0.43
After DAMF	0.79	15.3	0.32

TABLE VII: Quantitative analysis of cross-modal perception consistency before and after the DAMF module. CKA (Centered Kernel Alignment), KL Divergence, and JS Divergence are employed to measure the feature distribution alignment.

Fig. 6: feature maps visualizations in DAMF presented in sequence from left to right. (a) raw images, (b) ground truth masks, (c) visual feature maps before processing by DAMF, (d) audio feature maps before processing by DAMF, (e) visual feature maps after processing by DAMF, and (f) audio feature maps after processing by DAMF. The red boxes highlight the primary regions of modal attention, illustrating the significant spatial discrepancy before fusion and the improved cross-modal consensus after DAMF processing.

Fig. 7: Visualization comparisons of feature maps under different ablation settings within the DAMF module. The columns from left to right display: (a-b) Video and Audio features without the STC module; (c-d) Video and Audio features without Residual Multiplication; and (e-f) Video and Audio features processed by the full DAMF module.

Interaction	S4			MS3
Interaction	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
straight	83.9	91.2	87.6	71.2	79.6	74.8
add	84.3	91.5	87.9	71.6	79.9	75.2
$\odot$	85.5	92.0	88.8	72.3	80.8	76.6

TABLE VIII: Performance comparison of different cross-modal interaction strategies within the DAMF module. Three strategies including straight-through connection, element-wise addition, and the proposed residual multiplication (

\odot

) are evaluated on the S4 and MS3 datasets.

Test Condition	w/o SNRP			Ours
Test Condition	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
Original Audio	70.7	79.0	74.9	72.3	80.8	76.6
Brownian Noise	68.8	76.5	72.7	72.1	80.5	76.3
Moving Train	69.5	77.3	73.4	72.2	80.8	76.5

TABLE IX: Robustness evaluation of SDAVS with and without the SNRP module under different audio interference conditions. This table presents the segmentation performance on the MS3 dataset under three audio conditions: clean (original) audio, synthetic Brownian noise, and real-world train noise.

Fig. 8: Statistical analysis of signal-noise separation in SNRP. (a) Qualitative Feature Map Refinement: Feature maps from the final stage of SNRP (Ref. Fig. 5), showing refined spatial focus. (b) 3D Spatial Weight Statistical Analysis: 3D response maps revealing the flattening of background noise into a low-response floor and sharpening of signal peaks. (c) Channel-wise Weight Distribution: Violin plots of channel weights demonstrating the heterogeneous distribution and selective channel-wise gating.

Fig. 9: Comparison of model efficiency across state-of-the-art methods in terms of trainable parameters, computational complexity (FLOPs), and speed. Speed refers to the number of frames processed per second during inference in the same experimental settings.

Method	FLOPs (G) $\downarrow$	Trainable Params (M) $\downarrow$	FPS $\uparrow$
w/o SNRP	300.85	100.58	31.1
w/o DAMF	280.31	100.76	33.5
w/o STC	426.7	103	22
w/o RM	300.88	103.2	31.2
Ours	300.9	103.2	31.2

TABLE X: Component-wise computational efficiency analysis. We report FLOPs, Parameters, and Inference Speed (FPS) for the removal of SNRP, DAMF, and internal sub-modules (STC, RM) to comprehensively validate the architectural trade-offs between complexity and performance.

Method	FSLA	HOTA	mAP
AVSegformer [ 4 ]	35.7	55.7	35.7
COMBO [ 40 ]	39.5	57.4	37.5
Ours	41.5	60.3	39.8
AVISM [ 48 ]	42.8	61.7	40.6

TABLE XI: Performance comparison of Ours on the AVIS dataset

Method	mIoU $\uparrow$	F-score $\uparrow$	FDR $\downarrow$
Dataset: VPO-SS
TPAVI [ 1 ]	52.75	69.54	22.83
AVSegformer [ 4 ]	57.55	73.03	19.76
CAVP [ 49 ]	62.31	78.46	13.56
Ours	65.73	79.1	12.89
Dataset: VPO-MS

TABLE XII: Comparison results on VPO-SS, VPO-MS, and VPO-MSMI datasets.

Fig. 10: Failure cases: Our model demonstrates limitations when off-screen wolf howling coexists with silent dog, leading to incorrect segmentation.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
Sep 23, 2025Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Semantic Segmentation
Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
Aug 17, 2026Zhaojin Fu, Yuyang Hong, Qi Yang +4Audio-Visual Understanding Multimodal Fusion
School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing 100083, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Sep 23, 2025J/K move · Enter open · S save

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Kai Peng, Yunzhe Shen, Miao Zhang, Jia Wang+5

Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Audio-Visual Understanding Multimodal Semantic Segmentation Multimodal Fusion Frequency-Domain Feature Learning

Open paper AskPDF arXiv

Audio-visual segmentation (AVS) plays a critical role in multimodal machine learning by effectively integrating audio and visual cues to precisely segment objects or regions within visual scenes. Recent AVS methods have demonstrated significant improvements. However, they overlook the inherent frequency-domain contradictions between audio and visual modalities--the substantial noise often present in audio high-frequency signals vs. the structurally rich details in visual high-frequency signals. Ignoring these differences can result in suboptimal performance. In this paper, we rethink the AVS task from a deeper perspective by reformulating AVS task as a frequency-domain decomposition and recomposition problem. To this end, we introduce a novel Frequency-Aware Audio-Visual Segmentation (FAVS) framework consisting of two key modules: Frequency-Domain Enhanced Decomposer (FDED) module and Synergistic Cross-Modal Consistency (SCMC) module. FDED module employs a residual-based iterative frequency decomposition to discriminate modality-specific semantics and structural features, and SCMC module leverages a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing. Extensive experiments demonstrate that our FAVS framework achieves state-of-the-art performance on three benchmark datasets, and abundant qualitative visualizations further verify the effectiveness of the proposed FDED and SCMC modules. The project repository is available at https://github.com/happylife-pk/FAVS.

Abstract

Audio-Visual Understanding

Explore similar work

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling