Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

Kai Peng; Yunzhe Shen; Miao Zhang; Leiye Liu; Yidong Han; Wei Ji; Jingjing Li; Yongri Piao

cs.CVMar 15, 2026

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

Authors: Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Wei Ji, Jingjing Li, Yongri Piao,

Organizations: Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Abstract

The ability to capture and segment sounding objects in dynamic visual scenes is crucial for the development of Audio-Visual Segmentation (AVS) tasks. While significant progress has been made in this area, the interaction between audio and visual modalities still requires further exploration. In this work, we aim to answer the following questions: How can a model effectively suppress audio noise while enhancing relevant audio information? How can we achieve discriminative interaction between the audio and visual modalities? To this end, we propose SDAVS, equipped with the Selective Noise-Resilient Processor (SNRP) module and the Discriminative Audio-Visual Mutual Fusion (DAMF) strategy. The proposed SNRP mitigates audio noise interference by selectively emphasizing relevant auditory cues, while DAMF ensures more consistent audio-visual representations. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on benchmark AVS datasets, especially in multi-source and complex scenes. The code and model are available at https://github.com/happylife-pk/SDAVS.

Figures & tables

Fig. 1: In the first scene (a), the piano is the actual sounding object. However, during the piano performance, the interaction between the cat and the person interferes with the audio signal. In the second scene (b), the guitar is the true sounding object, while the person remains silent. Existing methods struggle to distinguish the correct sounding object.

Fig. 2: The overall pipeline of the proposed SDAVS, which includes the Selective Noise-Resilient Processor (SNRP) and Discriminative Audio-Visual Mutual Fusion (DAMF) modules. The SNRP module first filters noise and enhances audio-relevant features. The refined video features and original audio features are then fed into the DAMF module aligning their perception regions, which performs discriminative fusion using spatial-temporal-channel (STC) enhancement and bidirectional cross-modal attention. In DAMF, different icon shapes represent distinct perception regions per modality. After applying DAMF, the icons become more consistent, indicating improved cross-modal alignment.

Method	Backbone	S4			MS3
Method	Backbone	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
TPAVI [ 1 ]	PVT-v2	78.7	87.9	83.3	54.0	64.5	59.3
AVSC [ 36 ]	Swin-base	81.3	88.6	85.0	59.5	65.8	62.7
CATR [ 7 ]	PVT-v2	81.4	89.6	85.5	59.0	70.0	64.5
SelM [ 37 ]	PVT-v2	83.5	91.2	87.4	60.3	71.3	65.8
AQFormer [ 6 ]	PVT-v2	81.6	89.4	85.5	62.2	72.7	67.5

TABLE I: Comparison with state-of-the-art methods on the S4 and MS3 settings. The best results are highlighted in bold, while the second-best results are underlined.

\mathcal{J}\&\mathcal{F}_{m}

means the average value of

\mathcal{J}

and

\mathcal{F}_{m}

Method	Backbone	AVSS
Method	Backbone	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
TPAVI [ 1 ]	PVT-v2	29.8	35.2	32.5
CATR [ 7 ]	PVT-v2	32.8	38.5	35.7
AVSegFormer [ 4 ]	PVT-v2	37.3	42.8	40.1
BAVS [ 21 ]	Swin-base	33.6	37.5	35.6
SelM [ 37 ]	PVT-v2	41.3	46.9	44.1

TABLE II: Comparison with state-of-the-art methods on the AVSS settings. The best results are highlighted in bold, while the second-best results are underlined.

\mathcal{J}\&\mathcal{F}_{m}

means the average value of

\mathcal{J}

and

\mathcal{F}_{m}

. * denotes the two-stage methods.

Fig. 3: Qualitative comparison of AVSBench [ 1 ] , AVSegFormer [ 4 ] , AVSStone [ 19 ] and Ours on the AVSBench-object dataset. (a) demonstrates the accuracy of the model segmentation, and (b) demonstrates the model’s suppression of the incorrect segmentation, i.e., the segmentation of non-sounding objects.

Fig. 4: Qualitative comparison between AVSBench [ 1 ] , AVSegFormer [ 4 ] and Ours on the AVSS dataset, demonstrating our model’s excellent ability to interpret semantic information.

Module	S4			MS3
Module	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
w/o SNRP	84.1	90.8	87.5	70.7	79.0	74.9
w/o DAMF	83.2	90.5	86.9	69.1	78.2	73.7
SDAVS(Ours)	85.5	92.0	88.8	72.3	80.8	76.6

TABLE III: The impact of the proposed modules on the model’s effectiveness, and the absence of any module leads to performance degradation.

SNRP	S4			MS3
SNRP	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
w/o CFS	84.3	90.9	87.6	70.6	79.1	74.9
w/o SFS	84.6	91.3	88.0	70.5	79.1	74.8
SDAVS(Ours)	85.5	92.0	88.8	72.3	80.8	76.6

TABLE IV: Ablation study on SNRP components (CFS and SFS).

Fig. 5: feature maps visualizations in SNRP presented in sequence from left to right. (a) raw images, (b) ground truth masks, (c) feature maps before processing by SNRP, (d) feature maps after processing CFS, and (e) feature maps after processing SFS.

Position	S4			MS3
Position	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
Pre	85.5	92	88.8	72.3	80.8	76.6
After	84.3	90.6	87.5	71.1	78.7	74.9

TABLE V: Ablation study on the placement of the SNRP module. ’Pre’ indicates placing SNRP before the fusion stage, while ’After’ indicates placing it after the fusion stage.

DAMF	S4			MS3
DAMF	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
only A $\to$ V	83.4	90.9	87.2	69.6	78.8	74.2
only V $\to$ A	83.2	90.7	87.0	69.3	78.6	74.0
w/o STC	83.8	91.0	87.4	70.5	79.3	74.9
SDAVS(Ours)	85.5	92.0	88.8	72.3	80.8	76.6

TABLE VI: Ablation study on DAMF. A

\to

V means audio-prompt method and V

\to

A means video-prompt method.

Processing Stage	CKA $\uparrow$	KL $\downarrow$	JS $\downarrow$
before DAMF	0.33	22.1	0.43
After DAMF	0.79	15.3	0.32

TABLE VII: Quantitative analysis of cross-modal perception consistency before and after the DAMF module. CKA (Centered Kernel Alignment), KL Divergence, and JS Divergence are employed to measure the feature distribution alignment.

Fig. 6: feature maps visualizations in DAMF presented in sequence from left to right. (a) raw images, (b) ground truth masks, (c) visual feature maps before processing by DAMF, (d) audio feature maps before processing by DAMF, (e) visual feature maps after processing by DAMF, and (f) audio feature maps after processing by DAMF. The red boxes highlight the primary regions of modal attention, illustrating the significant spatial discrepancy before fusion and the improved cross-modal consensus after DAMF processing.

Fig. 7: Visualization comparisons of feature maps under different ablation settings within the DAMF module. The columns from left to right display: (a-b) Video and Audio features without the STC module; (c-d) Video and Audio features without Residual Multiplication; and (e-f) Video and Audio features processed by the full DAMF module.

Interaction	S4			MS3
Interaction	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
straight	83.9	91.2	87.6	71.2	79.6	74.8
add	84.3	91.5	87.9	71.6	79.9	75.2
$\odot$	85.5	92.0	88.8	72.3	80.8	76.6

TABLE VIII: Performance comparison of different cross-modal interaction strategies within the DAMF module. Three strategies including straight-through connection, element-wise addition, and the proposed residual multiplication (

\odot

) are evaluated on the S4 and MS3 datasets.

Test Condition	w/o SNRP			Ours
Test Condition	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
Original Audio	70.7	79.0	74.9	72.3	80.8	76.6
Brownian Noise	68.8	76.5	72.7	72.1	80.5	76.3
Moving Train	69.5	77.3	73.4	72.2	80.8	76.5

TABLE IX: Robustness evaluation of SDAVS with and without the SNRP module under different audio interference conditions. This table presents the segmentation performance on the MS3 dataset under three audio conditions: clean (original) audio, synthetic Brownian noise, and real-world train noise.

Fig. 8: Statistical analysis of signal-noise separation in SNRP. (a) Qualitative Feature Map Refinement: Feature maps from the final stage of SNRP (Ref. Fig. 5), showing refined spatial focus. (b) 3D Spatial Weight Statistical Analysis: 3D response maps revealing the flattening of background noise into a low-response floor and sharpening of signal peaks. (c) Channel-wise Weight Distribution: Violin plots of channel weights demonstrating the heterogeneous distribution and selective channel-wise gating.

Fig. 9: Comparison of model efficiency across state-of-the-art methods in terms of trainable parameters, computational complexity (FLOPs), and speed. Speed refers to the number of frames processed per second during inference in the same experimental settings.

Method	FLOPs (G) $\downarrow$	Trainable Params (M) $\downarrow$	FPS $\uparrow$
w/o SNRP	300.85	100.58	31.1
w/o DAMF	280.31	100.76	33.5
w/o STC	426.7	103	22
w/o RM	300.88	103.2	31.2
Ours	300.9	103.2	31.2

TABLE X: Component-wise computational efficiency analysis. We report FLOPs, Parameters, and Inference Speed (FPS) for the removal of SNRP, DAMF, and internal sub-modules (STC, RM) to comprehensively validate the architectural trade-offs between complexity and performance.

Method	FSLA	HOTA	mAP
AVSegformer [ 4 ]	35.7	55.7	35.7
COMBO [ 40 ]	39.5	57.4	37.5
Ours	41.5	60.3	39.8
AVISM [ 48 ]	42.8	61.7	40.6

TABLE XI: Performance comparison of Ours on the AVIS dataset

Method	mIoU $\uparrow$	F-score $\uparrow$	FDR $\downarrow$
Dataset: VPO-SS
TPAVI [ 1 ]	52.75	69.54	22.83
AVSegformer [ 4 ]	57.55	73.03	19.76
CAVP [ 49 ]	62.31	78.46	13.56
Ours	65.73	79.1	12.89
Dataset: VPO-MS

TABLE XII: Comparison results on VPO-SS, VPO-MS, and VPO-MSMI datasets.

Fig. 10: Failure cases: Our model demonstrates limitations when off-screen wolf howling coexists with silent dog, leading to incorrect segmentation.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Sep 23, 2025

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Audio-visual segmentation (AVS) plays a critical role in multimodal machine learning by effectively integrating audio and visual cues to precisely segment objects or regions within visual scenes. Recent AVS methods have demonstrated significant improvements. However, they overlook the inherent frequency-domain contradictions between audio and visual modalities--the substantial noise often present in audio high-frequency signals vs. the structurally rich details in visual high-frequency signals. Ignoring these differences can result in suboptimal performance. In this paper, we rethink the AVS task from a deeper perspective by reformulating AVS task as a frequency-domain decomposition and recomposition problem. To this end, we introduce a novel Frequency-Aware Audio-Visual Segmentation (FAVS) framework consisting of two key modules: Frequency-Domain Enhanced Decomposer (FDED) module and Synergistic Cross-Modal Consistency (SCMC) module. FDED module employs a residual-based iterative frequency decomposition to discriminate modality-specific semantics and structural features, and SCMC module leverages a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing. Extensive experiments demonstrate that our FAVS framework achieves state-of-the-art performance on three benchmark datasets, and abundant qualitative visualizations further verify the effectiveness of the proposed FDED and SCMC modules. The project repository is available at https://github.com/happylife-pk/FAVS.

Kai Peng, Yunzhe Shen, Miao Zhang +6

Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Audio-Visual Understanding Multimodal Semantic Segmentation Multimodal Fusion Frequency-Domain Feature Learning

Ask PDF

Dec 23, 2025

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Jingqi Tian, Yiheng Du, Haoji Zhang +6

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Ask PDF

Aug 17, 2026

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.

Zhaojin Fu, Yuyang Hong, Qi Yang +4

School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing 100083, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Audio-Visual Understanding Multimodal Fusion Audio-Visual Alignment

Ask PDF

Abstract

Audio-Visual Understanding

Explore similar work

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling