Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Kai Peng; Yunzhe Shen; Miao Zhang; Jia Wang; Leiye Liu; Wei Ji; Jingjing Li; Yongri Piao

cs.CVSep 23, 2025

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Authors: Kai Peng, Yunzhe Shen, Miao Zhang, Jia Wang, Leiye Liu, Wei Ji, Jingjing Li, Yongri Piao,

Organizations: Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Abstract

Audio-visual segmentation (AVS) plays a critical role in multimodal machine learning by effectively integrating audio and visual cues to precisely segment objects or regions within visual scenes. Recent AVS methods have demonstrated significant improvements. However, they overlook the inherent frequency-domain contradictions between audio and visual modalities--the substantial noise often present in audio high-frequency signals vs. the structurally rich details in visual high-frequency signals. Ignoring these differences can result in suboptimal performance. In this paper, we rethink the AVS task from a deeper perspective by reformulating AVS task as a frequency-domain decomposition and recomposition problem. To this end, we introduce a novel Frequency-Aware Audio-Visual Segmentation (FAVS) framework consisting of two key modules: Frequency-Domain Enhanced Decomposer (FDED) module and Synergistic Cross-Modal Consistency (SCMC) module. FDED module employs a residual-based iterative frequency decomposition to discriminate modality-specific semantics and structural features, and SCMC module leverages a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing. Extensive experiments demonstrate that our FAVS framework achieves state-of-the-art performance on three benchmark datasets, and abundant qualitative visualizations further verify the effectiveness of the proposed FDED and SCMC modules. The project repository is available at https://github.com/happylife-pk/FAVS.

Figures & tables

Fig. 1: Illustration of frequency-domain contradictions in AVS. Frequency spectra depict signal strength, with inner regions representing lower frequencies and outer regions corresponding to higher frequencies. In video (Top), high frequencies (outer regions) preserve critical structural details (e.g., object edges) for precise segmentation. Conversely, in audio (Bottom), high frequencies (outer regions) often contain substantial noise, while lower frequencies contain rich semantic information.

Fig. 2: Overview of the proposed FAVS framework and its key components: (a) Overview Architecture . The framework processes video and audio inputs through their corresponding encoders, and progressively refines visual and audio features via three stages of FDED and SCMC modules. The refined audio features are further used to generate audio queries, which are fused with learnable queries in the transformer decoder to predict the final segmentation masks. (b) FDED Module . This module transforms modality features into the frequency domain, performs residual radial decomposition to separate high-, mid-, low-, and residual-frequency bands, and applies modality-specific heterogeneous operations on the high-frequency bands, i.e., suppressing noisy audio high-frequency components while enhancing discriminative visual high-frequency structures, followed by inverse transformation and weighted reconstruction. (c) SCMC Module . This module employs a mixture-of-experts architecture to model complementary cross-modal patterns across the reconstructed frequency bands. An entropy-guided dynamic routing mechanism adaptively determines the number of activated experts and aggregates their outputs to restore suppressed features and enhance cross-modal consistency.

Results on 224 $\times$ 224 Image Size
Method	Reference	Backbone	S4			MS3
Method	Reference	Backbone	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
TPAVI	ECCV’22	PVT-v2	78.7	87.9	83.3	54.0	64.5	59.3
AQFormer	IJCAI’23	PVT-v2	81.6	89.4	85.5	61.1	72.1	66.6
CATR	ACMMM’23	PVT-v2	81.4	89.6	85.5	57.8	70.8	64.3
AVSC	ACMMM’23	Swin-Base	81.3	88.6	84.9	59.5	65.8	62.7

TABLE I: Comparison With State-of-the-Art Methods on the S4 and MS3 Datasets. The best results are highlighted in bold , and the second-best are underlined .

\mathcal{J}\&\mathcal{F}_{m}

means the average value of

\mathcal{J}

and

\mathcal{F}_{m}

. * Indicates results based on the evaluation criteria introduced by DDESeg [ 12 ] . Our corresponding result (FAVS * ) follows the same criteria for a fair comparison. –: unreported.

\dagger

METTLE uses

192\times 192

visual inputs.

Results on 224 $\times$ 224 Image Size
Method	Reference	Backbone	AVSS
Method	Reference	Backbone	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
TPAVI	ECCV’22	PVT-v2	29.8	35.2	32.5
CATR	ACMMM’23	PVT-v2	32.8	38.5	35.7
AVSegformer	AAAI’24	PVT-v2	36.7	42.0	39.4
BAVS	TMM’24	Swin-Base	33.6	37.5	35.6

TABLE II: Comparison with state-of-the-art methods on the AVSS dataset. The best results are highlighted in bold , and the second-best are underlined .

\mathcal{J}\&\mathcal{F}_{m}

means the average value of

\mathcal{J}

and

\mathcal{F}_{m}

. # denotes two-stage methods; * indicates results under the DDESeg evaluation protocol.

Method	S4		MS3		Params (M)	FLOPs (G)
Method	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	Params (M)	FLOPs (G)
Baseline	84.3	90.9	67.6	75.3	188.1	184.5
(+) FDED	+0.5	+0.5	+1.4	+1.5	192.2	203.8
(+) FDED (+) SCMC	+1.3	+1.3	+3.5	+3.4	199.5	210.0

TABLE III: Ablation Study of the Two Proposed Modules Included in FAVS. Params includes frozen parameters; FLOPs are per five-frame clip. Cost increments in the text are calculated before rounding.

Fig. 3: Qualitative comparison between AVSegFormer, COMBO and Ours on the three subsets of AVSBench dataset. (a) demonstrates the examples derived from S4, (b) demonstrates the examples derived from MS3, and (c) demonstrates the examples derived from AVSS.

Fig. 4: Feature maps visualizations in FDED presented in sequence from left to right. (a) raw images, (b) ground truth masks, (c) visual feature maps before processing by FDED, (d) visual feature maps after processing by FDED, (e) audio feature maps before processing by FDED, and (f) audio feature maps after processing by FDED.

Frequency bands	S4		MS3
Frequency bands	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$
Two	84.5	91.0	68.6	76.2
Three	84.8	91.4	69.0	76.8
Four	84.3	91.1	68.4	75.8

TABLE IV: Impact of frequency band number in the FDED module.

Fig. 5: Feature maps visualizations in SCMC presented in sequence from left to right. (a) raw images, (b) ground truth masks, (c) visual feature maps before processing by SCMC, (d) audio feature maps before processing by SCMC, (e) visual feature maps after processing by SCMC, and (f) audio feature maps after processing by SCMC.

Fig. 6: Effect of the number of experts in SCMC on S4 and MS3 datasets. The y-axis reports the average score $(\mathcal{J}+\mathcal{F}_{m})/2$ . Performance increases steadily from 1 to 4 experts and drops when using 5 experts, indicating that 4 experts provide the best balance between specialization and redundancy.

Fig. 7: The visual feature maps visualizations of the outputs from the four attention experts in SCMC. (a) raw images, (b) ground truth masks, (c)-(f) the visual feature maps from expert 1 to expert 4.

ID	H	M	L	P	S4			MS3
ID	H	M	L	P	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
A1	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	85.65	92.16	88.91	71.12	78.72	74.92
A2	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	0.00	0.00	0.00	$-1.77$	$-1.62$	$-1.70$
A3	$\times$	$\checkmark$	$\checkmark$	$\times$	$-0.09$	$-0.05$	$-0.07$	$-0.14$	$-0.28$	$-0.21$
A4	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$-0.04$	$-0.02$	$-0.03$	$-0.10$	$-0.47$	$-0.29$
A5	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$-0.04$	$-0.02$	$-0.03$	$-0.86$	$-0.68$	$-0.77$

TABLE V: Ablation of frequency bands and processing in FDED. A/V: audio/visual; H/M/L: high/mid/low bands; P: high-frequency processing. Bold: full-model scores (%); other entries: changes from the full model in percentage points.

$\tau_{1}$	$\tau_{2}$	S4			MS3
$\tau_{1}$	$\tau_{2}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}\&\mathcal{F}_{m}$
0.7	0.3	85.65	92.16	88.91	71.12	78.72	74.92
0.6	0.3	$-0.12$	$-0.06$	$-0.09$	$-0.10$	$-0.13$	$-0.12$
0.8	0.3	$-0.07$	$-0.03$	$-0.05$	$+0.06$	$-0.03$	$+0.02$
0.7	0.2	$-0.04$	$+0.02$	$-0.01$	$-0.16$	$-0.09$	$-0.13$
0.7	0.4	$-0.16$	$-0.10$	$-0.13$	$-0.28$	$-0.25$	$-0.27$

TABLE VI: Sensitivity to frequency boundaries in FDED. Bold: default scores (%); other entries: changes from the default in percentage points.

Fig. 8: The visualization presents experts’ ability to capture unique representations of audio features.

Fig. 9: The visualization presents the performance and computational efficiency of routing mechanisms. The x-axis denotes GFLOPs and the y-axis denotes the average score

(\mathcal{J}+\mathcal{F}_{m})/2

. Dynamic routing achieves a more favorable trade-off than fixed sparse routing while avoiding the highest computational cost of dense routing.

Threshold	$\mathcal{J}$ (%)	$\mathcal{F}_{m}$ (%)
0.4	85.58 ( $-0.02$ )	91.37 ( $-0.83$ )
0.5	85.60	92.20
0.6	84.93 ( $-0.67$ )	92.60 ( $+0.40$ )

TABLE VII: Prediction-threshold sensitivity on S4. Parentheses indicate changes from 0.5 in percentage points.

Decomposition	S4		MS3
Decomposition	$\mathcal{J}$	$\mathcal{F}_{m}$	$\mathcal{J}$	$\mathcal{F}_{m}$
Haar DWT	85.24 ( $-0.36$ )	91.96 ( $-0.24$ )	70.62 ( $-0.48$ )	78.35 ( $-0.35$ )
Fourier (Ours)	85.60	92.20	71.10	78.70

TABLE VIII: Comparison of decomposition strategies. Parentheses indicate changes from the proposed method in percentage points.

Noise	SNR (dB)	S4		MS3
Noise	SNR (dB)	Full	w/o $P$	Full	w/o $P$
Clean	–	85.61	85.60	71.12	69.35
Brownian	20	85.61	85.61	71.13	69.36
	10	85.61	85.61	71.33	69.50
	0	85.65	85.63	71.16	69.59
Moving train	20	85.63	85.62	71.00	69.69

TABLE IX: Audio robustness in

\mathcal{J}

(%).

P

: audio refinement in FDED. Unrelated audio uses SIR instead of SNR.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Mar 15, 2026

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

The ability to capture and segment sounding objects in dynamic visual scenes is crucial for the development of Audio-Visual Segmentation (AVS) tasks. While significant progress has been made in this area, the interaction between audio and visual modalities still requires further exploration. In this work, we aim to answer the following questions: How can a model effectively suppress audio noise while enhancing relevant audio information? How can we achieve discriminative interaction between the audio and visual modalities? To this end, we propose SDAVS, equipped with the Selective Noise-Resilient Processor (SNRP) module and the Discriminative Audio-Visual Mutual Fusion (DAMF) strategy. The proposed SNRP mitigates audio noise interference by selectively emphasizing relevant auditory cues, while DAMF ensures more consistent audio-visual representations. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on benchmark AVS datasets, especially in multi-source and complex scenes. The code and model are available at https://github.com/happylife-pk/SDAVS.

Kai Peng, Yunzhe Shen, Miao Zhang +6

Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Audio-Visual Understanding Multimodal Fusion

Ask PDF

Dec 23, 2025

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Jingqi Tian, Yiheng Du, Haoji Zhang +6

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Ask PDF

Aug 17, 2026

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.

Zhaojin Fu, Yuyang Hong, Qi Yang +4

School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing 100083, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Audio-Visual Understanding Multimodal Fusion Audio-Visual Alignment

Ask PDF

Abstract

Audio-Visual Understanding

Explore similar work

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling