Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

Zhaojin Fu; Yuyang Hong; Qi Yang; Zili Wang; Kun Ding; Shiming Xiang; Bin Fan

cs.CVAug 17, 2026

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

Authors: Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan

Organizations: School of Intelligent Science and Technology, University of Science and Technology Beijing, Beijing 100083, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China

Abstract

Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.

Figures & tables

Fig. 1: (a) Representative AVS methods that align audio and visual features. (b) Illustrates that humans exploit depth cues together with audio evidence to disambiguate sounding objects. (c) Presents our core idea, which aims to perform the AVS task in a human-like way.

Fig. 2: Overview of the proposed DGCM-AVS. (a) DADM extracts different frequency components from image and depth features to improve intra-object consistency and enhance inter-object distinction. (b) DGPF uses depth information as a bridge to progressively complete the Audio Depth Symbiotic stage (A-D Symbiotic) and Target Search stages, achieving cross-modal feature alignment. (c) Feature Calibrator further strengthens the target-related feature responses.

Fig. 3: Frequency-domain comparison of RGB and depth. The figure shows log power spectra, zoomed views of low/high-frequency regions, and edge responses reconstructed from the mid-to-high frequency band.

Fig. 4: Visualization of Intermediate Features in DADM. We visualize the intermediate features from the low-frequency branch $\mathcal{F}_{v}^{\prime}$ , the high-frequency branch $\mathcal{F}_{d}-\mathcal{F}_{d}^{\prime}$ , and the final fused feature $\mathcal{F}_{o}$ . We also show the model predictions on the MS3 and AVSS benchmarks, together with the corresponding GT.

Fig. 5: Depth Bridge for DGPF. Serving as a structural cue, depth embeds features in A–D symbiosis and facilitates audio–visual alignment during Target Search.

		S4		MS3		AVSS
Method	Backbone	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$
AVSBench [ 4 ]	ResNet-50	72.8	84.8	47.9	57.8	20.2	25.2
AVS-BiGen [ 38 ]	ResNet-50	74.1	85.4	45.0	56.8	-	-
BAVS [ 39 ]	ResNet-50	78.0	85.3	50.2	62.4	24.7	29.6
AVSegFormer [ 5 ]	ResNet-50	76.4	86.7	53.8	65.6	26.6	31.5
SelM [ 13 ]	ResNet-50	76.6	86.2	54.5	65.6	31.9	37.2

TABLE I: Quantitative results on the Single Sound Source Segmentation (S4) and Multiple Sound Source Segmentation (MS3), and AVSBench-Semantic (AVSS). Bold indicates the best result, and underline indicates the second best.

Fig. 6: Qualitative comparisons on S4. (a) audio, (b) frames, (c) AVSBench [ 4 ] , (d) AVSegFormer [ 5 ] , (e) COMBO [ 7 ] , (f) DGCM-AVS (Ours), (g) GT. The green denotes missed regions and red indicates over-segmented areas relative to GT.

Fig. 7: Qualitative comparisons on MS3. (a) audio, (b) frames, (c) AVSBench [ 4 ] , (d) AVSegFormer [ 5 ] , (e) COMBO [ 7 ] , (f) DGCM-AVS (Ours), (g) GT. The green denotes missed regions and red indicates over-segmented areas relative to GT.

Fig. 8: Qualitative comparisons on AVSS. We provide a full visualization of the semantic segmentation performed on 10-frame sequences. The letter labels represent, in order: (a) audio, (b) frames, (c) AVSBench [ 4 ] , (d) AVSegFormer [ 5 ] , (e) COMBO [ 7 ] , (f) DGCM-AVS (Ours), (g) GT.

Method	Audio Backbone	Backbone	$\mathcal{M}_{\mathcal{J}}$ (S4)	$\mathcal{M}_{\mathcal{F}}$ (S4)	Params	FLOPs	Inference Time
AVSBench [ 4 ]	VGGish	ResNet-50	72.8	84.8	90.7M	166.6G	17.7ms
AVSegFormer [ 5 ]	VGGish	ResNet-50	76.4	86.7	150.9M	210.1G	176.5ms
DGCM-AVS (Ours)	VGGish	ResNet-50	83.3	91.3	224.3M	183.4G	60.7ms

TABLE II: Efficiency and Performance Comparison on S4. Bold indicates the best result.

Fig. 9: Failure cases of DGCM-AVS on S4, MS3, and AVSS (left to right). For each example, we show (a) audio, (b) frames, (c) DGCM-AVS (ours), and (d) GT.

Fig. 10: Visualization of the ablation study to show the efficacy of each proposed component.

Method	S4		AVSS
Method	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$
shared encoder	82.9	91.0	39.4	43.3
w/o depth	81.1	90.0	37.0	40.9
w/o DADM	83.2	90.9	38.7	42.4
DGCM-AVS	83.3	91.3	39.6	43.5

TABLE III: Validation of DADM and Weights on ResNet-50.

Method	S4		AVSS
Method	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$
$\text{DADM}_{k_{l}=7,k_{h}=5}$	83.1	91.1	39.2	43.1
$\text{DADM}_{k_{l}=7,k_{h}=3}$	82.7	90.8	38.9	42.6
$\text{DADM}_{k_{l}=5,k_{h}=3}$	83.3	91.3	39.6	43.5

TABLE IV: Validation the

k_{h}

and

k_{l}

window size of DADM.

Method	S4		AVSS
Method	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$
only AV	82.9	90.9	38.7	42.4
w/o audio	82.4	91.1	38.0	41.8
w/o FC	82.9	91.2	39.2	43.0
DGCM-AVS	83.3	91.3	39.6	43.5

TABLE V: Validation of DGPF structures on ResNet-50.

Method	S4		AVSS
Method	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$	$\mathcal{M}_{\mathcal{J}}$	$\mathcal{M}_{\mathcal{F}}$
+ DP	82.0	90.5	37.7	41.7
+ DA	83.2	90.9	37.6	41.7
+ DA-V2	83.3	91.3	39.6	43.5

TABLE VI: The impact of different depth information.

Fig. 11: The impact of depth information. (a) frames, (b) DP results, (c) AVS results based on DP , (d) DA-V2 results, (e) AVS results based on DA-V2.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
Sep 23, 2025Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Semantic Segmentation
Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada
SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention
Oct 11, 2026Arnav Sharma, Anand Mishra, Renu RameshanAudio-Visual Understanding Slot Attention
Indian Institute of Technology Jodhpur · Vehant Technologies Pvt Ltd

Dec 23, 2025J/K move · Enter open · S save

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang+5

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Open paper AskPDF arXiv

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Abstract

Audio-Visual Understanding

Explore similar work

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention