Binaural Audio-Visual Instance Segmentation

Saijun Wang; Guanfeng Tang; Hongbo Zhao; Zhicheng Lei; Yutong Zhang; Wei Ye; Rui Fan

cs.CVSep 26, 2026

Binaural Audio-Visual Instance Segmentation

Authors: Saijun Wang, Guanfeng Tang, Hongbo Zhao, Zhicheng Lei, Yutong Zhang, Wei Ye, Rui Fan

Organizations: College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China

Abstract

Audio-visual segmentation (AVS) aims to segment sounding objects at the pixel level by integrating auditory and visual cues. However, existing methods are predominantly developed under the monaural setting and primarily rely on cross-modal semantic correspondence, which limits their ability to distinguish visually similar instances of the same semantic class. In contrast, humans naturally exploit binaural hearing, where interaural differences and direction-dependent acoustic filtering introduced by the head and pinnae provide physically grounded spatial cues for accurate sound source localization. Motivated by this observation, we introduce binaural audio-visual instance segmentation (BiAVIS), a new task that leverages synchronized binaural audio and video frames to segment sounding instances. To advance research on this task, we establish two benchmarks by manually annotating an existing binaural audio-visual dataset and collecting a new real-world dataset, BiAVIS-Bench, in more challenging and diverse scenarios. The proposed BiAVIS model (BiAVISM) features an audio-only sound source localization network that directly projects binaural spatial cues onto the image plane to learn spatial and semantic priors for sounding instances. A query-level audio-visual fusion strategy is subsequently introduced to inject these informative priors into the instance segmentation decoder. Extensive experiments conducted on the two proposed benchmarks demonstrate the superior performance of the BiAVISM over previous monaural AVS methods, especially in resolving instance-level intra-class ambiguity. On the more challenging BiAVIS-Bench, BiAVISM achieves 26.65 mAP and 45.90 FSLA, corresponding to relative improvements of approximately 35.35% over AVISM in mean average precision (mAP) and 11.62% over ACVIS in frame-level sound localization accuracy (FSLA).

Figures & tables

Figure 1: Comparison between AVISM and the proposed BiAVISM. AVISM conditions queries with monaural audio, leading to intra-class instance ambiguity. In contrast, BiAVISM exploits spatial priors, enabling segmentation of the truly sounding instance.

Figure 2: Overview of the proposed BiAVIS framework. The Audio Encoder denotes the complete binaural feature extraction pipeline, consisting of two channel-specific spectrogram encoders followed by a binaural-difference encoder. The audio-only SSL branch predicts sounding probability and class probability maps, while the visual branch follows a Mask2Former-style segmentation decoder. Binaural priors are injected via query-level cross-attention, where object queries are conditioned with binaural features to form audio-aware queries for segmentation prediction. The model is trained with segmentation, class-wise localization, and audio-visual consistency losses.

Figure 3: Dataset collection setup and category statistics. (a) Our binaural audio-visual recording system, consisting of a ZED 2i binocular camera and a Mic.x Binaural I microphone mounted on a custom rig. (b) Instance counts per category in BiAVIS-Bench. (c) Instance counts per category in FAIR-Play.

Figure 4: Qualitative comparison of sounding-object segmentation results across diverse scenes.

Task	Networks	Audio	FAIR-Play					BiAVIS-Bench
Task	Networks	Audio	mAP	FSLA	FSLAn	FSLAs	FSLAm	mAP	FSLA	FSLAn	FSLAs	FSLAm
VIS	Mask2Former [ 9 ] (CVPR’22)	✗	35.71	60.56	3.52	72.38	44.23	22.32	40.22	10.58	45.07	16.74
	MP-Former [ 40 ] (CVPR’23)	✗	26.43	54.17	0.00	73.98	24.40	18.89	19.82	2.88	21.95	10.89
	PolarNeXt [ 32 ] (CVPR’25)	✗	30.41	36.55	0.00	51.12	14.45	15.72	28.61	2.88	33.32	4.71
AVSS	COMBO [ 38 ] (CVPR’24)	✓	33.27	56.27	17.65	63.86	45.91	19.00	38.49	9.62	43.46	13.88
	AVSegFormer [ 12 ] (AAAI’24)	✓	27.36	44.88	76.47	59.75	18.36	15.47	28.61	4.81	32.30	11.15

Table 1: Quantitative comparison of different models from related tasks on the FAIR-Play and BiAVIS-Bench test sets.

Gaussian	$\mathcal{L}_{\mathrm{mse}}$	$\mathcal{L}_{\mathrm{js}}$	CC $\uparrow$	SIM $\uparrow$	EMD $\downarrow$
✗	✓	✗	0.01	0.02	5.35
✗	✓	✓	0.06	0.05	4.20
✓	✓	✗	0.31	0.24	3.41
✓	✓	✓	0.37	0.33	2.87

Table 2: Ablation study on the proposed localization loss conducted on the BiAVIS-Bench dataset.

$\mathcal{L}_{\mathrm{c}}$	$\mathcal{L}_{\mathrm{avc}}$	mAP $\uparrow$	FSLA $\uparrow$	FSLAn $\uparrow$	FSLAs $\uparrow$	FSLAm $\uparrow$
✗	✗	21.65	30.90	0.00	34.09	19.55
✗	✓	13.12	33.58	4.81	37.76	14.51
✓	✗	20.81	32.73	3.85	36.46	16.87
✓	✓	26.65	45.90	60.58	48.14	24.78

Table 3: Ablation study on the proposed class and audio-visual consistency loss conducted on the BiAVIS-Bench dataset.

Dataset	Audio	mAP $\uparrow$	FSLA $\uparrow$	FSLAn $\uparrow$	FSLAs $\uparrow$	FSLAm $\uparrow$
FAIR-Play	avg	35.14	54.90	88.24	63.29	39.01
	L/L	33.80	54.96	84.71	63.53	39.02
	R/R	37.27	55.21	87.06	64.46	38.02
	L/R	37.63	62.90	89.41	68.63	51.85
BiAVIS-Bench	avg	22.95	43.07	32.69	47.74	13.92
	L/L	21.59	42.43	33.65	47.81	17.84

Table 4: Ablation study on binaural versus monauralized audio inputs.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention
Oct 11, 2026Arnav Sharma, Anand Mishra, Renu RameshanAudio-Visual Understanding Slot Attention
Indian Institute of Technology Jodhpur · Vehant Technologies Pvt Ltd
Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation
Aug 4, 2026Leiye Liu, Miao Zhang, Jiahong Jiang +7Video Object Segmentation Instance Segmentation
Dalian University of Technology Dalian, Liaoning, China · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Yale University New Haven, Connecticut, USA

Dec 23, 2025J/K move · Enter open · S save

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang+5

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Open paper AskPDF arXiv

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Abstract

Audio-Visual Understanding

Explore similar work

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention

Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation