Authors: Saijun Wang, Guanfeng Tang, Hongbo Zhao, Zhicheng Lei, Yutong Zhang, Wei Ye, Rui Fan
Organizations: College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China
Audio-visual segmentation (AVS) aims to segment sounding objects at the pixel level by integrating auditory and visual cues. However, existing methods are predominantly developed under the monaural setting and primarily rely on cross-modal semantic correspondence, which limits their ability to distinguish visually similar instances of the same semantic class. In contrast, humans naturally exploit binaural hearing, where interaural differences and direction-dependent acoustic filtering introduced by the head and pinnae provide physically grounded spatial cues for accurate sound source localization. Motivated by this observation, we introduce binaural audio-visual instance segmentation (BiAVIS), a new task that leverages synchronized binaural audio and video frames to segment sounding instances. To advance research on this task, we establish two benchmarks by manually annotating an existing binaural audio-visual dataset and collecting a new real-world dataset, BiAVIS-Bench, in more challenging and diverse scenarios. The proposed BiAVIS model (BiAVISM) features an audio-only sound source localization network that directly projects binaural spatial cues onto the image plane to learn spatial and semantic priors for sounding instances. A query-level audio-visual fusion strategy is subsequently introduced to inject these informative priors into the instance segmentation decoder. Extensive experiments conducted on the two proposed benchmarks demonstrate the superior performance of the BiAVISM over previous monaural AVS methods, especially in resolving instance-level intra-class ambiguity. On the more challenging BiAVIS-Bench, BiAVISM achieves 26.65 mAP and 45.90 FSLA, corresponding to relative improvements of approximately 35.35% over AVISM in mean average precision (mAP) and 11.62% over ACVIS in frame-level sound localization accuracy (FSLA).
Figures & tables
Figure 1: Comparison between AVISM and the proposed BiAVISM. AVISM conditions queries with monaural audio, leading to intra-class instance ambiguity. In contrast, BiAVISM exploits spatial priors, enabling segmentation of the truly sounding instance.
Figure 2: Overview of the proposed BiAVIS framework. The Audio Encoder denotes the complete binaural feature extraction pipeline, consisting of two channel-specific spectrogram encoders followed by a binaural-difference encoder. The audio-only SSL branch predicts sounding probability and class probability maps, while the visual branch follows a Mask2Former-style segmentation decoder. Binaural priors are injected via query-level cross-attention, where object queries are conditioned with binaural features to form audio-aware queries for segmentation prediction. The model is trained with segmentation, class-wise localization, and audio-visual consistency losses.
Figure 3: Dataset collection setup and category statistics. (a) Our binaural audio-visual recording system, consisting of a ZED 2i binocular camera and a Mic.x Binaural I microphone mounted on a custom rig. (b) Instance counts per category in BiAVIS-Bench. (c) Instance counts per category in FAIR-Play.
Figure 4: Qualitative comparison of sounding-object segmentation results across diverse scenes.
Task
Networks
Audio
FAIR-Play
BiAVIS-Bench
mAP
FSLA
FSLAn
FSLAs
FSLAm
mAP
FSLA
FSLAn
FSLAs
FSLAm
VIS
Mask2Former [ 9 ] (CVPR’22)
✗
35.71
60.56
3.52
72.38
44.23
22.32
40.22
10.58
45.07
16.74
MP-Former [ 40 ] (CVPR’23)
✗
26.43
54.17
0.00
73.98
24.40
18.89
19.82
2.88
21.95
10.89
PolarNeXt [ 32 ] (CVPR’25)
✗
30.41
36.55
0.00
51.12
14.45
15.72
28.61
2.88
33.32
4.71
AVSS
COMBO [ 38 ] (CVPR’24)
✓
33.27
56.27
17.65
63.86
45.91
19.00
38.49
9.62
43.46
13.88
AVSegFormer [ 12 ] (AAAI’24)
✓
27.36
44.88
76.47
59.75
18.36
15.47
28.61
4.81
32.30
11.15
Table 1: Quantitative comparison of different models from related tasks on the FAIR-Play and BiAVIS-Bench test sets.
Gaussian
Lmse
Ljs
CC ↑
SIM ↑
EMD ↓
✗
✓
✗
0.01
0.02
5.35
✗
✓
✓
0.06
0.05
4.20
✓
✓
✗
0.31
0.24
3.41
✓
✓
✓
0.37
0.33
2.87
Table 2: Ablation study on the proposed localization loss conducted on the BiAVIS-Bench dataset.
Lc
Lavc
mAP ↑
FSLA ↑
FSLAn ↑
FSLAs ↑
FSLAm ↑
✗
✗
21.65
30.90
0.00
34.09
19.55
✗
✓
13.12
33.58
4.81
37.76
14.51
✓
✗
20.81
32.73
3.85
36.46
16.87
✓
✓
26.65
45.90
60.58
48.14
24.78
Table 3: Ablation study on the proposed class and audio-visual consistency loss conducted on the BiAVIS-Bench dataset.
Dataset
Audio
mAP ↑
FSLA ↑
FSLAn ↑
FSLAs ↑
FSLAm ↑
FAIR-Play
avg
35.14
54.90
88.24
63.29
39.01
L/L
33.80
54.96
84.71
63.53
39.02
R/R
37.27
55.21
87.06
64.46
38.02
L/R
37.63
62.90
89.41
68.63
51.85
BiAVIS-Bench
avg
22.95
43.07
32.69
47.74
13.92
L/L
21.59
42.43
33.65
47.81
17.84
Table 4: Ablation study on binaural versus monauralized audio inputs.
Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/
Jingqi Tian, Yiheng Du, Haoji Zhang +6
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Audio-Visual Segmentation (AVS) requires predicting pixel-wise masks of sounding objects by grounding acoustic signals in visual space. In multi-source scenes, concurrent sound sources often remain entangled within shared visual representations, leading to merged masks or unstable specialization when using standard alignment-focused transformer architectures. To address this, we present SlotAVS, an object-centric framework driven by competitive bi-modal slot attention. While robust foundation models provide rich semantic priors, SlotAVS shifts the paradigm from global cross-modal fusion to explicit competitive binding. By initializing discrete object slots from temporal audio sequences and enforcing strict slot-wise competition over spatial visual tokens, our method structurally mitigates the blending of distinct acoustic entities. Further, to stabilize instance-level learning, we introduce a Hungarian matching strategy that optimally aligns slots with ground-truth objects using spatial and boundary overlap. Coupled with a numerically stable, confidence-aware mask fusion mechanism, SlotAVS robustly aggregates multi-source predictions. Extensive experiments on the AVSBench dataset demonstrate the efficacy of our approach, achieving 88.6% mIoU on the Single-Source (S4) split and 71.7% mIoU on the Multi-Source (MS3) split. These results show that explicit competitive binding is critical for complex audio-visual scene understanding and effectively leveraging foundation-model priors without relying solely on their capacity.
Arnav Sharma, Anand Mishra, Renu Rameshan
Indian Institute of Technology Jodhpur · Vehant Technologies Pvt Ltd
Audio-visual instance segmentation (AVIS) requires accurately identifying and tracking individual sounding objects with pixel-level masks. Existing methods struggle to match overlapping acoustic events with visual instances and handle asynchronous audio-visual dynamics. Therefore, two critical questions arise: how can a model establish precise correspondence between overlapping sound sources and visual instances, and how can a model maintain robust tracking when audio and visual signals are temporally misaligned?This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms. The Acoustic-Semantic Projector (ASP) disentangles mixed audio and establishes hierarchical correspondence from semantic to spatial domains. The Asynchronous Dynamics Modulator (ADM) adaptively adjusts state transitions via audio-modulated Mamba, prioritizing current information during dynamic variations and maintaining continuity in stable periods.Experiments on AVISeg show H2S achieves SOTA performance, attaining 48.54 mAP with a COCO pretrained ResNet50 and surpassing the previous by 7.8%. The code will be open-sourced once the paper is accepted. The source code will be publicly available at https://github.com/leiyeliu/H2S.
Leiye Liu, Miao Zhang, Jiahong Jiang +7
Dalian University of Technology Dalian, Liaoning, China · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Yale University New Haven, Connecticut, USA