AVPS-360: A Spatial Audio-Visual Benchmark for Instance Segmentation and Tracking of Sound-Producing Persons

Yifu Sun; Parthasaarathy Sudarsanam; Tuomas Virtanen; Archontis Politis

cs.CVOct 29, 2026

AVPS-360: A Spatial Audio-Visual Benchmark for Instance Segmentation and Tracking of Sound-Producing Persons

Authors: Yifu Sun, Parthasaarathy Sudarsanam, Tuomas Virtanen, Archontis Politis

Organizations: Audio Research Group, Tampere University, Tampere, Finland

Abstract

Audio-visual instance segmentation (AVIS) segments and tracks individual sound-producing objects in videos. Existing benchmarks predominantly use monaural audio, which lacks inter-channel cues for spatially distinguishing concurrent sound sources. We introduce AVPS-360 (Audio-Visual Person Segmentation 360), a spatial audio-visual benchmark bridging AVIS and sound event localisation and detection (SELD). Built upon STARSS23, it pairs first-order Ambisonics audio with synchronised 360° video across 230 recordings totalling 10.5 hours. Recordings average 164.0 seconds. The dataset captures intermittent sound production in real-world interactions dominated by speech, alongside laughter, clapping, and footsteps. It provides 31,678 manually reviewed instance masks linked to source identities and sound-event labels. We also present a baseline that integrates spatial audio with visual features for frame-level segmentation and identity association across frames. Experiments on AVPS-360 show that existing AVIS methods struggle with detection and identity association in long recordings with intermittent sound production, while our baseline substantially improves performance in both aspects.

Figures & tables

Dataset	Domain	Video	Audio	#Videos	Len.	Total	#Frames	Masks	Cls	Track
AVSBench-O [ 1 ]	Common	perspective	mono	5,356	5.0 s	7.4 h	12,972	12,972	23	$\times$
AVSBench-S [ 2 ]	Common	perspective	mono	12,356	7.8 s	26.8 h	82,972	–	70	$\times$
RefAVS-Bench [ 3 ]	Common	perspective	mono	4,002	10.0 s	11.1 h	40,020	–	51	$\times$
AVISeg [ 4 ]	Common	perspective	mono	926	61.4 s	15.8 h	56,871	94,074	26	✓
AVTrack [ 5 ]	Human	perspective	mono †	871	54.0 s	13.1 h	47,202 ‡	36,216 ‡	1	✓
AVPS-360 (ours)	Human	panoramic	FOA	230	164.0 s	10.5 h	37,718	31,678	1	✓

Table 1: Comparison of audio-visual segmentation datasets. Len. denotes average recording duration; #Frames counts annotated frames; Cls denotes the number of object classes; Track indicates persistent instance identities across frames.

Figure 1: Annotation examples from AVPS-360 . Coloured masks denote sound-producing persons, illustrating occlusion (top) and sound-source movement (bottom). Colours indicate consistent person identities within each recording.

Figure 2: Baseline architecture. Audio guides frame-level segmentation, and the resulting instance queries are mapped to embeddings for cross-frame association. A memory bank provides historical embeddings for contrastive learning. $T$ denotes the number of sampled frames per training clip, and MA denotes momentum-averaged embeddings.

Panel of Figure 3 — Figure 3: Illustrations of AVPS-360 dataset statistics.

Method	Audio Enc	AP	HOTA	DetA	AssA	IDF1
AVISM [ 4 ]	VGGish	0.46	29.85	26.11	34.25	39.51
AVISM [ 4 ]	CNN–Conf.	0.72	31.42	27.84	35.55	40.72
ACVIS [ 11 ]	VGGish	0.68	26.50	19.98	35.29	31.42
ACVIS [ 11 ]	CNN–Conf.	1.58	25.63	19.59	33.67	29.87
Ours	CNN–Conf.	17.92	60.57	59.34	62.00	73.43

Table 2: Comparison of different AVIS methods on AVPS-360 .

Audio input	AP	HOTA	DetA	AssA	IDF1
None	0.51	34.98	31.28	39.17	47.33
W-zero	0.29	34.16	30.85	37.88	46.61
W-repeat	2.30	42.99	40.56	45.68	57.69
FOA	17.92	60.57	59.34	62.00	73.43

Table 3: Audio input ablation.

Backbone	Initialisation	AP	HOTA	DetA	AssA	IDF1
ResNet-50	IN+COCO	17.92	60.57	59.34	62.00	73.43
ResNet-50	IN+COCO+OVIS	19.03	61.14	60.08	62.38	72.00
Swin-L	IN+COCO	21.94	63.16	63.53	62.89	74.00

Table 4: Visual backbone and pre-training.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 156 papers

Binaural Audio-Visual Instance Segmentation
Sep 26, 2026Saijun Wang, Guanfeng Tang, Hongbo Zhao +4Audio-Visual Understanding Instance Segmentation
College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China
Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation
Aug 4, 2026Leiye Liu, Miao Zhang, Jiahong Jiang +7Video Object Segmentation Instance Segmentation
Dalian University of Technology Dalian, Liaoning, China · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Yale University New Haven, Connecticut, USA
AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes
Jun 1, 2026Yaoting Wang, Yun Zhou, Zipei Zhang +1Audio-Visual Understanding Instance Segmentation
Institute of Big Data, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.

Sep 26, 2026J/K move · Enter open · S save

Binaural Audio-Visual Instance Segmentation

Saijun Wang, Guanfeng Tang, Hongbo Zhao, Zhicheng Lei+3

College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China

Audio-Visual Understanding Instance Segmentation

Open paper AskPDF arXiv

Audio-visual segmentation (AVS) aims to segment sounding objects at the pixel level by integrating auditory and visual cues. However, existing methods are predominantly developed under the monaural setting and primarily rely on cross-modal semantic correspondence, which limits their ability to distinguish visually similar instances of the same semantic class. In contrast, humans naturally exploit binaural hearing, where interaural differences and direction-dependent acoustic filtering introduced by the head and pinnae provide physically grounded spatial cues for accurate sound source localization. Motivated by this observation, we introduce binaural audio-visual instance segmentation (BiAVIS), a new task that leverages synchronized binaural audio and video frames to segment sounding instances. To advance research on this task, we establish two benchmarks by manually annotating an existing binaural audio-visual dataset and collecting a new real-world dataset, BiAVIS-Bench, in more challenging and diverse scenarios. The proposed BiAVIS model (BiAVISM) features an audio-only sound source localization network that directly projects binaural spatial cues onto the image plane to learn spatial and semantic priors for sounding instances. A query-level audio-visual fusion strategy is subsequently introduced to inject these informative priors into the instance segmentation decoder. Extensive experiments conducted on the two proposed benchmarks demonstrate the superior performance of the BiAVISM over previous monaural AVS methods, especially in resolving instance-level intra-class ambiguity. On the more challenging BiAVIS-Bench, BiAVISM achieves 26.65 mAP and 45.90 FSLA, corresponding to relative improvements of approximately 35.35% over AVISM in mean average precision (mAP) and 11.62% over ACVIS in frame-level sound localization accuracy (FSLA).

Abstract

Audio-Visual Understanding

Explore similar work

Binaural Audio-Visual Instance Segmentation

Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation

AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes