QC-MI-OVSEL: Benchmarking Query-Conditioned Multi-Instance Open-Vocabulary Sound Event Localization

Yongqiang Chen; Peiwei Chang; Yuelan Cheng; Xiatian Zhu; Wenwu Wang

cs.SDOct 9, 2026

QC-MI-OVSEL: Benchmarking Query-Conditioned Multi-Instance Open-Vocabulary Sound Event Localization

Authors: Yongqiang Chen, Peiwei Chang, Yuelan Cheng, Xiatian Zhu, Wenwu Wang

Organizations: Centre for Vision, Speech and Signal Processing, University of Surrey, UK

Abstract

We study query-conditioned multi-instance open-vocabulary sound event localization (QC-MI-OVSEL), where a system must recover all active physical sources matching a natural-language category query, including when the queried category is absent. We introduce the QC-MI-OVSEL benchmark, a 40,000-scene first-order Ambisonics benchmark with 80,000 balanced present/absent query views over 83 classes, including 17 novel classes excluded from training and development. Each frame is annotated with the activity and 3-D direction of every query-matching source, with up to three simultaneous targets. We further propose Q-RSet, which uses query-conditioned source slots and responsibility-based evidence aggregation for source-set prediction. Experiments show improved source recovery, localization recall, cardinality estimation, exact-set recovery, and absent-query suppression over complementary baselines. Analysis indicates that novel categories, paraphrased queries, and higher source multiplicity mainly impair source discovery and set completion, while directional accuracy remains comparatively stable for successfully recovered sources.

Figures & tables

Figure 1: Query-conditioned multi-instance open-vocabulary sound event localization formulation.

Figure 2: Q-RSet architecture.

Model	F1 $\uparrow$	LR $\uparrow$	LE $\downarrow$	K-MAE $\downarrow$	Exact $\uparrow$	Abs-FPR $\downarrow$
OVSELD-Adapt	47.12	37.47	7.93	0.47	21.02	30.01
Q-MACCDOA	43.09	38.79	10.13	0.44	20.27	22.44
TQ-SEL-Adapt	26.91	37.45	11.21	0.94	13.32	96.13
Q-RSet	50.36	46.11	8.86	0.41	25.89	20.55

Table 1: Canonical test results on QC-MI-OVSEL benchmark, averaged over three seeds.

Generalization
Condition	F1 $\uparrow$	LR $\uparrow$	LE $\downarrow$	K-MAE $\downarrow$	Exact $\uparrow$	Abs-FPR $\downarrow$
Base-ID	57.07	56.56	8.74	0.33	31.84	18.15
Novel-ID	34.27	26.90	8.84	0.60	16.65	28.32
Acoustic-OOD	56.92	55.94	8.79	0.32	31.82	17.48
Generalized	49.87	45.12	9.10	0.40	23.29	18.23
Language robustness

Table 2: Generalization and language robustness of Q-RSet.

$K_{\max}$	F1 $\uparrow$	LR $\uparrow$	LE $\downarrow$	K-MAE $\downarrow$	Exact $\uparrow$
1	56.04	52.14	8.16	0.43	39.27
2	52.72	44.23	8.90	0.69	23.25
3	54.80	46.17	9.22	0.89	14.89

Table 3: Q-RSet performance by maximum simultaneously active target count.

Decoder depth
Model	F1 $\uparrow$	LR $\uparrow$	LE $\downarrow$	K-MAE $\downarrow$	Exact $\uparrow$	Abs-FPR $\downarrow$
3 layers	47.23	43.45	9.26	0.44	22.41	20.58
6 layers	49.28	44.87	9.00	0.42	24.75	20.88
12 layers	50.36	46.11	8.86	0.41	25.89	20.55
Component ablations
Q-RSet	50.36	46.11	8.86	0.41	25.89	20.55

Table 4: Architecture analysis and component ablations of Q-RSet.

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 215 papers

AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360 $^\circ$ Videos
Oct 25, 2026Qingyu Luo, Peng Zhang, Wenwu Wang +1Spatial Audio Audio-Visual Understanding
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, U.K.
Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
Jun 12, 2026Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida +6Audio-Language Model Evaluation Audio Representation Learning
1POSTECH · 2Sony AI · 4Sungkyunkwan University +2
Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
Aug 10, 2026Zhi Zeng, Cheng Zhang, Zesheng Yang +9Unified Multimodal Models Audio-Visual QA
Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2

Oct 25, 2026J/K move · Enter open · S save

AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360 $^\circ$ Videos

Qingyu Luo, Peng Zhang, Wenwu Wang, Philip J. B. Jackson

Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, U.K.

Spatial Audio Audio-Visual Understanding

Open paper AskPDF arXiv

Spatial audio is central to immersive 360

^\circ

media, yet existing audio-visual spatialization renders the scene as a whole, leaving no control over an individual source. We introduce Audio-Visual Query-conditioned Target Sound Spatialization (AV-QTSS): given a monaural multi-source mixture, a 360

^\circ

video and a class query from the 13-class DCASE sound-event ontology, it generates the first-order Ambisonics (FOA) signal of the queried source alone. The omnidirectional mixture contains no explicit spatial cues, so direction has to be inferred from the video, where several objects may sound at once. We propose a query-conditioned dense binding localizer that predicts a spherical probability map over the whole scene, coupled to the renderer so both are optimized jointly. Experiments with one to four concurrent sources show improved localization and spatial accuracy relative to established methods and ablated variants, and the localizer transfers to real 360

^\circ

recordings after adaptation.

Abstract

Audio Understanding

Explore similar work

AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360∘^\circ∘ Videos

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360 $^\circ$ Videos