AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360$^\circ$ Videos

Qingyu Luo; Peng Zhang; Wenwu Wang; Philip J. B. Jackson

eess.ASOct 25, 2026

AV-QTSS: Class-Query Target Sound Spatialization in Multi-Source 360 $^\circ$ Videos

Authors: Qingyu Luo, Peng Zhang, Wenwu Wang, Philip J. B. Jackson

Organizations: Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, U.K.

Abstract

Spatial audio is central to immersive 360 $^\circ$ media, yet existing audio-visual spatialization renders the scene as a whole, leaving no control over an individual source. We introduce Audio-Visual Query-conditioned Target Sound Spatialization (AV-QTSS): given a monaural multi-source mixture, a 360 $^\circ$ video and a class query from the 13-class DCASE sound-event ontology, it generates the first-order Ambisonics (FOA) signal of the queried source alone. The omnidirectional mixture contains no explicit spatial cues, so direction has to be inferred from the video, where several objects may sound at once. We propose a query-conditioned dense binding localizer that predicts a spherical probability map over the whole scene, coupled to the renderer so both are optimized jointly. Experiments with one to four concurrent sources show improved localization and spatial accuracy relative to established methods and ablated variants, and the localizer transfers to real 360 $^\circ$ recordings after adaptation.

Figures & tables

Figure 1: (a) Scene-level spatialization. (b) Class-query target-level spatialization.

Figure 2: The framework of the proposed AV-QTSS system.

Model	Reconstruction Metrics				Spatial Metrics			Subjective Metrics
	$\boldsymbol{\ell_{2}(\times 10^{3})\downarrow}$	$\boldsymbol{\mathcal{L}_{\mathrm{mag}}\downarrow}$	$\boldsymbol{\mathcal{L}_{\mathrm{phs}}\downarrow}$	$\boldsymbol{\mathcal{L}_{\mathrm{MRS}}\downarrow}$	$\boldsymbol{\Delta_{\mathrm{abs}}\theta\downarrow}$	$\boldsymbol{\Delta_{\mathrm{abs}}\phi\downarrow}$	$\boldsymbol{\Delta_{\mathrm{ang}}\downarrow}$	MOS-C $\boldsymbol{\uparrow}$	MOS-S $\boldsymbol{\uparrow}$
Comparison to Separator Methods
No-Separator	21.7	1.50	1.75	4.70	29.5°	19.1°	39.2°	2.33	2.95
USS [ 13 ]	3.66	0.82	1.71	3.43	19.4°	12.4°	26.4°	3.05	3.36
Ablation on Localizer Components
No-Binding [ 35 ]	4.07	0.62	1.70	3.20	43.9°	13.5°	47.4°	3.85	2.44

Table 1: Comparison of Separator and Renderer methods, and ablation of Localizer components, on AV-QTSS-Syn. Except for the training-free Ambi-Encode, all variants are retrained under the same schedule. Best in bold , second best underlined .

Dataset	Localizer	$K{=}1$	$K{=}2$	$K{=}3$	$K{=}4$
AV-QTSS-Syn	No-Binding [ 35 ]	97.0	76.6	65.5	55.1
	No-Class	97.1	78.5	69.9	60.4
	QDB	97.9	90.6	89.6	82.7
STAIRS26	No-Binding [ 35 ]	38.8	44.0	47.1	19.7
	No-Class	42.3	58.1	57.9	20.1
	QDB	55.0	69.4	60.2	29.6

Table 2: QDB Localizer ablation on AV-QTSS-Syn and on real STAIRS26 recordings after adaptation, by number of concurrent sources

K

(Acc@30 ∘ %

\uparrow

Figure 3: Qualitative results. Rows: input scene, mixture, predicted target, rendered FOA energy map. Boxes mark the annotated position of each queried source.

Spatial Audio

Spatial audio studies how sound is captured, represented, generated, and perceived with direction, distance, motion, and room acoustics.

See all 100 papers

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
Jun 12, 2026Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida +6Audio-Language Model Evaluation Audio Representation Learning
1POSTECH · 2Sony AI · 4Sungkyunkwan University +2
DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos
Apr 3, 2026Ziyu Luo, Lin Chen, Qiang Qu +2Spatial Audio Audio-Video Generation
School of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing, China · School of Computer Science, The University of Sydney, Sydney, NSW, Australia · School of Software, Shandong University, Jinan, China
SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning
Oct 12, 2026Chenlong Wei, Hanling Wang, Mingjin Lou +6Spatial Audio Spatial Reasoning Benchmarks
Xi’an Jiaotong-Liverpool University · University of California, San Diego · Imperial College London +2

Jun 12, 2026J/K move · Enter open · S save

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida, Kim Sung-Bin+5

1POSTECH · 2Sony AI · 4Sungkyunkwan University+2

Audio-Language Model Evaluation Audio Representation Learning Audio Reasoning Audio Understanding Audio-Language Models Spatial Audio

Open paper AskPDF arXiv

Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.

Abstract

Spatial Audio

Explore similar work

Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources

DynFOA: Generating First-Order Ambisonics with Conditional Diffusion for Dynamic and Acoustically Complex 360-Degree Videos

SonicWorld: Source-Centric Latent Dynamics for Spatial Audio Reasoning