Spatial audio understanding aims to enable machines to interpret complex auditory scenes, particularly when sound sources move over time. In this work, we study Spatial Audio Question Answering (Spatial AQA) with a focus on movement reasoning, where a model must infer object motion, position, and directional changes directly from stereo audio. First, we introduce a movement-centric spatial audio augmentation framework that synthesizes diverse motion patterns from isolated mono audio events, enabling controlled and scalable training data generation. Second, we propose an end-to-end multimodal finetuning approach with a thinking mode, which allows audio-language models to produce explicit intermediate reasoning steps before predicting an answer. Third, we investigate the impact of query-conditioned source separation as a preprocessing stage and compare three inference regimes: no masking, an audio grounding model (AGM), and ground-truth masks. Our results show that reasoning amplifies the benefits of source separation, with thinking mode showing significant improvement of +5.1% when a single event is present in the question. These findings highlight the interplay between movement modeling, reasoning, and separation quality, offering new insights for advancing spatial audio understanding.
Figures & tables
Figure 1: Our proposed spatial audio language model architecture with thinking mode. AGM refers to audio grounding model.
Figure 2: Spatial audio dataset generation
Metric
Score
Answer Accuracy
4.93
Reasoning Quality
4.94
Groundedness
4.97
Table 1: Human evaluation of the Spatial AQA dataset (1-5 scale; higher is better) across 200 randomly selected samples.
Category
Question Type
Example Question
Answer
Trajectories
yes_no
Is the Croak sound moving from left to right?
Yes
Radial changes
yes_no
Are the waves moving closer to the listener as they sweep across?
Yes
Comparative diagnostics
multiple_choice
Which source sweeps across a wider left → right range? Options: [Blender, Car, Both equally, Neither]
Blender
Temporal sequence
open
Narrate the overall choreography: where does each source start, how does it move, and where does it finish?
The whale begins on the left and moves a moderate amount further left while approaching the listener; the lawn mower starts on the right, makes a wide sweep across the center to the left while coming closer and then moving away, and finishes on the left — resulting in the lawn mower crossing through the middle and the whale remaining on the left throughout.
Relative motion between events
yes_no
Do scissors and cattle sound move toward the same region (converge) as they progress?
Yes
Natural perception
open
Which source feels most noticeable near the middle of the scene and why?
The Applause feels most noticeable near the middle because it crosses the center with a moderate sweep, passing through that region, whereas the Camera stays on the left and does not pass through the middle.
Table 2: Example QA pairs by category
BAT
Thinking
Non-Thinking
Δ Interaction (pp)
Question Type
NoMask
AGM
GT
NoMask
AGM
GT
AGM
GT
Yes_No
41.7
72.1
73.8
76.3
72.6
72.8
72.3
+1.5
+4.5
Multiple Choice
0.3
53.1
52.9
52.8
50.4
50.3
50.3
− 0.1
− 0.2
Open
0.00
41.0
41.4
41.8
41.2
41.0
41.5
+0.6
+0.5
Overall
14.7
54.3
55.0
56.1
54.1
54.0
54.1
+0.8
+1.8
Table 3: Accuracy ( %). ΔInteraction measures how much more (or less) thinking helps under a method compared to NoMask: ΔInteraction=(AccTmethod−AccNmethod)−(AccTNoMask−AccNNoMask) . Acc is accuracy, T refers to thinking and N refers to non-thinking
Overlap
NoMask
AGM
GT
No overlap
55.0
56.4
57.9
Overlap
53.7
53.7
54.3
Table 4: Accuracy by overlap category (%) for thinking mode.
Audio Events
NoMask
AGM
GT
Complete match
48.4
48.6
49.2
1 Event
64.3
66.2
68.7
2 events
54.1
53.7
53.8
Table 5: Accuracy by audio events (%) for thinking mode.
Error Type
Non-Thinking
Thinking
Lateral Direction
18.5
15.4
Attribute Swapping
15.7
14.4
Source Misidentification
14.5
13.3
Distance / Depth
14.2
10.3
Crossing Center
11.7
8.9
Temporal Sequence
7.8
5.5
Table 6: Error category comparison for GT masking: Thinking vs Non-Thinking. Values are in %. The lower error percentage the better.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Thinking
Non-Thinking
Metric
NoMask
AGM
GT
NoMask
AGM
GT
Accuracy (%)
54.3
55.0
56.1
54.1
54.0
54.1
Thinking Avg (1-5)
3.507
3.522
3.549
0.0
0.0
0.0
Rationale Avg (1-5)
4.156
4.168
4.179
4.182
4.182
4.180
Appendix
Table 7: Overall core metrics. Accuracies are %; Thinking/Rationale are 1–5 scales. GT refers to ground - truth masking based source separation, NoMask refers to without using source separation, AGM refers to audio grounding model based source separation.
Thinking
Non-Thinking
Δ Interaction (pp)
QType
Audio Events
NoMask
AGM
GT
NoMask
AGM
GT
AGM
GT
Multiple Choice
Complete Match
53.2
53.0
53.9
50.6
50.0
51.0
+0.4
+0.3
Multiple Choice
1 Event
57.0
56.3
52.7
52.7
54.2
50.2
− 2.2
− 1.8
Multiple Choice
2 Events
51.1
50.9
47.7
48.7
50.0
47.0
− 1.5
− 1.7
Open
Complete Match
35.6
36.1
36.2
36.1
35.5
36.3
+1.1
+0.4
Open
1 Event
52.8
53.2
54.2
52.3
52.3
52.7
+0.4
+1.0
Appendix
Table 8: Question Type × Audio Events. Accuracies are %; Δ Interaction is defined as (AccTmethod−AccNmethod)−(AccTNoMask−AccNNoMask) .
Thinking
Non-Thinking
Δ Interaction (pp)
QType
Overlap
NoMask
AGM
GT
NoMask
AGM
GT
AGM
GT
Yes No
No Overlap
72.2
74.4
78.3
72.6
73.0
72.2
+1.8
+6.5
Yes No
Overlap
72.0
73.2
74.4
72.6
72.6
72.4
+1.2
+2.6
Multiple Choice
No Overlap
53.2
55.3
54.2
49.2
47.6
48.4
+3.7
+1.8
Multiple Choice
Overlap
53.0
50.5
51.4
51.6
53.0
52.2
− 3.9
− 2.2
Open
No Overlap
42.5
43.0
43.8
41.8
41.2
41.9
+1.1
+1.2
Appendix
Table 9: Question Type × Overlap. Accuracies are %;
Question
Model Responses
Which sound spans a wider left–right range? Choices : [Whimper (dog), Tick-tock, They span the same range]
Ground truth answer: Whimper (dog) NoMask without thinking mode
Thinking:
Answer: Tick-tock
Rationale: Tick-tock is described as a wide sweep while the dog’s movement is moderate, so Tick-tock spans more left–right range. NoMask with thinking mode
Thinking: Step 1: Compare the span categories of both sounds. Solution: Both the whimper and tick-tock are described with a wide lateral span. Step 2: Decide which is wider based on those descriptions. Solution: Since both are labeled wide, they span the same wide left–right range. Final reasoning: Both sources have wide sweeps, so neither spans a wider left–right range than the other.
Answer: They span the same range
Appendix
Table 10: Selected samples to illustrate model reasoning and responses across masking regimes
Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.
Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83% average semantic accuracy across the four levels, compared with 37.28% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.
Zhi Zeng, Cheng Zhang, Zesheng Yang +9
Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2
Dynamic spatial audio reasoning requires a model to associate changing acoustic observations with the correct source and recover its spatial state at a queried time or interval, a challenge intensified by overlapping speech and source motion. To study this problem, we introduce Sonic-Bench, comprising 2.8K first-order Ambisonics clips and 28.8K target-specific question-answer pairs across 480 simulated indoor scenes. The benchmark covers localization, distance estimation, and motion reasoning in single- and dual-source settings. We further propose SonicWorld, which integrates physical, learned acoustic, and semantic features into persistent source states through learned transitions and recurrent updates, and projects them into an Omni LLM for spatial reasoning. Compared with Spatial-Omni on the Sonic-Bench test split, SonicWorld improves direction accuracy from 31.98% to 34.69% and reduces azimuth error from 57.22° to 48.01°. In zero-shot transfer to Spatial LibriSpeech, SonicWorld retains strong localization performance, achieving 73.98% direction accuracy and 19.29° azimuth error, compared with 27.14% and 70.74° for Spatial-Omni.
Chenlong Wei, Hanling Wang, Mingjin Lou +6
Xi’an Jiaotong-Liverpool University · University of California, San Diego · Imperial College London +2