Organizations: Xi’an Jiaotong-Liverpool University · University of California, San Diego · Imperial College London · Lyuliang University · Southeast University
Dynamic spatial audio reasoning requires a model to associate changing acoustic observations with the correct source and recover its spatial state at a queried time or interval, a challenge intensified by overlapping speech and source motion. To study this problem, we introduce Sonic-Bench, comprising 2.8K first-order Ambisonics clips and 28.8K target-specific question-answer pairs across 480 simulated indoor scenes. The benchmark covers localization, distance estimation, and motion reasoning in single- and dual-source settings. We further propose SonicWorld, which integrates physical, learned acoustic, and semantic features into persistent source states through learned transitions and recurrent updates, and projects them into an Omni LLM for spatial reasoning. Compared with Spatial-Omni on the Sonic-Bench test split, SonicWorld improves direction accuracy from 31.98% to 34.69% and reduces azimuth error from 57.22° to 48.01°. In zero-shot transfer to Spatial LibriSpeech, SonicWorld retains strong localization performance, achieving 73.98% direction accuracy and 19.29° azimuth error, compared with 27.14% and 70.74° for Spatial-Omni.
Figures & tables
Figure 1: Task overview of Sonic-Bench. Given an FOA recording, the task is to infer a specified source’s spatial attributes at a queried time or its movement direction over a queried interval. Time-aligned source trajectories provide ground-truth answers, illustrated here with two moving sources and a fixed receiver.
Figure 2: SonicWorld architecture. Physical, neural, and frozen-BEATs semantic evidence are concatenated before recurrent source binding. Smoothed states and decoded geometry are projected into spatial tokens, which join audio and question tokens at the language model.
Single source
Dual source
Model
Dir. ↑
Az. ↓
El. ↓
Dist. ↓
Mot. ↑
Dir. ↑
Az. ↓
El. ↓
Dist. ↓
Mot. ↑
Closed-source models
Gemini 3 Flash
12.81
88.55
4.40
1.11
7.81
15.47
88.23
4.49
2.71
8.28
Gemini 3.1 Pro
9.06
90.87
15.78
1.86
12.19
14.53
90.32
28.88
2.03
14.38
Open-source models
Qwen2.5-Omni-7B
12.50
91.18
84.91
98.83
51.88
12.66
88.33
83.58
34.30
48.59
Table 1: Experimental results on Sonic-Bench. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B. Dir./Mot. are label accuracies (%); Az./El. are MAE in degrees and Dist. is MAE in meters. Best values in each column are bold.
SpatialSceneQA
SLS
Model
Az. ↓
El. ↓
Dir. ↑
Az. ↓
El. ↓
Dist. ↓
Open-source models
Qwen2.5-Omni-7B
60.47
80.14
12.69
90.26
88.47
55.04
InternOmni
80.54
78.05
12.38
89.69
57.59
98.83
Phi-4-MM-5.6B
61.86
18.31
12.61
89.43
7.08
1.99
Audio Flamingo 3
132.66
24.92
11.87
89.92
28.08
33.40
Table 2: Zero-shot transfer to SpatialSceneQA and SLS. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B.
Physical
Neural
BEATs
BiGRU
Dir. ↑
Mot. ↑
Az20 ↑
Az. ↓
El. ↓
Dist. ↓
4
8
8
4
28.33
52.40
25.73
55.68
0.97
0.62
4
4
8
4
28.65
52.19
26.88
55.59
0.99
0.62
4
4
4
8
32.19
51.98
33.13
48.11
0.96
0.63
4
4
4
4
34.69
53.44
34.90
48.01
0.92
0.59
Table 3: Module ablations with Qwen2.5-Omni-7B. 4 denotes enabled and 8 disabled.
Variant
Dir. ↑
Mot. ↑
Az20 ↑
Az. ↓
El. ↓
Dist. ↓
w/o transition MLP
33.44
51.46
32.71
49.94
0.96
0.64
w/o prior-state supervision
24.48
51.67
22.19
61.25
1.05
0.65
w/o masked-state loss
33.02
52.81
30.21
50.66
0.98
0.64
w/o persistent source slots
33.65
52.81
25.94
54.13
1.09
0.68
Full model
34.69
53.44
34.90
48.01
0.92
0.59
Table 4: Ablations of source-state modeling with Qwen2.5-Omni-7B.
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Unit
Train
Test
Total
Scenes
400
80
480
Single-source clips
800
160
960
Dual-source clips
1,600
320
1,920
Source trajectories
4,000
800
4,800
Question–answer pairs
24,000
4,800
28,800
Audio duration
6 h 40 min
1 h 20 min
8 h
Appendix
Table A.1 : Sonic-Bench statistics by split. Each clip contains two target query times and five questions per query time.
Resource
Format
Scale
Motion
Spatial supervision
Queries
SpatialSoundQA [ 7 ]
Binaural
872K QA
Static
Static source geometry
Source attributes / relations
SpatialSceneQA [ 9 ]
FOA, RGB-D
61K samples
Static
Static 3D poses
Source identity / visual grounding
SAVVY-Bench [ 12 ]
7-ch audio, video
1,522 QA
Static / Moving
Camera-derived speaker tracks
Sound event-anchored
STAR-Bench [ 5 ]
Binaural
2,353 QA
Static / Moving
Motion / trajectory QA labels
Motion-pattern reasoning
SO-Bench [ 10 ]
FOA
7K QA
Static / Moving
Linked SO-Dataset tracks
Task-dependent source / time
Sonic-Bench (ours)
FOA
28.8K QA
Static / Moving
Full-clip source tracks
Target- and time-specific
Appendix
Table A.2 : Comparison with spatial-audio resources. Scale uses each resource’s reporting unit; Spatial supervision refers to geometry available during construction or training.
Parameter
Setting
Duration / sample rate
10 s / 16 kHz
Geometric sampling
12 Hz; 121 frames (including endpoints)
Receiver height
1.50 m above NavMesh
Source height
1.55 m above NavMesh
Source–receiver range
1.5–4 m along the path
Moving-path length
2–5 m
Appendix
Table A.3 : Scene and acoustic generation parameters.
Figure A.1 : Target-bearing sectors and clip distribution. Each sector spans 45∘ and contains 12.5% of the clips in each split.
Task
Example question
Answer
Position direction
At 3.00 s, which direction is the female voice coming from?
Front
Azimuth
At 6.00 s, what is the azimuth of the female voice?
−9.47∘
Elevation
At 4.50 s, what is the vertical angle of the male voice?
+2.25∘
Distance
At 7.00 s, how far away is the male voice?
3.12 m
Movement direction
Between 1.50 and 2.50 s, which direction did the female source move relative to the listener’s fixed orientation?
Front-Right
Appendix
Table A.5 : Examples of target-specific spatial questions.
Quantity
Statistic
Train
Test
Source range (m)
Minimum
1.50
1.50
Maximum
4.00
4.00
Mean
2.85
2.82
Median
2.90
2.87
Elevation ( ∘ )
Minimum
-16.76
-13.45
Maximum
19.94
20.30
Appendix
Table A.6 : Source geometry and trajectory statistics.
Subset
S
F
FL
L
BL
B
BR
R
FR
Total
Single
160
15
30
14
23
23
21
13
21
320
Dual
320
55
34
42
37
46
38
40
28
640
Total
480
70
64
56
60
69
59
53
49
960
Appendix
Table A.7 : Test movement-direction distribution. Counts denote questions in the single- and dual-source subsets. S denotes Stationary; F/B/L/R denote Front/Back/Left/Right, with combined letters indicating diagonal directions. Stationary labels account for half of each subset.
Figure A.2 : Target geometry at query times. Distributions use 4,800 training and 960 test query times, expressed as percentages within each split. The upper panels group azimuth into 45∘ bins starting at 0∘ and distance into 0.5 m bands from 1.5 to 4.0 m. The lower panels show elevation below 0∘ , in [0∘,2∘) , and at least 2∘ .
Layer
Channels
Kernel
Stride
1
1→48
10
5
2
48→96
7
4
3
96→128
5
4
4
128→192
5
4
Appendix
Table B.1 : Shared waveform encoder. Each convolution uses one-group normalization and GELU.
Input
Dimensions
Prior, observation, recurrent state, refined state, observation minus prior
5×256
Direction anchor; predicted direction and distance
Table B.2 : Projector input per source and time step.
Module
Linear widths
Frame encoder
1295→512→512
Local temporal compressor
2048→2048→512
Language-space projector
512→2048→3584
Appendix
Table B.3 : Projector MLPs. Each uses layer normalization at its input and output, with SiLU between the two linear layers.
Objective
Weights
Trajectory supervision
Direction 1; log-distance 1; presence 1; uncertainty 0.05; prior 0.20
Masked center prediction
Direction 1; log-distance 0.3; presence 0.2
Projector training
Answer cross-entropy 1; geometric alignment 0.5
Projector + LoRA
Answer cross-entropy 1; geometric alignment 0.25
Appendix
Table C.1 : Loss coefficients.
Representation
Shape
FOA waveform, [W,Y,Z,X]
[B,4,160000]
Physical input features
[B,13,64,121]
Physical tokens
[B,121,64,192]
Neural tokens / semantic tokens
[B,121,1,192]
Combined acoustic tokens
[B,121,66,192]
Slot–token weights
[B,3,66]
Appendix
Table C.2 : Model representations from input to output. B denotes batch size. Slot–token weights include two source slots and one background slot; subsequent source representations retain only the two source slots.
Stage
Epochs
Accum.
Updates
Trainable modules and peak learning rate
Source states
25
2
30,000
Physical and neural branches, semantic pooling, recurrent filter, bidirectional GRU, state heads: 10−4
Projector
5
8
15,000
Projector and alignment heads: 10−4
Projector + LoRA
3
8
9,000
Projector : 3×10−5 ; LoRA: 5×10−5
GPUs
3× NVIDIA RTX 4090; 6× NVIDIA RTX 3090
Appendix
Table C.3 : SonicWorld training stages. Updates count optimizer steps.
Backbone
Dir. ↑
Az. ↓
El. ↓
Dist. ↓
Mot. ↑
Single source
Qwen2.5-Omni-7B
50.00 ( +37.50 )
30.85 ( −60.33 )
0.91 ( −84.00 )
0.56 ( −98.27 )
52.50 ( +0.62 )
InternOmni
51.88 ( +39.07 )
36.49 ( −57.23 )
0.92 ( −21.73 )
0.61 ( −39.97 )
55.94 ( +47.81 )
Phi-4-MM-5.6B
48.75 ( +35.31 )
32.03 ( −49.18 )
0.92 ( −2.37 )
0.53 ( −1.40 )
53.44 ( +47.19 )
Dual source
Qwen2.5-Omni-7B
27.03 ( +14.37 )
56.59 ( −31.74 )
0.93 ( −82.65 )
0.60 ( −33.70 )
53.91 ( +5.32 )
Appendix
Table D.1 : Effect of SonicWorld on different backbones on Sonic-Bench. Values in parentheses indicate changes relative to the corresponding original backbone without SonicWorld.
SpatialSceneQA
SLS
Backbone
Az. ↓
El. ↓
Dir. ↑
Az. ↓
El. ↓
Dist. ↓
Qwen2.5-Omni-7B
21.35 ( −39.12 )
20.51 ( −59.63 )
73.98 ( +61.29 )
19.29 ( −70.97 )
7.09 ( −81.38 )
0.89 ( −54.16 )
InternOmni
22.57 ( −57.97 )
20.58 ( −57.47 )
66.80 ( +54.42 )
22.20 ( −67.49 )
7.08 ( −50.51 )
1.24 ( −97.59 )
Phi-4-MM-5.6B
19.79 ( −42.07 )
20.34 ( +2.03 )
73.44 ( +60.83 )
18.22 ( −71.21 )
7.95 ( +0.87 )
0.81 ( −1.18 )
Appendix
Table D.2 : Effect of SonicWorld on zero-shot transfer across backbones. Values in parentheses indicate changes relative to the corresponding original backbone without SonicWorld.
Single source
Dual source
Model
Az20 ↑
El1 ↑
D0.5 ↑
Az20 ↑
El1 ↑
D0.5 ↑
Closed-source models
Gemini 3 Flash
10.00
30.31
25.94
11.72
27.34
26.41
Gemini 3.1 Pro
12.19
31.88
1.88
12.03
18.59
6.25
Open-source models
Qwen2.5-Omni-7B
14.06
1.88
0.00
11.72
1.09
0.00
Appendix
Table D.3 : Localization tolerance accuracies. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B. Az20, El1, and D0.5 denote azimuth, elevation, and distance accuracies within 20∘ , 1∘ , and 0.5 m, respectively.
Sonic- Bench
Spatial SceneQA
SLS
Evaluated clips
480
622
8,512
Single-source clips
160
247
8,512
Dual-source clips
320
375
0
Motion state
Stationary / Moving
Stationary
Stationary
Questions per evaluated clip
10
2
4
Single-source questions
1,600
494
34,048
Appendix
Table D.4 : Evaluation-set scale and selection. Sonic-Bench uses its full test set; SpatialSceneQA and SLS use the selected test subsets. Dir., Az., El., Dist., and Mot. denote direction, azimuth, elevation, distance, and movement direction.
Single source
Dual source
Model
Az. ↓
El. ↓
Az20 ↑
El1 ↑
Az. ↓
El. ↓
Az20 ↑
El1 ↑
Open-source audio models
Qwen2.5-Omni-7B
20.54
100.49
53.85
0.00
86.77
66.73
23.20
0.00
InternOmni
75.35
83.90
5.26
0.00
83.89
74.26
4.53
0.00
Phi-4-MM-5.6B
78.49
16.42
10.93
0.40
47.69
21.14
30.13
0.00
Audio Flamingo 3
117.98
27.04
14.17
0.40
145.38
22.64
3.47
0.00
Appendix
Table D.5 : Single- and dual-source SpatialSceneQA results. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B.
Model
Dir. ↑
Az. ↓
El. ↓
Dist. ↓
Open-source models
Qwen2.5-Omni-7B
12.01
90.68
14.40
19.61
InternOmni
12.76
89.19
54.55
7.12
Phi-4-MM-5.6B
12.48
90.42
7.04
0.96
Audio Flamingo 3
12.31
89.46
16.39
3.11
Kimi-Audio-7B-Instruct
11.83
90.81
8.59
2.81
Appendix
Table D.6 : Results on SLS with official spatial interference. Evaluation covers 8,512 clips and 34,048 questions. Spatial baselines and SonicWorld use Qwen2.5-Omni-7B. Dir. is accuracy (%); Az./El. are MAE in degrees and Dist. is MAE in meters. Best values are bold.
Figure E.1 : Movement predictions with a stationary interferer.
Figure E.2 : Movement predictions for a male target with a moving interferer.
Figure E.3 : Decoded states of a single moving source. Solid lines show decoded trajectories and dashed lines show ground truth. The scene view shows the spatial paths; the time series show receiver-centered front and left coordinates and source distance. Circles and triangles mark trajectory starts and ends.
Figure E.4 : Decoded states of two moving sources. Colors distinguish the sources. Solid lines show decoded trajectories and dashed lines show ground truth.
Figure E.5 : Decoded states of a moving source and a stationary source. Colors distinguish the sources. Solid lines show decoded trajectories and dashed lines show ground truth.
Variant
ADE ↓
ID retention ↑
Stationary interferer
w/o transition MLP
0.82
0.00
w/o masked-state loss
0.66
9.09
Full SonicWorld
0.52
30.30
Moving interferer
w/o transition MLP
0.33
14.81
Appendix
Table E.1 : Source-state continuity across speech pauses. ADE (m) and ID retention (%) are evaluated after speech resumes, with pre-pause slot assignments fixed. ID retention uses distance thresholds of 0.5 m and 0.25 m for stationary and moving interferers, respectively.
Figure E.6 : Tracking across a speech pause with a stationary interferer. Blue/gold/green show the decoded target trajectory before/during/after the pause; A/B mark pause onset/resumption. Dark dashed/purple dotted lines show target/interferer ground truth. Right panels show speech activity, receiver-centered front coordinate, and target position error.
Figure E.7 : Tracking across a speech pause with a moving interferer. Both speakers continue moving during the target’s 1.08 s speech pause. Colors and markers follow Figure E.6 . The coordinate panel shows the receiver-centered left position.
Sound events are entities with semantic identities, locations, and trajectories, but current audio-language models usually reason about clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To address this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and questions about what is sounding, where it is, how it moves, and how sources relate. We further propose ST-Audio Encoder, a time-resolved FOA audio encoder that learns event semantics together with source trajectories, and ST-AudioLM, which connects the audio tokens from the encoder to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the semantic-localization tradeoff and yields stronger reasoning performance than static spatial and localization-oriented baselines.
Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83% average semantic accuracy across the four levels, compared with 37.28% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.
Zhi Zeng, Cheng Zhang, Zesheng Yang +9
Xi’an Jiaotong University · The Hong Kong Polytechnic University · National University of Singapore +2
Spatial audio understanding aims to enable machines to interpret complex auditory scenes, particularly when sound sources move over time. In this work, we study Spatial Audio Question Answering (Spatial AQA) with a focus on movement reasoning, where a model must infer object motion, position, and directional changes directly from stereo audio. First, we introduce a movement-centric spatial audio augmentation framework that synthesizes diverse motion patterns from isolated mono audio events, enabling controlled and scalable training data generation. Second, we propose an end-to-end multimodal finetuning approach with a thinking mode, which allows audio-language models to produce explicit intermediate reasoning steps before predicting an answer. Third, we investigate the impact of query-conditioned source separation as a preprocessing stage and compare three inference regimes: no masking, an audio grounding model (AGM), and ground-truth masks. Our results show that reasoning amplifies the benefits of source separation, with thinking mode showing significant improvement of +5.1% when a single event is present in the question. These findings highlight the interplay between movement modeling, reasoning, and separation quality, offering new insights for advancing spatial audio understanding.