Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled 2×2 experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40% of its trainable parameters.
Figures & tables
Figure 1: Spatial and general-audio understanding with AURA. Left: illustrative questions and responses covering general-audio understanding, source descriptions, and geometric relations. Source coordinates in responses are model outputs. Right: performance overview for AURA and the evaluated spatial baselines; AURA attains results comparable to dedicated state-of-the-art systems on spatial audio understanding and generalizes well to general-audio understanding. Matterport3D imagery is subject to the Matterport3D Terms of Use .
Figure 2: Layer-wise similarity analysis of RVQ representations. Top: schematic illustration of acoustic variables on Matterport3D scene wc2JMjhGNzB . (a,d) Interaural similarity for simulated (fixed source and BRIR) and real (MRSSpeech) binaural audio; the RVQ vectors diverge markedly as the layer index increases. (b,c) Room variation: right-ear similarity when the same audio is rendered at another placement in the same building or with an impulse response from a different building; the similarity of the RVQ vectors still decreases clearly with depth. (e,f) Full versus early-reflection-only and direct-path-only impulse responses for the same placement, and the additional reduction from removing early reflections. These comparisons reveal increasing sensitivity to spatially relevant acoustic changes in deeper RVQ layers. Scene data under the Matterport3D Terms of Use .
Direction ↑
Dist.
Features
DoA
L/R
F/B
U/D
MAE ↓
Single source
Reference
15.6
50.9
50.9
59.0
1.249
Monaural
39.5
74.0
71.7
72.1
1.051
Binaural
51.1
93.0
72.9
73.4
1.029
Dual source (queried source)
Table 1: Linear probes of spatial attributes from the frozen RVQ representation. The linear classifiers clearly exceed the chance-level reference, showing that the RVQ representation carries information for understanding the spatial relations in audio. Binaural audio further outperforms monaural audio, most markedly on left/right discrimination, where the second channel removes most of the remaining error (26.0% to 7.0%), showing that the interaural cross-channel difference is what makes left/right reliably decodable. Reference rows are constant predictions without audio.
Figure 3: Overall results of the 2×2 controlled experiment on the dual-source direction questions. The four-slot answer-only baseline (red) fails to exceed the majority baseline on left/right discrimination, whereas every other intervention clearly surpasses the baseline level. The best value for each metric is in bold and the second best is underlined; the majority baseline is not ranked. H1: scene supervision; H2: six-slot representation with interaural difference slots; AO: answer-only supervision. Each pie spans 100 points: the grey sector is the majority-baseline level, the blue sector the gain over it, and the number below the pie the metric value (%); for DER ( ∗ ) lower is better, so the pie shows 100−DER .
Figure 4: Dual-source direction accuracy stratified by whether the non-queried source lies on the same side as the queried source along each axis. Except for the full intervention, the direction gains of every intervention are confined to the same-side stratum (open markers); only the full intervention effectively improves the opposite-side questions (filled markers), which require binding the semantic and acoustic attributes to the queried source, lifting them above the majority-baseline level. n is the number of questions per stratum. H1: scene supervision; H2: six-slot representation; AO: answer-only supervision.
Figure 5: AURA architecture. (a) Existing dedicated spatial pathways and general-audio pathways are separate and rely on different feature spaces, so features cannot be shared for unified perception. (b) AURA provides a shared, frozen general-audio codec representation that unifies perception in a single audio feature space. (c) Details of the AURA pipeline: for each ear, RVQ layers L0–L2 and L3–L7 are summed into content-oriented ( S ) and acoustic-oriented ( A ) bands, and adding the band-wise interaural difference yields six slots. A shared projector maps these slots to audio tokens for the language model, which generates a scene description and, for relation questions, a geometric analysis before the final answer.
Figure 6: AURA training-target examples for scene supervision (top) and geometric supervision (bottom). The question is shown in gray; the remaining text is the model’s target.
Model
Params.
Single Source
Dual Source
Relation
DoA ↑
DER ↓
DoA ↑
DER ↓
Dir. BA ↑
Dist. BA ↑
Random
–
12.57
67.33
12.41
67.46
50.00
50.00
Majority baseline
–
15.57
49.31
13.62
71.34
50.00
50.67
Qwen2-Audio
–
8.85
80.03
9.74
80.27
49.32
46.14
Qwen2.5-Omni
–
10.61
69.50
12.02
79.26
49.55
54.30
BAT
161.04M
72.28
64.55
36.64
70.98
68.54
71.35
Table 2: Main results: system-level comparison on SpatialSoundQA. AURA performs comparably to current dedicated spatial audio understanding systems on SpatialSoundQA and achieves the best results on single-source DER, dual-source DoA and DER, and the relative-direction relation questions. All spatial metrics are reported in percentage points. The first group lists references: the random predictor, the majority-class baseline, and the zero-shot general-purpose models. The best result in each column is in bold and the second best is underlined; reference rows are not ranked.
Model
MMAU ↑
MMAR ↑
AIR-Bench ↑
Qwen2-Audio
55.80
40.00
48.95
Qwen2.5-Omni
65.20
49.20
53.89
BAT
33.20
31.10
26.52
DSpAST-BAT †
31.30
29.80
26.99
OWL (Stage 2)
34.90
29.20
25.38
AURA
44.50
34.80
32.71
Table 3: General-audio accuracy (%). AURA generalizes well to general-audio understanding and outperforms the current spatial understanding systems. The two general-purpose models are zero-shot references trained with large-scale audio instruction data and are not ranked; among the spatial systems, the best result is in bold and the second best underlined.
Stage
Training data
Samples
Single Source
Dual Source
Relation
DoA ↑
DER ↓
DoA ↑
DER ↓
Dir. BA ↑
Dist. BA ↑
Random
–
–
12.57
67.33
12.41
67.46
50.00
50.00
Majority baseline
–
–
15.57
49.31
13.62
71.34
50.00
50.67
Stage 1
single-source
0.33M
71.59
41.03
–
–
–
–
Stage 2
+ dual-source
0.85M
72.09
42.77
46.15
64.74
–
–
Stage 3 (AURA)
+ relation questions
1.94M
73.58
42.69
48.21
64.72
68.86
70.65
Table 4: Results on SpatialSoundQA at the end of each curriculum stage, in percent. Each stage is initialized from the previous one; “Samples” is the cumulative number of training samples. A dash means the task has not been introduced at that stage and was not evaluated. The majority baseline answers the majority class on each axis and the best constant distance.
Direction ↑
Dist.
Temporal summary
DoA
L/R
F/B
U/D
MAE ↓
Single source
Reference
15.6
50.9
50.9
59.0
1.249
Frame-level
43.4
93.0
70.4
65.5
1.152
Global mean
51.2
93.0
72.9
73.4
1.029
Global max
50.3
92.2
72.2
73.7
1.014
Table 5: Effect of the temporal pooling scheme on the linear readability of the features. Reference rows are the majority-class baseline. Frame-level uses every frame as a probe row and averages the frame scores over the clip; global mean and global max pool each slot over the whole clip; AURA is the windowed, band-specific scheme ( w=4 , max for content-oriented and mean for acoustic-oriented slots) followed by averaging over windows, so that each clip is one vector. Protocol, data, and columns as in Table 1 . The best value in each column is in bold and the second best is underlined.
Direction ↑
Dist.
Zeroed slot
DoA
L/R
F/B
U/D
DER ↓
Single source
Full model
73.58
93.95
86.84
85.29
42.7
w/o LS
60.27
85.43
81.04
78.44
53.7
w/o LA
71.72
93.34
85.85
84.03
43.4
w/o RS
64.29
88.56
83.60
79.64
51.0
Table 6: Slot-wise ablation: zeroing each of the six slots in turn at inference in the final AURA model, on the single-source and dual-source SSQA evaluation sets (direction accuracies and DER in percent). The zeroed token stays in the sequence at its position. All six slots of AURA contribute noticeably to the final performance. The best value in each column is in bold and the second best is underlined.
Figure 7: Text-to-audio attention across layers. Attention concentrates on more specific frames with depth and, depending on the question, is finally directed to different slots. Columns show the first (layer 0 ), an intermediate (layer 15 ), and the final (layer 35 ) transformer layer after prefill. Attention is averaged over heads. In each panel, the wide map sums attention over the six slots at each audio frame and the narrow map sums it over frames for each slot; rows are text tokens. Each map is normalized to its own maximum.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Parameters
Stage 1, alignment
Trained: projector Samples: 55,008 single-source Steps: 1,719 Global batch size: 32 Schedule: cosine with 100 warmup steps Learning rate: 10−4
Stage 1, joint
Trained: projector and LoRA Samples: 278,784 single-source Steps: 17,424 Global batch size: 16 Schedule: cosine with 100 warmup steps Learning rate: 10−4
Stage 2
Trained: projector and LoRA Samples: 514,784 single- and dual-source Steps: 32,174 Global batch size: 16 Schedule: cosine with 100 warmup steps Learning rate: 10−4
Stage 3
Trained: projector and LoRA Samples: 872,312 single-source, dual-source, and relation Steps: 54,520 Global batch size: 16 Schedule: cosine with 100 warmup steps Learning rate: 10−4
Appendix
Table 7: Training stages of AURA. Each stage covers one epoch.
Scene
Question type
Train
Eval
Single source
Direction and distance
139,392
14,828
Classification
139,392
14,828
Dual source
Direction and distance
118,000
10,640
Classification
118,000
10,640
Relation
Binary direction
120,002
12,002
Binary distance
125,800
10,600
Appendix
Table 8: Number of training and evaluation questions per type.
Type
Question
Answer
Direction and distance
How would you describe the location of the speech’s sound in terms of direction and distance?
right, front, below; 0.5m
Classification
Enumerate the sound occurrences in the audio clip.
accelerating, revving, vroom; car; vehicle
Binary direction
Are you able to detect the orchestra’s sound coming from the right and the cricket’s sounds from the left?
Yes
Binary distance
Is the sound of mechanisms further from you than male singing when considering the direct paths?
Yes
Non-binary direction
Do you know if the tuning fork’s sound comes from the left or right side of the sound made by the hum?
right
Non-binary distance
What is the distance between the sound of the crumpling, crinkling and the sound of the music?
2m
Appendix
Table 9: Example questions and answers from the original SSQA dataset.
Figure 8: Layer-wise similarity on reconstruction-only codecs. Left: DAC; right: EnCodec. Top: interaural similarity for simulated (fixed source and BRIR) and real (MRSSpeech) binaural audio; the RVQ vectors diverge markedly as the layer index increases. Bottom: room variation: right-ear similarity when the same audio is rendered at another placement in the same building or with an impulse response from a different building; the similarity of the RVQ vectors still decreases clearly with depth. These comparisons reveal that the increasing sensitivity to interaural and room changes in deeper RVQ layers is not specific to X-Codec.
Direction ↑
Dist.
Features
L/R+F/B
L/R
F/B
U/D
MAE ↓
Reference
50.6
61.6
56.1
–
0.515
Monaural
62.9
77.8
72.3
–
0.496
Binaural
67.2
84.5
74.6
–
0.487
Appendix
Table 10: Linear probes of spatial attributes on real binaural recordings (MRSSpeech), with the features of Table 1 . Test speakers and every drama they appear in are excluded from training, and monaural features use the left ear. L/R+F/B requires both axes to be correct. Up/down has no entries because every annotated source lies on one side of the listener, so the axis has a single class. Reference rows are constant predictions without audio.
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
Sound events have semantic identities, locations, and trajectories, yet existing audio-language models typically encode clips as global event content. Conversely, sound event localization models track source directions over time but offer limited semantic coverage for language reasoning. To bridge this gap, we introduce ST-AudioQA, a spatio-temporal audio QA dataset and benchmark built from first-order ambisonic (FOA) renderings of static and moving sound sources. Each scene provides source identity, activity, direction, distance, and motion metadata, enabling dense trajectory supervision and QA about what is sounding, where it is, how it moves, and how sources relate. We further introduce ST-Audio Encoder, which jointly learns event semantics and time-resolved source trajectories, and ST-AudioLM, which exposes these audio tokens to an LLM for spatio-temporal audio QA. Experiments show that this representation improves the balance between semantic preservation and localization and yields stronger reasoning than static spatial and localization-oriented baselines. Beyond the controlled benchmark, ST-AudioLM shows zero-shot transfer to real recordings, with further gains after light adaptation. Project page: https://st-audiolm.github.io/.
Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.