Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
Figures & tables
Figure 1: SpheraAudio overview. The audio encoder AuT and spatial encoder Spatial AuT extract complementary features from the same recording. Separate projectors produce audio tokens and spatial tokens, which are paired in time and interleaved. SMRoPE encodes their temporal correspondence and modality identities for the language model.
Figure 2: Schematic training framework. Spatial-encoder pretraining learns temporally resolved geometry. Projector alignment connects spatial features to the pretrained language representation. Joint SFT & RL combines spatial and acoustic evidence through supervised answer learning and group-relative optimization with verifiable outcome rewards.
Figure 3: SpheraQA. The task taxonomy spans single-attribute perception, cross-source relations, and spatial reasoning. Examples connect source observations to comparisons and geometric operations.
Task / benchmark
N
Spatial- Omni-7B
Gemini 3.1 Pro
SpheraAudio
Δ (pp)
(a) SpheraQA by question type
Azimuth
1,563
22.39
9.72
55.60
+33.21
Elevation
1,556
35.86
32.46
69.34
+33.48
Distance
1,564
49.55
55.75
74.81
+25.26
Source count
1,107
46.52
54.29
60.70
+14.18
Lateral position
1,315
29.66
44.64
41.52
+11.86
Table 1: Spatial understanding (%). SpheraQA reports whole-answer accuracy, with Spatial Reasoning pooling six reasoning question types. SO-Bench reports the mean per-question score. Gemini receives only W . N counts questions and Δ is SpheraAudio minus Spatial-Omni-7B in percentage points before rounding. Best scores are in bold.
Spatial MAE ↓
Speech
Spatial Acc. (%) ↑
Variant
Azimuth ( ∘ )
Elevation ( ∘ )
Distance (m)
AISHELL CER (%) ↓
Left/right
Motion state
Trajectory
Element-wise addition
13.47
7.93
1.32
5.34
91.19
82.85
64.60
Token interleaving
4.79
5.97
1.20
2.81
94.82
87.23
65.49
Table 4: Component ablations. (a) Token fusion in single-source scenes: spatial mean absolute error (MAE), AISHELL CER, and task accuracy. (b) Named-source direction under token-stream shifts. (c) Timed and untimed queries under SMRoPE and sequential RoPE. Panels (b,c) evaluate a fixed SFT checkpoint on identical confirmation recordings, with N questions per task (Appendix C.1 ).
Task
N
SFT
RL
Δ
TWNM / stereo transfer: Acc. (%)
Distance estimation
43
41.86
62.79
+20.93
Distance comparison
60
46.67
51.67
+5.00
Azimuth direction
43
83.72
86.05
+2.33
Lateral position (stereo)
73
91.78
93.15
+1.37
Motion (stereo)
272
37.13
37.87
+0.74
Table 5: Task-level gains after multiple-choice RL. Each supervised policy is compared with its direct RL continuation. Values are TWNM and stereo accuracy or SO-Bench mean per-question score (%); Δ is the percentage-point gain before rounding.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Recording
Front-end operation
AuT signal
Spatial AuT signals
Tokens / chunk
Mono
Audio pathway
a
—
13
Stereo
Mid–side transform
Wm
[Wm,Ym,0,0]
26
FOA
Native first-order components
W
[W,Y,Z,X]
26
Higher-order Ambisonics
External first-order extraction
W
[W,Y,Z,X]
26
Calibrated array
External array-to-Ambisonics encoding
W
[W,Y,Z,X]
26
Appendix
Table 6: Input routing and projected token counts. Waveform signals are shown before spectrogram conversion; a denotes the mono waveform. Counts refer to a complete one-second chunk and exclude text and boundary tokens.
Setting
Value
Updates
1000
Questions per update
32
Responses per question
8
Learning rate (constant)
5×10−7
Adam coefficients
(0.9,0.999)
Weight decay
0.01
Appendix
Table 10: GRPO configuration for the single outcome-based RL stage. The final SFT policy initializes training and remains fixed as the KL reference.
Source
Recordings
N
Simulated scenes
6,828
7,047
LOCATA
63
1,140
STARSS23
168
12,305
Appendix
Table 11: Recording sources in SpheraQA. Recordings are counted uniquely within each source, and N denotes the number of questions.
Question type
Answer format
N
L1 Perception
Azimuth
Numeric angle
1,563
Elevation
Numeric angle
1,556
Distance
Numeric distance
1,564
Source count
Integer
1,107
Lateral position
Direction category
1,315
Appendix
Table 12: SpheraQA question types, answer formats, and counts across the L1–L3 hierarchy, corresponding to Table 5.1 .
Benchmark
N
Metric
Result (%)
SpheraQA
25
Whole-answer Acc.
28.00
SO-Bench
25
Mean Score
34.47
TWNM
25
Option Acc.
60.00
Stereo
25
Acc.
0.00
Appendix
Table 17: Waveform-analysis diagnostic on 25 selected questions per benchmark.
Figure 4: Responses to sound-field transformations on 223 fixed azimuth questions. (a) MAE across all nine yaw rotations. (b) Left–right reflection and removal of directional channels. Rotation and reflection use transformed reference directions; channel removal uses original reference directions. The same supervised model is evaluated across all conditions.
Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding. We propose Spatial-Omni, a lightweight method that implements SO-Encoder to inject First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality, without modifying their original audio encoders. SO-Encoder provides spatial tokens with limited additional context cost and improves spatial audio understanding through efficient staged training. To support training and evaluation, we construct SO-Dataset, SO-QA, and SO-Bench from open-source data, real recordings, and simulations, containing 400K FOA spatial audio clips and 2.1M spatial question answering pairs. SO-Bench covers 16 spatial audio understanding subtasks, including basic detection and location estimation, spatial relation understanding, and complex spatial reasoning. Experiments show that Spatial-Omni outperforms existing open-source Large Audio-Language Models (LALMs) and Omni LLM models on spatial audio understanding tasks while retaining a reasonable level of general audio understanding. Code and data are available at https://github.com/dieKarotte/Spatial-Omni.
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled 2×2 experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40% of its trainable parameters.
Boyan Han, Xiaoyan Wang, Zhe Sun +6
Southeast University · Rutgers University · University of California, Merced
Large audio-language models (LALMs) have achieved strong performance in general audio understanding, yet most are designed for monaural input and discard the inter-channel cues essential for spatial perception. In contrast, existing spatial audio-language models are purpose-built for spatial tasks and fail to capitalize on the general understanding capabilities of monaural LALMs. We present MiDashengLM-Spatial, the first open-source end-to-end unified audio-language model, to our knowledge, which supports both general audio understanding and spatial awareness within a single architecture. It extends MiDashengLM with a spatial audio encoder, Spatial-Dasheng, integrated through a hierarchical semantic-to-spatial conditioning module that injects intermediate semantic representations into the spatial branch at multiple depths while preserving the original semantic pathway. To provide spatial audio-language supervision at scale, we develop a data synthesis pipeline that renders diverse spatial acoustic scenes with scene-level spatial descriptions and question-answer pairs. Experiments show that Spatial-Dasheng achieves strong performance on sound event localization and detection in real-world scenes, and that MiDashengLM-Spatial substantially outperforms existing LALMs on spatial understanding and reasoning benchmarks. Meanwhile, it remains competitive with state-of-the-art 8B-scale LALMs on diverse monaural benchmarks, demonstrating that spatial awareness can be acquired without compromising general audio understanding. The source code and model checkpoint are available at https://github.com/xiaomi-research/midashenglm-spatial and https://huggingface.co/mispeech/midashenglm-spatial.