AURA: Auditory Spatial Intelligence via a Unified Representation from a General-Audio Encoder through Complete Alignment
Organizations: Southeast University · Rutgers University · University of California, Merced
Abstract
Answering spatial questions about multi-source scenes requires a model both to identify each sound and to locate it. Existing spatial audio-language models obtain spatial information from a dedicated encoder, which separates spatial and semantic processing into distinct feature spaces. We study whether a general-audio codec trained without spatial supervision can serve both purposes. A controlled layer-wise analysis shows that the frozen codec retains spatial information, with lateral cues carried by the difference between the two ear channels. This information, however, is not readily accessible: in dual-source scenes, a model trained on these features predicts left/right at chance and, on the remaining axes, assigns the detected direction to the queried and the non-queried source equally often. We attribute this failure to an audio--text information asymmetry, in which the audio contains both sources while the answer describes only the queried one. A controlled experiment shows that either completing the scene description or exposing the inter-channel difference lifts left/right accuracy above chance, whereas binding directions to the queried source requires both. Building on these findings, AURA keeps the codec and the language model frozen and exceeds the strongest dedicated system on SpatialSoundQA by 10.31 points in dual-source direction accuracy, with about 40% of its trainable parameters.
Figures & tables
| Direction | Dist. | ||||
| Features | DoA | L/R | F/B | U/D | MAE |
| Single source | |||||
| Reference | 15.6 | 50.9 | 50.9 | 59.0 | 1.249 |
| Monaural | 39.5 | 74.0 | 71.7 | 72.1 | 1.051 |
| Binaural | 51.1 | 93.0 | 72.9 | 73.4 | 1.029 |
| Dual source (queried source) | |||||
| Model | Params. | Single Source | Dual Source | Relation | |||
|---|---|---|---|---|---|---|---|
| DoA | DER | DoA | DER | Dir. BA | Dist. BA | ||
| Random | – | 12.57 | 67.33 | 12.41 | 67.46 | 50.00 | 50.00 |
| Majority baseline | – | 15.57 | 49.31 | 13.62 | 71.34 | 50.00 | 50.67 |
| Qwen2-Audio | – | 8.85 | 80.03 | 9.74 | 80.27 | 49.32 | 46.14 |
| Qwen2.5-Omni | – | 10.61 | 69.50 | 12.02 | 79.26 | 49.55 | 54.30 |
| BAT | 161.04M | 72.28 | 64.55 | 36.64 | 70.98 | 68.54 | 71.35 |
| Model | MMAU | MMAR | AIR-Bench |
|---|---|---|---|
| Qwen2-Audio | 55.80 | 40.00 | 48.95 |
| Qwen2.5-Omni | 65.20 | 49.20 | 53.89 |
| BAT | 33.20 | 31.10 | 26.52 |
| DSpAST-BAT † | 31.30 | 29.80 | 26.99 |
| OWL (Stage 2) | 34.90 | 29.20 | 25.38 |
| AURA | 44.50 | 34.80 | 32.71 |
| Stage | Training data | Samples | Single Source | Dual Source | Relation | |||
|---|---|---|---|---|---|---|---|---|
| DoA | DER | DoA | DER | Dir. BA | Dist. BA | |||
| Random | – | – | 12.57 | 67.33 | 12.41 | 67.46 | 50.00 | 50.00 |
| Majority baseline | – | – | 15.57 | 49.31 | 13.62 | 71.34 | 50.00 | 50.67 |
| Stage 1 | single-source | 0.33M | 71.59 | 41.03 | – | – | – | – |
| Stage 2 | + dual-source | 0.85M | 72.09 | 42.77 | 46.15 | 64.74 | – | – |
| Stage 3 (AURA) | + relation questions | 1.94M | 73.58 | 42.69 | 48.21 | 64.72 | 68.86 | 70.65 |
| Direction | Dist. | ||||
| Temporal summary | DoA | L/R | F/B | U/D | MAE |
| Single source | |||||
| Reference | 15.6 | 50.9 | 50.9 | 59.0 | 1.249 |
| Frame-level | 43.4 | 93.0 | 70.4 | 65.5 | 1.152 |
| Global mean | 51.2 | 93.0 | 72.9 | 73.4 | 1.029 |
| Global max | 50.3 | 92.2 | 72.2 | 73.7 | 1.014 |
| Direction | Dist. | ||||
| Zeroed slot | DoA | L/R | F/B | U/D | DER |
| Single source | |||||
| Full model | 73.58 | 93.95 | 86.84 | 85.29 | 42.7 |
| w/o | 60.27 | 85.43 | 81.04 | 78.44 | 53.7 |
| w/o | 71.72 | 93.34 | 85.85 | 84.03 | 43.4 |
| w/o | 64.29 | 88.56 | 83.60 | 79.64 | 51.0 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Parameters |
|---|---|
| Stage 1, alignment | Trained: projector Samples: single-source Steps: Global batch size: Schedule: cosine with warmup steps Learning rate: |
| Stage 1, joint | Trained: projector and LoRA Samples: single-source Steps: Global batch size: Schedule: cosine with warmup steps Learning rate: |
| Stage 2 | Trained: projector and LoRA Samples: single- and dual-source Steps: Global batch size: Schedule: cosine with warmup steps Learning rate: |
| Stage 3 | Trained: projector and LoRA Samples: single-source, dual-source, and relation Steps: Global batch size: Schedule: cosine with warmup steps Learning rate: |
| Scene | Question type | Train | Eval |
|---|---|---|---|
| Single source | Direction and distance | 139,392 | 14,828 |
| Classification | 139,392 | 14,828 | |
| Dual source | Direction and distance | 118,000 | 10,640 |
| Classification | 118,000 | 10,640 | |
| Relation | Binary direction | 120,002 | 12,002 |
| Binary distance | 125,800 | 10,600 |
| Type | Question | Answer |
|---|---|---|
| Direction and distance | How would you describe the location of the speech’s sound in terms of direction and distance? | right, front, below; 0.5m |
| Classification | Enumerate the sound occurrences in the audio clip. | accelerating, revving, vroom; car; vehicle |
| Binary direction | Are you able to detect the orchestra’s sound coming from the right and the cricket’s sounds from the left? | Yes |
| Binary distance | Is the sound of mechanisms further from you than male singing when considering the direct paths? | Yes |
| Non-binary direction | Do you know if the tuning fork’s sound comes from the left or right side of the sound made by the hum? | right |
| Non-binary distance | What is the distance between the sound of the crumpling, crinkling and the sound of the music? | 2m |
| Direction | Dist. | ||||
|---|---|---|---|---|---|
| Features | L/R+F/B | L/R | F/B | U/D | MAE |
| Reference | 50.6 | 61.6 | 56.1 | – | 0.515 |
| Monaural | 62.9 | 77.8 | 72.3 | – | 0.496 |
| Binaural | 67.2 | 84.5 | 74.6 | – | 0.487 |
Audio-Language Models
Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.
See all 324 papers