The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
Organizations: School of Intelligence Science and Technology, Peking University, Beijing, China
Abstract
Large audio-language models have made rapid progress in recognizing what is present in an audio clip, but spatial audio-language understanding still lacks a clear task interface. A model must also decide where sound events occur, which semantic and spatial attributes belong to the same auditory object, how multiple objects are arranged, and whether a scene-level answer is physically plausible. We formalize this capability as audio scene analysis (ASA), a three-level problem spanning atomic perception, relational integration, and cognitive reasoning. We propose The World is Not Mono (TWNM), a framework that equips audio-language models with explicit spatial evidence. TWNM uses physically grounded First-Order Ambisonics (FOA) simulation for controllable supervision, learns slot-regularized spatial representations from multichannel audio, fuses them with semantic audio features, and trains with a progressive curriculum ending in preference optimization over metadata-derived correct answers and auxiliary format/evidence rewards. To operationalize ASA, we build a controlled benchmark from scene metadata, covering localization, attribute binding, spatial comparison, scene abduction, and counterfactual reasoning. On this benchmark, TWNM achieves 70.8% overall accuracy, 66.4% on spatial-family tasks, and 79.76% on mixed scene-level QA under exact multiple-choice question answering (MCQA) scoring. We also audit monaural and binaural reference systems as diagnostic references with explicit audit labels, since they differ in spatial input, training interface, and output format. The supported claim is that a clearly defined ASA hierarchy, FOA-conditioned spatial representations, and metadata-grounded training enable controlled, auditable spatial audio-language reasoning, with STARSS23 providing a limited real-recording diagnostic.
Figures & tables
| Domain | Method | Count Acc. (%) | DOA MAE ( ∘ ) | Dist. MAE (m) |
|---|---|---|---|---|
| Simulation | EINV2 | 76.8 | 30.4 | 0.642 |
| Sp-ACCDOA | 64.9 | 26.0 | 0.506 | |
| TWNM Encoder | 89.4 | 7.9 | 0.270 | |
| STARSS23 | EINV2 | 58.5 | 35.6 | 0.399 |
| Sp-ACCDOA | 54.1 | 33.0 | 0.434 | |
| TWNM Encoder (Sim-to-Real) | 81.7 | 22.5 | 0.380 |
| Model / Stage | Input | Audit | ASA Competency Levels | Overall (%) | ||
|---|---|---|---|---|---|---|
| Random | – | Analytic | 25.00 | 25.00 | 25.00 | 25.00 |
| BAT-SFT ( Zheng et al., 2025 ) | Binaural | Semantic judge | 26.23 | 21.51 | 27.08 | 25.20 |
| Qwen2-Audio-7B ( Chu et al., 2024 ) | Mono | Semantic judge | 52.40 | 42.40 | 48.80 | 48.40 |
| Qwen2.5-Omni-7B ( Xu et al., 2025 ) | Mono | Exact avg. | 58.44 | 49.64 | 66.97 | 58.85 |
| Step-Audio-R1.1 | Mono | Exact parse | 59.22 | 48.03 | 71.13 | 60.10 |
| Model | Spatial families (%) | Semantic families (%) |
|---|---|---|
| Qwen2-Audio-7B | 42.63 | 55.63 |
| Step-Audio-R1.1 | 45.32 | 78.60 |
| Qwen3-Omni-Flash | 46.58 | 71.85 |
| Qwen2.5-Omni-7B | 47.30 | 73.31 |
| GPT-audio | 50.36 | 68.69 |
| Gemini-3.1-Pro Preview | 49.28 | 78.60 |
| Level | Task family | BAT-SFT | TWNM-Align | TWNM-SFT | TWNM-SAPO |
|---|---|---|---|---|---|
| Count | 36.36 | 65.45 | 67.27 | 60.00 | |
| Acoustics | 7.27 | 61.82 | 78.18 | 69.09 | |
| Distance | 25.45 | 38.18 | 56.36 | 70.91 | |
| Event class | 18.18 | 61.82 | 58.18 | 83.64 | |
| Speech | 21.82 | 41.82 | 80.00 | 60.00 | |
| Azimuth | 27.27 | 54.55 | 52.73 | 54.55 |
| Stage | All | |||
|---|---|---|---|---|
| TWNM-Align | 53.25 | 29.39 | 43.15 | 43.20 |
| TWNM-SFT | 65.19 | 70.61 | 51.19 | 62.00 |
| TWNM-SAPO | 63.64 | 69.89 | 79.76 | 70.80 |
| Run | (%) | (%) | (%) | Overall (%) | |
|---|---|---|---|---|---|
| Qwen2.5-Omni-7B | 500 | 41.90 | 40.48 | 36.89 | 40.20 |
| TWNM-SFT | 500 | 41.90 | 38.10 | 47.54 | 42.00 |
| TWNM-SAPO | 500 | 45.24 | 38.69 | 49.18 | 44.00 |
| TWNM-SAPO, Gaussian noise | 500 | 21.90 | 25.60 | 32.79 | 25.80 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus | Role and construction |
|---|---|
| Full-simulation corpus | 50,000 scenes and 75,150 source instances for controlled spatial encoder training and analysis. Dry events are sampled from FSD50K development audio [ Fonseca et al., 2022 ] ; labels are mapped to the official 200-class vocabulary, with multi-label event annotations preserved. |
| SFT render corpus | 50,000 scenes and 84,781 source instances for supervised audio-language training, producing 45,000 instruction-tuning examples after QA filtering. Source domains are sampled with event/speech/music weights of 5:3:2 from ClothoV2, LibriTTS-R, and SongDescriber. |
| SAPO render corpus | 50,000 scenes and 84,866 source instances for preference-optimization rollouts. It uses the same public source pools and simulator policy as SFT, but separate rendered scenes and RTSDs. |
| Benchmark render corpus | 10,000 scenes and 16,744 source instances for held-out ASA benchmark construction. The final 1,000-question test set is selected from this corpus and is excluded from all SFT and SAPO batches. |
| Setting | Phase I: synthetic warm-up | Phase II: sim-to-real adaptation |
|---|---|---|
| Data | 50k full-simulation corpus | Simulation plus STARSS23 FOA clips |
| Sample rate / clip length | 44.1 kHz / 1.0 s | 44.1 kHz / 1.0 s |
| Max sources | 3 | 3 |
| Optimizer / LR | AdamW / | AdamW / |
| Losses | Direction, distance, class, activity, count, room area, absorption | Direction, distance, activity, and count on real clips; full losses on synthetic clips |
| Hyperparameter | Stage 2 | Stage 3 | Stage 4 |
|---|---|---|---|
| Trainable modules | Dense hybrid projector | Projector and decoder LoRA | SAPO LoRA adapters |
| Learning rate | |||
| LoRA Rank ( ) | N/A | 8 | 8 |
| LoRA Alpha ( ) | N/A | 32 | 32 |
| Level | Task family | Specific tasks | Example question template |
|---|---|---|---|
| Numeracy | Count objects | “How many independent sound sources are audible in the clip?” | |
| Semantics | Identify event class | “Which of the following sounds is present in the audio?” | |
| Identify speech transcript | “What specific phrase was spoken in the audio?” | ||
| Spatial physics | Localize azimuth | “What is the horizontal direction of the event?” | |
| Localize elevation | “Is the event coming from above, below, or eye level?” | ||
| Estimate distance | “Approximately how many meters away is the event?” |
| Run | Audio input | Overall | |||
|---|---|---|---|---|---|
| Question-only | Four-channel silence | 43.12 | 34.05 | 50.60 | 43.10 |
| Audio-mismatch | Deranged FOA | 35.32 | 34.77 | 50.89 | 40.40 |
| W-only | kept, zeroed | 38.96 | 30.47 | 54.46 | 41.80 |
| Task family | Question-only | Audio-mismatch | W-only |
|---|---|---|---|
| Causal intent | 53.57 | 53.57 | 51.79 |
| Source removal | 37.50 | 35.71 | 50.00 |
| Rotate | 64.29 | 64.29 | 64.29 |
| Multi-hop | 39.29 | 39.29 | 35.71 |
| Physical consistency | 62.50 | 60.71 | 62.50 |
| Scene abduction | 46.43 | 51.79 | 62.50 |
| Model / audit | Input | (%) | (%) | (%) | Overall (%) |
|---|---|---|---|---|---|
| Qwen2-Audio-7B, exact option | Mono | 41.04 | 31.18 | 30.95 | 34.90 |
| Qwen2-Audio-7B, semantic judge | Mono | 52.40 | 42.40 | 48.80 | 48.40 |
| Qwen2.5-Omni-7B, semantic judge | Mono | 49.90 | 41.90 | 51.00 | 48.00 |
| Qwen2.5-Omni-7B, exact default order | Mono | 61.30 | 47.67 | 68.45 | 59.90 |
| Qwen2.5-Omni-7B, exact shuffled order | Mono | 55.58 | 51.61 | 65.48 | 57.80 |
| Qwen2.5-Omni-7B, exact order average | Mono | 58.44 | 49.64 | 66.97 | 58.85 |
| Task family | Type | Accuracy (%) |
|---|---|---|
| Localize Azimuth Direction | Spatial perception | 50.91 |
| Bind Identity, Location, Distance | Spatial binding | 41.07 |
| Compare Object Distances | Spatial comparison | 42.86 |
| Relative Direction Between Objects | Spatial comparison | 38.18 |
| Counterfactual Rotate | Spatial counterfactual | 51.79 |
| Multi-hop Composed Query | Spatial reasoning | 46.43 |
| Variant | (%) | (%) | (%) | Overall (%) |
|---|---|---|---|---|
| P0: Single MLP | 37.76 | 27.11 | 51.64 | 39.20 |
| P1: Dual Tower | 44.88 | 34.55 | 59.10 | 46.40 |
| P2: Dense Hybrid | 56.43 | 41.88 | 56.55 | 52.10 |
| P2 No-Attractor | 45.09 | 36.23 | 52.08 | 44.50 |
| P2 No-Attractor-Encoder | 42.55 | 39.33 | 54.80 | 43.90 |
| Reward setting | (%) | (%) | (%) | Overall (%) |
|---|---|---|---|---|
| Full SAPO (reduced) | 56.10 | 50.00 | 66.10 | 54.00 |
| No format reward | 61.40 | 44.70 | 59.60 | 53.60 |
| Correctness-only reward | 43.20 | 41.00 | 52.90 | 51.00 |
| No evidence/reference reward | 56.80 | 46.20 | 50.00 | 50.50 |
| No correctness reward | 58.10 | 55.30 | 47.20 | 46.20 |
| Threshold | 1-src (%, N=113) | 2-src (%, N=61) | 3-src (%, N=26) | Overall (%) |
|---|---|---|---|---|
| 0.10 | 0.00 | 100.00 | 0.00 | 30.50 |
| 0.45 | 90.30 | 41.00 | 0.00 | 63.50 |
| 0.75 | 100.00 | 0.00 | 0.00 | 56.50 |
| Run | (%) | (%) | (%) | Overall (%) | |
|---|---|---|---|---|---|
| BAT-SFT | 464 | 35.12 | 27.34 | 33.33 | 32.33 |
| Qwen2.5-Omni full | 500 | 41.90 | 40.48 | 36.89 | 40.20 |
| TWNM-SFT | 500 | 41.90 | 38.10 | 47.54 | 42.00 |
| TWNM-SAPO full | 500 | 45.24 | 38.69 | 49.18 | 44.00 |
| TWNM-SFT, Gaussian noise | 500 | 22.38 | 17.86 | 31.15 | 23.00 |
| TWNM-SAPO, Gaussian noise | 500 | 21.90 | 25.60 | 32.79 | 25.80 |
| Model / input | Accuracy (%) | Difference [95% CI] (pp) |
|---|---|---|
| Question-only | 28.2 | – |
| Trained -only, SFT | 49.3 | – |
| Full FOA, SFT | 62.0 | vs. -only |
| Full FOA, SAPO | 70.8 | vs. SFT |
| Aspect | Agree / Total | Agreement (%) | |
|---|---|---|---|
| Selected answer | 153/180 | 85.0 | .81 |
| Question correctness | 162/180 | 90.0 | .65 |
| Answer uniqueness | 171/180 | 95.0 | .72 |
| Audio–question consistency | 158/180 | 87.8 | .70 |
Audio Understanding
Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.
See all 225 papers