One Waveform, Two Stories: Benchmarking Speech-Acoustic Conflict in Audio-Language Models
Organizations: The University of Osaka · University of Wisconsin-Madison
Abstract
Audio-language models (ALMs) are evaluated on their understanding of both speech and environmental sounds. What remains underexplored is how they respond when spoken claims contradict acoustic evidence within the same recording. We introduce SAC-Bench, a benchmark of intra-audio speech--acoustic conflict comprising 616,031 unique clips from 188 sound-event classes across seven semantic, temporal, and spatial tasks. It pairs environmental recordings with compatible or contradictory synthesized spoken claims and includes audio-only controls where applicable. Across five open-weight ALMs evaluated on semantic and temporal tasks, high audio-only accuracy often coexists with large losses under conflicting claims; Event Identity accuracy is 24.8--46.5 percentage points lower in conflict than compatible conditions. Temporal Order failures persist even when speech follows the complete event sequence, so simultaneous speech--event masking cannot fully explain them. A spatial case study with Spatial-Omni finds lower motion accuracy associated with both spoken claims and the speech source's physical trajectory. Prompt interventions sometimes restore acoustic accuracy, but can also reduce claim-aligned answers without improving grounding or worsen performance. SAC-Bench thus separates acoustic competence, conflict robustness, and intervention effects.
Figures & tables
| Speech–acoustic event conflict | Audio | QA pairs | |||||||||||||
| Benchmark | Grounding tasks | P | I | St | Or | Lc | Sm | Rm | Other conf. | SF ctrl. | 3-src GT | Total | Eval | Total | Eval |
| CompA ( Ghosh et al., 2024 ) | Order, attributes (600 inst.) | – | – | – | – | – | – | – | – | – | – | 1,300 | 1,300 | n/a | n/a |
| AHa-Bench † ( Cheng et al., 2026 ) | Presence, duration, order, distance | – | – | – | – | – | – | – | – | – | 396 | 396 | 906 | 906 | |
| HalluAudio ( Zhao et al., 2026 ) | Presence, overlap, order | – | – | – | – | – | – | – | – | – | – | n/r | n/r | 5,720 | 5,720 |
| SO-Bench ( Zhu et al., 2026 ) | Onset, location, distance, motion | – | – | – | – | – | – | – | – | – | – | 7,877 | 7,877 | 7,877 | 7,877 |
| AHA-Eval ( Seth et al., 2026 ) | Absent-event queries | – | – | – | – | – | – | – | – | n/r | n/r | 6.5K | 6.5K | ||
| Task | Dev | Eval | Total |
|---|---|---|---|
| Presence | 160,663 | 81,892 | 242,555 |
| Identity | 160,658 | 81,892 | 242,550 |
| Temporal State | 2,884 | 1,520 | 4,404 |
| Temporal Order | 4,304 | 2,258 | 6,562 |
| Static Location | 36,346 | 28,200 | 64,546 |
| Spherical Motion | 17,620 | 14,638 | 32,258 |
| Model | Audio-only | Compatible | Conflict | Gap | SFR |
|---|---|---|---|---|---|
| Event Presence (BAcc) | |||||
| MOSS-Audio-8B | 81.2 | 78.5 | 65.7 | 12.9 | 67.5 |
| Audio Flamingo Next | 84.0 | 85.2 | 71.1 | 14.1 | 67.8 |
| Qwen3-Omni | 91.3 | 89.0 | 79.3 | 9.7 | 64.2 |
| Step-Audio-2-mini | 85.8 | 79.9 | 67.8 | 12.0 | 74.7 |
| Spatial-Omni (SO-30B) | 73.1 | 72.8 | 69.3 | 3.5 | 64.6 |
| Task | Chance | Audio-only | Speech (NS) | Speech (S) |
|---|---|---|---|---|
| Location | 2.6 | 31.3 | 21.0 | 21.5 |
| Motion | 25.0 | 50.5 | 35.0 | 33.2 |
| Radial | 50.0 | 50.3 | 49.2 | 49.7 |
| Acoustic score (%) | SFR (%) | ||||||
| Model | Task | Neu | AG | CA | Neu | AG | CA |
| MOSS-Audio-8B | Presence | 65.7 | 68.6 | 64.9 | 67.5 | 51.8 | 24.1 |
| MOSS-Audio-8B | Identity | 29.1 | 32.9 | 39.2 | 60.9 | 51.8 | 38.1 |
| Audio Flamingo Next | Identity | 35.6 | 38.5 | 37.5 | 71.5 | 59.1 | 64.1 |
| Qwen3-Omni | Identity | 39.8 | 49.6 | 59.3 | 70.5 | 54.7 | 40.9 |
| Step-Audio-2-mini | Identity | 32.9 | 33.5 | 33.0 | 80.8 | 78.7 | 79.4 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Configuration | Value |
|---|---|
| CLAP checkpoint | 630k-audioset-best.pt |
| CLAP label-text template | Corrected human-readable label |
| Temporal-order sampling seed | 42 |
| GPT-4o model version | gpt-4o-2024-11-20 |
| F5-TTS checkpoint | F5TTS_Base |
| M-AILABS reference recording | elliot_miller |
| Task | Compatible | Conflict |
|---|---|---|
| Presence (+) | target | none / near / far |
| Presence ( ) | none / near / far | target |
| Identity (+) | target | none / near / far |
| Identity ( ) | none / near / far | target |
| Temporal State | matched | reversed |
| Temporal order | matched | reversed |
| Parameter | Value |
|---|---|
| Speech–event gap | 0.2 s |
| Inter-event gap | 0.2 s |
| Speech position for Temporal order | after sequence |
| Semantic SER settings | dB |
| Acoustic environment | free-field |
| Spatial formats | binaural, FOA |
| Task | Unique clips | Hours |
|---|---|---|
| Presence | 242,555 | 781.8 |
| Identity | 242,550 | 786.6 |
| Temporal State | 4,404 | 24.2 |
| Temporal Order | 6,562 | 54.9 |
| Static Location | 64,546 | 194.6 |
| Spherical Motion | 32,258 | 142.1 |
| Checkpoint | Backend version | Precision | Max. tokens | Revision |
|---|---|---|---|---|
| MOSS-Audio-8B-Instruct | vLLM: 0.31.0 | BF16 | 256 | 66326e6 |
| Spatial-Omni-SO30B | PyTorch: 2.9.0 | BF16 | 256 | ba33a4c |
| audio-flamingo-next-hf | Transformers: 5.19.0 | BF16 | 256 | 5634886 |
| Qwen3-Omni-30B-A3B-Instruct | vLLM: 0.9.3 | BF16 | 256 | 26291f7 |
| Step-Audio-2-mini | vLLM: 0.1.dev1+g9ecabf22f | BF16 | 256 | e36fdd5 |
Audio-Language Model Evaluation
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
See all 225 papers