Audio-language models (ALMs) are evaluated on their understanding of both speech and environmental sounds. What remains underexplored is how they respond when spoken claims contradict acoustic evidence within the same recording. We introduce SAC-Bench, a benchmark of intra-audio speech--acoustic conflict comprising 616,031 unique clips from 188 sound-event classes across seven semantic, temporal, and spatial tasks. It pairs environmental recordings with compatible or contradictory synthesized spoken claims and includes audio-only controls where applicable. Across five open-weight ALMs evaluated on semantic and temporal tasks, high audio-only accuracy often coexists with large losses under conflicting claims; Event Identity accuracy is 24.8--46.5 percentage points lower in conflict than compatible conditions. Temporal Order failures persist even when speech follows the complete event sequence, so simultaneous speech--event masking cannot fully explain them. A spatial case study with Spatial-Omni finds lower motion accuracy associated with both spoken claims and the speech source's physical trajectory. Prompt interventions sometimes restore acoustic accuracy, but can also reduce claim-aligned answers without improving grounding or worsen performance. SAC-Bench thus separates acoustic competence, conflict robustness, and intervention effects.
Figures & tables
Figure 1: Overview of SAC-Bench. A single audio waveform contains spoken claims and environmental acoustic evidence that may imply conflicting interpretations. SAC-Bench evaluates this conflict across four task groups: event presence, event identity, temporal structure, and spatial configuration.
Speech–acoustic event conflict
Audio
QA pairs
Benchmark
Grounding tasks
P
I
St
Or
Lc
Sm
Rm
Other conf.
SF ctrl.
3-src GT
Total
Eval
Total
Eval
CompA ( Ghosh et al., 2024 )
Order, attributes (600 inst.)
–
–
–
–
–
–
–
–
–
–
1,300
1,300
n/a
n/a
AHa-Bench † ( Cheng et al., 2026 )
Presence, duration, order, distance
–
–
–
–
–
–
–
∘
–
–
396
396
906
906
HalluAudio ( Zhao et al., 2026 )
Presence, overlap, order
–
–
–
–
–
–
–
–
–
–
n/r
n/r
5,720
5,720
SO-Bench ( Zhu et al., 2026 )
Onset, location, distance, motion
–
–
–
–
–
–
–
–
–
–
7,877
7,877
7,877
7,877
AHA-Eval ( Seth et al., 2026 )
Absent-event queries
∙
–
–
–
–
–
–
–
∘
–
n/r
n/r
∼ 6.5K
∼ 6.5K
Table 1: Comparison of audio-language benchmarks in speech–acoustic conflict coverage and dataset scale.
Figure 2: SAC-Bench construction pipeline, from environmental audio selection and spoken claim synthesis to conflict generation, audio rendering, and evaluation.
Figure 3: Taxonomic and semantic coverage of sound-event labels in AudioSet, FSD50K, and SAC-Bench. (a) Label distribution across ten ontology categories with fractional assignments. (b) UMAP projection of label-name embeddings.
Task
Dev
Eval
Total
Presence
160,663
81,892
242,555
Identity
160,658
81,892
242,550
Temporal State
2,884
1,520
4,404
Temporal Order
4,304
2,258
6,562
Static Location
36,346
28,200
64,546
Spherical Motion
17,620
14,638
32,258
Table 2: Unique audio clips by task and FSD50K split. The final row reports the deduplicated total across all tasks.
Model
Audio-only
Compatible
Conflict
Gap
SFR
Event Presence (BAcc)
MOSS-Audio-8B
81.2
78.5
65.7
12.9
67.5
Audio Flamingo Next
84.0
85.2
71.1
14.1
67.8
Qwen3-Omni
91.3
89.0
79.3
9.7
64.2
Step-Audio-2-mini
85.8
79.9
67.8
12.0
74.7
Spatial-Omni (SO-30B)
73.1
72.8
69.3
3.5
64.6
Table 3: Neutral-prompt acoustic performance and SFR (%). Presence uses recording-level BAcc; other tasks use accuracy. Gap = compatible minus conflict (pp). SFR uses only eligible conflict questions.
Figure 4: SFR versus speech-to-event ratio (SER) under neutral prompting. Shading indicates source-cluster bootstrap 95% confidence intervals; dashed lines indicate uniform-choice references.
Figure 5: Acoustic accuracy under compatible and conflicting claims for (a) Temporal Order and (b) Temporal State. Error bars show 95% source-cluster bootstrap confidence intervals ( n=760 per model–condition cell).
Task
Chance
Audio-only
Speech (NS)
Speech (S)
Location
2.6
31.3
21.0
21.5
Motion
25.0
50.5
35.0
33.2
Radial
50.0
50.3
49.2
49.7
Table 4: Spatial-Omni acoustic accuracy (%) on eligible FOA inputs under neutral prompting. NS/S denote non-spatial/spatialized speech; both columns pool compatible and conflicting conditions.
Figure 6: Claim and speaker cues in Spatial-Omni’s FOA motion task. (a,b) Matched acoustic accuracy when claim C and speech trajectory S match ground truth A or an alternative B . (c) Attraction to conflicting C and S relative to audio-only controls. Error bars show 95% class-bootstrap confidence intervals; dashed lines mark 25% chance in (a,b) and zero attraction in (c).
Figure 7: Paired changes in acoustic performance and SFR under prompt interventions relative to neutral prompting. Rightward indicates reduced SFR; upward indicates improved acoustic performance. Presence uses recording-level BAcc. Model–task comparisons without eligible matched pairs are omitted.
Acoustic score (%)
SFR (%)
Model
Task
Neu
AG
CA
Neu
AG
CA
MOSS-Audio-8B
Presence
65.7
68.6
64.9
67.5
51.8
24.1
MOSS-Audio-8B
Identity
29.1
32.9
39.2
60.9
51.8
38.1
Audio Flamingo Next
Identity
35.6
38.5
37.5
71.5
59.1
64.1
Qwen3-Omni
Identity
39.8
49.6
59.3
70.5
54.7
40.9
Step-Audio-2-mini
Identity
32.9
33.5
33.0
80.8
78.7
79.4
Table 5: Prompt-policy performance on representative conflict tasks (%). Neu: neutral; AG: acoustic-grounded; CA: conflict-aware. Presence uses recording-level BAcc; other tasks use accuracy.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
CLAP checkpoint
630k-audioset-best.pt
CLAP label-text template
Corrected human-readable label
Temporal-order sampling seed
42
GPT-4o model version
gpt-4o-2024-11-20
F5-TTS checkpoint
F5TTS_Base
M-AILABS reference recording
elliot_miller
Appendix
Table 6: Dataset-generation configuration.
Task
Compatible
Conflict
Presence (+)
target
none / near / far
Presence ( − )
none / near / far
target
Identity (+)
target
none / near / far
Identity ( − )
none / near / far
target
Temporal State
matched
reversed
Temporal order
matched
reversed
Appendix
Table 7: Semantic compatibility conditions. In spatial tasks, semantic compatibility requires the spoken claim C to match the acoustic ground truth A , independent of speaker geometry S .
Parameter
Value
Speech–event gap
0.2 s
Inter-event gap
0.2 s
Speech position for Temporal order
after sequence
Semantic SER settings
−10,−5,0,5,10 dB
Acoustic environment
free-field
Spatial formats
binaural, FOA
Appendix
Table 8: Audio rendering parameters used in dataset generation.
Task
Unique clips
Hours
Presence
242,555
781.8
Identity
242,550
786.6
Temporal State
4,404
24.2
Temporal Order
6,562
54.9
Static Location
64,546
194.6
Spherical Motion
32,258
142.1
Appendix
Table 9: Unique audio clips and path-deduplicated duration by task. Task rows overlap; the total is their union, not their sum.
Checkpoint
Backend version
Precision
Max. tokens
Revision
MOSS-Audio-8B-Instruct
vLLM: 0.31.0
BF16
256
66326e6
Spatial-Omni-SO30B
PyTorch: 2.9.0
BF16
256
ba33a4c
audio-flamingo-next-hf
Transformers: 5.19.0
BF16
256
5634886
Qwen3-Omni-30B-A3B-Instruct
vLLM: 0.9.3
BF16
256
26291f7
Step-Audio-2-mini
vLLM: 0.1.dev1+g9ecabf22f
BF16
256
e36fdd5
Appendix
Table 10: Model checkpoints and inference configurations for the reported experiments.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.