SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

Laya Iyer; Angelina Wang; Sanmi Koyejo

cs.SDMar 10, 2026

SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

Authors: Laya Iyer, Angelina Wang, Sanmi Koyejo

Organizations: Stanford University · Cornell Tech

Abstract

Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.

Figures & tables

Benchmark	Type	Multi turn	Primary Focus	Clips ( $\approx$ )	#Tasks	Bkgd Sound	Noise Loc.	Cross Ling.	Vocal Chars	Latency
SCENEBench	MC+FRQ	✓	Audio & speech understanding	16k	4	✓	✓	✓	✓	✓
AudioBench Wang et al. (2025)	MC+FRQ	✗	ASR, scene & voice	100k	8	✓	✗	✗	✗	✗
MMAU Sakshi et al. (2024)	MC	✗	Multi - task reasoning	10k	27	✓	✗	✗	✗	✗
AIR - Bench Yang et al. (2024)	MC+FRQ	✓	Generative comprehension	21k	20	✗	✗	✗	✗	✗
MARBLE Yuan et al. (2023)	MC	✗	Music classification	25.9k	18	✗	✗	✗	✗	✗
Clotho - AQA Lipping et al. (2022)	FRQ	✗	Environmental QA	1.9k	1	✓	✗	✗	✗	✗

Table 1: Benchmark comparison with standardized size and coverage. Type : MC = multiple choice, FRQ = free response, CLS = classification, Audit = dataset audit. Clips (

\approx

) reports total items when stated or derivable; #Tasks counts distinct task families (— where not reported). Coverage columns indicate whether each suite evaluates that dimension.

Figure 1: Summary radar charts across tasks. Each axis is a task-specific category; values are percentages (or mean similarity %). Legend shown once: GPT-4o (blue), Gemini (orange), Qwen2 (green), Flamingo (pink), Desta (gray).

Panel of Figure 2 — Figure 2: Ambient–sound accuracy across models ( $N{=}4000$ clips). “Specific Type*” treats FR1–correct as FR2–correct. Baselines for simple chance are 25% (MC; 1 of 4).

Model	Correct	Accuracy (%)	95% CI
Gemini	981	16.35	[15.4, 17.3]
GPT-4o	658	10.97	[10.20, 11.78]
Flamingo	481	8.02	[ 7.49, 8.58]
Desta	362	6.03	[ 5.51, 6.65]
Qwen	447	7.45	[ 6.81, 8.14]

Table 2: Motion FR1 (general description) — accuracy with Wilson 95% CIs;

N=6000

per model.

Model	Correct	Accuracy (%)	95% CI
Qwen	3434	57.23	[55.98, 58.48]
GPT-4o	2938	48.97	[47.70, 50.23]
Gemini	1934	32.23	[31.06, 33.43]
Desta	1344	22.40	[21.36, 23.47]
Flamingo	1209	20.15	[19.11, 21.23]

Table 3: Motion FR2 (direction) — accuracy with Wilson 95% CIs;

N=6000

per model. FR2 counts FR1-correct as FR2-correct by design.

Figure 3: Cross-linguistic transcription accuracy (mean $\pm$ 95% CI) across models and languages. Languages: Spanish ( es ), Hindi ( hi ), Portuguese ( pt ), Mandarin Chinese ( zh-CN ). Per-language clip counts are $N_{\text{es}}{=}1010$ , $N_{\text{hi}}{=}1034$ , $N_{\text{pt}}{=}1052$ , $N_{\text{zh-CN}}{=}884$ . Error bars are normal-approximation CIs over per-clip similarity scores.

Model	Cough	Cry	Laugh	Sneeze	Yawn
GPT-4o	91.5	72.1	87.6	71.6	87.9
	$\pm$ 2.2	$\pm$ 2.1	$\pm$ 1.9	$\pm$ 5.8	$\pm$ 4.4
Gemini	48.1	65.2	65.3	14.0	7.5
	$\pm$ 3.9	$\pm$ 2.2	$\pm$ 2.8	$\pm$ 4.5	$\pm$ 3.6
Qwen2	74.1	69.0	97.2	79.2	86.4
	$\pm$ 3.4	$\pm$ 2.2	$\pm$ 1.0	$\pm$ 5.2	$\pm$ 4.6

Table 4: Vocal characterizer accuracy (percent). Entries are mean

\pm

95% CI.

Figure 4: Model Rankings on Synthetic and Natural Data. Although per-task model rankings differ between synthetic and human-recorded evaluations, models that perform well on synthetic data tend to remain strong on average across natural recordings.

Error type	Bkgd. Sound	Noise Loc.	Cross Ling.	Vocal chars.
Omission	18%	43%	–	1%
Over-general label	4%	–	–	–
Misattribution	3%	21%	–	94%
Direction swap	–	8%	–	–
Dropping language	–	–	64%	–
Partial re-translation	–	–	36%	–

Table 5: Distribution of error types by task (percent of error cases in the aggregated dataset). Entries are percentages conditioned on an error in that task, not on all clips. Cells marked “–” denote that the error category is not applicable to that task.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model (local)	Median (s)	Mean (s)	p90 (s)
Flamingo	2.32	2.35	2.98
Qwen	6.04	6.54	10.25
Desta	14.53	27.96	72.16
GPT-4o	n/a	n/a	n/a
Gemini	n/a	n/a	n/a

Appendix

Table 6: Motion-task effective latency (local inference). Values are per-clip medians, means, and 90th-percentiles in seconds; API models (GPT-4o, Gemini) are not available in this run.

Language (N)	Model	Mean $\pm$ 95% CI (%)
Spanish (es, $N=1010$ )	GPT–4o	93.9 [93.4, 94.4]
	Gemini	93.3 [92.9, 93.7]
	Flamingo	90.1 [89.5, 90.7]
	Qwen2	68.7 [68.0, 69.4]
	Desta	49.0 [47.4, 50.5]
Hindi (hi, $N=1034$ )	Gemini	78.2 [77.5, 78.8]

Appendix

Table 7: Cross–linguistic transcription (mean similarity

\pm

95% CI, %). All models are shown for each language; the best mean per language is bolded.

Task	Model	Accuracy (%)	95% CI
FR1	GPT-4o	60.0	[38, 79]
FR1	Gemini	45.0	[25, 66]
FR1	Flamingo	70.0	[47, 87]
FR1	Desta	0.0	[0, 17]
FR1	Qwen	50.0	[29, 71]
FR2	GPT-4o	70.0	[47, 87]

Appendix

Table 8: Task 1: Background Sound Understanding — accuracy with Wilson 95% CIs;

N=20

per model.

Task	Model	Accuracy (%)	95% CI
FR1	GPT-4o	5.0	[1, 24]
FR1	Gemini	5.0	[1, 24]
FR1	Flamingo	0.0	[0, 17]
FR1	Desta	0.0	[0, 17]
FR1	Qwen	5.0	[1, 24]
FR2	GPT-4o	20.0	[8, 42]

Appendix

Table 9: Task 2: Noise Localization — accuracy with Wilson 95% CIs;

N=20

per model.

Lang Pair	Model	Accuracy (%)	95% CI
ENG-ZH	GPT-4o	57.5	[35.3, 74.9]
ENG-ZH	Gemini	44.6	[31.8, 56.7]
ENG-ZH	Flamingo	63.6	[44.8, 80.7]
ENG-ZH	Desta	26.8	[10.0, 45.4]
ENG-ZH	Qwen	44.7	[32.1, 55.7]
ENG-SA	GPT-4o	25.6	[7.8, 48.6]

Appendix

Table 10: Task 3: Cross-Linguistic Evaluation — mean similarity accuracy with 95% CIs on human recordings (

N=20

per model).

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 212 papers

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models
Jun 24, 2026Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif +6Audio-Language Model Evaluation Audio Understanding
University of California Irvine · University of Illinois Chicago · Kennesaw State University
ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions
Jul 20, 2026Fernando López, Ana Ayala, Guillermo Segovia +4Audio-Language Model Evaluation Spoken Language Understanding
Telefónica Innovación Digital · Universidad Autónoma de Madrid · Universidad Complutense de Madrid
GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models
Jun 6, 2026Ryner Tan, Wenxuan ZhangAudio-Language Model Evaluation Cross-Cultural Language Model Evaluation
Singapore University of Technology and Design

Jun 24, 2026J/K move · Enter open · S save

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif, Yutong Song+5

University of California Irvine · University of Illinois Chicago · Kennesaw State University

Audio-Language Model Evaluation Audio Understanding Audio-Language Models

Open paper AskPDF arXiv

Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.

Abstract

Audio Understanding

Explore similar work

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models