SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

Laya Iyer; Angelina Wang; Sanmi Koyejo

cs.SDMar 10, 2026

SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

Authors: Laya Iyer, Angelina Wang, Sanmi Koyejo

Organizations: Stanford University · Cornell Tech

Abstract

Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.

Figures & tables

Benchmark	Type	Multi turn	Primary Focus	Clips ( $\approx$ )	#Tasks	Bkgd Sound	Noise Loc.	Cross Ling.	Vocal Chars	Latency
SCENEBench	MC+FRQ	✓	Audio & speech understanding	16k	4	✓	✓	✓	✓	✓
AudioBench Wang et al. (2025)	MC+FRQ	✗	ASR, scene & voice	100k	8	✓	✗	✗	✗	✗
MMAU Sakshi et al. (2024)	MC	✗	Multi - task reasoning	10k	27	✓	✗	✗	✗	✗
AIR - Bench Yang et al. (2024)	MC+FRQ	✓	Generative comprehension	21k	20	✗	✗	✗	✗	✗
MARBLE Yuan et al. (2023)	MC	✗	Music classification	25.9k	18	✗	✗	✗	✗	✗
Clotho - AQA Lipping et al. (2022)	FRQ	✗	Environmental QA	1.9k	1	✓	✗	✗	✗	✗

Table 1: Benchmark comparison with standardized size and coverage. Type : MC = multiple choice, FRQ = free response, CLS = classification, Audit = dataset audit. Clips (

\approx

) reports total items when stated or derivable; #Tasks counts distinct task families (— where not reported). Coverage columns indicate whether each suite evaluates that dimension.

Figure 1: Summary radar charts across tasks. Each axis is a task-specific category; values are percentages (or mean similarity %). Legend shown once: GPT-4o (blue), Gemini (orange), Qwen2 (green), Flamingo (pink), Desta (gray).

Panel of Figure 2 — Figure 2: Ambient–sound accuracy across models ( $N{=}4000$ clips). “Specific Type*” treats FR1–correct as FR2–correct. Baselines for simple chance are 25% (MC; 1 of 4).

Model	Correct	Accuracy (%)	95% CI
Gemini	981	16.35	[15.4, 17.3]
GPT-4o	658	10.97	[10.20, 11.78]
Flamingo	481	8.02	[ 7.49, 8.58]
Desta	362	6.03	[ 5.51, 6.65]
Qwen	447	7.45	[ 6.81, 8.14]

Table 2: Motion FR1 (general description) — accuracy with Wilson 95% CIs;

N=6000

per model.

Model	Correct	Accuracy (%)	95% CI
Qwen	3434	57.23	[55.98, 58.48]
GPT-4o	2938	48.97	[47.70, 50.23]
Gemini	1934	32.23	[31.06, 33.43]
Desta	1344	22.40	[21.36, 23.47]
Flamingo	1209	20.15	[19.11, 21.23]

Table 3: Motion FR2 (direction) — accuracy with Wilson 95% CIs;

N=6000

per model. FR2 counts FR1-correct as FR2-correct by design.

Figure 3: Cross-linguistic transcription accuracy (mean $\pm$ 95% CI) across models and languages. Languages: Spanish ( es ), Hindi ( hi ), Portuguese ( pt ), Mandarin Chinese ( zh-CN ). Per-language clip counts are $N_{\text{es}}{=}1010$ , $N_{\text{hi}}{=}1034$ , $N_{\text{pt}}{=}1052$ , $N_{\text{zh-CN}}{=}884$ . Error bars are normal-approximation CIs over per-clip similarity scores.

Model	Cough	Cry	Laugh	Sneeze	Yawn
GPT-4o	91.5	72.1	87.6	71.6	87.9
	$\pm$ 2.2	$\pm$ 2.1	$\pm$ 1.9	$\pm$ 5.8	$\pm$ 4.4
Gemini	48.1	65.2	65.3	14.0	7.5
	$\pm$ 3.9	$\pm$ 2.2	$\pm$ 2.8	$\pm$ 4.5	$\pm$ 3.6
Qwen2	74.1	69.0	97.2	79.2	86.4
	$\pm$ 3.4	$\pm$ 2.2	$\pm$ 1.0	$\pm$ 5.2	$\pm$ 4.6

Table 4: Vocal characterizer accuracy (percent). Entries are mean

\pm

95% CI.

Figure 4: Model Rankings on Synthetic and Natural Data. Although per-task model rankings differ between synthetic and human-recorded evaluations, models that perform well on synthetic data tend to remain strong on average across natural recordings.

Error type	Bkgd. Sound	Noise Loc.	Cross Ling.	Vocal chars.
Omission	18%	43%	–	1%
Over-general label	4%	–	–	–
Misattribution	3%	21%	–	94%
Direction swap	–	8%	–	–
Dropping language	–	–	64%	–
Partial re-translation	–	–	36%	–

Table 5: Distribution of error types by task (percent of error cases in the aggregated dataset). Entries are percentages conditioned on an error in that task, not on all clips. Cells marked “–” denote that the error category is not applicable to that task.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model (local)	Median (s)	Mean (s)	p90 (s)
Flamingo	2.32	2.35	2.98
Qwen	6.04	6.54	10.25
Desta	14.53	27.96	72.16
GPT-4o	n/a	n/a	n/a
Gemini	n/a	n/a	n/a

Appendix

Table 6: Motion-task effective latency (local inference). Values are per-clip medians, means, and 90th-percentiles in seconds; API models (GPT-4o, Gemini) are not available in this run.

Language (N)	Model	Mean $\pm$ 95% CI (%)
Spanish (es, $N=1010$ )	GPT–4o	93.9 [93.4, 94.4]
	Gemini	93.3 [92.9, 93.7]
	Flamingo	90.1 [89.5, 90.7]
	Qwen2	68.7 [68.0, 69.4]
	Desta	49.0 [47.4, 50.5]
Hindi (hi, $N=1034$ )	Gemini	78.2 [77.5, 78.8]

Appendix

Table 7: Cross–linguistic transcription (mean similarity

\pm

95% CI, %). All models are shown for each language; the best mean per language is bolded.

Task	Model	Accuracy (%)	95% CI
FR1	GPT-4o	60.0	[38, 79]
FR1	Gemini	45.0	[25, 66]
FR1	Flamingo	70.0	[47, 87]
FR1	Desta	0.0	[0, 17]
FR1	Qwen	50.0	[29, 71]
FR2	GPT-4o	70.0	[47, 87]

Appendix

Table 8: Task 1: Background Sound Understanding — accuracy with Wilson 95% CIs;

N=20

per model.

Task	Model	Accuracy (%)	95% CI
FR1	GPT-4o	5.0	[1, 24]
FR1	Gemini	5.0	[1, 24]
FR1	Flamingo	0.0	[0, 17]
FR1	Desta	0.0	[0, 17]
FR1	Qwen	5.0	[1, 24]
FR2	GPT-4o	20.0	[8, 42]

Appendix

Table 9: Task 2: Noise Localization — accuracy with Wilson 95% CIs;

N=20

per model.

Lang Pair	Model	Accuracy (%)	95% CI
ENG-ZH	GPT-4o	57.5	[35.3, 74.9]
ENG-ZH	Gemini	44.6	[31.8, 56.7]
ENG-ZH	Flamingo	63.6	[44.8, 80.7]
ENG-ZH	Desta	26.8	[10.0, 45.4]
ENG-ZH	Qwen	44.7	[32.1, 55.7]
ENG-SA	GPT-4o	25.6	[7.8, 48.6]

Appendix

Table 10: Task 3: Cross-Linguistic Evaluation — mean similarity accuracy with 95% CIs on human recordings (

N=20

per model).

Audio Understanding

Audio understanding enables machines to recognize speech, sounds, music, speakers, and events, then answer questions grounded in what they hear.

See all 212 papers

Jun 24, 2026

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.

Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif +6

University of California Irvine · University of Illinois Chicago · Kennesaw State University

Audio-Language Model Evaluation Audio Understanding Audio-Language Models

Ask PDF

Jul 20, 2026

ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

As large audio language models (LALMs) advance, robust evaluation frameworks have become essential. In this context, Spanish speech understanding under realistic acoustic conditions has received particularly little attention. We introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate LALMs across heterogeneous acoustic conditions and reasoning abilities. ESCUCHA comprises 1,000 human-curated questions paired with audio, totaling 162.9 hours sourced directly ``from the wild'' rather than drawn from existing datasets, with durations ranging from a few seconds to over 80 minutes. The benchmark emphasizes reasoning, spanning 9 perceptual and 10 reasoning categories, and it captures linguistic diversity through multiple Spanish accents and non-normative speech. ESCUCHA further includes multi-audio questions, spoken questions, and audio instructions, and it flags which questions support open-ended evaluation. Benchmarking several state-of-the-art multimodal and speech models reveals substantial performance gaps relative to trained humans.

Fernando López, Ana Ayala, Guillermo Segovia +4

Telefónica Innovación Digital · Universidad Autónoma de Madrid · Universidad Complutense de Madrid

Audio-Language Model Evaluation Spoken Language Understanding Audio QA Speech Processing

Ask PDF

Jun 6, 2026

GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models

Large Audio-Language Models (LALMs) integrate audio perception and language understanding within a unified framework, enabling a wide range of real-world applications. Despite recent advances, evaluation for LALMs remains heavily underspecified relative to real-world requirements: most lack true linguistic and cultural authenticity, while others fail to capture acoustic realism. To bridge this gap, we propose GlobeAudio, a multilingual and multicultural benchmark designed to evaluate naturalistic audio understanding. GlobeAudio consists of 5,637 multiple-choice questions across six typologically diverse languages, expertly crafted by native speakers grounded on naturally occurring audio. In order to do well, models must possess higher-level auditory reasoning skills and culturally grounded interpretation. We systematically evaluate representative closed-source and open-source LALMs, as well as cascaded ASR-LLM pipelines. Our experiments reveal substantial performance gaps under natural acoustic conditions, particularly for open-source models and low-resource languages. These findings highlight critical limitations of current LALMs and underscore the importance of naturalistic audio evaluation for future audio-language systems. GlobeAudio can be found at https://huggingface.co/datasets/iNLP-Lab/GlobeAudio .

Ryner Tan, Wenxuan Zhang

Singapore University of Technology and Design

Audio-Language Model Evaluation Cross-Cultural Language Model Evaluation Audio Understanding

Ask PDF

Abstract

Audio Understanding

Explore similar work

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models