WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

Jaeyeon Kim; Heeseung Yun; Sang Hoon Woo; Chao-Han Huck Yang; Gunhee Kim

cs.SDAug 28, 2025

WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

Authors: Jaeyeon Kim, Heeseung Yun, Sang Hoon Woo, Chao-Han Huck Yang, Gunhee Kim

Organizations: Carnegie Mellon University · Seoul National University · NVIDIA

Abstract

Large audio-language models (LALMs) extend language understanding into the auditory domain, yet their ability to perform low-level listening, such as pitch and duration detection, remains underexplored. However, low-level listening is critical for real-world, out-of-distribution tasks where models must reason about unfamiliar sounds based on fine-grained acoustic cues. To address this gap, we introduce the World-of-Whale benchmark (WoW-Bench) to evaluate low-level auditory perception and cognition using marine mammal vocalizations. We use marine mammal vocalizations as out-of-distribution sound events to better assess models' low-level listening and so that the models do not rely on prior knowledge of the sound events. WoW-bench is composed of a Perception benchmark for categorizing novel sounds and a Cognition benchmark, inspired by Bloom's taxonomy, to assess the abilities to remember, understand, apply, and analyze sound events. For the Cognition benchmark, we additionally introduce distractor questions to evaluate whether models are truly solving problems through listening rather than relying on other heuristics. Experiments with state-of-the-art LALMs show performance far below human levels, indicating a need for stronger auditory grounding in LALMs.

Figures & tables

Figure 1: World-of-Whale benchmark aims to evaluate low-level listening capabilities of LALMs using marine mammal vocalizations, which are rarely represented in conventional datasets and span a broad acoustic range. LALMs struggle with Cognition questions that humans can reliably solve by using low-level auditory perception.

Panel of Table 1 — Table 1: Comparison to existing LALM benchmarks. Low-level Listening indicates that the dataset evaluates a model’s ability to process fine-grained acoustic attributes prior to semantic categorization. indicates partial assessment of the perceptual capabilities like low/high pitch. Casual indicates the dataset corpora covers general sound events like human and music, e.g. , Gemmeke et al. (2017) ; Kim et al. (2019) ; Drossos et al. (2020) ; Ju et al. (2024) . ∗ denotes the number of general audio-related questions in each test set.

Model	Params	Perception				Cognition									Total	MMAU
Model	Params	S	V	B	All	R	U	AF	AD	AA	AV	ND	D	All	Total	MMAU
Random	-	27.3	22.7	22.0	24.0	24.8	25.8	22.0	23.6	28.0	25.3	26.6	23.3	24.9	24.8	–
SALMONN ( 2024 )	13B	24.2	14.4	8.0	15.5	26.8	18.8	32.0	26.4	0.0	3.3	21.9	2.7	18.0	17.6	42.1
LTU ( 2024 )	7B	28.3	29.9	24.0	27.4	22.8	30.8	8.4	20.4	22.0	32.0	19.2	36.0	22.6	23.4	20.7
LTU-AS ( 2023 )	7B	24.2	20.6	22.0	22.3	20.0	23.3	18.0	22.0	21.6	26.6	24.3	12.3	21.9	22.0	–
GAMA ( 2024a )	7B	21.2	29.9	24.0	25.0	23.6	22.5	21.6	22.0	20.4	23.2	14.8	51.3	22.2	22.7	30.7

Table 2: Evaluation results on the WoW-Bench. Each acronym stands for S pecies, V ocalization, B oth (Perception), R emember, U nderstand, A pply- F requency, A pply- D uration, A nalyze- A coustics, A nalyze- V ocalization, N on- D istractor, D istractor (Cognition), respectively. Last two columns report the overall score and their performance on the Sound Test subset from MMAU-v05.15.25 ( 2025 ) as reference. For each column, the highest score is indicated in bold , while the second-highest score is underlined .

Figure 2: Performance distribution across cognition question types, grouped by the presence of dataset and distractor condition. Each bar represents the interquartile range (Q1 to Q3) of model performance. The three horizontal lines of each bar correspond to the first quartile (Q1), median (Q2), and third quartile (Q3). The × marker indicates the mean.

Figure 3: Distribution of models’ performance regarding (a) benchmark type, (b) distractor presence, and (c) multi-clip, where we observe the Pearson correlation of 0.75, -0.51, and 0.47, respectively.

Figure 4: Qualitative examples of model responses to (a) a Perception/Vocalization question and (b) a Cognition/Apply-Frequency question.

Figure 5: Qualitative example of a model response to (c) a distractor question for Cognition/Apply-Frequency task.

Category	NR	S	L	C	Total
Count	16	16	2	10	44

Table 3: Counts of error types in 44 incorrectly answered Remember -type distractor questions. NR denotes No Reasoning; S denotes Semantic Categorization; L denotes Low-Level Grounding; C denotes Direct Comparison

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Influence of audio property on accuracy, namely (a) sampling rate and (b) duration.

Model Name	Remember	Understand	Apply-Frequency	Apply-Duration	Average
Audio Flamingo 2 (WoW-Bench)	5.0	30.5	32.5	29.5	24.4
Audio Flamingo 2 (ESC-50)	24.0	48.0	32.0	28.0	33.0
Qwen2.5-Omni (WoW-Bench)	21.5	46.3	34.5	42.0	36.1
Qwen2.5-Omni (ESC-50)	20.0	76.0	53.0	40.0	47.3
Gemini 1.5 Pro (WoW-Bench)	34.5	33.7	32.5	37.5	34.6
Gemini 1.5 Pro (ESC-50)	44.0	80.0	44.0	44.0	53.0

Appendix

Table 4: Evaluation results on non-distractor Cognition questions from WoW-Bench and ESC-50 based set.

(a) Perception			(b) Non-distractor			(c) Distractor
Word	Fraction	Accuracy	Word	Fraction	Accuracy	Word	Fraction	Accuracy
hum	25/40	0.6250	longest	152/182	0.8352	wide	29/52	0.5577
chirp	29/54	0.5370	reference	103/134	0.7687	across	20/40	0.5000
modulated	19/36	0.5278	duration	186/265	0.7019	consistent	12/29	0.4138
present	26/55	0.4727	silence	189/278	0.6799	present	11/32	0.3438
continuous	35/75	0.4667	end	91/146	0.6233	modulating	11/32	0.3438

Appendix

Table 5: Keyword-based accuracy for the Gemini 2.5 Flash model. The top group displays exemplar keywords with high accuracies, while the bottom group displays keywords with low accuracies. Each cell shows the fraction of correct responses when the specified keyword appears in the model’s answer, broken down by question type: (a) Perception tasks, (b) Cognition tasks without distractors, and (c) Cognition tasks with distractors.

Model	Perception				Cognition									Total
Model	S	V	B	All	R	U	AF	AD	AA	AV	ND	D	All	Total
Gemini 2.5 Flash	27.3	54.6	30.0	37.2	64.4	39.2	36.0	66.4	34.0	42.7	54.5	18.3	47.2	45.5
GPT-4o-Audio	19.2	38.1	19.0	25.3	14.0	26.3	21.2	16.0	24.8	25.3	23.0	14.0	21.2	21.9

Appendix

Table 6: Evaluation result of Gemini 2.5 Flash and GPT-4o-Audio on the WoW-Bench.

Task	Subtask	ND	D	Total
Perception	Species	99	-	99
	Vocalization	97	-	97
	Both	100	-	100
Cognition	Remember	200	50	250
	Understand	190	50	240
	Apply-Frequency	200	50	250

Appendix

Table 7: Number of questions in WoW-Bench by task type. ND denotes non-distractor questions, and D denotes distractor questions.

Figure 7: Example questions from each task type in WoW-Bench, shown alongside spectrograms of the corresponding input audio.

Model	Original	Shuffle 1	Shuffle 2	STD
Audio Flamingo 2	31.9	30.7	31.0	0.666
Qwen2-Audio-Instruct	32.7	33.4	32.1	0.636
Gemini 2.0 Flash	35.0	36.8	36.3	0.958

Appendix

Table 8: Model performance across different shuffle configurations.

Category	Gemini 2.5 Flash (%)	Non-Expert (%)	Audio Expert (%)	Human Total (%)
Perception	41.67	46.67	33.33	40.00
Cognition: Non-distractor	48.33	62.67	78.67	70.67
Cognition: Distractor	25.00	65.00	88.33	76.67
Remember	57.14	97.14	97.14	97.14
Understand	42.86	57.14	77.14	67.14
Apply–Frequency	21.43	65.71	85.71	75.71

Appendix

Table 9: Detailed human evaluation results across the various task categories.

	Accuracy	Chance ( $\bar{P}_{e}$ )	kappa ( $\kappa$ )
Perception	0.4444	0.5062	0.0775
Cognition	0.7421	0.6172	0.2329
Total	0.6759	0.5619	0.2476

Appendix

Table 10: Inter-rater agreement measured by Fleiss’ kappa.

Figure 8: Multiple-choice questionnaire interface used for human evaluation.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition
Oct 12, 2026Zhe Sun, Liang An, Jiayu Yao +9Underwater Acoustics Audio-Language Model Evaluation
Southeast University
VocalGrad: Evaluating Acoustic Perception in Audio Language Models
Oct 16, 2026Koki Ryu, Hitomi YanakaAudio-Language Model Evaluation Audio Understanding
The University of Tokyo · Riken · Tohoku University
How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Mar 19, 2026Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13Audio Understanding Audio-Language Model Evaluation
National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan

Oct 12, 2026J/K move · Enter open · S save

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian+8

Southeast University

Underwater Acoustics Audio-Language Model Evaluation

Open paper AskPDF arXiv

Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing UA-Bench, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0%--13.1% leaf accuracy, against a random reference of 7.7%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.

Abstract

Audio-Language Model Evaluation

Explore similar work

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

VocalGrad: Evaluating Acoustic Perception in Audio Language Models

How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation