cs.SDOct 12, 2026

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

Authors: Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian, Jianing Cai, Ziwei Huang, Qing Fan, Zhiyuan Yang, 

Organizations: Southeast University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing UA-Bench, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0%--13.1% leaf accuracy, against a random reference of 7.7%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.

Figures & tables

Figure 1: UA-Bench at a glance. An underwater platform (AUV, ROV, or submarine) equipped with an intelligent passive sonar system must interpret the sound field around it. We build UA-Bench, the first underwater acoustic understanding benchmark, to evaluate large audio-language models (LALMs) on this task. Despite their strong general-purpose audio understanding, the best of the nine evaluated LALMs reaches only 13.1% accuracy at the finest level (L3), compared with 7.7% for the random-guess reference, which shows that current LALMs are far from ready to listen to the ocean.
Figure 2: Overview of UA-Bench. Detection pulses, communication waveforms, and ship-radiated noise are synthesized under traceable source and propagation conditions. Each record includes a three-level label and a reference dialogue that proceeds from source category to leaf class and then requests an explanation. The benchmark measures recognition accuracy, separates routing errors from downstream errors, and compares performance across acoustic conditions. It also checks whether each explanation agrees with the predicted label.
ResourceSignal focusScalePrimary targetLALMHier.Dialogue/QASource meta.Prop. ctrl.
Audio-language benchmarks
AIR-Bench [ yang2024air ]Speech, sound, music21k QFound. + chat✓✗✓✗✗
MMAU [ sakshi2025mmau ]Speech, sound, music10k QAPercept. + reason.✓✗✓✗✗
Dynamic-SUPERB P2 [ huang2025dynamic ]Speech, music, sound180 tasksCls. / reg. / gen.∼\bm{\sim}✗∼\bm{\sim}∼\bm{\sim}✗
AudioBench [ wang2025audiobench ]Speech, voice, scenes100k+ samp.8 tasks / 26 sets✓✗∼\bm{\sim}∼\bm{\sim}✗
WoW-Bench [ kim2025wowbench ]Marine mammal calls1,777 QPercept. + cognit.✓✗✓✗✗
Table 1: Comparison with representative audio-language benchmarks and underwater acoustic resources.
L1L2L3 classes
Table 2: UA-Bench label hierarchy. A source category, signal family, and specific class form each label path.
More in this topic

Underwater Acoustics

See all 19 papers

Explore similar work

CardsList
Oct 15, 2026

From Signals to Evidence: Interpretable Underwater Acoustic Target Sensing with Large Language Models - Part I: Framework and Recognition

Underwater acoustic target detection and recognition are essential to marine surveillance and intelligent ocean sensing. Despite substantial advances in deep learning, existing systems typically produce binary detection decisions, categorical labels, or scalar confidence scores, offering limited insight into the physical evidence underlying their outputs. Large language models (LLMs) offer an opportunity to move underwater acoustic sensing toward interpretable, evidence-grounded understanding; however, generic audio encoders struggle to capture the distinctive physical characteristics of underwater acoustics, and physically annotated data are scarce. In this paper, the first of a two-part series, we propose a from-signals-to-evidence framework that integrates physics-aware acoustic encoding with a pretrained LLM, organized around a unified feature-evidence-decision chain with a shared reasoning core. An open input layer accommodates heterogeneous information-raw waveforms, time-frequency representations, environmental and geographic data, and textual physical priors-unifying detection and recognition within a common formulation. In Part I, we instantiate this framework for target recognition through an Underwater Acoustic Physics-aware Transformer (UPT), which tokenizes complementary acoustic representations and learns task-disentangled physical representations through four physics-oriented learning tasks. We develop a physics-parameterized data engine that provides traceable supervision at two levels of granularity and introduce an attribute-level evaluation protocol to assess the physical fidelity of generated descriptions. Experiments on simulated recordings of four vessels demonstrate accurate vessel recognition and reliable estimation of shaft frequency, blade-passing frequency, and machinery-related attributes.

Yuge Zhang, Jie Chen, Zhengqiao Zhao +4

School of Artificial Intelligence, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China · School of Ocean Engineering and Technology, Sun Yat-sen University, Zhuhai, 519000, Guangdong, China · School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China +1

Underwater AcousticsAudio Representation LearningPhysics-Informed MLAudio-Language Models
AskPDF
Mar 10, 2026

SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.

Laya Iyer, Angelina Wang, Sanmi Koyejo

Stanford University · Cornell Tech

Audio UnderstandingAudio-Language Model EvaluationSpoken Language Understanding
AskPDF
Sep 27, 2026

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims. Existing audio hallucination benchmarks mainly measure response correctness, leaving it unclear whether an LALM hallucinates or simply fails to understand the audio. We challenge correctness-based evaluation by defining two hallucination categories: (i) context, where claims are not grounded in the audio; and (ii) knowledge, where claims about audio-related topics lack support from externally verifiable facts. We introduce MISHAP-Bench, a comprehensive benchmark with 12,000 challenging open-ended question-audio pairs and a rigorous evaluation pipeline covering both categories. To evaluate open-ended responses, we propose a groundedness judge that uses reference rubrics and judge prompts guided by human annotations. We extensively evaluate ten state-of-the-art LALMs and show that hallucination remains substantial. Even a frontier model such as Gemini 3.7 Flash reaches a hallucination rate of 36.5%. We further adapt and benchmark four mitigation methods from multiple domains for LALMs. Despite some improvements, effective hallucination mitigation remains an open challenge. Finally, we call on the community to evaluate hallucination and benchmark mitigation methods with MISHAP-Bench.

Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen +1

University of Neuchâtel · TU Dortmund University · RWTH Aachen University +1

Audio-Language Model EvaluationAudio-Language Model Hallucination DetectionLanguage Model Generation EvaluationAudio-Language ModelsHallucination Mitigation
AskPDF