UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

Zhe Sun; Liang An; Jiayu Yao; Ronglai Qian; Jianing Cai; Ziwei Huang; Qing Fan; Zhiyuan Yang

cs.SDOct 12, 2026

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

Authors: Zhe Sun, Liang An, Jiayu Yao, Ronglai Qian, Jianing Cai, Ziwei Huang, Qing Fan, Zhiyuan Yang,

Organizations: Southeast University

Abstract

Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domains remains largely open. Vessel monitoring, maritime safety, and ocean sensing all depend on recognizing underwater signals, but to our knowledge LALMs have not been systematically evaluated on them. Annotated recordings are costly to collect, and a faithful synthetic alternative has to bring together many interacting factors: the structure of detection pulses, communication waveforms, and ship-radiated noise, and the sound-speed profile, geometry, and multipath propagation through which each reaches the receiver. We address this by proposing UA-Bench, to our knowledge the first large-scale benchmark for evaluating LALMs on underwater acoustic signal recognition. It comprises 375,051 audio records that couple parameterized sources with modeled propagation, so that every record can be traced to the conditions that produced it. The records span these three signal families, and each carries a three-level label hierarchy (13 leaf classes), a templated multi-turn dialogue, and source and channel metadata. We evaluate on a balanced 2,600-record subset with a gated coarse-to-fine protocol that locates each error at source routing, fine-grained classification, or output format, and add a paired silent-audio control to test whether the waveform is actually used. Nine open LALMs, evaluated without task-specific adaptation, reach only 2.0%--13.1% leaf accuracy, against a random reference of 7.7%. The waveform helps source routing far more than leaf recognition (for one model, +29.5 versus +2.3 points over silence), and just four models gain reliably at the leaf level, by at most 4.7 points. UA-Bench thus gives a reference point for these models, and its released training partition supports follow-up adaptation work.

Figures & tables

Figure 1: UA-Bench at a glance. An underwater platform (AUV, ROV, or submarine) equipped with an intelligent passive sonar system must interpret the sound field around it. We build UA-Bench, the first underwater acoustic understanding benchmark, to evaluate large audio-language models (LALMs) on this task. Despite their strong general-purpose audio understanding, the best of the nine evaluated LALMs reaches only 13.1% accuracy at the finest level (L3), compared with 7.7% for the random-guess reference, which shows that current LALMs are far from ready to listen to the ocean.

Figure 2: Overview of UA-Bench. Detection pulses, communication waveforms, and ship-radiated noise are synthesized under traceable source and propagation conditions. Each record includes a three-level label and a reference dialogue that proceeds from source category to leaf class and then requests an explanation. The benchmark measures recognition accuracy, separates routing errors from downstream errors, and compares performance across acoustic conditions. It also checks whether each explanation agrees with the predicted label.

Audio-language benchmarks
Resource	Signal focus	Scale	Primary target	LALM	Hier.	Dialogue/QA	Source meta.	Prop. ctrl.
AIR-Bench [ yang2024air ]	Speech, sound, music	21k Q	Found. + chat	✓	✗	✓	✗	✗
MMAU [ sakshi2025mmau ]	Speech, sound, music	10k QA	Percept. + reason.	✓	✗	✓	✗	✗
Dynamic-SUPERB P2 [ huang2025dynamic ]	Speech, music, sound	180 tasks	Cls. / reg. / gen.	$\bm{\sim}$	✗	$\bm{\sim}$	$\bm{\sim}$	✗
AudioBench [ wang2025audiobench ]	Speech, voice, scenes	100k+ samp.	8 tasks / 26 sets	✓	✗	$\bm{\sim}$	$\bm{\sim}$	✗
WoW-Bench [ kim2025wowbench ]	Marine mammal calls	1,777 Q	Percept. + cognit.	✓	✗	✓	✗	✗

Table 1: Comparison with representative audio-language benchmarks and underwater acoustic resources.

L3 classes

Table 2: UA-Bench label hierarchy. A source category, signal family, and specific class form each label path.

Underwater Acoustics

See all 19 papers

From Signals to Evidence: Interpretable Underwater Acoustic Target Sensing with Large Language Models - Part I: Framework and Recognition
Oct 15, 2026Yuge Zhang, Jie Chen, Zhengqiao Zhao +4Underwater Acoustics Audio Representation Learning
School of Artificial Intelligence, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China · School of Ocean Engineering and Technology, Sun Yat-sen University, Zhuhai, 519000, Guangdong, China · School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China +1
SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases
Mar 10, 2026Laya Iyer, Angelina Wang, Sanmi KoyejoAudio Understanding Audio-Language Model Evaluation
Stanford University · Cornell Tech
MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models
Sep 27, 2026Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen +1Audio-Language Model Evaluation Audio-Language Model Hallucination Detection
University of Neuchâtel · TU Dortmund University · RWTH Aachen University +1

Oct 15, 2026J/K move · Enter open · S save

From Signals to Evidence: Interpretable Underwater Acoustic Target Sensing with Large Language Models - Part I: Framework and Recognition

Yuge Zhang, Jie Chen, Zhengqiao Zhao, Xiling Yao+3

School of Artificial Intelligence, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China · School of Ocean Engineering and Technology, Sun Yat-sen University, Zhuhai, 519000, Guangdong, China · School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, Shaanxi 710129, China+1

Underwater Acoustics Audio Representation Learning Physics-Informed ML Audio-Language Models

Open paper AskPDF arXiv

Underwater acoustic target detection and recognition are essential to marine surveillance and intelligent ocean sensing. Despite substantial advances in deep learning, existing systems typically produce binary detection decisions, categorical labels, or scalar confidence scores, offering limited insight into the physical evidence underlying their outputs. Large language models (LLMs) offer an opportunity to move underwater acoustic sensing toward interpretable, evidence-grounded understanding; however, generic audio encoders struggle to capture the distinctive physical characteristics of underwater acoustics, and physically annotated data are scarce. In this paper, the first of a two-part series, we propose a from-signals-to-evidence framework that integrates physics-aware acoustic encoding with a pretrained LLM, organized around a unified feature-evidence-decision chain with a shared reasoning core. An open input layer accommodates heterogeneous information-raw waveforms, time-frequency representations, environmental and geographic data, and textual physical priors-unifying detection and recognition within a common formulation. In Part I, we instantiate this framework for target recognition through an Underwater Acoustic Physics-aware Transformer (UPT), which tokenizes complementary acoustic representations and learns task-disentangled physical representations through four physics-oriented learning tasks. We develop a physics-parameterized data engine that provides traceable supervision at two levels of granularity and introduce an attribute-level evaluation protocol to assess the physical fidelity of generated descriptions. Experiments on simulated recordings of four vessels demonstrate accurate vessel recognition and reliable estimation of shaft frequency, blade-passing frequency, and machinery-related attributes.

Abstract

Underwater Acoustics

Explore similar work

From Signals to Evidence: Interpretable Underwater Acoustic Target Sensing with Large Language Models - Part I: Framework and Recognition

SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models