Audio Understanding

Follow this topic Ask about this topic

Audio understanding studies how machines can identify and interpret speech, music, environmental sounds, speakers, and the relationships among events in a recording. It matters for applications such as voice assistants, accessibility, health monitoring, and ecological research, where systems must respond to what was actually heard. Common approaches include self-supervised audio encoders trained with masked prediction, contrastive audio-text models, and audio-language models adapted through instruction tuning; retrieval and temporal-grounding methods help handle long recordings and locate relevant events. Models perform well on many standard tasks, but can rely on text or visual cues instead of sound, and remain unreliable on noisy, multilingual, long-form, and fine-grained temporal understanding.

Latest papers

217 papers26 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model Evaluation Audio Representation Learning
University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1
Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering
Oct 14, 2025Tsung-En Lin, Kuan-Yi Lee, Hung-Yi LeeAudio QA Hallucination in Language Models
National Taiwan University, Taipei, Taiwan · ASUS Open Cloud Infrastructure Software Center, Taipei, Taiwan
Do Modern Video-LLMs Need to Listen? A Benchmark Audit and Scalable Remedy
Sep 22, 2025Geewook Kim, Minjoon SeoVLM Evaluation Audio-Visual Understanding
NAVER Cloud AI, South Korea · KAIST AI, South Korea
BabyHuBERT: Multilingual Self-Supervised Learning for Segmenting Speakers in Child-Centered Long-Form Recordings
Sep 18, 2025Théo Charlot, Tarek Kunze, Maxime Poli +3Speech Foundation Models Speech Processing
LSCP, DEC, ENS, EHESS, CNRS, PSL University, France · Laboratoire d’Informatique et Syst`emes, Universit´e Aix-Marseille, CNRS, France
WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations
Aug 28, 2025Jaeyeon Kim, Heeseung Yun, Sang Hoon Woo +2Audio-Language Model Evaluation Audio Understanding
Carnegie Mellon University · Seoul National University · NVIDIA
OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
Jul 18, 2025Shikhar Bharadwaj, Samuele Cornell, Kwanghee Choi +4Audio Representation Learning Audio QA
Carnegie Mellon University, USA · National Institute of Advanced Industrial Science and Technology (AIST), Japan
Audio-Language Models for Audio-Centric Tasks: A Systematic Survey
Jan 25, 2025Yi Su, Jisheng Bai, Qisheng Xu +2Audio Representation Learning Audio Understanding
College of Computer Science and Technology, National University of Defense Technology, Changsha, China · School of Communications and Information Engineering, Xi’an University of Posts and Telecommunications, Xi’an, China

Nov 20, 2025J/K move · Enter open · S save

Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo, Yiwen Shao+2

University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University+1

Audio-Language Model Evaluation Audio Representation Learning Audio Understanding Audio-Language Models Audio Captioning Multimodal Contrastive Learning

Open paper AskPDF arXiv

Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-language models can build effective general-purpose audio encoders, nor a systematic understanding of how pretraining objectives behave across diverse tasks and scales. We identify three key barriers: limited scale of audio-text corpora, limited coverage of audio attributes in existing caption corpora, and lack of systematic exploration and evaluation. To fill this gap, we present the first principled empirical study of ALP. We first introduce CaptionStew, a 10.7M caption dataset aggregating open-source audio-text corpora across multiple domains and captioning focuses. We then conduct the first comprehensive evaluation comparing contrastive and captioning objectives for learning audio representation across speech, music, and environmental sound tasks. Our results not only demonstrate that ALP yields competitive, transferable representations, but reveal critical trade-offs: contrastive learning offers superior data efficiency, while captioning exhibits better scalability. Furthermore, we find that the benefits of supervised initialization often diminish at larger scales, challenging common practices. By grounding these claims in empirical evidence, we establish a viable pathway toward general-purpose audio representation learning, guiding future research.