Audio Understanding

Follow this topic Ask about this topic

Audio understanding studies how machines can identify and interpret speech, music, environmental sounds, speakers, and the relationships among events in a recording. It matters for applications such as voice assistants, accessibility, health monitoring, and ecological research, where systems must respond to what was actually heard. Common approaches include self-supervised audio encoders trained with masked prediction, contrastive audio-text models, and audio-language models adapted through instruction tuning; retrieval and temporal-grounding methods help handle long recordings and locate relevant events. Models perform well on many standard tasks, but can rely on text or visual cues instead of sound, and remain unreliable on noisy, multilingual, long-form, and fine-grained temporal understanding.

Latest papers

217 papers26 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning
Mar 25, 2026Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda +3Audio Representation Learning Self-Supervised Pre-Training
Tokyo Metropolitan University, Tokyo, Japan · NTT Inc., Atsugi, Japan
When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removal
Mar 20, 2026Yen-Ting Piao, Jay Chiehen Liao, Wei-Tang Chien +5Audio-Language Model Evaluation LLM Evaluation
National Taiwan University, Taiwan
How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation
Mar 19, 2026Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13Audio Understanding Audio-Language Model Evaluation
National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan
Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Models
Mar 19, 2026Yuchen Su, Shaoxin Zhong, Yonghua Zhu +6Audio-Language Model Evaluation Audio Understanding
School of Computer Science, University of Auckland, New Zealand · Singapore University of Technology and Design, Singapore · School of Statistics and Data Science, Nankai University, China +1
DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models
Mar 17, 2026Jiaqi Xiong, Yunjia Qi, Qi Cao +6Audio-Language Model Evaluation Audio-Language Models
University of Oxford · XJTLU · HNU +3
PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark
Mar 15, 2026Mohammad Javad Ranjbar Kalahroodi, Mohammad Amini, Parmis Bathayan +2Audio-Language Model Evaluation Audio Reasoning
School of Electrical and Computer Engineering, University of Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran
Causal Tracing of Audio-Text Fusion in Large Audio Language Models
Mar 14, 2026Wei-Chih Chen, Chien-yu Huang, Hung-yi LeeLLM Interpretability Audio Understanding
National Taiwan University, Taiwan · Carnegie Mellon University, United States
SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases
Mar 10, 2026Laya Iyer, Angelina Wang, Sanmi KoyejoAudio Understanding Audio-Language Model Evaluation
Stanford University · Cornell Tech
MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models
Mar 10, 2026Chih-Kai Yang, Yun-Shao Tsai, Yu-Kai Guo +7Audio-Language Model Evaluation Audio Understanding
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan
The First Environmental Sound Deepfake Detection Challenge: Benchmarking Robustness, Evaluation, and Insights
Mar 5, 2026Han Yin, Yang Xiao, Rohan Kumar Das +2Audio Deepfake Detection Robustness Audio Deepfake Detection
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea · University of Melbourne, Australia · Fortemedia Singapore, Singapore +2
Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Models
Mar 5, 2026Han Yin, Yang Xiao, Younghoo Kwon +2Audio Understanding Audio-Language Models
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea · University of Melbourne, Australia
When Scaling Fails: Mitigating Audio Perception Decay of LALMs via Multi-Step Perception-Aware Reasoning
Feb 28, 2026Ruixiang Mao, Xiangnan Ma, Dan Chen +12Audio Reasoning Audio Understanding
Northeastern University,China · NiuTrans Research.
DashengTokenizer: One layer is enough for unified audio understanding and generation
Feb 27, 2026Heinrich Dinkel, Xingwei Sun, Gang Li +8Audio Tokenization Audio Representation Learning
MiLM Plus, Xiaomi Inc., Beijing, China
SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents
Feb 26, 2026Zeyu Xie, Chenxing Li, Qiao Jin +6Neural Vocoders Text-to-Audio Generation
Peking University, Shenzhen · Tencent AI Lab, Shenzhen · Shanghai AI Lab, Shanghai +1
UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation
Feb 25, 2026Yuxuan Chen, Peize He, Haoyuan Yu +1Audio Representation Learning Audio Understanding
Jilin University · University of Electronic Science and Technology of China · Hunan University +1
A Knowledge-Driven Approach to Music Segmentation, Music Source Separation and Cinematic Audio Source Separation
Feb 25, 2026Chun-wei Ho, Sabato Marco Siniscalchi, Kai Li +1Audio Source Separation Music Source Separation
Georgia Institute of Technology, USA · University of Palermo, Italy · Dolby Laboratories, China
AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration
Feb 23, 2026Henry Zhong, Jörg M. Buchholz, Julian Maclaren +2Clustering LLM-Assisted Annotation
Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia
Re-purposing Multimodal Large Language Models for Audio-Text Retrieval
Feb 20, 2026Jilan Xu, Carl Thomé, Danijela Horak +2Multimodal Large Language Models Audio Understanding
Visual Geometry Group, University of Oxford · Epidemic Sound · School of Artificial Intelligence, Shanghai Jiao Tong University
AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing
Feb 19, 2026William Chen, Prem Seetharaman, Rithesh Kumar +4Audio-Language Models Audio Understanding
Adobe Research · Language Technologies Institute, Carnegie Mellon University · OpenAI
MAEB: Massive Audio Embedding Benchmark
Feb 17, 2026Adnan El Assadi, Isaac Chung, Chenghao Xiao +15Audio Understanding Audio-Language Model Evaluation
Carleton University · Zendesk · Durham University +7
Event-Grounded Question Answering over Long Audio via Structured Retrieval
Feb 16, 2026Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada +2Multimodal RAG Audio QA
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA
Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio Understanding Audio-Language Models
1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research
RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity
Feb 4, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio-Language Model Evaluation Audio QA
University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal Understanding Multimodal Grounding
Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Jan 25, 2026Xilin Jiang, Qiaolin Wang, Junkai Wu +30Audio-Visual Understanding Multimodal Large Language Models
Columbia University · University of Washington · University of Illinois Urbana-Champaign +5
Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs
Jan 18, 2026Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar ChowdhurySpoken Language Understanding Speech Processing
Qatar Computing Research Institute, Qatar
HeartMuLa: A Family of Open Sourced Music Foundation Models
Jan 15, 2026Dongchao Yang, Yuxin Xie, Yuguo Yin +26Text-to-Music Generation Neural Audio Codecs
TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding
Jan 11, 2026Mingyue Huo, Yiwen Shao, Yuheng ZhangMulti-Speaker ASR Audio Understanding
University of Illinois Urbana-Champaign · Johns Hopkins University
FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
Jan 8, 2026Junseok Lee, Chang-Jae ChunMultimodal Large Language Models Speech Processing
OKESTRO · Sejong University
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model Evaluation LLM Evaluation
AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1

Mar 25, 2026J/K move · Enter open · S save

Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen+2

Tokyo Metropolitan University, Tokyo, Japan · NTT Inc., Atsugi, Japan

Audio Representation Learning Self-Supervised Pre-Training Masked Autoencoders Audio Understanding

Open paper AskPDF arXiv

Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning.