Audio Understanding

Ask about this topic

Overview

Audio understanding studies how machines can identify and interpret speech, music, environmental sounds, speakers, and the relationships among events in a recording. It matters for applications such as voice assistants, accessibility, health monitoring, and ecological research, where systems must respond to what was actually heard. Common approaches include self-supervised audio encoders trained with masked prediction, contrastive audio-text models, and audio-language models adapted through instruction tuning; retrieval and temporal-grounding methods help handle long recordings and locate relevant events. Models perform well on many standard tasks, but can rely on text or visual cues instead of sound, and remain unreliable on noisy, multilingual, long-form, and fine-grained temporal understanding.

Latest papers

CardsList

217 papers26 papers in the last four weeks, up 86% on the four weeks before. 0.2% of all new papers.

  1. MixCards: Evaluating Structured Descriptions of Multi-Source Sound Scenes

    Oct 28, 2026Sripathi Sridhar, Yujin Kim, Giovana Morais +6Audio-Language Model EvaluationAudio Understanding

    New Jersey Institute of Technology · New York University · Adobe Research +1

  2. Beyond Answer Changes: Matched Same-Class Controls for Local Audio Evidence

    Oct 27, 2026Yunuo Wang, Maoyuan Chen, Yuanjin Huang +1Audio UnderstandingAudio-Language Model Evaluation

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · School of Computer Science, Wuhan University, Wuhan, China · National University of Singapore, Singapore +1

  3. Correspondence Is Not Transfer: A Pairing-Controlled Study of Clinical Audio-Metadata Alignment

    Oct 26, 2026Liu He, Xi Wang, Jiahong Yuan +1Cross-Modal AlignmentAudio Understanding

    University of Science and Technology of China, China · Independent Researcher · University of Cambridge, UK

  4. Interpretable Singing Emotion Recognition Using Acoustic and Vocal Technique Features

    Oct 23, 2026Fotini Kiriakopoulou, Vassilis Lyberatos, Spyridon Kantarelis +2Emotion RecognitionAudio Understanding

    National Technical University of Athens, Athens, Greece

  5. Qwen-Audio-3.1-ASR Technical Report

    Oct 20, 2026Chuanmeng Bian, Daren Chen, Peixin Chen +45Audio-Language ModelsAutomatic Speech Recognition

  6. Multilingual Voicemail and Answering Machine Detection via Pretrained Audio Embeddings and Gradient Boosting Machines

    Oct 19, 2026Kemal Altwlkany, Muamer ParicAudio ClassificationAudio Understanding

    Infobip · Faculty of Science, University of Sarajevo

  7. VocalGrad: Evaluating Acoustic Perception in Audio Language Models

    Oct 16, 2026Koki Ryu, Hitomi YanakaAudio-Language Model EvaluationAudio Understanding

    The University of Tokyo · Riken · Tohoku University

  8. MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

    Oct 16, 2026Kun Liu, Lyra TaoAudio-Language Model EvaluationMusic Understanding

  9. MedSonor: Learning to Use Clinical Audio in Medical LLMs

    Oct 14, 2026Geyang Yu, Junying Chen, Benyou WangMedical LLMsAudio-Language Models

    The Chinese University of Hong Kong, Shenzhen

  10. AMemOps: Evaluating Memory Updates in Long Audio Streams

    Oct 13, 2026Peiwei Chang, Li Zhang, Mark D. Plumbley +2Audio UnderstandingAssociative Memory

    Centre for Vision, Speech and Signal Processing, University of Surrey, UK · Department of Computer Science, Royal Holloway, University of London, UK · Department of Informatics, King’s College London, UK

  11. ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

    Oct 12, 2026Ashutosh Chaubey, Jinxi Guo, Minh Tran +2Temporal Audio GroundingAudio Understanding

    Meta Superintelligence Labs · University of Southern California

  12. SpectralBridge: Adding High-Frequency Capability to Large Audio-Language Models

    Oct 12, 2026Sergei Polezhaev, Saba Anjum, Michael GormishAudio-Language ModelsAudio Understanding

    Nebius AI

  13. PitchIn: Natural-Language Vocal Coaching from Unstructured Community Feedback

    Oct 12, 2026Hayeon Bang, Hounsu Kim, Joonyong Park +2Audio-Language ModelsAudio Understanding

    KAIST, Republic of Korea · The University of Tokyo, Japan · ETRI, Republic of Korea

  14. The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations

    Oct 12, 2026Liumeng Xue, Yuang Cao, Jingbin Hu +12Nonverbal Vocalization GenerationAudio Understanding

    School of Intelligence Science and Technology, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University · Shenzhen Pimei Technology +5

  15. QC-MI-OVSEL: Benchmarking Query-Conditioned Multi-Instance Open-Vocabulary Sound Event Localization

    Oct 9, 2026Yongqiang Chen, Peiwei Chang, Yuelan Cheng +2Audio UnderstandingSpatial Audio

    Centre for Vision, Speech and Signal Processing, University of Surrey, UK

  16. MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness

    Oct 8, 2026Jinbo Hu, Hang Su, Lichun Fan +10Audio-Language ModelsAudio Understanding

    Xiaomi Inc., China

  17. CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

    Oct 7, 2026Marcel Gibier, Thomas Thebaud, Olivier Boëffard +1Audio UnderstandingAudio Captioning

    Inria, Paris, France · AMIAD, France

  18. ARIA: Audio-Driven Melody-Tone Relation Modeling for Cantonese Lyric Authoring

    Oct 6, 2026Shengyu Li, Jinting Wang, Li LiuAudio UnderstandingText Generation

    The Hong Kong University of Science and Technology (Guangzhou)

  19. SkillFormer: Skill-Decomposed Adaptation for Audio Language Models

    Oct 6, 2026Lee Seung-woo, Bowen Qi, Kim Min-jun +1Multi-Task LearningAudio Understanding

    Pusan National University · Shanghai Jiao Tong University · Hanyang University

  20. Logbook: Extremely Long-form Audio Event Understanding

    Oct 5, 2026Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk +7Audio-Language Model EvaluationAudio Understanding

    UT Austin, USA · Meta Reality Labs, USA

  21. Revisiting Frame-Wise Saliency for Audio Moment Retrieval

    Oct 5, 2026Tatsuya Komatsu, Hokuto MunakataDetection TransformerAudio Understanding

    LY Corporation, Tokyo, Japan

  22. Hallucination Reduction for LLM-Based Audio Understanding via Multimodal Direct Preference Optimization

    Oct 2, 2026Bebe Cosgrove, Aaron Isidore Grace, Weiran WangAudio UnderstandingAudio-Language Models

    Smith College, Northampton, USA · University of Iowa, Iowa City, USA

  23. Revisiting Input Time-frequency Representations in Multi-pitch Estimation for Vocal Ensembles

    Oct 2, 2026Junyoung Koh, Hao-Wen DongMusic Information RetrievalAudio Understanding

    Yonsei University · University of Michigan

  24. Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript

    Oct 2, 2026Jie Jin, Ziyin Ma, Min Yin +4Audio QAAudio Understanding

    Adventists.ai, Melbourne, Australia · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Future Design Lab, Innovation Center of Yangtze River Delta, Zhejiang University, Jiaxing, China +1

  25. LAST: Looped Audio Spectrogram Transformer

    Oct 1, 2026Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina +1Audio UnderstandingRecurrent Transformers

    Georgia Institute of Technology · University of California San Diego · Google DeepMind

  26. Multi-agent Auditory Scene Analysis: Improved Localization Speed and Robustness by Multi-beamformed Speech Quality Feedback

    Sep 30, 2026Caleb RasconMulti-Agent System OptimizationDirection-of-Arrival Estimation

    Instituto de Investigaciones en Matematicas Aplicadas y en Sistemas, Universidad Nacional Autonoma de Mexico, Circuito Escolar 3000, Coyoacan, 03740, Ciudad de Mexico, Mexico.

  27. UniAE-MoE: A Unified Audio Encoder via Mixture of Experts

    Sep 30, 2026Shengbo Cai, Zhisheng Zhang, Zichao Nie +3Audio Representation LearningMixture of Experts

    Tsinghua University, China

  28. Audio Token Attention Is Predictable Before the Language Model Runs

    Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token CompressionAudio Understanding

    The University of Texas at Austin

  29. Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions

    Sep 28, 2026Michel Olvera, Paraskevas Stamatiadis, Changhong Wang +1Audio-Language Model EvaluationAudio Reasoning

    S2A, LTCI, IDS · S2A, IDS

  30. NSV-Shift: A Contrastive Benchmark for Non-Speech Vocalization Understanding and Response Adaptation in Speech-to-Speech Models

    Sep 27, 2026Ziwei ChenSpeech Generation EvaluationAudio Understanding

    University of California San Diego