Audio-Language Models

Ask about this topic

Overview

Audio-language models learn to connect speech, music, and environmental sounds with language so they can answer questions, describe events, retrieve audio, and generate speech or music. Common approaches align audio and text with contrastive dual encoders, or connect audio encoders to language models through projectors, discrete tokens, or continuous representations; training often combines large-scale pretraining with instruction tuning and preference or reinforcement learning. Benchmarks increasingly test temporal grounding, overlapping sounds, multilingual speech, and conversational or clinical settings, alongside standard captioning and question-answering tasks. A central challenge is ensuring answers actually depend on acoustic evidence rather than text priors, while improving robustness, faithful reasoning, and efficiency.

Latest papers

CardsList

303 papers55 papers in the last four weeks, up 22% on the four weeks before. 0.3% of all new papers.

  1. Disentangling Reasoning in Large Audio-Language Models for Ambiguous Emotion Prediction

    Mar 9, 2026Xiaofeng Yu, Jiaheng Dong, Jean Honorio +3Speech Emotion RecognitionEmotion Recognition

    University of Auckland, New Zealand · The University of Melbourne, Australia · University of Birmingham, United Kingdom

  2. Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings

    Mar 9, 2026Fan Wu, Tsai-Ning Wang, Nicolas Zumarraga +8Audio-Text AlignmentClinical QA

    Agentic Systems Lab, ETH Zurich Switzerland · Eindhoven University of Technology Netherlands · Centre for Digital Health Interventions, ETH Zurich +2

  3. Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

    Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language ModelsAudio-Language Model Evaluation

    Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

  4. RAMoEA-QA: Hierarchical Specialization for Robust Respiratory Audio Question Answering

    Mar 6, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio QAMedical QA

    University of Cambridge, United Kingdom · Tsinghua University, China · University of Calabria, Italy

  5. PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

    Mar 5, 2026Yuanjian Chen, Yang Xiao, Han Yin +3Audio ReasoningAudio-Language Model Evaluation

    Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea +1

  6. TW-Sound580K: A Regional Audio-Text Dataset with Verification-Guided Curation for Localized Audio-Language Modeling

    Mar 5, 2026Hao-Hui Xie, Ho-Lam Chung, Yi-Cheng Lin +4Audio-Language ModelsSpeech Processing

    National Taiwan University · Shanghai Jiao Tong University

  7. Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Models

    Mar 5, 2026Han Yin, Yang Xiao, Younghoo Kwon +2Audio UnderstandingAudio-Language Models

    School of Electrical Engineering, KAIST, Daejeon, Republic of Korea · University of Melbourne, Australia

  8. A Sensitivity Analysis of Multi-Event Audio Grounding in Audio LLMs

    Mar 4, 2026Taehan Lee, Jaehan Jung, Hyukjun LeeAudio-Language Model EvaluationAudio-Language Models

    Sogang University, South Korea

  9. When Scaling Fails: Mitigating Audio Perception Decay of LALMs via Multi-Step Perception-Aware Reasoning

    Feb 28, 2026Ruixiang Mao, Xiangnan Ma, Dan Chen +12Audio ReasoningAudio Understanding

    Northeastern University,China · NiuTrans Research.

  10. StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks

    Feb 27, 2026Yishan Wang, Tsai-Ning Wang, Mathias Funk +1Audio-Language ModelsRespiratory Sound Classification

    Eindhoven University of Technology

  11. Evaluation of Audio Language Models for Fairness, Safety, and Security

    Feb 25, 2026Ranya Aloufi, Srishti Gupta, Soumya Shaw +2Audio-Language Model EvaluationAudio-Language Models

  12. SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

    Feb 25, 2026Paweł Pozorski, Jakub Muszyński, Maria GanzhaShapley Value AttributionAudio-Language Models

    Warsaw University of Technology, Warsaw, Poland

  13. Re-purposing Multimodal Large Language Models for Audio-Text Retrieval

    Feb 20, 2026Jilan Xu, Carl Thomé, Danijela Horak +2Multimodal Large Language ModelsAudio Understanding

    Visual Geometry Group, University of Oxford · Epidemic Sound · School of Artificial Intelligence, Shanghai Jiao Tong University

  14. AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

    Feb 19, 2026William Chen, Prem Seetharaman, Rithesh Kumar +4Audio-Language ModelsAudio Understanding

    Adobe Research · Language Technologies Institute, Carnegie Mellon University · OpenAI

  15. Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens

    Feb 18, 2026Potsawee Manakul, Woody Haosheng Gan, Martijn Bartelds +3Audio-Language ModelsAudio Generation

    Stanford University · SCB 10X · University of Southern California +2

  16. Spatial Audio Question Answering and Reasoning on Dynamic Source Movements

    Feb 18, 2026Arvind Krishna Sridhar, Yinyi Guo, Erik VisserSpatial AudioAudio Reasoning

    Qualcomm Technologies Inc. San Diego, CA

  17. Event-Grounded Question Answering over Long Audio via Structured Retrieval

    Feb 16, 2026Kartik Hegde, Arvind Krishna Sridhar, Naveen Vakada +2Multimodal RAGAudio QA

    Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA

  18. Hello-Chat: Towards Realistic Social Audio Interactions

    Feb 16, 2026Yueran Hou, Peilei Jia, Zihan Sun +5Expressive Speech SynthesisAudio-Language Models

  19. Encode Once, Decode Never: Reusing Audio LM Internals for Efficient Temporal Localization

    Feb 10, 2026Joseph An, Phillip Keung, Jiaqi Wang +2Temporal Audio GroundingAudio-Language Models

    University of Washington · Allen Institute for Artificial Intelligence

  20. Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions

    Feb 5, 2026Jinchuan Tian, Haoran Wang, Bo-Hao Su +14Audio UnderstandingAudio-Language Models

    1Language Technologies Institute, Carnegie Mellon University · 2LY Corporation · 3NVIDIA Research

  21. RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

    Feb 4, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio-Language Model EvaluationAudio QA

    University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3

  22. Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs

    Jan 18, 2026Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar ChowdhurySpoken Language UnderstandingSpeech Processing

    Qatar Computing Research Institute, Qatar

  23. HeartMuLa: A Family of Open Sourced Music Foundation Models

    Jan 15, 2026Dongchao Yang, Yuxin Xie, Yuguo Yin +26Text-to-Music GenerationNeural Audio Codecs

  24. ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

    Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model EvaluationLLM Evaluation

    AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1

  25. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model EvaluationAudio Representation Learning

    University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1

  26. Silence is Golden: Mitigating Hallucinations in Large Audio-Language Models via Layer-Weighted Vector Steering

    Oct 14, 2025Tsung-En Lin, Kuan-Yi Lee, Hung-Yi LeeAudio QAHallucination in Language Models

    National Taiwan University, Taipei, Taiwan · ASUS Open Cloud Infrastructure Software Center, Taipei, Taiwan

  27. VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

    Oct 13, 2025Jiliang Hu, Wenfu Wang, Zuchao Li +6Audio-Language Model EvaluationVoice Agent Evaluation

    School of Artificial Intelligence, Wuhan University, Wuhan, China · Tencent AI Lab, Beijing, China

  28. Robustness assessment of large audio language models in multiple-choice evaluation

    Oct 6, 2025Fernando López, Santosh Kesiraju, Jordi LuqueAudio-Language Model EvaluationAudio QA

    Scientific Research, Telef´onica Innovaci´on Digital, Spain · Brno University of Technology, Czech Republic

  29. Hearing the Order: Investigating Position Bias in Large Audio-Language Models

    Oct 1, 2025Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei +3Audio-Language Model EvaluationAudio-Language Models

    National Taiwan University

  30. Aligning Audio Captions with Human Preferences

    Sep 18, 2025Kartik Hegde, Rehana Mahfuz, Yinyi Guo +1Reward ModelingAudio-Language Models

    Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA