Audio-Language Models

Ask about this topic

Overview

Audio-language models learn to connect speech, music, and environmental sounds with language so they can answer questions, describe events, retrieve audio, and generate speech or music. Common approaches align audio and text with contrastive dual encoders, or connect audio encoders to language models through projectors, discrete tokens, or continuous representations; training often combines large-scale pretraining with instruction tuning and preference or reinforcement learning. Benchmarks increasingly test temporal grounding, overlapping sounds, multilingual speech, and conversational or clinical settings, alongside standard captioning and question-answering tasks. A central challenge is ensuring answers actually depend on acoustic evidence rather than text priors, while improving robustness, faithful reasoning, and efficiency.

Latest papers

CardsList

303 papers55 papers in the last four weeks, up 22% on the four weeks before. 0.3% of all new papers.

  1. MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model

    May 5, 2026Jingyao GongSpeech Foundation ModelsUnified Multimodal Models

    Independent Researcher

  2. ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

    May 5, 2026Honglei Zhang, Yuting Chen, Chenpeng Hu +2Audio ReasoningAudio Understanding

    School of Software, Nanjing University Nanjing, China · School of Software, Northwestern Polytechnical University Xi’an, China · College of Computing and Data Science, Nanyang Technological University Singapore, Singapore

  3. When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition

    May 4, 2026Pehuén Moure, Niclas Pokel, Bilal Bounajma +4Audio-Language Model EvaluationASR Evaluation

    Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland · Department of Mathematics, ETH Zurich, Switzerland · Department of Computational Linguistics, University of Zurich, Switzerland

  4. GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models

    May 1, 2026Zuyao You, Zhesong Yu, Mingyu Liu +3Music UnderstandingMultimodal Pretraining

    1Fudan University · 2ByteDance

  5. Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

    Apr 28, 2026Chun-Yi Kuan, Wei-Ping Huang, Hung-yi LeeSemantic EntropyAudio Understanding

    Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan

  6. Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

    Apr 27, 2026NVIDIA, :, Amala Sanjay Deshmukh +216Efficient Multimodal InferenceMultimodal Large Language Models

  7. All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

    Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model EvaluationModality Reliance in Multimodal Models

    National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

  8. HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models

    Apr 26, 2026Peize He, Yaodi Luo, Xiaoqian Liu +7Audio Token CompressionSelf-Attention

    1EPIC Lab, Shanghai Jiao Tong University · 2DAIL Tech · 3Northeastern University +2

  9. Au-M-ol: A Unified Model for Medical Audio and Language Understanding

    Apr 25, 2026Meizhu Liu, Nistha Mitra, Paul Li +3Medical Speech RecognitionSpeech Processing

    Oracle AI Science · Neuramill

  10. Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training

    Apr 23, 2026Yoshinori NomuraLanguage Model PretrainingAudio-Language Models

    Mirage Mountain Technologies Inc.

  11. SpeechParaling-Bench: A Comprehensive Benchmark for Paralinguistic-Aware Speech Generation

    Apr 22, 2026Ruohan Liu, Shukang Yin, Tao Wang +6Audio-Language Model EvaluationLLM Evaluation

    1Nanjing University · 2Xiaomi

  12. HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models

    Apr 21, 2026Feiyu Zhao, Yiming Chen, Wenhuan Lu +3Audio-Language Model EvaluationAudio-Language Model Hallucination Detection

    College of Intelligence and Computing, Tianjin University, China · ASUS Intelligent Cloud Services, Singapore

  13. Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval

    Apr 20, 2026HaeJun Yoo, Yongseop Shin, Insung Lee +2Audio-Language Model EvaluationMultimodal IR

    Sogang University

  14. Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models

    Apr 20, 2026Xiang He, Chenxing Li, Jinting Wang +5RL for Language Model ReasoningAudio Reasoning

    Tencent AI Lab, Beijing, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

  15. Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment

    Apr 20, 2026Woody Haosheng Gan, William Held, Diyi YangAudio-Language Model EvaluationHuman Preference Modeling

    University of Southern California · Stanford University · OpenAthena

  16. VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

    Apr 19, 2026Yi-Cheng Lin, Yusuke Hirota, Sung-Feng Huang +1Audio-Language Model EvaluationGender Bias in Language Models

    Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · NVIDIA Research NVIDIA Taipei, Taiwan · AI Center of Research Excellence National Taiwan University Taipei, Taiwan

  17. Benign Fine-Tuning Breaks Safety Alignment in Audio LLMs

    Apr 17, 2026Jaechul Roh, Virat Shejwalkar, Amir HoumansadrLanguage Model Safety EvaluationLLM Fine-Tuning

    University of Massachusetts Amherst

  18. NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages

    Apr 17, 2026Marie Maltais, Yejin Jeon, Min Ma +7Speech TranslationAudio-Language Models

    Mila - Quebec AI Institute · McGill University, Canada · Google DeepMind +6

  19. Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularization

    Apr 17, 2026Habibeh Naderi, Behrouz Haji Soleimani, Stan MatwinCross-Modal LearningModality Imbalance

    Dalhousie University, Halifax NS, Canada

  20. TinyMU: A Compact Audio-Language Model for Music Understanding

    Apr 17, 2026Xiquan Li, Aurian Quelennec, Slim EssidMusic UnderstandingSmall Language Models

    LTCI, T´el´ecom Paris, Institut Polytechnique de Paris · Shanghai Jiao Tong University

  21. Listen, Pause, and Reason: Toward Perception-Grounded Hybrid Reasoning for Audio Understanding

    Apr 16, 2026Jieyi Wang, Yazhe Niu, Dexuan Xu +1Audio QAAudio Reasoning

    Shanghai AI Laboratory · Peking University · CUHK MMLab +1

  22. Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection

    Apr 16, 2026Meng Chen, Kun Wang, Li Lu +2Audio-Language ModelsLLM Security

    The State Key Laboratory of Blockchain and Data Security, Zhejiang University · Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security · National University of Singapore +1

  23. Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models

    Apr 16, 2026Yanda Li, Yuhan Liu, Zirui Song +3Audio UnderstandingAudio-Language Models

    Mohamed bin Zayed University of Artificial Intelligence, UAE · Beijing Jiaotong University, China

  24. Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt

    Apr 15, 2026Yanfeng Shi, Pengfei Cai, Jun Liu +5Audio-Language ModelsTemporal Audio Grounding

    National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China · ICT Cluster, Singapore Institute of Technology, Singapore

  25. SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

    Apr 14, 2026Luoyi Sun, Xiao Zhou, Zeqian Li +3Temporal Audio GroundingAudio-Language Models

    Zhejiang University Hangzhou, Zhejiang, China · SAI, Shanghai Jiao Tong University Shanghai, China

  26. Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

    Apr 14, 2026Linhao Zhang, Yuhan Song, Aiwei Liu +6Audio UnderstandingAudio-Language Models

    Basic Model Technology Center, WeChat AI, Tencent Inc. · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University

  27. Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

    Apr 13, 2026Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar +15Audio UnderstandingAudio-Language Models

    NVIDIA, USA · University of Maryland, USA

  28. Whisper-AuT: Domain-Adapted Audio Encoder for Efficient Audio-LLM Training

    Apr 12, 2026Jielin Qiu, Ming Zhu, Wenting Zhao +11Audio Representation LearningDomain Adaptation

    Salesforce AI Research

  29. GRM: Utility-Aware Jailbreak Attacks on Audio LLMs via Gradient-Ratio Masking

    Apr 10, 2026Yunqiang Wang, Hengyuan Na, Di Wu +2LLM Jailbreak AttacksAdversarial Perturbations

    Sun Yat-Sen University Guangzhou Yunshan Research Institute of Artificial Intelligence Security Guangzhou, China · Sun Yat-Sen University Guangzhou, China

  30. Noise-Aware In-Context Learning for Hallucination Mitigation in ALLMs

    Apr 10, 2026Qixuan Huang, Khalid Zaman, Masashi UnokiAudio-Language ModelsAudio-Language Model Hallucination Detection

    Graduate School of Advanced Science and Technology, Japan Advanced Institute of Science and Technology 1-1 Asahidai, Nomi, Ishikawa 923–1292 Japan