Audio-Language Model Evaluation

Ask about this topic

Overview

The field measures how well models use audio to answer questions, follow spoken instructions, describe sounds, and reason about events over time. Researchers build benchmarks for speech, music, environmental sound, long recordings, multilingual settings, and safety, using multiple-choice and open-ended tasks alongside human ratings. Controlled tests—such as removing or altering audio, changing answer order, or making speech conflict with text—help reveal shortcuts, bias, and whether a model’s answer is actually grounded in sound; uncertainty estimates and model-based judges can make evaluation more scalable. A central challenge is that strong benchmark scores may coexist with weak acoustic grounding, temporal reasoning, and robustness to changes in wording, speakers, or recording conditions.

Latest papers

CardsList

206 papers33 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

  1. Multi-Source Evidence Fusion for Audio Question Answering

    Mar 18, 2026Aivo Olev, Tanel AlumäeAudio QAAudio Reasoning

    Tallinn University of Technology, Estonia

  2. DEAF: A Benchmark for Diagnostic Evaluation of Acoustic Faithfulness in Audio Language Models

    Mar 17, 2026Jiaqi Xiong, Yunjia Qi, Qi Cao +6Audio-Language Model EvaluationAudio-Language Models

    University of Oxford · XJTLU · HNU +3

  3. PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark

    Mar 15, 2026Mohammad Javad Ranjbar Kalahroodi, Mohammad Amini, Parmis Bathayan +2Audio-Language Model EvaluationAudio Reasoning

    School of Electrical and Computer Engineering, University of Tehran, Iran · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran

  4. Do What I Say: A Spoken Prompt Dataset for Instruction-Following

    Mar 10, 2026Maike Züfle, Sara Papi, Fabian Retkowski +5Audio-Language Model EvaluationSpeech Language Models

    Karlsruhe Institute of Technology, Germany · Fondazione Bruno Kessler, Italy · ACC Cyfronet AGH, Poland +2

  5. SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases

    Mar 10, 2026Laya Iyer, Angelina Wang, Sanmi KoyejoAudio UnderstandingAudio-Language Model Evaluation

    Stanford University · Cornell Tech

  6. MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models

    Mar 10, 2026Chih-Kai Yang, Yun-Shao Tsai, Yu-Kai Guo +7Audio-Language Model EvaluationAudio Understanding

    Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

  7. How Contrastive Decoding Enhances Large Audio Language Models

    Mar 10, 2026Tzu-Quan Lin, Wei-Ping Huang, Yi-Cheng Lin +1Contrastive DecodingAudio-Language Models

    Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

  8. Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

    Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language ModelsAudio-Language Model Evaluation

    Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

  9. PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio

    Mar 5, 2026Yuanjian Chen, Yang Xiao, Han Yin +3Audio ReasoningAudio-Language Model Evaluation

    Harbin University of Science and Technology, China · The University of Melbourne, Australia · KAIST, South Korea +1

  10. A Sensitivity Analysis of Multi-Event Audio Grounding in Audio LLMs

    Mar 4, 2026Taehan Lee, Jaehan Jung, Hyukjun LeeAudio-Language Model EvaluationAudio-Language Models

    Sogang University, South Korea

  11. Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

    Mar 2, 2026Ming-Hao Hsu, Xiaohai Tian, Jun Zhang +3Speech Language ModelsAudio-Language Model Evaluation

    School of Data Science, The Chinese University of Hong Kong, Shenzhen · ByteDance

  12. Voices of Civilizations: A Multilingual QA Benchmark for Global Music Understanding

    Feb 28, 2026Shangda Wu, Ziya Zhou, Yongyi Zang +4Music UnderstandingAudio-Language Model Evaluation

    Independent Researcher · The Hong Kong University of Science and Technology, Hong Kong SAR, China · The Chinese University of Hong Kong, Hong Kong SAR, China

  13. AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

    Feb 27, 2026Jielin Qiu, Jianguo Zhang, Zixiang Chen +13Audio-Language Model EvaluationAudio Captioning

    Salesforce AI Research

  14. Evaluation of Audio Language Models for Fairness, Safety, and Security

    Feb 25, 2026Ranya Aloufi, Srishti Gupta, Soumya Shaw +2Audio-Language Model EvaluationAudio-Language Models

  15. The Cascade Equivalence Hypothesis: When Do Speech LLMs Behave Like ASR→\rightarrowLLM Pipelines?

    Feb 19, 2026Jayadev BillaSpeech Language ModelsAudio-Language Model Evaluation

    San Jose CA, USA.

  16. MAEB: Massive Audio Embedding Benchmark

    Feb 17, 2026Adnan El Assadi, Isaac Chung, Chenghao Xiao +15Audio UnderstandingAudio-Language Model Evaluation

    Carleton University · Zendesk · Durham University +7

  17. RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

    Feb 4, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio-Language Model EvaluationAudio QA

    University of Cambridge · Cambridge, United Kingdom · Tsinghua University +3

  18. SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

    Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal UnderstandingMultimodal Grounding

    Vector Institute for Artificial Intelligence, MaRS Centre, Toronto, ON M5G 1L7, Canada · University of Groningen, Nijenborgh 4, 9747 AG Groningen, Netherlands · York University, 4700 Keele Street, Toronto, ON M3J 1P3, Canada

  19. ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

    Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model EvaluationLLM Evaluation

    AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1

  20. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model EvaluationAudio Representation Learning

    University of Texas at Austin · Tencent AI Lab Seattle · Zhejiang University +1

  21. VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

    Oct 13, 2025Jiliang Hu, Wenfu Wang, Zuchao Li +6Audio-Language Model EvaluationVoice Agent Evaluation

    School of Artificial Intelligence, Wuhan University, Wuhan, China · Tencent AI Lab, Beijing, China

  22. Robustness assessment of large audio language models in multiple-choice evaluation

    Oct 6, 2025Fernando López, Santosh Kesiraju, Jordi LuqueAudio-Language Model EvaluationAudio QA

    Scientific Research, Telef´onica Innovaci´on Digital, Spain · Brno University of Technology, Czech Republic

  23. Hearing the Order: Investigating Position Bias in Large Audio-Language Models

    Oct 1, 2025Yu-Xiang Lin, Chen-An Li, Sheng-Lun Wei +3Audio-Language Model EvaluationAudio-Language Models

    National Taiwan University

  24. Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

    Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2MLLM EvaluationSarcasm Detection

    University of Groningen, The Netherlands

  25. AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs

    Sep 9, 2025Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte +9Audio-Language Model EvaluationAudio-Language Models

    ServiceNow · University of Texas at Austin

  26. WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

    Aug 28, 2025Jaeyeon Kim, Heeseung Yun, Sang Hoon Woo +2Audio-Language Model EvaluationAudio Understanding

    Carnegie Mellon University · Seoul National University · NVIDIA