Audio-Language Model Evaluation

Overview

The field measures how well models use audio to answer questions, follow spoken instructions, describe sounds, and reason about events over time. Researchers build benchmarks for speech, music, environmental sound, long recordings, multilingual settings, and safety, using multiple-choice and open-ended tasks alongside human ratings. Controlled tests—such as removing or altering audio, changing answer order, or making speech conflict with text—help reveal shortcuts, bias, and whether a model’s answer is actually grounded in sound; uncertainty estimates and model-based judges can make evaluation more scalable. A central challenge is that strong benchmark scores may coexist with weak acoustic grounding, temporal reasoning, and robustness to changes in wording, speakers, or recording conditions.

Latest papers

CardsList

225 papers23 papers in the last four weeks, down 38% on the four weeks before. 0.2% of all new papers.

  1. MAEB: Massive Audio Embedding Benchmark

    Feb 17, 2026Adnan El Assadi, Isaac Chung, Chenghao Xiao +15Audio UnderstandingAudio-Language Model Evaluation

  2. RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

    Feb 4, 2026Gaia A. Bertolino, Yuwei Zhang, Tong Xia +2Audio-Language Model EvaluationAudio QA

  3. The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfer

    Jan 30, 2026Yupeng Chen, Junchi Yu, Aoxi Liu +3MLLM SafetyAdversarial Attacks on Unified Multimodal Models

  4. SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

    Jan 29, 2026Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum +2Multimodal UnderstandingMultimodal Grounding

  5. The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models

    Jan 6, 2026Yuhuan You, Lai Wei, Tianshu QuAudio UnderstandingSpatial Audio

  6. ORCA: Open-ended Response Correctness Assessment for Audio Question Answering

    Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model EvaluationLLM Evaluation

  7. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    Nov 20, 2025Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3Audio-Language Model EvaluationAudio Representation Learning

  8. VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

    Oct 13, 2025Jiliang Hu, Wenfu Wang, Zuchao Li +6Audio-Language Model EvaluationVoice Agent Evaluation

  9. Robustness assessment of large audio language models in multiple-choice evaluation

    Oct 6, 2025Fernando López, Santosh Kesiraju, Jordi LuqueAudio-Language Model EvaluationAudio QA

  10. Evaluating Multimodal Large Language Models on Spoken Sarcasm Detection

    Sep 18, 2025Zhu Li, Xiyuan Gao, Yuqing Zhang +2MLLM EvaluationSarcasm Detection

  11. AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs

    Sep 9, 2025Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte +9Audio-Language Model EvaluationAudio-Language Models

  12. WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations

    Aug 28, 2025Jaeyeon Kim, Heeseung Yun, Sang Hoon Woo +2Audio-Language Model EvaluationAudio Understanding