Audio-Language Model Evaluation

Overview

The field measures how well models use audio to answer questions, follow spoken instructions, describe sounds, and reason about events over time. Researchers build benchmarks for speech, music, environmental sound, long recordings, multilingual settings, and safety, using multiple-choice and open-ended tasks alongside human ratings. Controlled tests—such as removing or altering audio, changing answer order, or making speech conflict with text—help reveal shortcuts, bias, and whether a model’s answer is actually grounded in sound; uncertainty estimates and model-based judges can make evaluation more scalable. A central challenge is that strong benchmark scores may coexist with weak acoustic grounding, temporal reasoning, and robustness to changes in wording, speakers, or recording conditions.

Latest papers

CardsList

225 papers23 papers in the last four weeks, down 38% on the four weeks before. 0.2% of all new papers.

  1. Does the Voice Matter? Evaluating Frustration Detection on Deployed Clinical Voice-Agent Calls

    Nov 16, 2026Annika Michael, Zachary Ellis, Yuxiang Zhou +4Voice Agent EvaluationClinical Speech Processing

  2. Clinician Assessment of Laryngeal Disease with Auditory AI before Laryngoscopy: A Paired Reader Study

    Nov 15, 2026Siyuan Hou, Shuting Yang, Zengrui Jin +13Clinical Decision SupportClinical Prediction

  3. Three Invisible Failure Modes in Speech-Based Age Estimation for Child Safety

    Nov 13, 2026Christopher M. Stewart, Kelsey Kraus, J. Elliott Casal +1Language Model Safety EvaluationLLM Reliability

  4. Right Answers, Inconsistent Evidence: Source-Level Counterfactual Auditing of Large Audio-Language Models

    Nov 11, 2026Yi Zhang, Yi Li, Hongwei DuAudio-Language Model EvaluationSound Event Detection

  5. AuDiR: Diagnosing Audio Reasoning with Rule and Percept Interventions

    Nov 9, 2026Chengwei Wei, Yi Ma, Nancy F. ChenAudio ReasoningAudio-Language Model Evaluation

  6. Beyond the Transcript: Measuring and Improving the Use of Acoustic Information in Audio Language Models

    Nov 8, 2026Linkai Peng, Baorian Nuchged, Yuyang YaoAudio-Language ModelsAudio-Language Model Evaluation

  7. LongCap: Hour-Scale Timestamped Audio Captioning via Chunk-wise Processing

    Nov 8, 2026Jinchuan Tian, Sen Chen, Wenbin Huang +7Audio CaptioningAudio-Language Models

  8. Machine Hearing: Can Audio LLMs Generalize Beyond Human Auditory Perception?

    Nov 7, 2026Liwei Liu, Jun Tao, Xingyuan Li +8Audio-Language ModelsAudio Understanding

  9. Do Multi-Turn Conversational Capabilities Carry Over from Text to Speech? A Parallel Multilingual Benchmark

    Nov 3, 2026Jiaming Zhou, Pei Zhang, Shiwan Zhao +3Audio-Language Model EvaluationSpeech Language Models

  10. PianoBench: Evaluating Musical Perception and Reasoning in Language Models

    Nov 2, 2026Louis Bradshaw, Irmak Bukey, Daniel Chenyu Lin +2Audio-Language Model EvaluationMLLM Reasoning

  11. Voice-Native LLMs Hear Frustration but Do Not Listen to It

    Nov 2, 2026Hanjun Luo, Yuhan Lu, Hanan SalamAudio-Language Model EvaluationSpeech Language Models

  12. One Waveform, Two Stories: Benchmarking Speech-Acoustic Conflict in Audio-Language Models

    Oct 30, 2026Wenbin Wang, Yuheng Li, Koichiro YoshinoAudio-Language Model EvaluationAudio-Language Models

  13. Scalable Audio-LLM System for Speech MOS Evaluation: A Diagnostic Case Study and Adaptation Roadmap

    Oct 30, 2026Dimitrios Kokkinis, Panagiotis Kakoulidis, Nikolaos Ellinas +10Speech Quality AssessmentTTS Evaluation

  14. Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models

    Oct 29, 2026Fengji Jing, Guy LabanMultimodal Emotion RecognitionAudio-Language Model Evaluation

  15. MixCards: Evaluating Structured Descriptions of Multi-Source Sound Scenes

    Oct 28, 2026Sripathi Sridhar, Yujin Kim, Giovana Morais +6Audio-Language Model EvaluationAudio Understanding

  16. Beyond Answer Changes: Matched Same-Class Controls for Local Audio Evidence

    Oct 27, 2026Yunuo Wang, Maoyuan Chen, Yuanjin Huang +1Audio UnderstandingAudio-Language Model Evaluation

  17. MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model

    Oct 22, 2026Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen +1Audio-Text AlignmentAutomated Evaluation

  18. MixJudge: Learning a Language-Queryable Critic for Vocal Mixing

    Oct 22, 2026Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh +1Audio-Language ModelsAudio Editing

  19. AT-Differ: Dataset for Audio-Text Difference Captioning

    Oct 21, 2026Akihiro Fujii, Yuki Okamoto, Yuki Saito +1Audio-Text AlignmentAudio-Language Model Evaluation

  20. Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

    Oct 21, 2026Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi +2Audio-Language Model EvaluationSpeech Language Models

  21. Heard But Not Said: Prompting Understates Temporal Information of Musical Moments in Frozen Large Audio-Language Models

    Oct 21, 2026Eunsik ShinAudio-Language ModelsAudio-Language Model Evaluation

  22. Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis

    Oct 21, 2026Rupali Rajesh, Sethuraman T, Savya Khosla +4Tool-Augmented ReasoningMusic Understanding

  23. Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits

    Oct 19, 2026Ankur Banerjee, Vishnu Guntakala, Danity Pike +3Audio-Language ModelsAudio-Language Model Evaluation

  24. VocalGrad: Evaluating Acoustic Perception in Audio Language Models

    Oct 16, 2026Koki Ryu, Hitomi YanakaAudio-Language Model EvaluationAudio Understanding

  25. ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

    Oct 16, 2026Ming-Hao Hsu, Shujie Liu, Yuxuan Hu +4Audio-Language ModelsSpeech Language Models

  26. TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

    Oct 15, 2026Giovana Morais, Yujin Kim, Atilay Kucukoglu +1Audio-Language Model EvaluationAudio-Language Models

  27. Qwen-Audio-Jev: A Unified Interface for Audio Decisions

    Oct 15, 2026Peixin Chen, Zhigao Chen, Zhiyun Fan +19Audio-Language ModelsAudio-Language Model Evaluation

  28. A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

    Oct 15, 2026Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu +4Audio CaptioningAudio-Language Model Evaluation