Audio-Language Model Evaluation

Ask about this topic

Momentum

25 papers in the last four weeks, up 108% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 125

All topics
CardsList

  1. CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

    Oct 7, 2026Marcel Gibier, Thomas Thebaud, Olivier Boëffard +1Audio UnderstandingAudio Captioning

    Inria, Paris, France · AMIAD, France

  2. Logbook: Extremely Long-form Audio Event Understanding

    Oct 5, 2026Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk +7Audio-Language Model EvaluationAudio Understanding

    UT Austin, USA · Meta Reality Labs, USA

  3. RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments

    Oct 1, 2026Peihao Chen, Qing Wang, Lichun Fan +12Audio-Language Model EvaluationAudio QA

    University of Science and Technology of China, China · Xiaomi Corporation, China · DataoceanAI, China +2

  4. SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models

    Sep 30, 2026Rong Wan, Suliu Qin, Jiaxi Li +5Audio-Language Model EvaluationAudio Deepfake Detection

    University of Surrey · Singapore University of Technology and Design · Guangxi University +1

  5. Role-guided Speaker Deletion Verification in Clinical Psychiatry Speech Recordings with Audio Language Models

    Sep 29, 2026Joseph T Colonel, Daniel Katzman, Kelsey Kirker +6Audio-Language Model EvaluationSpeaker Verification

    Icahn School of Medicine at Mount Sinai, Department of Psychiatry, New York, NY, USA

  6. When Capabilities Fail to Compose: Diagnosing the Compositionality Gap in Large Audio-Language Models

    Sep 29, 2026Chien-Feng Liu, Chih-Kai Yang, Bo-Han Feng +3Audio-Language Model EvaluationAudio QA

    National Taiwan University · ASUS Open Cloud Infrastructure Software Center · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)

  7. HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models

    Sep 28, 2026Shen Yan, Duc Le, Irina-Elena VelicheAudio-Language Model EvaluationAudio-Language Models

    Meta Superintelligence Labs

  8. Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions

    Sep 28, 2026Michel Olvera, Paraskevas Stamatiadis, Changhong Wang +1Audio-Language Model EvaluationAudio Reasoning

    S2A, LTCI, IDS · S2A, IDS

  9. SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences

    Sep 28, 2026Mingyue Huo, Shivam Mehta, Bhavin Jawade +2Audio-Language Model EvaluationHuman Preference Modeling

    University of Illinois Urbana-Champaign · Netflix

  10. MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

    Sep 27, 2026Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen +1Audio-Language Model EvaluationAudio-Language Model Hallucination Detection

    University of Neuchâtel · TU Dortmund University · RWTH Aachen University +1

  11. Audio LLMs Know When They Can't Hear You

    Sep 24, 2026Amirhosein Javadi, Richa Dixit, Mehrdad Farajtabar +3Audio-Language Model EvaluationLanguage Model Self-Assessment

    Apple · University of California San Diego

  12. To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

    Sep 24, 2026Debajyoti Mazumder, Mamta, Abhirama Subramanyam PenamakuriAudio-Language Model EvaluationRetrieval-Augmented Generation

    IISER Bhopal · King’s College London · MBZUAI

  13. Broadening Uncertainty Estimation for Audio Question Answering Across Methods, Formats, and Inputs

    Sep 24, 2026Aaron Isidore Grace, Weiran WangAudio-Language Model EvaluationAudio QA

    David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada · Department of Computer Science, University of Iowa, Iowa City, IA, USA

  14. Spooftral: Can Voxtral Audio-Language Model Detect Speech Spoofing?

    Sep 23, 2026Avishai Weizman, Yehuda Ben-Shimol, Itshak LapidotAudio-Language Model EvaluationAudio Deepfake Detection

    School of Electrical and Computer Engineering, Ben-Gurion University of the Negev, Israel · Department of Electrical Engineering, Afeka the Academic College of Engineering, Israel · Avignon University, LIA, France

  15. MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents

    Sep 21, 2026Chenxu Xiong, Dongming Shen, Yuzhi Tang +3Audio-Language Model EvaluationVoice Agent Evaluation

    Boson AI Santa Clara, CA 95054

  16. Text Scores Do Not Establish Performance on Lexically Non-Diagnostic Speech Tasks: A Qwen2-Audio Quantization Case Study

    Sep 20, 2026Mengzhe Geng, Jinxi Ji, Junhao XuAudio-Language Model EvaluationMixed-Precision Quantization

    National Research Council Canada · The Hong Kong Polytechnic University · The Chinese University of Hong Kong

  17. MuLA-Bench: A Multilingual Long-Form Audio Understanding Benchmark via Multi-Tier Auditing

    Sep 20, 2026Zeyu Yang, Xinyu Zhang, Zibo Bi +5Audio-Language Model EvaluationAudio Understanding

    The Chinese University of Hong Kong, Shenzhen · Alibaba Token Hub, Alibaba Group · Xi’an Jiaotong University

  18. Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs

    Sep 19, 2026Jing Peng, Zichao Nie, Zhisheng Zhang +2Audio-Language Model EvaluationAudio Representation Learning

    Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

  19. Correlation-Guided Encoder Selection for Multi-Encoder Large Audio-Language Models

    Sep 16, 2026Pei-Jun Liao, Hung-Shin Lee, Wenze Ren +3Audio-Language Model EvaluationAudio-Language Models

    Inst. of Information Science, Academia Sinica, Taiwan · Grad. Inst. of Electrical Engineering, National Taiwan University, Taiwan · Grad. Inst. of AI Interdisciplinary Applied Technology, National Taiwan Normal University, Taiwan +3

  20. CLASH: Counterfactual Auditing of Lexical and Prosodic Reliance in Spoken Sarcasm Detection

    Sep 15, 2026Qiyang Sun, Xudong Li, Yupei Li +4Audio-Language Model EvaluationCounterfactual Evaluation

    Imperial College London · New York University · Technical University of Munich +3

  21. MUUNRiver-Bench: Diagnosing Relation-Dependent Music Retrieval with Multimodal Instructions

    Sep 14, 2026Zhancheng Guo, Congren Dai, Shangda Wu +4Audio-Language Model EvaluationAudio-Text Retrieval

    Central Conservatory of Music, Beijing, China · Tsinghua University, Beijing, China

  22. Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

    Sep 12, 2026Chibuzor Okocha, Christan Earl GrantAudio-Language Model EvaluationCode-Switching Speech Recognition

    Department of Computer & Information Science & Engineering University of Florida, Gainesville, FL, USA

  23. SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ

    Sep 11, 2026Huy Hoang Le, Long-Bao Nguyen, Minh Tri DaoAudio-Language Model EvaluationSpoken Language Understanding

    CAKE by VPBank, Vietnam

  24. Nuha-Speech: Building General-Purpose Arabic Speech-LLMs

    Sep 11, 2026Yingzhi Wang, Reem Alhazzani, Muhammad AlqurishiAudio-Language Model EvaluationAudio QA

    Elm Company, KSA

  25. Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

    Sep 9, 2026Yupei Li, Qiyang Sun, Mohamed Mady +4Audio-Language Model EvaluationAudio Reasoning

    Imperial College London, UK · Technische Universität München München, Germany · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, AE +2

  26. SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia

    Sep 9, 2026Jingyi Liao, Wenyu Zhang, Zhuohan Liu +6Audio-Language Model EvaluationASR Evaluation

    Institute of Advanced Intelligence and Computing, A*STAR · Nanyang Technological University · Center for AI Safety +1

  27. Reasoning Beyond Transcription: Audio Language Models on Child Stuttering Speech

    Sep 7, 2026Chibuzor Okocha, Christan Grant, Zoey LiuAudio-Language Model EvaluationSpoken Language Understanding

    University of Florida, USA

  28. Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

    Sep 1, 2026Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh +1Audio-Language Model EvaluationAudio-Language Models

    Language Technologies Institute Carnegie Mellon University

  29. VoiceLongMemEval: Do Assistants Remember How You Sounded?

    Sep 1, 2026Ramit Pahwa, Parivesh Priye, Apoorva BeeduAudio-Language Model EvaluationLong-Context Language Model Evaluation

    Rivian VW Tech

  30. When Models Hear What They Expect: Diagnosing Prosodic Heuristics in Multimodal Sarcasm Detection

    Aug 31, 2026Yongjian Chen, Pengfei Wei, Yiqun Sun +2Audio-Language Model EvaluationPragmatic Reasoning in Language Models

    Magellan Technology Research Institute (MTRI) · Center for Language and Cognition, University of Groningen, the Netherlands

  31. VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

    Aug 28, 2026Luc Debaupte, Tyler Baumgartner, Brandon Tai +3Audio-Language Model EvaluationSpeech Emotion Recognition

    Besimple AI, San Mateo, CA

  32. Auditing generative audio calls for known-task audio-llm evaluation

    Aug 28, 2026Mengzhe GengAudio-Language Model EvaluationAudio Classification

    National Research Council Canada

  33. SURE-Voice: A Front-End Baseline for Speech-Evidence Filtering in Speech LLMs

    Aug 27, 2026Mengzhe GengAudio-Language Model EvaluationHallucination Detection

    National Research Council Canada

  34. LongAudioSpan: Spanning the Duration and Depth of Audio Comprehension

    Aug 26, 2026Wen Huang, Yunfei Chu, Meng Gao +2Audio-Language Model EvaluationAudio Reasoning

    Qwen Team, Alibaba Group · Tsinghua University · The Chinese University of Hong Kong

  35. Lost in Speech: Trilingual Spoken Hallucination Detection Across Audio and Transcripts

    Aug 25, 2026Meruyert Aristombayeva, Jason S. Lucas, Chaewan Chun +1Audio-Language Model EvaluationHallucination Detection

    Satbayev University, Almaty, Kazakhstan · University of Colorado Boulder, Boulder, CO, USA · The Pennsylvania State University, University Park, PA, USA

  36. MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models

    Aug 23, 2026Yize Li, Ningyuan Yang, Sile Yin +6Audio-Language Model EvaluationAudio Reasoning

    Northeastern University · Stony Brook University · Bose Corporation

  37. The SLT 2026 SmartGlasses Challenge: Benchmarking Egocentric Multi-Talker Speech Recognition and Understanding with Audio-Language Models

    Aug 12, 2026Dehui Gao, Zhixian Zhao, Zhennan Lin +14Audio-Language Model EvaluationASR Evaluation

    Huawei · AIShell Inc · Shanghai Jiao Tong University +4

  38. Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

    Aug 10, 2026Oluwanifemi Bamgbose, Simon Rosen, Jash Shah +6Audio-Language Model EvaluationHuman Preference Evaluation

    ServiceNow

  39. From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs

    Aug 10, 2026Yuanhe Zhang, Weiliu Wang, Jie Ren +7Audio-Language Model EvaluationAudio-Language Models

    Beijing University of Posts and Telecommunications · Institute of Information Engineering, Chinese Academy of Sciences · Nanyang Technological University +3

  40. Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation

    Aug 7, 2026Kevin Miller, Arjun Chandra, Venkatesh SaligramaAudio-Language Model EvaluationLanguage Model Auditing

    Boston University

  41. Spoken Function Calling: A New Perspective on Spoken Language Understanding for Large Audio Language Models

    Aug 5, 2026Yuezhang Peng, Yuxin Liu, Changfeng Gao +4Audio-Language Model EvaluationSpoken Language Understanding

    Shanghai Jiao Tong University Shanghai, China · Token Foundry, Alibaba Group Hangzhou, China · Shanghai Jiao Tong University, Shanghai Innovation Institute Shanghai, China

  42. Separating Decision-Rule Misalignment from Readout-Coverage Limitations in Speech Language Models

    Aug 3, 2026Linkai Peng, Baorian NuchgedAudio-Language Model EvaluationSpeech Emotion Recognition

    Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT 06269, USA · Department of Linguistics, The University of Texas at Austin, Austin, TX 78712, USA

  43. Can Foundation Models Hear What Made That Sound? A Tiered Benchmark of Audio-Language Models and Traditional Classifiers for Closed-Set Sound Source Identification

    Aug 3, 2026Sajjad Abdoli, Ghassan Al-Sumaidaee, Ahmad ElShiekh +1Audio-Language Model EvaluationAudio-Language Models

    Perle

  44. TORUS: A Test of Rendering-Understanding Self-Coherence for Unified Audio Models

    Jul 30, 2026Aryan Vijay Bhosale, Harshit Rajgarhia, Abhishek Mukherji +1Audio-Language Model EvaluationAudio Editing

    Centific Global Solutions Inc. · University of Maryland

  45. MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning

    Jul 29, 2026Weijie Wu, Junbo Li, Lin Li +2Audio-Language Model EvaluationAudio Captioning

    School of Informatics, Xiamen University, China · DiDi Global Inc., Beijing, China · School of Electronic Science and Engineering, Xiamen University, China

  46. Prosody-driven Jailbreaks in Audio LLMs: A Controlled Study and Mechanistic Analysis

    Jul 29, 2026Jiachen Qian, Junyu LiAudio-Language Model EvaluationLLM Safety Benchmarks

    City University of Hong Kong · City University of Hong Kong (Dongguan)

  47. ESCUCHA: A Spanish Speech Benchmark for Heterogeneous Acoustic Conditions

    Jul 20, 2026Fernando López, Ana Ayala, Guillermo Segovia +4Audio-Language Model EvaluationSpoken Language Understanding

    Telefónica Innovación Digital · Universidad Autónoma de Madrid · Universidad Complutense de Madrid

  48. Large Audio Language Models for Spoofing-Aware Speaker Verification

    Jul 16, 2026Sofya Savelyeva, Mariia Perunova, Evgeny Kushnir +3Audio-Language Model EvaluationAudio Deepfake Detection

    Applied AI Institute · MIRAI · AXXX, HSE +3

  49. Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

    Jul 15, 2026Joonyong Park, David M. Chan, Yuki Saito +1Audio-Language Model EvaluationLLM Auditing

    Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Berkeley AI Research (BAIR), University of California, Berkeley, CA, USA

  50. Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models

    Jul 15, 2026Chun-Yi Kuan, Siwon Kim, Byeonggeun Kim +7Audio-Language Model EvaluationText-to-Audio Generation

    National Taiwan University, Taiwan · Amazon, USA

  51. The Sound of Absence: Audio-Language Embedding Models Struggle with Negation

    Jul 14, 2026Chun-Yi Kuan, Hung-yi LeeAudio-Language Model EvaluationAudio-Language Models

    ♡Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · ♣Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan

  52. Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models

    Jul 11, 2026Yun-Shao Tsai, Chun-Wei Chen, Chee-En Yu +2Audio-Language Model EvaluationSpeech Language Models

    Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · Graduate Institute of Electrical Engineering National Taiwan University Taipei, Taiwan · Artificial Intelligence Center of Research Excellence National Taiwan University Taipei, Taiwan

  53. A Reliability Assessment of LALM Audio Judges for Full-Duplex Voice Agents

    Jul 8, 2026A. Sayyad, J. Emmons, S. Jones +2Audio-Language Model EvaluationVoice Agent Evaluation

    Salesforce Applied AI Research, eVerse team

  54. Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music

    Jul 7, 2026Tomáš Sourada, Katia Vendrame, Jan HajičAudio-Language Model EvaluationSynthetic Benchmark Generation

    Charles University · Brno University of Technology

  55. SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

    Jul 6, 2026Thomas Thebaud, Yuzhe Wang, Hao Zhang +5Audio-Language Model EvaluationLanguage Model Generation Evaluation

    ∗Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA · †Amazon Inc., Seattle, WA, USA

  56. Adaptive Perturbation Selection for Contrastive Audio Decoding

    Jun 30, 2026Aaron Isidore Grace, Zhouyuan Huo, Weiran WangAudio-Language Model EvaluationAudio-Language Model Hallucination Detection

    Department of Computer Science University of Iowa Iowa City, IA, USA · Google Mountain View, CA, USA

  57. Beyond Binary Instrument QA: Probing Instrument Grounding in Music Audio-Language Models

    Jun 30, 2026Yujun Lee, Joonhyeok Shin, Hyoeun Kim +1Audio-Language Model EvaluationAudio QA

    Sungkyunkwan University

  58. RedVox: Safety and Fairness Gaps in Speech Models Across Languages

    Jun 25, 2026Beatrice Savoldi, Sara Papi, Wafa Aissa +2Audio-Language Model EvaluationAI Safety Evaluation

    Fondazione Bruno Kessler, Italy

  59. FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following

    Jun 25, 2026Zhihang Xie, Marco Gaido, Sara Papi +2Audio-Language Model EvaluationSpeech Processing

    Fondazione Bruno Kessler · University of Trento

  60. Real-Time Voice AI Hears but Does Not Listen

    Jun 24, 2026Martijn Bartelds, Federico Bianchi, James ZouAudio-Language Model EvaluationVoice Agent Evaluation

    Together AI · Stanford University