Audio-Language Model Evaluation

Ask about this topic

Overview

The field measures how well models use audio to answer questions, follow spoken instructions, describe sounds, and reason about events over time. Researchers build benchmarks for speech, music, environmental sound, long recordings, multilingual settings, and safety, using multiple-choice and open-ended tasks alongside human ratings. Controlled tests—such as removing or altering audio, changing answer order, or making speech conflict with text—help reveal shortcuts, bias, and whether a model’s answer is actually grounded in sound; uncertainty estimates and model-based judges can make evaluation more scalable. A central challenge is that strong benchmark scores may coexist with weak acoustic grounding, temporal reasoning, and robustness to changes in wording, speakers, or recording conditions.

Latest papers

CardsList

206 papers33 papers in the last four weeks, level with the four weeks before. 0.2% of all new papers.

  1. Beyond Answer Changes: Matched Same-Class Controls for Local Audio Evidence

    Oct 27, 2026Yunuo Wang, Maoyuan Chen, Yuanjin Huang +1Audio UnderstandingAudio-Language Model Evaluation

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · School of Computer Science, Wuhan University, Wuhan, China · National University of Singapore, Singapore +1

  2. MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model

    Oct 22, 2026Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen +1Audio-Text AlignmentAutomated Evaluation

    GICE, National Taiwan University · AI-CoRE, National Taiwan University · Taiwan AILabs

  3. MixJudge: Learning a Language-Queryable Critic for Vocal Mixing

    Oct 22, 2026Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh +1Audio-Language ModelsAudio Editing

    Independent Researcher · Music Informatics Group, Georgia Institute of Technology · Music and AI Lab, National Taiwan University +1

  4. AT-Differ: Dataset for Audio-Text Difference Captioning

    Oct 21, 2026Akihiro Fujii, Yuki Okamoto, Yuki Saito +1Audio-Text AlignmentAudio-Language Model Evaluation

    The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan

  5. Measuring Local Acoustic Contribution in Speech LLMs via Word-Span Counterfactual Interventions

    Oct 21, 2026Hardik B. Sailor, Qiongqiong Wang, Dheemant Rastogi +2Audio-Language Model EvaluationSpeech Language Models

    †Institute of Advanced Intelligence and Computing, Agency for Science, Technology and Research, Singapore · ⋆Nanyang Technological University, Singapore

  6. Heard But Not Said: Prompting Understates Temporal Information of Musical Moments in Frozen Large Audio-Language Models

    Oct 21, 2026Eunsik ShinAudio-Language ModelsAudio-Language Model Evaluation

    Music and Audio Research Group (MARG), Department of Intelligence and Information, Seoul National University

  7. Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis

    Oct 21, 2026Rupali Rajesh, Sethuraman T, Savya Khosla +4Music UnderstandingAudio-Language Model Evaluation

    IIT Madras · UIUC · Google +2

  8. Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits

    Oct 19, 2026Ankur Banerjee, Vishnu Guntakala, Danity Pike +3Audio-Language ModelsAudio-Language Model Evaluation

    Independent Researcher · Brown University · University of Birmingham

  9. VocalGrad: Evaluating Acoustic Perception in Audio Language Models

    Oct 16, 2026Koki Ryu, Hitomi YanakaAudio-Language Model EvaluationAudio Understanding

    The University of Tokyo · Riken · Tohoku University

  10. ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

    Oct 16, 2026Ming-Hao Hsu, Shujie Liu, Yuxuan Hu +4Audio-Language ModelsSpeech Language Models

    School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research Asia, Hong Kong, China. · Microsoft Research, Redmond, WA, USA.

  11. MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

    Oct 16, 2026Kun Liu, Lyra TaoAudio-Language Model EvaluationMusic Understanding

  12. TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

    Oct 15, 2026Giovana Morais, Yujin Kim, Atilay Kucukoglu +1Audio-Language Model EvaluationAudio-Language Models

    Music and Audio Research Lab, NYU · Integrated Design and Media, NYU

  13. Qwen-Audio-Jev: A Unified Interface for Audio Decisions

    Oct 15, 2026Peixin Chen, Zhigao Chen, Zhiyun Fan +19Audio-Language ModelsAudio-Language Model Evaluation

    Token Foundry, Alibaba Group

  14. A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

    Oct 15, 2026Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu +4Audio CaptioningAudio-Language Model Evaluation

    Center for Language and Speech Processing, Johns Hopkins University, USA · Otter AI · The University of Texas at Dallas, USA +1

  15. ACR-Net: Mitigating Semantic Dominance via Contrastive Acoustic-Semantic Decoupling

    Oct 14, 2026Mengke Zhang, Yanda Shao, Tianhe Wu +1Speech Emotion RecognitionAudio-Language Model Evaluation

    Beijing University of Posts and Telecommunications, China

  16. Audio System One Model: Typed Probabilistic Perception of Emotion in Speech

    Oct 14, 2026Jaime Bellver-Soler, Iván Martín-Fernández, Anmol Guragain +1Speech Emotion RecognitionSpeech Foundation Models

    Speech Technology and Machine Learning Group (THAU), Information Processing and Telecommunications Center, E.T.S.I. de Telecomunicación, Universidad Politécnica de Madrid, Spain

  17. What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning Capabilities

    Oct 14, 2026Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu +5Audio-Language ModelsAudio-Language Model Evaluation

    National Taiwan University · NVIDIA · Independent Researcher +1

  18. ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

    Oct 12, 2026Ashutosh Chaubey, Jinxi Guo, Minh Tran +2Temporal Audio GroundingAudio Understanding

    Meta Superintelligence Labs · University of Southern California

  19. UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

    Oct 12, 2026Zhe Sun, Liang An, Jiayu Yao +9Underwater AcousticsAudio-Language Model Evaluation

    Southeast University

  20. ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context

    Oct 12, 2026Mingwei Fu, Zheshu Song, Zhikai Zhou +3ASR EvaluationAudio-Visual Speech Recognition

    University of Science and Technology of China · Tenlai Team, China

  21. What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings

    Oct 9, 2026Mateusz Modrzejewski, Tomasz RadzikowskiAudio-Language Model EvaluationLLM-as-a-Judge

    Institute of Computer Science, Warsaw University of Technology

  22. STEMMA: Song-to-Stem Multi-Audio Reasoning for Large Audio Language Models

    Oct 8, 2026Hoyeol Sohn, Wonil Kim, Keunhyoung Kim +7Audio ReasoningMusic Understanding

    Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea · Neutune, Seoul, Republic of Korea

  23. BEANS-Next and ROOTS: Broadening Audio-Language Capabilities for Bioacoustics

    Oct 7, 2026Christos Plachouras, David Robinson, Marius Miron +17BioacousticsAudio-Language Models

    Earth Species Project · Queen Mary University of London

  24. CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

    Oct 7, 2026Marcel Gibier, Thomas Thebaud, Olivier Boëffard +1Audio UnderstandingAudio Captioning

    Inria, Paris, France · AMIAD, France

  25. Logbook: Extremely Long-form Audio Event Understanding

    Oct 5, 2026Kwanghee Choi, Suwon Shon, Dmitriy Serdyuk +7Audio-Language Model EvaluationAudio Understanding

    UT Austin, USA · Meta Reality Labs, USA

  26. Where Does the Audio Jailbreak Live? A Controlled Frequency-Depth Audit of AdvWave-P on Qwen2-Audio

    Oct 4, 2026Boyuan Chen, Minseok Kim, Sohaila Abdulsattar +4Jailbreak AttacksAdversarial Perturbations

    eBRAIN Lab, Division of Engineering, New York University Abu Dhabi, UAE · Tandon School of Engineering, New York University, NY, USA

  27. Extending Music Annotation Schemas: Zero-Shot Prediction or Few-Shot Adaptation?

    Oct 2, 2026Christos Plachouras, Emmanouil Benetos, Johan PauwelsMusic Information RetrievalAudio-Language Model Evaluation

    School of Electronic Engineering and Computer Science, Queen Mary University of London, UK

  28. RMS-AQA: A Two-Stage Spatial Audio Question Answering Benchmark for Real-World Domestic Environments

    Oct 1, 2026Peihao Chen, Qing Wang, Lichun Fan +12Audio-Language Model EvaluationAudio QA

    University of Science and Technology of China, China · Xiaomi Corporation, China · DataoceanAI, China +2

  29. SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models

    Sep 30, 2026Rong Wan, Suliu Qin, Jiaxi Li +5Audio-Language Model EvaluationAudio Deepfake Detection

    University of Surrey · Singapore University of Technology and Design · Guangxi University +1

  30. Role-guided Speaker Deletion Verification in Clinical Psychiatry Speech Recordings with Audio Language Models

    Sep 29, 2026Joseph T Colonel, Daniel Katzman, Kelsey Kirker +6Audio-Language Model EvaluationSpeaker Verification

    Icahn School of Medicine at Mount Sinai, Department of Psychiatry, New York, NY, USA