Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and nonstandardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our empirical studies demonstrate that AU-Harness achieves 77% reduction in evaluation processing time over existing evaluation toolkits, enabling large-scale LALM evaluations previously considered impractical. We provide standardized evaluation protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.
Figures & tables
EvalKit
HF Support
vLLM Support
Multi-task Parallel
Multi-turn
Customizable
RTF ↓
SPS ↑
AudioBench Wang et al. (2025a)
✓
✗
✗
✗
✗
101.29 ± 6.37
0.34 ± 0.02
Kimi-Eval Ding et al. (2025)
✓
✗
✗
✗
✗
54.13 ± 0.07
0.63 ± 0.01
VoiceBench Chen et al. (2024)
✓
✗
✗
✗
✗
129.67 ± 5.30
0.29 ± 0.03
LMMs-Eval Zhang et al. (2025b)
✓
✓
✓
✗
✗
20.84 ± 0.62
1.63 ± 0.05
AHELM Lee et al. (2025)
✓
✓
✗
✓
✗
57.86 ± 14.69
0.62 ± 0.18
AU-Harness
✓
✓
✓
✓
✓
10.29 ± 0.61
3.32 ± 0.15
Table 1: Feature comparison of contemporary LALM evaluation toolkits. We evaluate key technical capabilities across existing frameworks: vLLM integration for efficient batching, HuggingFace (HF) model support, multi-turn dialogue support for conversational scenarios, parallel processing support for multi-task concurrent evaluation, and configurable customizations for flexible evaluation design. Throughput is further reported on MELD-Emotion Poria et al. (2019) via Samples Per Second (SPS ↑ ) and Real-Time Factor (RTF ↓ ) averaged across 3 runs to empirically validate efficiency claims beyond feature support alone. Refer to § 3.1 for further details. Our framework provides comprehensive support across most dimensions.
Figure 1: Architecture overview of AU-Harness evaluation framework. Our system comprises three core components: (1) Config module for hierarchical task configuration and standardized prompting, (2) Central Request Controller (CRC) managing token-based concurrency limits across all engines with adaptive retry mechanisms, and (3) Concurrent Engines executing parallel model evaluation with dataset sharding. The Central Request Controller maintains a global Token Pool accessible to all engines, enabling efficient resource utilization and scalable throughput. Multiple concurrent connections between the controller and inference models illustrate parallel request dispatch, with each engine supporting the evaluation of multiple models on targeted datasets.
Figure 2: Compositional task configuration in AU-Harness, illustrated on SpokenWOZ. A shared base config defines all invariant components — processors, instruction prompt, generation parameters, and metrics — inherited by all task-specific variants. Each variant requires modifying only a single targeted field, such as input modality or transcription strategy, enabling new evaluation scenarios to be instantiated in minutes without any pipeline re-engineering. This compositional design is representative of AU-Harness’s broader configuration system, where comprehensive coverage of diverse tasks and settings is achieved through minimal, targeted modifications; facilitating rapid, reproducible experimentation at scale.
Models
Speech
Audio & Music
Task Category
PR
ASR
Para.
S&L
SLU
HD
SE
Safety
Multi-turn
IF
AU
MU
Dataset
voxangeles
Librispeech
IEMOCAP
SR
BBA
StutterDetect
NoiseDetect
Advbench
MT-Bench
SpokenWoz
IFEval
AudioCaps
ChoMusic
Metrics
LB (↑)
WER (↓)
LB (↑)
LB (↑)
LBBA(↑)
LB (↑)
LB (↑)
SafetyJudge (↑)
MTJudge (↑)
JGA (↑)
IFScore (↑)
LB (↑)
LB (↑)
Small-sized Audio Language Models (<5B parameters)
Voxtral-Mini-3B
0.10
2.10
54.90
45.80
43.50
12.90
14.50
78.50
65.88
24.92
40.02
14.96
45.40
Qwen2.5-Omni-3B
1.20
8.09
81.50
55.90
44.80
58.40
58.40
97.30
59.81
31.30
35.82
42.82
53.50
Table 2: LALM performance on audio tasks. We evaluate representative LALMs from different spectra: Open-source LALMs(small-sized, medium-sized, large-sized), Proprietary LALMs and Cascaded System LALMs across representative task categories. Metrics include LLM-as-judge evaluations using GPT-4o-mini and task-specific automatic metrics. PR : Phoneme Recognition, Para. : Paralinguistics, ASR : Automatic Speech Recognition, S&L : Speaker & Language, SLU : Spoken Language Understanding, HD : Hearing Disorder, SE : Speech Enhancement, IF : Instruction Following, AU : Audio Understanding, MU : Music Understanding, Refer to Appendix A.1 for benchmark, model and metric abbreviations together with their detailed explanations. Bold : highest; underline : second highest. *Performance affected by Azure OpenAI content filtering. † denotes results obtained prior to the API discontinuation date
Figure 3: Efficiency comparison across evaluation frameworks and runtime scenarios. (a) Real-time Factor ( ↓ better, log-scale) and (b) SPS ( ↑ better, log-scale) measured across three datasets (MELD-Emotion, LibriSpeech-test-clean, ClothoAQA) and three runtime conditions: Individual (dataset-specific), Sequential (worst-case serialized execution), Parallel (optimal concurrent execution) and Total (complete execution). AU-Harness consistently outperforms existing toolkits across all scenarios, with most significant gains in parallel and total execution, demonstrating effective utilization of concurrent processing.
Figure 4: Total runtime efficiency analysis across evaluation frameworks. Scatter plot comparing frameworks under optimal parallel execution conditions, plotting Real-time Factor (x-axis, ↓ better) against Samples Processed per Second (y-axis, ↑ better) on a log-log scale to span the wide dynamic range of both metrics (up to ∼ 63 × across frameworks). Our framework (top-left-most cluster) achieves superior performance in both dimensions at 8.79 s/s and 1.0% RTF, demonstrating the effectiveness of token-based request scheduling, dataset sharding, and vLLM integration for large-scale LALM evaluation.
EvalKit
RTF ( ↓ )
SPS ( ↑ )
LMMS-Eval-HF (Optimal)
3.11 ± 0.04
3.39 ± 0.12
LMMS-Eval-vLLM (Optimal)
1.87 ± 0.04
4.83 ± 0.10
LMMS-Eval-vLLM (Reality)
3.46 ± 0.22
2.62 ± 0.17
AU-Harness (Optimal)
1.14 ± 0.16
8.32 ± 0.97
AU-Harness (Reality)
1.69 ± 0.08
5.48 ± 0.27
Table 3: Parallel runtime efficiency comparison between AU-Harness and LMMS-Eval. We conduct controlled experiments following the setups presented in Section 5.1 . As both LMMS-Eval and AU-Harness support multi-task parallel evaluation, besides the Optimal parallel runtime, we report Reality parallel runtime variant where additional realistic overheads are taken into account during evaluation. Clarifications are further provided in Appendix A.2 . Bold and underline represent the best and second best performance respectively
Figure 5: SpokenWOZ multi-turn DST — JGA vs. turn index across input-format configurations for Qwen3-Omni-30B-A3B-Thinking. Each panel groups evaluation configurations enabled by AU-Harness by scoring strategy: Standard Configurations (left) vary the current-turn modality across text-only (baseline), last user turn in audio, and all user turns in audio, evaluated against the cumulative dialogue state; Standard Configurations with GT Prior State (right) apply the same modality variants but inject the oracle prior state before each turn, scoring only the per-turn delta. Grey bars denote the mean accumulated slot count per turn (left panel) and the mean number of new slots introduced per turn (right panel), both on the right axis. Further details in Section A.9 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Task Category
Task Type
Dataset Name
Task
Description
License
Speech Recognition
ASR
AISHELL-1
1
High-quality Mandarin speech recognition dataset
Apache 2.0
ASR
AMI Meeting Corpus
2
Multispeaker meeting recordings for ASR and diarization
CC BY 4.0
ASR
CallHome
5
Conversational speech corpus across multiple languages
LDC User Agreement for Non-Members
ASR
Common Voice
100
Crowdsourced multilingual speech dataset from Mozilla
CC0 1.0 Universal
ASR
FLEURS EN-US
102
Multilingual speech dataset for ASR and translation
CC BY 4.0
ASR
GigaSpeech
1
Large-scale audio and transcription corpus for end-to-end ASR
Apache 2.0
Appendix
Table 5: Comprehensive Audio and Speech Datasets Overview. Listing of 61 datasets across 6 task categories: Speech Recognition, Paralinguistics, Audio Understanding, Spoken Language Understanding, Spoken Language Reasoning, and Safety & Security.
MELD-Emotion
Librispeech-clean
ClothoAQA
# Samples
500
500
500
Audio Duration (seconds)
1,476
3,780
11,376
Evaluation Metric
LLM-Judge
WER
LLM-Judge
Appendix
Table 6: Experimental setup for efficiency comparison across evaluation frameworks. We conduct controlled experiments using 500 samples from three diverse datasets: MELD-Emotion (short emotional speech), LibriSpeech-clean (medium-length read speech), and ClothoAQA (long-form descriptive audio). Total audio duration varies from 1,476 to 11,376 seconds, enabling assessment across different audio characteristics and evaluation modalities (LLM-judge vs. traditional metrics).
Figure 6: Inference efficiency ablations in AU-Harness. We examine three factors: (a) impact of batch size on execution time, (b) throughput gains from parallel execution, and (c) latency reduction through replica scaling.
Figure 7: Config UI. A user-friendly web interface for configuring and running audio model evaluations with the AU-Harness framework.
Figure 8: Summary . The Summary tab is the landing view 8 . It lists the run’s top-3 strengths and top-3 weaknesses, each tagged with whether the comparison baseline is intra-run ("vs run peer") or the published leaderboard ("vs leaderboard"). A coverage matrix below the headline lists every dataset that produced at least one valid score, mapped to its category and the downstream tabs in which it appears. Top-3 status is a presentation cap, not a coverage filter — a dataset can be analyzed elsewhere even if it does not surface in the headline lists.
Figure 9: Config . Config is the reproducibility surface. It records each model’s inference parameters (batch size, audio chunk length, timeout, retry budget, post-request delay) and the judge configuration (judge model, provider, concurrency, sampling temperature). Per-task cards report the generation metric, decoding parameters, and a representative instruction drawn from the dataset, so that any score downstream can be traced back to a concrete request configuration. When a task-config file is not bundled, the card falls back to the run-level defaults and is labeled accordingly.
Figure 10: Categories. Categories groups every (dataset, metric, model) result under one of the AU-Harness category labels — Audio Understanding, Paralinguistics, Safety and Security, Speech Disorder / Enhancement, Speech Recognition, Spoken Language Reasoning, and Spoken Language Understanding. Within each block, rows are sorted by score and color-coded against the category mean; each block closes with a category-average row aggregating over the (dataset × model) entries in scope. The view is intended to expose category-level strengths that a flat leaderboard obscures.
Figure 11: Patterns. The Patterns tab is the per-task error-mining surface. For each (task, model) pair, AU-Harness partitions samples into low-score and high-score buckets (counts shown in the header) and computes the prompt terms and judge-rationale phrases via LDA that are disproportionately overrepresented in the low bucket. Low-score prompt terms are tokens drawn from the dataset’s user-side instruction; judge low-score phrases are tokens from the judge model’s free-text justification on failing samples. Together they provide a coarse but fast signal of where a model is breaking down — vocabulary it cannot ground, content it refuses, output format it mis-aligns - without requiring per-sample inspection.
Figure 12: Multi-turn. Multi-turn targets long-conversation degradation, currently instrumented on SpokenWOZ-style dialogue state tracking with joint goal accuracy, slot accuracy, and slot F1. For each (task, metric, model) triple the tab plots mean score per turn, summarizes early (turns 0–3), middle (4–9), and late (10+) buckets, and renders a failure-turn histogram showing which turn indices contribute the most zero-score samples. Bucket cells are tinted by absolute level, making floor effects in the late bucket visually distinguishable from a smooth glide.
Figure 13: Compare. Compare is a head-to-head view that accepts up to four model selections and renders per-task scores side by side. A Dir column encodes metric direction (↑ higher is better; ↓ lower is better, e.g. word error rate), and the leading score per row is bolded. The header line summarizes wins by aggregating direction-aware comparisons across the tasks where every selected model has a valid score; tasks with any missing score are excluded from the win count but remain visible in the table.
Figure 14: Ops Health. Ops health separates inference reliability from model quality. For each (dataset, metric, model) row it reports the total request count, hard-failure count, failure rate, mean wall-clock wait per request, and the most common error class. Highlighted rows surface transient API outages, rate-limit storms, or per-model timeout misconfigurations that would otherwise be mis-read as quality regressions in the other tabs.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. However, subtle changes, such as shifting the order of choices, result in substantially different results. Existing MCQA frameworks do not account for this variability and report a single accuracy number per benchmark or category. We dive into the MCQA evaluation framework and conduct a systematic study spanning three benchmarks (MMAU, MMAR and MMSU) and four models: Audio Flamingo 2, Audio Flamingo 3, Qwen2.5-Omni-7B-Instruct, and Kimi-Audio-7B-Instruct. Our findings indicate that models are sensitive not only to the ordering of choices, but also to the paraphrasing of the question and the choices. Finally, we propose a simpler evaluation protocol and metric that account for subtle variations and provide a more detailed evaluation report of LALMs within the MCQA framework.
Fernando López, Santosh Kesiraju, Jordi Luque
Scientific Research, Telef´onica Innovaci´on Digital, Spain · Brno University of Technology, Czech Republic
Large Audio-Language Models (LALMs) integrate audio perception and language understanding within a unified framework, enabling a wide range of real-world applications. Despite recent advances, evaluation for LALMs remains heavily underspecified relative to real-world requirements: most lack true linguistic and cultural authenticity, while others fail to capture acoustic realism. To bridge this gap, we propose GlobeAudio, a multilingual and multicultural benchmark designed to evaluate naturalistic audio understanding. GlobeAudio consists of 5,637 multiple-choice questions across six typologically diverse languages, expertly crafted by native speakers grounded on naturally occurring audio. In order to do well, models must possess higher-level auditory reasoning skills and culturally grounded interpretation. We systematically evaluate representative closed-source and open-source LALMs, as well as cascaded ASR-LLM pipelines. Our experiments reveal substantial performance gaps under natural acoustic conditions, particularly for open-source models and low-resource languages. These findings highlight critical limitations of current LALMs and underscore the importance of naturalistic audio evaluation for future audio-language systems. GlobeAudio can be found at https://huggingface.co/datasets/iNLP-Lab/GlobeAudio .
While multi-audio understanding is critical for large audio-language models (LALMs), it remains underexplored. We introduce MUGEN, a comprehensive benchmark evaluating this capability across speech, general audio, and music. Our experiments reveal consistent weaknesses in multi-audio settings, and performance degrades sharply as the number of concurrent audio inputs increases, identifying input scaling as a fundamental bottleneck. We further investigate training-free strategies and observe that Audio-Permutational Self-Consistency, which diversifies the order of audio candidates, helps models form more robust aggregated predictions, yielding up to 6.28% accuracy gains. Combining this permutation strategy with Chain-of-Thought further improves performance to 6.74%. These results expose blind spots in current LALMs and provide a foundation for evaluating complex auditory comprehension.
Chih-Kai Yang, Yun-Shao Tsai, Yu-Kai Guo +7
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan