cs.SDSep 9, 2025
SaveAU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs
Organizations: ServiceNow · University of Texas at Austin
Abstract
Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and nonstandardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our empirical studies demonstrate that AU-Harness achieves 77% reduction in evaluation processing time over existing evaluation toolkits, enabling large-scale LALM evaluations previously considered impractical. We provide standardized evaluation protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.
Figures & tables
| EvalKit | HF Support | vLLM Support | Multi-task Parallel | Multi-turn | Customizable | RTF | SPS |
| AudioBench Wang et al. (2025a) | ✓ | ✗ | ✗ | ✗ | ✗ | 101.29 6.37 | 0.34 0.02 |
| Kimi-Eval Ding et al. (2025) | ✓ | ✗ | ✗ | ✗ | ✗ | 54.13 0.07 | 0.63 0.01 |
| VoiceBench Chen et al. (2024) | ✓ | ✗ | ✗ | ✗ | ✗ | 129.67 5.30 | 0.29 0.03 |
| LMMs-Eval Zhang et al. (2025b) | ✓ | ✓ | ✓ | ✗ | ✗ | 20.84 0.62 | 1.63 0.05 |
| AHELM Lee et al. (2025) | ✓ | ✓ | ✗ | ✓ | ✗ | 57.86 14.69 | 0.62 0.18 |
| AU-Harness | ✓ | ✓ | ✓ | ✓ | ✓ | 10.29 0.61 | 3.32 0.15 |
Table 1: Feature comparison of contemporary LALM evaluation toolkits. We evaluate key technical capabilities across existing frameworks: vLLM integration for efficient batching, HuggingFace (HF) model support, multi-turn dialogue support for conversational scenarios, parallel processing support for multi-task concurrent evaluation, and configurable customizations for flexible evaluation design. Throughput is further reported on MELD-Emotion Poria et al. (2019) via Samples Per Second (SPS ) and Real-Time Factor (RTF ) averaged across 3 runs to empirically validate efficiency claims beyond feature support alone. Refer to § 3.1 for further details. Our framework provides comprehensive support across most dimensions.
Figure 1: Architecture overview of AU-Harness evaluation framework. Our system comprises three core components: (1) Config module for hierarchical task configuration and standardized prompting, (2) Central Request Controller (CRC) managing token-based concurrency limits across all engines with adaptive retry mechanisms, and (3) Concurrent Engines executing parallel model evaluation with dataset sharding. The Central Request Controller maintains a global Token Pool accessible to all engines, enabling efficient resource utilization and scalable throughput. Multiple concurrent connections between the controller and inference models illustrate parallel request dispatch, with each engine supporting the evaluation of multiple models on targeted datasets.
Figure 2: Compositional task configuration in AU-Harness, illustrated on SpokenWOZ. A shared base config defines all invariant components — processors, instruction prompt, generation parameters, and metrics — inherited by all task-specific variants. Each variant requires modifying only a single targeted field, such as input modality or transcription strategy, enabling new evaluation scenarios to be instantiated in minutes without any pipeline re-engineering. This compositional design is representative of AU-Harness’s broader configuration system, where comprehensive coverage of diverse tasks and settings is achieved through minimal, targeted modifications; facilitating rapid, reproducible experimentation at scale.
| Models | Speech | Audio & Music | |||||||||||
| Task Category | PR | ASR | Para. | S&L | SLU | HD | SE | Safety | Multi-turn | IF | AU | MU | |
| Dataset | voxangeles | Librispeech | IEMOCAP | SR | BBA | StutterDetect | NoiseDetect | Advbench | MT-Bench | SpokenWoz | IFEval | AudioCaps | ChoMusic |
| Metrics | LB (↑) | WER (↓) | LB (↑) | LB (↑) | LBBA(↑) | LB (↑) | LB (↑) | SafetyJudge (↑) | MTJudge (↑) | JGA (↑) | IFScore (↑) | LB (↑) | LB (↑) |
| Small-sized Audio Language Models (<5B parameters) | |||||||||||||
| Voxtral-Mini-3B | 0.10 | 2.10 | 54.90 | 45.80 | 43.50 | 12.90 | 14.50 | 78.50 | 65.88 | 24.92 | 40.02 | 14.96 | 45.40 |
| Qwen2.5-Omni-3B | 1.20 | 8.09 | 81.50 | 55.90 | 44.80 | 58.40 | 58.40 | 97.30 | 59.81 | 31.30 | 35.82 | 42.82 | 53.50 |
Table 2: LALM performance on audio tasks. We evaluate representative LALMs from different spectra: Open-source LALMs(small-sized, medium-sized, large-sized), Proprietary LALMs and Cascaded System LALMs across representative task categories. Metrics include LLM-as-judge evaluations using GPT-4o-mini and task-specific automatic metrics. PR : Phoneme Recognition, Para. : Paralinguistics, ASR : Automatic Speech Recognition, S&L : Speaker & Language, SLU : Spoken Language Understanding, HD : Hearing Disorder, SE : Speech Enhancement, IF : Instruction Following, AU : Audio Understanding, MU : Music Understanding, Refer to Appendix A.1 for benchmark, model and metric abbreviations together with their detailed explanations. Bold : highest; underline : second highest. *Performance affected by Azure OpenAI content filtering. † denotes results obtained prior to the API discontinuation date
Figure 3: Efficiency comparison across evaluation frameworks and runtime scenarios. (a) Real-time Factor ( better, log-scale) and (b) SPS ( better, log-scale) measured across three datasets (MELD-Emotion, LibriSpeech-test-clean, ClothoAQA) and three runtime conditions: Individual (dataset-specific), Sequential (worst-case serialized execution), Parallel (optimal concurrent execution) and Total (complete execution). AU-Harness consistently outperforms existing toolkits across all scenarios, with most significant gains in parallel and total execution, demonstrating effective utilization of concurrent processing.
Figure 4: Total runtime efficiency analysis across evaluation frameworks. Scatter plot comparing frameworks under optimal parallel execution conditions, plotting Real-time Factor (x-axis, better) against Samples Processed per Second (y-axis, better) on a log-log scale to span the wide dynamic range of both metrics (up to 63 across frameworks). Our framework (top-left-most cluster) achieves superior performance in both dimensions at 8.79 s/s and 1.0% RTF, demonstrating the effectiveness of token-based request scheduling, dataset sharding, and vLLM integration for large-scale LALM evaluation.
| EvalKit | RTF ( ) | SPS ( ) |
| LMMS-Eval-HF (Optimal) | 3.11 0.04 | 3.39 0.12 |
| LMMS-Eval-vLLM (Optimal) | 1.87 0.04 | 4.83 0.10 |
| LMMS-Eval-vLLM (Reality) | 3.46 0.22 | 2.62 0.17 |
| AU-Harness (Optimal) | 1.14 0.16 | 8.32 0.97 |
| AU-Harness (Reality) | 1.69 0.08 | 5.48 0.27 |
Table 3: Parallel runtime efficiency comparison between AU-Harness and LMMS-Eval. We conduct controlled experiments following the setups presented in Section 5.1 . As both LMMS-Eval and AU-Harness support multi-task parallel evaluation, besides the Optimal parallel runtime, we report Reality parallel runtime variant where additional realistic overheads are taken into account during evaluation. Clarifications are further provided in Appendix A.2 . Bold and underline represent the best and second best performance respectively
Figure 5: SpokenWOZ multi-turn DST — JGA vs. turn index across input-format configurations for Qwen3-Omni-30B-A3B-Thinking. Each panel groups evaluation configurations enabled by AU-Harness by scoring strategy: Standard Configurations (left) vary the current-turn modality across text-only (baseline), last user turn in audio, and all user turns in audio, evaluated against the cumulative dialogue state; Standard Configurations with GT Prior State (right) apply the same modality variants but inject the oracle prior state before each turn, scoring only the per-turn delta. Grey bars denote the mean accumulated slot count per turn (left panel) and the mean number of new slots introduced per turn (right panel), both on the right axis. Further details in Section A.9 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Task Category | Task Type | Dataset Name | Task | Description | License |
| Speech Recognition | ASR | AISHELL-1 | 1 | High-quality Mandarin speech recognition dataset | Apache 2.0 |
| ASR | AMI Meeting Corpus | 2 | Multispeaker meeting recordings for ASR and diarization | CC BY 4.0 | |
| ASR | CallHome | 5 | Conversational speech corpus across multiple languages | LDC User Agreement for Non-Members | |
| ASR | Common Voice | 100 | Crowdsourced multilingual speech dataset from Mozilla | CC0 1.0 Universal | |
| ASR | FLEURS EN-US | 102 | Multilingual speech dataset for ASR and translation | CC BY 4.0 | |
| ASR | GigaSpeech | 1 | Large-scale audio and transcription corpus for end-to-end ASR | Apache 2.0 |
Table 5: Comprehensive Audio and Speech Datasets Overview. Listing of 61 datasets across 6 task categories: Speech Recognition, Paralinguistics, Audio Understanding, Spoken Language Understanding, Spoken Language Reasoning, and Safety & Security.
| MELD-Emotion | Librispeech-clean | ClothoAQA | |
| # Samples | 500 | 500 | 500 |
| Audio Duration (seconds) | 1,476 | 3,780 | 11,376 |
| Evaluation Metric | LLM-Judge | WER | LLM-Judge |
Table 6: Experimental setup for efficiency comparison across evaluation frameworks. We conduct controlled experiments using 500 samples from three diverse datasets: MELD-Emotion (short emotional speech), LibriSpeech-clean (medium-length read speech), and ClothoAQA (long-form descriptive audio). Total audio duration varies from 1,476 to 11,376 seconds, enabling assessment across different audio characteristics and evaluation modalities (LLM-judge vs. traditional metrics).
Figure 6: Inference efficiency ablations in AU-Harness. We examine three factors: (a) impact of batch size on execution time, (b) throughput gains from parallel execution, and (c) latency reduction through replica scaling.
Figure 7: Config UI. A user-friendly web interface for configuring and running audio model evaluations with the AU-Harness framework.
Figure 8: Summary . The Summary tab is the landing view 8 . It lists the run’s top-3 strengths and top-3 weaknesses, each tagged with whether the comparison baseline is intra-run ("vs run peer") or the published leaderboard ("vs leaderboard"). A coverage matrix below the headline lists every dataset that produced at least one valid score, mapped to its category and the downstream tabs in which it appears. Top-3 status is a presentation cap, not a coverage filter — a dataset can be analyzed elsewhere even if it does not surface in the headline lists.
Figure 9: Config . Config is the reproducibility surface. It records each model’s inference parameters (batch size, audio chunk length, timeout, retry budget, post-request delay) and the judge configuration (judge model, provider, concurrency, sampling temperature). Per-task cards report the generation metric, decoding parameters, and a representative instruction drawn from the dataset, so that any score downstream can be traced back to a concrete request configuration. When a task-config file is not bundled, the card falls back to the run-level defaults and is labeled accordingly.
Figure 10: Categories. Categories groups every (dataset, metric, model) result under one of the AU-Harness category labels — Audio Understanding, Paralinguistics, Safety and Security, Speech Disorder / Enhancement, Speech Recognition, Spoken Language Reasoning, and Spoken Language Understanding. Within each block, rows are sorted by score and color-coded against the category mean; each block closes with a category-average row aggregating over the (dataset × model) entries in scope. The view is intended to expose category-level strengths that a flat leaderboard obscures.
Figure 11: Patterns. The Patterns tab is the per-task error-mining surface. For each (task, model) pair, AU-Harness partitions samples into low-score and high-score buckets (counts shown in the header) and computes the prompt terms and judge-rationale phrases via LDA that are disproportionately overrepresented in the low bucket. Low-score prompt terms are tokens drawn from the dataset’s user-side instruction; judge low-score phrases are tokens from the judge model’s free-text justification on failing samples. Together they provide a coarse but fast signal of where a model is breaking down — vocabulary it cannot ground, content it refuses, output format it mis-aligns - without requiring per-sample inspection.
Figure 12: Multi-turn. Multi-turn targets long-conversation degradation, currently instrumented on SpokenWOZ-style dialogue state tracking with joint goal accuracy, slot accuracy, and slot F1. For each (task, metric, model) triple the tab plots mean score per turn, summarizes early (turns 0–3), middle (4–9), and late (10+) buckets, and renders a failure-turn histogram showing which turn indices contribute the most zero-score samples. Bucket cells are tinted by absolute level, making floor effects in the late bucket visually distinguishable from a smooth glide.
Figure 13: Compare. Compare is a head-to-head view that accepts up to four model selections and renders per-task scores side by side. A Dir column encodes metric direction (↑ higher is better; ↓ lower is better, e.g. word error rate), and the leading score per row is bolded. The header line summarizes wins by aggregating direction-aware comparisons across the tasks where every selected model has a valid score; tasks with any missing score are excluded from the win count but remain visible in the table.
Figure 14: Ops Health. Ops health separates inference reliability from model quality. For each (dataset, metric, model) row it reports the total request count, hard-failure count, failure rate, mean wall-clock wait per request, and the most common error class. Highlighted rows surface transient API outages, rate-limit storms, or per-model timeout misconfigurations that would otherwise be mis-read as quality regressions in the other tabs.
More in this topic
Audio-Language Model Evaluation
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
See all 206 papers