cs.CLMar 29, 2026
SaveHumMusQA: A Human-written Music Understanding QA Benchmark Dataset
Organizations: Universitat Pompeu Fabra · Universitat Autònoma de Barcelona
Abstract
The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.
Figures & tables
| Accuracy with SD (%) | Consistency (%) | |||||||
|---|---|---|---|---|---|---|---|---|
| Model | All | Low | Med. | High | All | Low | Med. | High |
| qwen2.5-omni-7b Xu et al. (2025) | 64.3 ± 2.9 | 73.9 ± 3.1 | 58.1 ± 2.5 | 53.2 ± 3.9 | 66.6 | 70.5 | 61.3 | 67.9 |
| music-flamingo Ghosh et al. (2025a) | 58.5 ± 1.6 | 60.9 ± 2.1 | 58.1 ± 1.8 | 53.2 ± 4.1 | 39.1 | 41.7 | 38.7 | 30.2 |
| audio-flamingo-3 Goel et al. (2025) | 58.1 ± 3.2 | 62.7 ± 3.3 | 54.7 ± 3.8 | 54.1 ± 1.5 | 55.0 | 55.4 | 56.5 | 50.9 |
| gpt-audio OpenAI (2025b) | 57.7 ± 1.5 | 65.3 ± 3.9 | 55.1 ± 1.9 | 43.6 ± 1.8 | 45.6 | 45.3 | 43.5 | 47.2 |
| gemini-2.5-flash Google (2025) | 55.9 ± 1.2 | 64.4 ± 1.5 | 48.4 ± 3.6 | 50.5 ± 2.7 | 35.0 | 40.3 | 31.5 | 28.3 |
Table 1: Accuracy & consistency scores for systems across all benchmark questions, overall and by difficulty level. Accuracy is averaged over four runs with randomized answer orderings (standard deviation shown). Consistency measures the percentage of questions where the model produced identical answers across all four runs, indicating robustness to answer position bias.
Figure 1: Accuracy across different categories in the benchmark. Categories accounting for of questions in the dataset are excluded from the chart. Specifically, these are Historical and Cultural Context , Musical Texture , Lyrics , Structure , and Functional Context .
| Accuracy with SD (%) | |||
|---|---|---|---|
| Model | Real | Noise | Silence |
| qwen2.5-omni-7b | 64.3 ± 2.9 | 46.2 ± 1.3 | 47.0 ± 2.0 |
| musicflamingo | 58.5 ± 1.6 | 42.9 ± 3.6 | 42.2 ± 3.0 |
| gemini-2.5-flash | 55.9 ± 1.2 | 37.8 ± 1.5 | 39.8 ± 1.6 |
Table 2: Accuracy scores with true audio compared to fake audio (Gaussian noise or silence).
| Model | Backbone & Architecture | Training data | Reasoning Mechanism & Variants | Context Length | Max Audio | Capabilities | Openness |
|---|---|---|---|---|---|---|---|
| audio-flamingo-3 | Qwen-2.5-7B with AF-Whisper unified audio encoder | AudioSkills-XL, LongAudio-XL, AF-Think, AF-Chat ( 9.5M samples). | Optional short chain-of-thought (on-demand, not default) • AF3 (Base) - Single-turn Inference • AF3 + Think - Inference with short CoT prefixes • AF3-Chat - multi-turn, multi-audio, voice-to-voice interaction • AF3 (Long-Audio reasoning) - long-audio reasoning | 8,192 tokens | Up to 10 minutes | automatic speech recognition, audio question answering, audio reasoning, sound event recognition, music understanding, long-audio understanding, multi-turn audio dialogue, voice-to-voice interaction | Fully open (research) |
| audsemthinker | Qwen2.5-Omni-7B fine-tuned | AUDSEM dataset ( 900k instances) from YouTube. | Explicit structured semantic reasoning over sound • Think - <think> <answer> • Think + Semantic - <think> <semantic_elements> <answer> | 32,768 tokens (inherited from Qwen2.5-Omni) | Variable-length audio segments ( 3 seconds, context-limited) | semantic audio reasoning, sound event understanding, audio question answering, audio captioning, audio-based multiple-choice reasoning, creative audio-conditioned text generation | Open |
| gemini-2.5-flash | Sparse Mixture-of-Experts (MoE) multimodal transformer | Proprietary Data | Dynamic internal reasoning with user-controllable thinking budget | 1,000,000 tokens | Up to 3 hours of audio/video | automatic speech recognition, speech translation, audio question answering, audio summarization, audio-visual reasoning, long-context audio understanding, native audio dialogue, text-to-speech | Closed / Proprietary |
| gpt-audio-2025-08-28 | Not available | Not available | Not available | 128,000 tokens | Not available | general audio model; accepts audio input and output; | Closed |
| music-flamingo-hf | Enhanced AF3 backbone with time-aware embeddings (RoTE) | MF-Skills (4M+) and MF-Think (300k) music datasets. | Explicit music-theory chain-of-thought (optional, structured) | 24,000 tokens | Up to 20 minutes | music captioning, music question answering, music theory reasoning, harmonic analysis, structural music analysis, lyric grounding, cultural music analysis, long-form song understanding | Fully open (research) |
| qwen2.5-omni-7b | Qwen-2.5-7B Thinker-Talker end-to-end multimodal architecture | Large-scale multimodal pretraining ( 1.2T tokens) from open sources. | Implicit internal reasoning | 32,768 tokens | Long streaming audio (context-limited, no fixed minute cap) | automatic speech recognition, speech-to-text translation, audio question answering, audio reasoning, sound classification, music classification, audio-visual reasoning, streaming speech interaction | Open weights |
Table 4: Comparison of models used in the study