cs.IRApr 8, 2026
SaveJamendo-MT-QA: A Benchmark for Multi-Track Comparative Music Question Answering
Organizations: Yonsei University · KRAFTON · MODULABS MAAP · University of Oxford · Sungkyul University · George Mason University · Onoma AI
Abstract
Recent work on music question answering (Music-QA) has primarily focused on single-track understanding, where models answer questions about an individual audio clip using its tags, captions, or metadata. However, listeners often describe music in comparative terms, and existing benchmarks do not systematically evaluate reasoning across multiple tracks. Building on the Jamendo-QA dataset, we introduce Jamendo-MT-QA, a dataset and benchmark for multi-track comparative question answering. From Creative Commons-licensed tracks on Jamendo, we construct 36,519 comparative QA items over 12,173 track pairs, with each pair yielding three question types: yes/no, short-answer, and sentence-level questions. We describe an LLM-assisted pipeline for generating and filtering comparative questions, and benchmark representative audio-language models using both automatic metrics and LLM-as-a-Judge evaluation.
Figures & tables
Figure 1: Overview of the Jamendo-MT-QA construction pipeline. Starting from the Jamendo-QA dataset, we first generate rich single-track captions using Music Flamingo, expand them into single-track QA pairs using GPT-5.1, and construct comparative QA groups consisting of three question types (yes/no, short-answer, sentence-level) for each track pair via an LLM-assisted generation pipeline. Finally, we apply human evaluation and LLM-as-a-Judge assessment for quality control and filtering, resulting in the final Jamendo-MT-QA dataset.
Figure 2: Qualitative example of a Stage 3 comparative QA group generated by GPT-5 mini for the same track pair. The generator produces three comparative question types (yes/no, short-answer, sentence-level) and provides explicitly step-by-step reasoning, which improves interpretability and makes the reasoning field more suitable as a supervision signal for downstream training.
| Metric | R1 | R2 | R3 | R4 | Human Mean | GPT-5 Mini Mean | MAD |
|---|---|---|---|---|---|---|---|
| Correctness | 4.93 | 4.66 | 4.64 | 5.00 | 4.79 | 4.87 | 0.32 |
| Comparative Validity | 4.93 | 4.75 | 4.83 | 4.99 | 4.83 | 4.61 | 0.44 |
| Reasoning Quality | 4.87 | 4.71 | 4.67 | 5.00 | 4.78 | 4.37 | 0.56 |
| Difficulty | 1.89 | 2.58 | 3.12 | 2.66 | 2.25 | 2.17 | 0.57 |
Table 1: Human evaluation results on 300 sampled QA items using four annotators (R1–R4), together with LLM-as-a-Judge scores on the same subset. We report human mean scores and mean absolute deviation (MAD) to characterize aggregate-level agreement between human and LLM judgments.
Figure 3: Baseline evaluation setups for Jamendo-MT-QA. Left: caption-based single-audio baselines (e.g., Music Flamingo, MU-LLaMA), where each track is independently captioned and an LLM performs comparative reasoning over the two captions plus the question. Right: multi-audio baselines (e.g., GPT-4o(-mini) Audio, Qwen2-Audio, Qwen3-Omni), which take two audio inputs and the question to produce an answer end-to-end.
| Model | Type | Yes/No | Short | Sentence-level | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Acc. | Acc. | BLEU | R-1 | R-2 | R-L | BERT-F1 | GPT | Claude | ||
| Music Flamingo | Cap | 77.4% | 89.7% | 4.00 | 27.5 | 4.6 | 24.7 | 0.8786 | 3.24 | 3.87 |
| Qwen2-Audio | Cap | 37.4% | 39.1% | 1.88 | 19.3 | 1.8 | 17.4 | 0.8489 | 1.49 | 1.53 |
| MU-LLaMA | Cap | 20.6% | 55.3% | 2.39 | 23.8 | 3.2 | 16.5 | 0.8572 | 2.36 | 2.01 |
| Qwen2-Audio | Multi | 50.9% | 80.2% | 2.09 | 21.0 | 3.4 | 14.6 | 0.8472 | 1.37 | 1.62 |
| Qwen3-Omni | Multi | 62.9% | 80.3% | 3.58 | 29.3 | 6.6 | 20.4 | 0.8632 | 3.11 | 3.48 |
Table 2: Results on the full benchmark of 12,173 track pairs.
| Model | Type | Yes/No | Short | Sentence-level | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Acc. | Acc. | BLEU | R-1 | R-2 | R-L | BERT-F1 | GPT | Claude | ||
| Music Flamingo | Cap | 82.1% | 88.1% | 2.06 | 25.3 | 5.9 | 19.6 | 0.875 | 3.25 | 3.97 |
| Qwen2-Audio | Cap | 26.9% | 55.5% | 0.58 | 15.3 | 1.6 | 12.1 | 0.845 | 1.37 | 1.60 |
| MU-LLaMA | Cap | 23.6% | 50.5% | 0.68 | 16.1 | 1.9 | 13.3 | 0.853 | 1.92 | 2.13 |
| Qwen2-Audio | Multi | 34.4% | 77.7% | 0.57 | 14.3 | 1.4 | 11.3 | 0.845 | 1.63 | 1.75 |
| Qwen3-Omni | Multi | 59.7% | 75.5% | 1.22 | 19.9 | 3.9 | 16.0 | 0.860 | 2.57 | 3.25 |
Table 3: Results on the subset of 2,010 track pairs.
| Model | Comp.Col. | Attr.Hal. | Gran.Mis. |
|---|---|---|---|
| Music Flamingo | 35.3% | 64.0% | 0.7% |
| MU-LLaMA | 37.3% | 60.7% | 2.0% |
| GPT-4o-Audio | 56.7% | 35.3% | 8.0% |
| Qwen3-Omni | 23.4% | 75.9% | 0.7% |
Table 4: Error type distribution across models on sentence-level questions (LLM judge score 3). Classified by GPT-4o-mini.
Figure 4: Heatmap visualization of error type distributions across models. Values indicate the percentage of each error category among incorrectly answered sentence-level questions (LLM judge score 3).
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Annotator | Mean | Std |
|---|---|---|
| Annotator A | 4.43 | 0.65 |
| Annotator B | 4.47 | 0.62 |
| Annotator C | 4.60 | 0.71 |
| Annotator D | 4.85 | 0.48 |
| Average | 4.59 | – |
Table 5: Human evaluation results for audio–text alignment of enriched captions in Stage 1, reported on a 1–5 scale.
| Musical Attribute | Coverage | Count |
|---|---|---|
| Instrumentation | 99.9% | 7,325 / 7,335 |
| Production style | 99.7% | 7,316 / 7,335 |
| Vocal characteristics | 99.7% | 7,316 / 7,335 |
| Mood / atmosphere | 98.4% | 7,216 / 7,335 |
| Tempo | 96.7% | 7,095 / 7,335 |
| Genre | 96.3% | 7,061 / 7,335 |
Table 6: Caption attribute coverage across all 7,335 Music Flamingo captions.
| Score | Description |
|---|---|
| 0 | Completely wrong or irrelevant |
| 1 | Mostly wrong with very minor correct elements |
| 2 | Partially correct but missing key points |
| 3 | Correct on main points but missing details or minor errors |
| 4 | Mostly correct with only minor omissions |
| 5 | Fully correct, capturing all key comparative information |
Table 7: Scoring rubric used by LLM judges for sentence-level comparative answers.
| Metric | GPT GPT | GPT Claude | |
|---|---|---|---|
| Correctness | 4.729 | 4.741 | 0.318 |
| Comp. Validity | 4.721 | 4.735 | 0.264 |
| Reasoning Qual. | 4.721 | 4.735 | 0.264 |
Table 8: Cross-model self-preference analysis (GPT evaluator, ). No significant bias detected ( ).
| Question Type | Evaluation Method | Score Range |
|---|---|---|
| Yes/No | Rule-based matching | 0 or 1 |
| Short-answer | Rule-based matching (Track A/B) | 0 or 1 |
| Sentence-level | LLM-as-a-Judge | 0–5 |
| Open-ended | LLM-as-a-Judge | 0–5 |
Table 9: Evaluation methods used for different question types.
| Verification Check | Tested | Pass |
|---|---|---|
| Answer format validity (non-empty) | 36,519 | 100.0% |
| Yes/No: valid “yes”/“no” answer | 12,173 | 100.0% |
| Short Answer: contains track name | 12,173 | 100.0% |
| Sentence: answer 20 characters | 12,173 | 100.0% |
| All 3 QA types present per pair | 12,173 | 100.0% |
| Tempo answer correct (SA) | 1,627 | 100.0% |
Table 10: Full dataset quality verification (12,173 pairs).
| Statistic | Value |
|---|---|
| Total music pairs | 12,173 |
| Total QA pairs | 36,519 |
| Unique genre pairs | 789 |
Table 11: Basic statistics of the multi-track comparative QA dataset.
| Genre | Count | Ratio |
|---|---|---|
| rock | 4,890 | 20.09% |
| indie | 3,424 | 14.06% |
| pop | 2,527 | 10.38% |
| folk | 1,611 | 6.62% |
| electronic | 1,446 | 5.94% |
| rap | 1,358 | 5.58% |
Table 12: Top genres across all tracks in the dataset.
Figure 5: Genre composition and genre-pair distribution in Jamendo-MT-QA. The inner ring shows the marginal distribution of individual genres across all tracks, while the outer ring visualizes the frequency of genre–genre pairings used for comparative question generation. For clarity, only the top-10 most frequent genres are shown in the visualization; therefore, percentages may differ from table-based statistics computed over all tracks.
| Genre 1 | Genre 2 | Count |
|---|---|---|
| indie | rock | 674 |
| pop | rock | 497 |
| rock | rock | 488 |
| indie | pop | 367 |
| folk | rock | 331 |
| electronic | rock | 313 |
Table 13: Most frequent genre–genre pairs in the dataset.
| Category | Count | Ratio |
|---|---|---|
| Same-genre pairs | 1,065 | 8.75% |
| Different-genre pairs | 11,108 | 91.25% |
Table 14: Distribution of same-genre and different-genre track pairs.
| Topic | Count | Ratio |
|---|---|---|
| vocal | 15,509 | 42.47% |
| tempo | 13,509 | 36.99% |
| instrument | 13,384 | 36.65% |
| production | 12,301 | 33.68% |
| key | 11,086 | 30.36% |
| mood | 4,646 | 12.72% |
Table 15: Distribution of question topics across the dataset.
| Question Type | Count | Avg. Difficulty |
|---|---|---|
| Yes/No | 12,173 | 2.5 |
| Short-Answer | 12,173 | 2.5 |
| Sentence-Level | 12,173 | 2.9 |
| Total | 36,519 | 2.6 |
Table 16: Distribution of question types and their average difficulty scores from Stage 4 human evaluation. Difficulty is reported on a 1–5 scale, where higher values indicate more challenging comparative reasoning.
| QA Type | Metadata | Caption | Random |
|---|---|---|---|
| Yes/No | 0.163 | 0.885 | 0.440 |
| Short Answer | 0.084 | 0.808 | 0.527 |
| Sentence (F1) | 0.141 | 0.276 | 0.062 |
| Overall | 0.129 | 0.656 | 0.343 |
Table 17: Metadata-only vs. caption-only vs. random baseline on 500 QA pairs. Metadata provides only genre, speed, and vocal gender.
| Model | Temp. | Top-p | Top-k | Max Tokens |
|---|---|---|---|---|
| MU-LLaMA | 0.6 | 0.8 | – | 512 |
| Music Flamingo | 0.0 (greedy) | – | – | 512 |
| Qwen3-Omni | 0.01 | 0.1 | 1 | 256 |
| GPT-4o-mini | 0.0 (greedy) | – | – | 512 |
Table 18: Decoding parameters used for baseline inference and LLM-as-a-Judge evaluation.
Figure 6: Qualitative example of Stage 3 comparative QA generation using Qwen3-32B . The short-answer questions occasionally contain uncommon or subjective word choices, and sentence-level answers often closely mirror the reasoning text. Red highlights mark cases where reasoning and answers are nearly identical, reducing diversity and usefulness for training.
Figure 7: Qualitative example of Stage 3 comparative QA generation using InternLM3-8B . The model frequently produces reasoning that closely resembles the question itself, and we observe occasional incorrect answers in the short-answer type. These issues indicate weaker grounding and reliability for comparative QA generation. Red highlights denote incorrect or weakly supported outputs.
Figure 8: Qualitative example of Stage 3 comparative QA generation using Gemma 3 12B . The model generally produces reasonable questions and answers; however, the reasoning is sometimes less detailed or less step-wise, which reduces interpretability for dataset construction and downstream supervision. Red highlights indicate parts where the reasoning lacks sufficient specificity.
Figure 9: Qualitative example of Stage 3 comparative QA generation using Claude 4.5 Haiku . The model typically produces strong comparative questions and fluent sentence-level answers. However, compared to GPT-5 mini, the reasoning is less consistently structured in explicit step-by-step form, which motivated our preference for GPT-5 mini in final dataset construction.
| Model | Error Type | Question (Sentence-level) | Reference | Model Output |
|---|---|---|---|---|
| GPT-4o-Audio | (A) Comp.Col. | In one sentence, how do the vocal style and emotional tone differ between Track A and Track B? | Track A features a restrained, airy vocal delivery with a more melancholic tone, while Track B has a fuller vocal presence and a warmer, more uplifting feel. | Track A and Track B both present expressive vocals and emotional atmospheres. Track A leans into a softer, intimate delivery, while Track B maintains a similarly emotive style but with a slightly different texture overall. |
| GPT-4o-Audio | (B) Attr.Hal. | In one sentence, compare the rhythmic groove and instrumentation between Track A and Track B. | Track A has a steady mid-tempo groove with prominent electric guitars and live drums, whereas Track B uses a lighter rhythmic feel with more electronic percussion and synth layers. | Track A features a hip-hop inspired beat with heavy bass and sampled percussion, while Track B relies on acoustic strumming and orchestral strings, creating a more cinematic rhythm section. |
| Music Flamingo | (B) Attr.Hal. | In one sentence, how do the primary instruments differ between Track A and Track B? | Track A is driven by electric guitars and live drums, whereas Track B emphasizes synth textures and programmed percussion. | Track A is led by piano and orchestral strings, while Track B centers on brass sections and saxophone riffs. |
| Music Flamingo | (A) Comp.Col. | In one sentence, compare the mood and atmosphere of Track A and Track B. | Track A feels darker and more introspective, while Track B is brighter and more upbeat. | Both tracks create strong atmospheres and convey distinctive moods through their instrumentation and arrangement. |
| MU-LLaMA | (A) Comp.Col. | How do the key, tempo, instrumentation, and vocal delivery of each track create their distinct emotional atmospheres? | Track A uses contrasting musical elements to produce a dramatic, cinematic feel, while Track B creates an uplifting trance atmosphere. | Both Track A and Track B share an upbeat and energetic quality, but they create distinct emotional atmospheres through their instrumentation and vocal delivery. |
| MU-LLaMA | (B) Attr.Hal. | In one sentence, how do the two tracks differ in genre, tempo, vocal timbre, and production approach? | Track A is an introspective, guitar-driven Indie/Alternative Rock song at about 100 BPM with a melancholic male tenor. | Track A has an indie genre with a weird and tribal feel, likely slower in tempo, while Track B is an upbeat electronic fusion of traditional Indian and Western styles. |
Table 19: Representative error examples for sentence-level comparative questions. Each row shows one incorrectly answered item (LLM-judge score ), together with its assigned error type. Comp.Col.: Comparative Collapse; Attr.Hal.: Attribute Hallucination.