Jamendo-MT-QA: A Benchmark for Multi-Track Comparative Music Question Answering

Junyoung Koh; Jaeyun Lee; Soo Yong Kim; Gyu Hyeong Choi; Jung In Koh; Jordan Phillips; Yeonjin Lee; Min Song

cs.IRApr 8, 2026

Jamendo-MT-QA: A Benchmark for Multi-Track Comparative Music Question Answering

Authors: Junyoung Koh, Jaeyun Lee, Soo Yong Kim, Gyu Hyeong Choi, Jung In Koh, Jordan Phillips, Yeonjin Lee, Min Song

Organizations: Yonsei University · KRAFTON · MODULABS MAAP · University of Oxford · Sungkyul University · George Mason University · Onoma AI

Abstract

Recent work on music question answering (Music-QA) has primarily focused on single-track understanding, where models answer questions about an individual audio clip using its tags, captions, or metadata. However, listeners often describe music in comparative terms, and existing benchmarks do not systematically evaluate reasoning across multiple tracks. Building on the Jamendo-QA dataset, we introduce Jamendo-MT-QA, a dataset and benchmark for multi-track comparative question answering. From Creative Commons-licensed tracks on Jamendo, we construct 36,519 comparative QA items over 12,173 track pairs, with each pair yielding three question types: yes/no, short-answer, and sentence-level questions. We describe an LLM-assisted pipeline for generating and filtering comparative questions, and benchmark representative audio-language models using both automatic metrics and LLM-as-a-Judge evaluation.

Figures & tables

Figure 1: Overview of the Jamendo-MT-QA construction pipeline. Starting from the Jamendo-QA dataset, we first generate rich single-track captions using Music Flamingo, expand them into single-track QA pairs using GPT-5.1, and construct comparative QA groups consisting of three question types (yes/no, short-answer, sentence-level) for each track pair via an LLM-assisted generation pipeline. Finally, we apply human evaluation and LLM-as-a-Judge assessment for quality control and filtering, resulting in the final Jamendo-MT-QA dataset.

Figure 2: Qualitative example of a Stage 3 comparative QA group generated by GPT-5 mini for the same track pair. The generator produces three comparative question types (yes/no, short-answer, sentence-level) and provides explicitly step-by-step reasoning, which improves interpretability and makes the reasoning field more suitable as a supervision signal for downstream training.

Metric	R1	R2	R3	R4	Human Mean	GPT-5 Mini Mean	MAD
Correctness	4.93	4.66	4.64	5.00	4.79	4.87	0.32
Comparative Validity	4.93	4.75	4.83	4.99	4.83	4.61	0.44
Reasoning Quality	4.87	4.71	4.67	5.00	4.78	4.37	0.56
Difficulty	1.89	2.58	3.12	2.66	2.25	2.17	0.57

Table 1: Human evaluation results on 300 sampled QA items using four annotators (R1–R4), together with LLM-as-a-Judge scores on the same subset. We report human mean scores and mean absolute deviation (MAD) to characterize aggregate-level agreement between human and LLM judgments.

Figure 3: Baseline evaluation setups for Jamendo-MT-QA. Left: caption-based single-audio baselines (e.g., Music Flamingo, MU-LLaMA), where each track is independently captioned and an LLM performs comparative reasoning over the two captions plus the question. Right: multi-audio baselines (e.g., GPT-4o(-mini) Audio, Qwen2-Audio, Qwen3-Omni), which take two audio inputs and the question to produce an answer end-to-end.

Model	Type	Yes/No	Short	Sentence-level
		Acc.	Acc.	BLEU	R-1	R-2	R-L	BERT-F1	GPT	Claude
Music Flamingo	Cap	77.4%	89.7%	4.00	27.5	4.6	24.7	0.8786	3.24	3.87
Qwen2-Audio	Cap	37.4%	39.1%	1.88	19.3	1.8	17.4	0.8489	1.49	1.53
MU-LLaMA	Cap	20.6%	55.3%	2.39	23.8	3.2	16.5	0.8572	2.36	2.01
Qwen2-Audio	Multi	50.9%	80.2%	2.09	21.0	3.4	14.6	0.8472	1.37	1.62
Qwen3-Omni	Multi	62.9%	80.3%	3.58	29.3	6.6	20.4	0.8632	3.11	3.48

Table 2: Results on the full benchmark of 12,173 track pairs.

Model	Type	Yes/No	Short	Sentence-level
		Acc.	Acc.	BLEU	R-1	R-2	R-L	BERT-F1	GPT	Claude
Music Flamingo	Cap	82.1%	88.1%	2.06	25.3	5.9	19.6	0.875	3.25	3.97
Qwen2-Audio	Cap	26.9%	55.5%	0.58	15.3	1.6	12.1	0.845	1.37	1.60
MU-LLaMA	Cap	23.6%	50.5%	0.68	16.1	1.9	13.3	0.853	1.92	2.13
Qwen2-Audio	Multi	34.4%	77.7%	0.57	14.3	1.4	11.3	0.845	1.63	1.75
Qwen3-Omni	Multi	59.7%	75.5%	1.22	19.9	3.9	16.0	0.860	2.57	3.25

Table 3: Results on the subset of 2,010 track pairs.

Model	Comp.Col.	Attr.Hal.	Gran.Mis.
Music Flamingo	35.3%	64.0%	0.7%
MU-LLaMA	37.3%	60.7%	2.0%
GPT-4o-Audio	56.7%	35.3%	8.0%
Qwen3-Omni	23.4%	75.9%	0.7%

Table 4: Error type distribution across models on sentence-level questions (LLM judge score

<

3). Classified by GPT-4o-mini.

Figure 4: Heatmap visualization of error type distributions across models. Values indicate the percentage of each error category among incorrectly answered sentence-level questions (LLM judge score $<$ 3).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Annotator	Mean	Std
Annotator A	4.43	0.65
Annotator B	4.47	0.62
Annotator C	4.60	0.71
Annotator D	4.85	0.48
Average	4.59	–

Appendix

Table 5: Human evaluation results for audio–text alignment of enriched captions in Stage 1, reported on a 1–5 scale.

Musical Attribute	Coverage	Count
Instrumentation	99.9%	7,325 / 7,335
Production style	99.7%	7,316 / 7,335
Vocal characteristics	99.7%	7,316 / 7,335
Mood / atmosphere	98.4%	7,216 / 7,335
Tempo	96.7%	7,095 / 7,335
Genre	96.3%	7,061 / 7,335

Appendix

Table 6: Caption attribute coverage across all 7,335 Music Flamingo captions.

Score	Description
0	Completely wrong or irrelevant
1	Mostly wrong with very minor correct elements
2	Partially correct but missing key points
3	Correct on main points but missing details or minor errors
4	Mostly correct with only minor omissions
5	Fully correct, capturing all key comparative information

Appendix

Table 7: Scoring rubric used by LLM judges for sentence-level comparative answers.

Metric	GPT $\to$ GPT	GPT $\to$ Claude	$p$
Correctness	4.729	4.741	0.318
Comp. Validity	4.721	4.735	0.264
Reasoning Qual.	4.721	4.735	0.264

Appendix

Table 8: Cross-model self-preference analysis (GPT evaluator,

N{\approx}13{,}096

). No significant bias detected (

d<0.01

Question Type	Evaluation Method	Score Range
Yes/No	Rule-based matching	0 or 1
Short-answer	Rule-based matching (Track A/B)	0 or 1
Sentence-level	LLM-as-a-Judge	0–5
Open-ended	LLM-as-a-Judge	0–5

Appendix

Table 9: Evaluation methods used for different question types.

Verification Check	Tested	Pass
Answer format validity (non-empty)	36,519	100.0%
Yes/No: valid “yes”/“no” answer	12,173	100.0%
Short Answer: contains track name	12,173	100.0%
Sentence: answer $\geq$ 20 characters	12,173	100.0%
All 3 QA types present per pair	12,173	100.0%
Tempo answer correct (SA)	1,627	100.0%

Appendix

Table 10: Full dataset quality verification (12,173 pairs).

Statistic	Value
Total music pairs	12,173
Total QA pairs	36,519
Unique genre pairs	789

Appendix

Table 11: Basic statistics of the multi-track comparative QA dataset.

Genre	Count	Ratio
rock	4,890	20.09%
indie	3,424	14.06%
pop	2,527	10.38%
folk	1,611	6.62%
electronic	1,446	5.94%
rap	1,358	5.58%

Appendix

Table 12: Top genres across all tracks in the dataset.

Figure 5: Genre composition and genre-pair distribution in Jamendo-MT-QA. The inner ring shows the marginal distribution of individual genres across all tracks, while the outer ring visualizes the frequency of genre–genre pairings used for comparative question generation. For clarity, only the top-10 most frequent genres are shown in the visualization; therefore, percentages may differ from table-based statistics computed over all tracks.

Genre 1	Genre 2	Count
indie	rock	674
pop	rock	497
rock	rock	488
indie	pop	367
folk	rock	331
electronic	rock	313

Appendix

Table 13: Most frequent genre–genre pairs in the dataset.

Category	Count	Ratio
Same-genre pairs	1,065	8.75%
Different-genre pairs	11,108	91.25%

Appendix

Table 14: Distribution of same-genre and different-genre track pairs.

Topic	Count	Ratio
vocal	15,509	42.47%
tempo	13,509	36.99%
instrument	13,384	36.65%
production	12,301	33.68%
key	11,086	30.36%
mood	4,646	12.72%

Appendix

Table 15: Distribution of question topics across the dataset.

Question Type	Count	Avg. Difficulty
Yes/No	12,173	2.5
Short-Answer	12,173	2.5
Sentence-Level	12,173	2.9
Total	36,519	2.6

Appendix

Table 16: Distribution of question types and their average difficulty scores from Stage 4 human evaluation. Difficulty is reported on a 1–5 scale, where higher values indicate more challenging comparative reasoning.

QA Type	Metadata	Caption	Random
Yes/No	0.163	0.885	0.440
Short Answer	0.084	0.808	0.527
Sentence (F1)	0.141	0.276	0.062
Overall	0.129	0.656	0.343

Appendix

Table 17: Metadata-only vs. caption-only vs. random baseline on 500 QA pairs. Metadata provides only genre, speed, and vocal gender.

Model	Temp.	Top-p	Top-k	Max Tokens
MU-LLaMA	0.6	0.8	–	512
Music Flamingo	0.0 (greedy)	–	–	512
Qwen3-Omni	0.01	0.1	1	256
GPT-4o-mini	0.0 (greedy)	–	–	512

Appendix

Table 18: Decoding parameters used for baseline inference and LLM-as-a-Judge evaluation.

Figure 6: Qualitative example of Stage 3 comparative QA generation using Qwen3-32B . The short-answer questions occasionally contain uncommon or subjective word choices, and sentence-level answers often closely mirror the reasoning text. Red highlights mark cases where reasoning and answers are nearly identical, reducing diversity and usefulness for training.

Figure 7: Qualitative example of Stage 3 comparative QA generation using InternLM3-8B . The model frequently produces reasoning that closely resembles the question itself, and we observe occasional incorrect answers in the short-answer type. These issues indicate weaker grounding and reliability for comparative QA generation. Red highlights denote incorrect or weakly supported outputs.

Figure 8: Qualitative example of Stage 3 comparative QA generation using Gemma 3 12B . The model generally produces reasonable questions and answers; however, the reasoning is sometimes less detailed or less step-wise, which reduces interpretability for dataset construction and downstream supervision. Red highlights indicate parts where the reasoning lacks sufficient specificity.

Figure 9: Qualitative example of Stage 3 comparative QA generation using Claude 4.5 Haiku . The model typically produces strong comparative questions and fluent sentence-level answers. However, compared to GPT-5 mini, the reasoning is less consistently structured in explicit step-by-step form, which motivated our preference for GPT-5 mini in final dataset construction.

Model	Error Type	Question (Sentence-level)	Reference	Model Output
GPT-4o-Audio	(A) Comp.Col.	In one sentence, how do the vocal style and emotional tone differ between Track A and Track B?	Track A features a restrained, airy vocal delivery with a more melancholic tone, while Track B has a fuller vocal presence and a warmer, more uplifting feel.	Track A and Track B both present expressive vocals and emotional atmospheres. Track A leans into a softer, intimate delivery, while Track B maintains a similarly emotive style but with a slightly different texture overall.
GPT-4o-Audio	(B) Attr.Hal.	In one sentence, compare the rhythmic groove and instrumentation between Track A and Track B.	Track A has a steady mid-tempo groove with prominent electric guitars and live drums, whereas Track B uses a lighter rhythmic feel with more electronic percussion and synth layers.	Track A features a hip-hop inspired beat with heavy bass and sampled percussion, while Track B relies on acoustic strumming and orchestral strings, creating a more cinematic rhythm section.
Music Flamingo	(B) Attr.Hal.	In one sentence, how do the primary instruments differ between Track A and Track B?	Track A is driven by electric guitars and live drums, whereas Track B emphasizes synth textures and programmed percussion.	Track A is led by piano and orchestral strings, while Track B centers on brass sections and saxophone riffs.
Music Flamingo	(A) Comp.Col.	In one sentence, compare the mood and atmosphere of Track A and Track B.	Track A feels darker and more introspective, while Track B is brighter and more upbeat.	Both tracks create strong atmospheres and convey distinctive moods through their instrumentation and arrangement.
MU-LLaMA	(A) Comp.Col.	How do the key, tempo, instrumentation, and vocal delivery of each track create their distinct emotional atmospheres?	Track A uses contrasting musical elements to produce a dramatic, cinematic feel, while Track B creates an uplifting trance atmosphere.	Both Track A and Track B share an upbeat and energetic quality, but they create distinct emotional atmospheres through their instrumentation and vocal delivery.
MU-LLaMA	(B) Attr.Hal.	In one sentence, how do the two tracks differ in genre, tempo, vocal timbre, and production approach?	Track A is an introspective, guitar-driven Indie/Alternative Rock song at about 100 BPM with a melancholic male tenor.	Track A has an indie genre with a weird and tribal feel, likely slower in tempo, while Track B is an upbeat electronic fusion of traditional Indian and Western styles.

Appendix

Table 19: Representative error examples for sentence-level comparative questions. Each row shows one incorrectly answered item (LLM-judge score

<3

), together with its assigned error type. Comp.Col.: Comparative Collapse; Attr.Hal.: Attribute Hallucination.

Explore similar work

Mar 29, 2026

HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.

Benno Weck, Pablo Puentes, Andrea Poltronieri +2

Universitat Pompeu Fabra · Universitat Autònoma de Barcelona

Audio-Language Model Evaluation Audio Understanding Audio-Language Models Music Information Retrieval

Ask PDF

Oct 8, 2026

STEMMA: Song-to-Stem Multi-Audio Reasoning for Large Audio Language Models

Music understanding often requires comparing excerpts and reasoning about relationships among songs, sections, and stems. However, existing large audio-language models (LALMs) and music question-answering datasets typically operate on single recordings or compare independently sampled tracks with no known production relationship. We introduce STEMMA, a multi-audio music question-answering framework built around production provenance: whether excerpts originate from the same track or section, and which stems belong to which mixtures. Because such relations are sparse under conventional audio-first sampling, STEMMA adopts a relation-first construction strategy: it first specifies a target relation and then queries the catalog for excerpts that satisfy it and hard negatives that do not. Labels are determined directly from catalog provenance rather than generated by a language model from metadata. We build STEMMA-Bench for evaluation and a track-disjoint training set, STEMMA-Instruct. Fine-tuning two LALMs on STEMMA-Instruct improves multi-audio reasoning, with the largest gains on structural relations directly determined by the catalog, while preserving single-audio music understanding.

Hoyeol Sohn, Wonil Kim, Keunhyoung Kim +7

Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea · Neutune, Seoul, Republic of Korea

Audio Reasoning Audio-Language Models Audio-Language Model Evaluation Music Information Retrieval

Ask PDF

Feb 28, 2026

Voices of Civilizations: A Multilingual QA Benchmark for Global Music Understanding

We introduce Voices of Civilizations, the first multilingual QA benchmark for evaluating audio LLMs' cultural comprehension on full-length music recordings. Covering 380 tracks across 38 languages, our automated pipeline yields 1,190 multiple-choice questions through four stages - each followed by manual verification: 1) compiling a representative music list; 2) generating cultural-background documents for each sample in the music list via LLMs; 3) extracting key attributes from those documents; and 4) constructing multiple-choice questions probing language, region associations, mood, and thematic content. We evaluate models under four conditions and report per-language accuracy. Our findings demonstrate that even state-of-the-art audio LLMs struggle to capture subtle cultural nuances without rich textual context and exhibit systematic biases in interpreting music from different cultural traditions. The dataset is publicly available on Hugging Face to foster culturally inclusive music understanding research.

Shangda Wu, Ziya Zhou, Yongyi Zang +4

Independent Researcher · The Hong Kong University of Science and Technology, Hong Kong SAR, China · The Chinese University of Hong Kong, Hong Kong SAR, China

Audio-Language Model Evaluation Cultural Knowledge in Language Models Multilingual QA Cultural Bias in Language Models Music Information Retrieval

Ask PDF