Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models
Authors: Kun Fang, Ziyu Wang, Ichiro Fujinaga
Organizations: Schulich School of Music, McGill University, Montr´eal, QC, Canada · CIRMMT (Centre for Interdisciplinary Research in Music Media and Technology), Montr´eal, Canada · Courant Institute, New York University, New York, USA · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE
Large audio-language models (LALMs) demonstrate growing music-understanding capabilities, but whether their responses are grounded in acoustic evidence remains unclear. Musical language often involves abstract concepts whose acoustic evidence is difficult to define and evaluate precisely. We introduce MusicGroundingBench, a controlled benchmark of algorithmically generated piano audio with exact symbolic alignment, comprising three-note and two-bar settings. We evaluate two complementary capabilities: grounding, which localizes the acoustic evidence for a musical query, and understanding, which answers questions about the same excerpts. Our experiments show that cross-modal fine-tuning enables models to learn each capability, but adding grounding supervision does not consistently improve understanding across backbones. We further test whether understanding requires listening through audio-ablation controls that remove or replace the input audio, and use attention analysis to examine whether grounding supervision shifts attention toward note boundaries. Meanwhile, the two evaluated LALMs show limited zero-shot grounding even for basic musical concepts, highlighting grounded music understanding as an important open challenge.
Figures & tables
Figure 1: Overview of the MusicGroundingBench construction pipeline. We first specify the evaluation scope for MGBench-3N and MGBench-2B, generate symbolic MIDI, render aligned audio, and construct grounding and understanding QA pairs. The resulting data are then validated and divided into clip-disjoint splits. G and U denote grounding and understanding, respectively.
IoU≥0.5
IoU≥0.7
Model
IFR ↑
MergedIoU ↑
P ↑
R ↑
F1 ↑
On./Off. MAE ↓
P ↑
R ↑
F1 ↑
On./Off. MAE ↓
MGBench-3N Grounding Test Set (Count: 900)
GT-Note Baseline
0.9433
0.4353
0.7603
0.4979
0.6017
5.00 / 5.80
0.7487
0.4903
0.5926
0.70 / 0.90
Non-LLM (FiLM)
N/A
0.4839
0.2534
0.4176
0.3154
63.50 / 66.89
0.1483
0.2444
0.1846
31.01 / 25.30
Non-LLM (XAttn)
N/A
0.5615
0.5008
0.8296
0.6246
147.56 / 133.58
0.2226
0.3687
0.2776
86.77 / 75.10
Music Flamingo (Zero-shot)
0.7867
0.0666
0.0350
0.0243
0.0287
209.62 / 172.88
0.0109
0.0076
0.0089
130.86 / 81.11
Table 1: Grounding results on MGBench-3N and MGBench-2B grounding test sets. We report instruction following rate (IFR), MergedIoU computed over the unions of predicted and ground-truth time spans, and precision, recall, F1, onset/offset MAE under greedy matching at IoU thresholds of 0.5 and 0.7, where 0.7 is the stricter criterion. MAE is computed over matched spans only and reported in milliseconds. The GT-note baseline supplies zero-shot Qwen2.5-7B-Instruct with exact MIDI note identities and boundaries but no audio. External LALMs are evaluated zero-shot without MGBench-specific training. The remaining models receive task-specific supervision.
Accuracy ↑
Per-Concept Accuracies ↑
Model / Setting
IFR ↑
Global (1078)
Rhythm (161)
Tonality (161)
Interval (160)
BarComp. (160)
Harmony (138)
Repetition (160)
HarmonicPat. (138)
LALM Baselines (Zero-Shot)
Music Flamingo
0.7208
0.2542
0.3106
0.1553
0.1875
0.1938
0.3116
0.2313
0.4203
Qwen2-Audio-7B-Instruct
0.5779
0.1558
0.1180
0.1739
0.1000
0.1625
0.1232
0.1500
0.2754
LLaMA-3-8B (Ours)
Train on U
0.8989
0.6113
0.4969
0.5963
0.4313
0.4625
0.7391
0.6875
0.9275
Table 2: Understanding results on the MGBench-2B test set. Accuracy is computed over all test queries. Semantic correctness of instruction-following outputs is determined by the LLM judge. The final four rows report audio-input, text-only, and GT-note controls defined in the experimental setup. Numbers in parentheses in the column headers denote the number of test queries.
Figure 2: Attention analysis for Qwen3.5-9B-Base. Left: An illustrative layer-19 attention map for a grounding query with the answer 3.57–4.67 s. Middle: Attention per note around onsets/offsets across the grounding model’s eight full-attention layers. Right: The same measure during understanding for U , U+0.5G , and U+1.0G , averaged over all eight layers. Full and light shades denote evidence notes needed to determine the answer and the remaining notes, respectively.
Recent Large Audio-Language Models (LALMs) have demonstrated promising abilities in understanding musical content. However, whether their responses are grounded in the correct temporal regions of the audio remains underexplored. This limitation is particularly critical for music understanding, where key information often occurs as temporally localized events, such as instrument entries and rhythmic transitions. To address this gap, we introduce MusTBench, a music-expert-validated benchmark designed to evaluate temporal grounding in LALMs through five temporally grounded question-answering tasks. To further improve temporal grounding in existing models, we propose MusT, a novel four-stage temporal optimization recipe spanning music encoder adaptation, LLM adaptation, LLM supervised fine-tuning, and RL-based optimization. Experiments on MusTBench show that existing LALMs struggle with precise temporal grounding, while MusT brings significant improvements over strong baselines. These results establish temporal grounding as a key missing capability in current LALMs and position MusTBench as a challenging benchmark for future research in temporally grounded music understanding.
Daeyong Kwon, Qiyu Wu, Shinobu Kuriya +6
1Seoul National University · 2Sony Group Corporation · 3Sony AI
Recent music audio-language models achieve high accuracy on instrument question-answering benchmarks, but it remains unclear whether this reflects robust audio grounding or benchmark-specific shortcuts. In this paper, we introduce an OpenMIC-derived diagnostic benchmark sequence for instrument grounding in music audio-language models, extending binary instrument-presence QA to genre-prior-reduced examples, confusable instrument discrimination, longer audio context, and temporal localization. Across these settings, high binary QA accuracy often fails to predict model behavior: models can exhibit option-position bias, confusable-instrument errors, and temporal response bias. These results suggest that instrument grounding should be evaluated with multi-axis diagnostic benchmarks rather than a single aggregate accuracy.
The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.
Benno Weck, Pablo Puentes, Andrea Poltronieri +2
Universitat Pompeu Fabra · Universitat Autònoma de Barcelona