PianoBench: Evaluating Musical Perception and Reasoning in Language Models

Louis Bradshaw; Irmak Bukey; Daniel Chenyu Lin; John Thickstun; Chris Donahue

cs.SDNov 2, 2026

PianoBench: Evaluating Musical Perception and Reasoning in Language Models

Authors: Louis Bradshaw, Irmak Bukey, Daniel Chenyu Lin, John Thickstun, Chris Donahue

Organizations: Queen Mary University of London · Carnegie Mellon University · Cornell University

Abstract

Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.

Figures & tables

Figure 1: We release PianoBench, a benchmark for evaluating music perception and reasoning in LLMs. PianoBench is derived from PianoTalk, a training-scale (20k hours) dataset of interleaved commentary and music transcripts, which is derived from instructional piano videos. Evaluation results are on PianoBench using Gemma 4 12B (thinking) and Gemini 3.1 Pro Preview (medium).

Figure 2: PianoTalk data recipe. Starting from seed videos, we crawl related videos and YouTube channels. An LLM evaluates video metadata to select videos likely to contain both speech and piano. Audio from videos with captions is fetched and screened for piano content. Audio is converted into interleaved speech and music transcripts via source separation, automatic speech recognition (ASR) and automatic music transcription (AMT).

Figure 3: Distribution of interleaving scores across PianoTalk videos, measured from chronological speech and MIDI chunks.

Figure 4: Schematic construction of PianoBench (clockwise from top left). Speech and piano activity form a source timeline. An LLM with access to the source timeline proposes questions, answers, audio intervals, and distractors. A model ranks distractors without hearing the music. A human checks the passages and answer choices, retaining Q1 and rejecting Q2.

Model	No music	MIDI-text		Piano synth		Piano stem
	Acc.	Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$
Open-weight audio-language models
Qwen 3 Omni 30B-A3B (thinking)	$42.4$	$47.6$	$+5.2$	$48.4$	$+6.0$	$46.8$	$+4.4$
Qwen 3 Omni 30B-A3B (non-thinking)	$44.8$	$47.2$	$+2.4$	$50.8$	$+6.0$	$49.2$	$+4.4$
Gemma 4 12B (thinking)	$44.8$	$58.4$	$+13.6$	$46.4$	$+1.6$	$45.2$	$+0.4$
Gemma 4 12B (non-thinking) ∗	-	-	-	$\underline{51.2}$	-	$\mathbf{51.6}$	-

Table 1: Zero-shot accuracy (%) on the 250 PianoBench MCQs. Models are evaluated with the question and choices alone (no music), MIDI-text, Pianoteq-synthesized audio, and the source-separated piano stem, where supported.

\Delta

denotes the percentage-point change from that model’s no-music accuracy.

\dagger

: Audio Flamingo 3 and Music Flamingo had inconsistent instruction following on text-only inputs.

\ast

: Gemma 4 12B continued to produce chain-of-thought on MIDI-text and no-music inputs despite thinking being disabled; we therefore omit these results. Bold is best per column / section, underline is second best.

Figure 5: Construction of mid-training documents. Interleaved and separated training setups use the same speech and piano chunks, organized either chronologically in one interleaved document or as separate speech-only and piano-only documents.

Condition	No music	MIDI-text
	Acc.	Acc.	$\Delta$
Official Instruct	$46.4$	$50.0$	$+3.6$
Official Instruct + COT	$43.6$	$70.0$	$+26.4$
Base-SFT	$40.8$	$42.8$	$+2.0$
Speech only	$37.2$	$37.6$	$+0.4$
Separated	$42.4$	$42.4$	$+0.0$

Table 2: PianoBench accuracy (%) after mid-training and instruction tuning of Qwen 3.5 9B.

\Delta

is the gain over no music (pp). Official Instruct is Qwen/Qwen3.5-9B † † footnotemark: ; Base-SFT is its base variant after our SFT. Responses use greedy decoding without chain of thought, except for Official Instruct + COT.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Inference route	Temp.	Top- $p$	Thinking
Self-hosted
Qwen 3 Omni (thinking)	huggingface:Qwen/Qwen3-Omni-30B-A3B-Thinking	0.6	0.95	1024
Qwen 3 Omni (instruct)	huggingface:Qwen/Qwen3-Omni-30B-A3B-Instruct	0	1	–
Gemma 4 12B (thinking)	huggingface:google/gemma-4-12B-it	1	0.95	1024
Gemma 4 12B (non-thinking)	huggingface:google/gemma-4-12B-it	0	1	–
Nemotron Audex 30B-A3B	huggingface:nvidia/Nemotron-Labs-Audex-30B-A3B	0	1	–

Appendix

Table 3: Requested temperature, top-

p

, and thinking settings for the models evaluated on PianoBench. Model labels abbreviate those in Table 1 where the identity remains clear.

ID	Question	#Notes	Error
36	What rhythmic style characterizes the left-hand accompaniment?	255	No
99	What articulation best describes the notes in the short repeated figure?	59	No
128	What is the relationship between the low repeated notes and the melody?	113	No
138	What recurring pulse pattern supports the melody?	44	No
157	Where does the harmony move later in the clip relative to the opening chord?	8	Yes
182	How does the chord quality change from the first chord to the next?	3	Yes

Appendix

Table 4: Questions answered incorrectly by GPT 6 Astra with MIDI-text input. The 17 errors correspond to 93.2% accuracy on 250 questions. Question 564 has an incorrect human-selected reference answer rather than a transcription error: the notes C–E

\flat

–G–B form a minor–major seventh chord, not a minor seventh chord.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 215 papers

Aug 28, 2026

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance

Musicians commonly communicate musical intent through scores and performances. To investigate whether models can reason across both modalities, we introduce MuSP-Bench, a human-authored benchmark of 490 questions assessing understanding across Musical Scores and Performances, both individually and in combination. The benchmark spans interpretive and long-horizon reasoning across classical piano and orchestral works, represented through ABC notation, score images, MIDI, and audio. Our evaluation of frontier multimodal large language models reveals substantial difficulties in understanding musical scores and even greater challenges in reasoning about audio. The benchmark is available at https://musp.vaclis.net/.

Milan Liessens Dujardin, Song-Ze Yu, Kevin Miao

Bryel Labs · UC Berkeley

Music Understanding Multimodal Model Evaluation Multimodal Reasoning Audio Understanding Music Information Retrieval

Ask PDF

Aug 30, 2026

Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models

Large audio-language models (LALMs) demonstrate growing music-understanding capabilities, but whether their responses are grounded in acoustic evidence remains unclear. Musical language often involves abstract concepts whose acoustic evidence is difficult to define and evaluate precisely. We introduce MusicGroundingBench, a controlled benchmark of algorithmically generated piano audio with exact symbolic alignment, comprising three-note and two-bar settings. We evaluate two complementary capabilities: grounding, which localizes the acoustic evidence for a musical query, and understanding, which answers questions about the same excerpts. Our experiments show that cross-modal fine-tuning enables models to learn each capability, but adding grounding supervision does not consistently improve understanding across backbones. We further test whether understanding requires listening through audio-ablation controls that remove or replace the input audio, and use attention analysis to examine whether grounding supervision shifts attention toward note boundaries. Meanwhile, the two evaluated LALMs show limited zero-shot grounding even for basic musical concepts, highlighting grounded music understanding as an important open challenge.

Kun Fang, Ziyu Wang, Ichiro Fujinaga

Schulich School of Music, McGill University, Montr´eal, QC, Canada · CIRMMT (Centre for Interdisciplinary Research in Music Media and Technology), Montr´eal, Canada · Courant Institute, New York University, New York, USA +1

Music Understanding Audio-Language Models Audio-Language Model Evaluation Temporal Audio Grounding Music Information Retrieval

Ask PDF

Oct 21, 2026

Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis

Audio-language models remain unreliable at extracting basic musical properties from audio, motivating pipelines that place a music information retrieval (MIR) tool between the audio and the model. The tool reports measurements such as keys, chords, and structural similarity, and the model is expected to derive the answer. We test whether it can. Holding the musical information fixed, we vary derivation depth: the number of reasoning steps between the tool output and the answer. When the report states the conclusion, all five models score near 100%, and they follow it even when it is wrong. One step away, accuracy drops. On modulation, open models fall to 57-81%, although they answer that single step correctly when asked directly, and reach 0% at greater depths. Frontier models degrade less, but still degrade. Adding the audio does not recover the loss. The bottleneck in tool-augmented music analysis is not access to musical information but reasoning from it, so tool design must account for how much derivation it leaves to the model.

Rupali Rajesh, Sethuraman T, Savya Khosla +4

IIT Madras · UIUC · Google +2

Music Understanding Audio-Language Model Evaluation Audio Reasoning Music Information Retrieval

Ask PDF

Abstract

Audio-Language Model Evaluation

Explore similar work

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance

Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models

Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis