PianoBench: Evaluating Musical Perception and Reasoning in Language Models

Louis Bradshaw; Irmak Bukey; Daniel Chenyu Lin; John Thickstun; Chris Donahue

cs.SDNov 2, 2026

PianoBench: Evaluating Musical Perception and Reasoning in Language Models

Authors: Louis Bradshaw, Irmak Bukey, Daniel Chenyu Lin, John Thickstun, Chris Donahue

Organizations: Queen Mary University of London · Carnegie Mellon University · Cornell University

Abstract

Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.

Figures & tables

Figure 1: We release PianoBench, a benchmark for evaluating music perception and reasoning in LLMs. PianoBench is derived from PianoTalk, a training-scale (20k hours) dataset of interleaved commentary and music transcripts, which is derived from instructional piano videos. Evaluation results are on PianoBench using Gemma 4 12B (thinking) and Gemini 3.1 Pro Preview (medium).

Figure 2: PianoTalk data recipe. Starting from seed videos, we crawl related videos and YouTube channels. An LLM evaluates video metadata to select videos likely to contain both speech and piano. Audio from videos with captions is fetched and screened for piano content. Audio is converted into interleaved speech and music transcripts via source separation, automatic speech recognition (ASR) and automatic music transcription (AMT).

Figure 3: Distribution of interleaving scores across PianoTalk videos, measured from chronological speech and MIDI chunks.

Figure 4: Schematic construction of PianoBench (clockwise from top left). Speech and piano activity form a source timeline. An LLM with access to the source timeline proposes questions, answers, audio intervals, and distractors. A model ranks distractors without hearing the music. A human checks the passages and answer choices, retaining Q1 and rejecting Q2.

Model	No music	MIDI-text		Piano synth		Piano stem
	Acc.	Acc.	$\Delta$	Acc.	$\Delta$	Acc.	$\Delta$
Open-weight audio-language models
Qwen 3 Omni 30B-A3B (thinking)	$42.4$	$47.6$	$+5.2$	$48.4$	$+6.0$	$46.8$	$+4.4$
Qwen 3 Omni 30B-A3B (non-thinking)	$44.8$	$47.2$	$+2.4$	$50.8$	$+6.0$	$49.2$	$+4.4$
Gemma 4 12B (thinking)	$44.8$	$58.4$	$+13.6$	$46.4$	$+1.6$	$45.2$	$+0.4$
Gemma 4 12B (non-thinking) ∗	-	-	-	$\underline{51.2}$	-	$\mathbf{51.6}$	-

Table 1: Zero-shot accuracy (%) on the 250 PianoBench MCQs. Models are evaluated with the question and choices alone (no music), MIDI-text, Pianoteq-synthesized audio, and the source-separated piano stem, where supported.

\Delta

denotes the percentage-point change from that model’s no-music accuracy.

\dagger

: Audio Flamingo 3 and Music Flamingo had inconsistent instruction following on text-only inputs.

\ast

: Gemma 4 12B continued to produce chain-of-thought on MIDI-text and no-music inputs despite thinking being disabled; we therefore omit these results. Bold is best per column / section, underline is second best.

Figure 5: Construction of mid-training documents. Interleaved and separated training setups use the same speech and piano chunks, organized either chronologically in one interleaved document or as separate speech-only and piano-only documents.

Condition	No music	MIDI-text
	Acc.	Acc.	$\Delta$
Official Instruct	$46.4$	$50.0$	$+3.6$
Official Instruct + COT	$43.6$	$70.0$	$+26.4$
Base-SFT	$40.8$	$42.8$	$+2.0$
Speech only	$37.2$	$37.6$	$+0.4$
Separated	$42.4$	$42.4$	$+0.0$

Table 2: PianoBench accuracy (%) after mid-training and instruction tuning of Qwen 3.5 9B.

\Delta

is the gain over no music (pp). Official Instruct is Qwen/Qwen3.5-9B † † footnotemark: ; Base-SFT is its base variant after our SFT. Responses use greedy decoding without chain of thought, except for Official Instruct + COT.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Inference route	Temp.	Top- $p$	Thinking
Self-hosted
Qwen 3 Omni (thinking)	huggingface:Qwen/Qwen3-Omni-30B-A3B-Thinking	0.6	0.95	1024
Qwen 3 Omni (instruct)	huggingface:Qwen/Qwen3-Omni-30B-A3B-Instruct	0	1	–
Gemma 4 12B (thinking)	huggingface:google/gemma-4-12B-it	1	0.95	1024
Gemma 4 12B (non-thinking)	huggingface:google/gemma-4-12B-it	0	1	–
Nemotron Audex 30B-A3B	huggingface:nvidia/Nemotron-Labs-Audex-30B-A3B	0	1	–

Appendix

Table 3: Requested temperature, top-

p

, and thinking settings for the models evaluated on PianoBench. Model labels abbreviate those in Table 1 where the identity remains clear.

ID	Question	#Notes	Error
36	What rhythmic style characterizes the left-hand accompaniment?	255	No
99	What articulation best describes the notes in the short repeated figure?	59	No
128	What is the relationship between the low repeated notes and the melody?	113	No
138	What recurring pulse pattern supports the melody?	44	No
157	Where does the harmony move later in the clip relative to the opening chord?	8	Yes
182	How does the chord quality change from the first chord to the next?	3	Yes

Appendix

Table 4: Questions answered incorrectly by GPT 6 Astra with MIDI-text input. The 17 errors correspond to 93.2% accuracy on 250 questions. Question 564 has an incorrect human-selected reference answer rather than a transcription error: the notes C–E

\flat

–G–B form a minor–major seventh chord, not a minor seventh chord.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 215 papers

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance
Aug 28, 2026Milan Liessens Dujardin, Song-Ze Yu, Kevin MiaoMusic Understanding Multimodal Model Evaluation
Bryel Labs · UC Berkeley
Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models
Aug 30, 2026Kun Fang, Ziyu Wang, Ichiro FujinagaMusic Understanding Audio-Language Models
Schulich School of Music, McGill University, Montr´eal, QC, Canada · CIRMMT (Centre for Interdisciplinary Research in Music Media and Technology), Montr´eal, Canada · Courant Institute, New York University, New York, USA +1
Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis
Oct 21, 2026Rupali Rajesh, Sethuraman T, Savya Khosla +4Music Understanding Audio-Language Model Evaluation
IIT Madras · UIUC · Google +2

Aug 28, 2026J/K move · Enter open · S save

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance

Milan Liessens Dujardin, Song-Ze Yu, Kevin Miao

Bryel Labs · UC Berkeley

Music Understanding Multimodal Model Evaluation Multimodal Reasoning Audio Understanding Music Information Retrieval

Open paper AskPDF arXiv

Musicians commonly communicate musical intent through scores and performances. To investigate whether models can reason across both modalities, we introduce MuSP-Bench, a human-authored benchmark of 490 questions assessing understanding across Musical Scores and Performances, both individually and in combination. The benchmark spans interpretive and long-horizon reasoning across classical piano and orchestral works, represented through ABC notation, score images, MIDI, and audio. Our evaluation of frontier multimodal large language models reveals substantial difficulties in understanding musical scores and even greater challenges in reasoning about audio. The benchmark is available at https://musp.vaclis.net/.

Abstract

Audio-Language Model Evaluation

Explore similar work

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance

Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models

Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis