cs.SDNov 1, 2026

MEX: The Musical Expectation Benchmark Suite

Authors: Mathias Rose Bjare, Augustin Bouquillard, Giorgia Cantisani, Seung-Goo Kim, Stefan Lattner, Gerhard Widmer

Organizations: Johannes Kepler University, Linz, Austria · École polytechnique, Palaiseau, France · Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany · STMS, CNRS, IRCAM, Sorbonne Université, Paris, France · 

Abstract

We propose the Musical Expectation (MEX) benchmark suite for perceptually validating computational models of musical expectation, i.e., models that estimate music information dynamics (MID) measures of perceived musical expectations, such as surprisal or uncertainty from symbolic music or musical audio. MEX collects and extends tasks for evaluating such models previously proposed in music information retrieval and cognitive sciences using standard datasets, and evaluates whether: (1) proposed MID measures decrease with repetition due to in-context learning of the computational model, (2) peaks in such measures align with boundaries of music segments, (3) pitch surprisal and uncertainty estimated from audio are consistent with those modeled in event-based symbolic music, (4) proposed MID measures can predict electrophysiological responses to music. We stress that this benchmark suite does not provide an exhaustive computational characterization of music expectation. Rather, we envision this as an active, interdisciplinary initiative, with benchmarks evolving as our understanding of the underlying perceptual processes deepens. We provide open-access code https://github.com/musicperceptmodeling/mex to estimate surprisal and uncertainty from audio using baseline models, run the tasks on standard datasets, and include utilities for running the tasks with users' own data.

Figures & tables

Explore similar work

Jul 7, 2026

Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music

Music represents a cornerstone of human culture, existing digitally across diverse modalities, including audio, symbolic encodings (e.g., MIDI, MusicXML), and sheet music. Despite the advancement of Multimodal Large Language Models (MLLMs), current music benchmarks face three major limitations. First, large static benchmarks are resource-intensive to evaluate, and it remains unclear how their results transfer to diverse kinds of music beyond those included in the benchmark. Second, benchmarks claiming to measure "music understanding" often fail to require music perception. Third, they do not support systematic performance comparisons across musical modalities. To overcome these issues, we introduce the Music I Care About Meta-Benchmark (MusICA-MetaBench), a framework that automatically derives on-demand benchmarks directly from user-provided data. By leveraging structured symbolic representations (e.g., MusicXML) and our pre-defined question templates, we build multiple-choice question-answer pairs that probe music perception competencies, aligned with music pedagogy, across audio, music notation images, and symbolic files. We demonstrate our framework with the ChoraleBricks dataset, and experimentally determine benchmark sizes that ensure statistically reliable model comparisons for this setup. By comparing against text-only and white-noise baselines, we show our questions do measure music perception. Ultimately, MusICA-MetaBench represents a significant advancement in the cross-modal assessment of music perception for MLLMs. By proposing a dataset-specific benchmarking paradigm, it enables efficient on-demand evaluation of music perception capabilities.
Oct 16, 2026

MUB: An Identity-Masked Benchmark for Music Understanding in Multimodal Language Models

Evaluating musical perception in multimodal language models requires separating audio evidence from answers recoverable through metadata or external tools. We present the Music Understanding Benchmark (MUB), a locally assembled collection of 500 identity-masked recordings with existing annotations for tempo, key, time signature, and structural intervals. The collection covers all 24 major/minor key combinations, tempos of 47--180 beats per minute, and 4,676 annotated intervals. Original titles, artists, and platform identifiers are excluded from evaluation inputs. Relative to recent music question-answering and temporal-grounding work, MUB emphasizes scalar tempo prediction and full-recording interval outputs, together with a documented metadata-masking protocol. Saved baseline predictions yield 53.6% tempo accuracy for a simple DSP estimator and 98.4% time-signature accuracy for a constant 4/4 predictor. We identify important limitations, including an asymmetric legacy key metric, unverified independent annotation agreement, extreme metre imbalance, and unresolved recording-level rights clearance. No contemporary multimodal language model comparison has yet been completed. MUB is therefore an evaluation resource and protocol under validation, rather than a completed demonstration of model capabilities.
May 25, 2026

PitchBench: Measuring Pitch Hearing in Audio-Language Models

Audio-language models (ALMs) are increasingly used in real-world applications that require music understanding, including tutoring, captioning, recommendation, and production. Yet existing benchmarks rarely evaluate a fundamental component of this understanding: pitch hearing. Pitch underpins the identification of notes, chords, and melodies and the grounding of captions in musical events, but it remains unclear how reliably ALMs perceive fine-grained pitch across sound sources, acoustic conditions, and response formats. We introduce PitchBench, an evaluation suite comprising 28 tasks that assess absolute and relative pitch perception under variations in loudness, note duration, sound source, time stretching, background noise, and other acoustic conditions. Tasks range from identifying isolated pitches to tracking voices in four-part musical textures. Evaluating frontier ALMs, we find that pitch hearing remains highly unreliable. Models perform poorly across settings, with accuracy varying sharply depending on the notation or prompt format, indicating instruction drift. Specialized DSP and neural pitch baselines achieve mean scores of approximately 73.2% and 75.2%, respectively, compared with approximately 53.9% for the strongest ALM. We release PitchBench as an evaluation benchmark, a training dataset, and a Python package with data generation tools to support future research on pitch-aware audio-language modeling.