Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.
Figures & tables
Model
Architecture
L
H
Params
Masked Modeling (MM)
MERT-v1-95M
Transformer Enc.
12
768
95 M
MERT-v1-330M
Transformer Enc.
24
1024
330 M
MusicFM MSD
Conformer Enc.
12
1024
330 M
MuQ iter
Conformer Enc.
12
1024
310 M
OMAR-RQ multifeature
Conformer Enc.
24
1024
580 M
Table 1: Models included in our study, grouped by pre-training paradigm. L : number of layers; H : hidden dim. † Hierarchical Swin Transformer; H doubles across stages.
Figure 1: (a) Mean within-model Spearman ρ between intrinsic-metric and downstream-performance layer profiles, averaged by pre-training paradigm. † Weakens or flips sign under depth control (partial Spearman) or across models. (b) Top-3 proxy-guided layer-selection mean gap (pp ↓ ) for masked models. The best intrinsic-metric proxy varies by task.
Figure 2: In autoregressive models, intrinsic dimension peaks within the shaded intermediate-depth band where performance on semantic and rhythm tasks is highest.
How can we understand what a music foundation model has learned internally? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as structured relations rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, we shift from feature identification to structure-based analysis, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.
Liwei Lin, Gus Xia
Computer Science Department, NYU Shanghai · Machine Learning Department, MBZUAI
This study analyzes the intermediate representations of music foundation models (MFMs) and reports the geometric structures used to represent pitch information. By inputting isolated musical notes into trained MFMs and analyzing their principal components, we reveal that the representations form a helical structure reflecting the octave periodicity of pitch. Furthermore, we show that the clarity and geometry of this helical structure vary not only across models but also with the acoustic properties of the input signals. Our analysis provides a novel approach for clarifying the internal mechanisms of MFMs.
Hayato Yagi, Shinnosuke Takamichi, Rin Sato +2
Keio University, Japan · The University of Tokyo, Japan · Waseda University, Japan
Self-supervised music foundation models underperform on key detection, which requires pitch-sensitive representations. In this work, we present the first systematic study showing that the design of self-supervised pretraining directly impacts pitch sensitivity, and demonstrate that masked contrastive embeddings uniquely enable state-of-the-art (SOTA) performance in key detection in the supervised setting. First, we discover that linear evaluation after masking-based contrastive pretraining on Mel spectrograms leads to competitive performance on music key detection out of the box. This leads us to train shallow but wide multi-layer perceptrons (MLPs) on features extracted from our base model, leading to SOTA performance without the need for sophisticated data augmentation policies. We further analyze robustness and show empirically that the learned representations naturally encode common augmentations. Our study establishes self-supervised pretraining as an effective approach for pitch-sensitive MIR tasks and provides insights for designing and probing music foundation models.
Ori Yonay, Tracy Hammond, Tianbao Yang
Department of Computer Science and Engineering, Texas A&M University