Pitch is a fundamental aspect of auditory perception. Pitch perception is commonly described across two perceptual dimensions: pitch height is the sense that tones with varying frequencies seem to be higher or lower, and chroma equivalence is the cyclical similarity of notes octaves, corresponding to a doubling of fundamental frequency. Existing research is divided on whether chroma equivalence is a learned percept that varies according to musical experience and culture, or is an innate percept that develops automatically. Building on a recent framework that proposes to use ANNs to ask 'why' questions about the brain, we evaluated recent auditory ANNs using representational similarity analysis to test the emergence of pitch height and chroma equivalence in their learned representations. Additionally, we fine-tuned two models, Wav2Vec 2.0 and Data2Vec, on a self-supervised learning task using speech and music, and a supervised music transcription task. We found that all models exhibited varying degrees of pitch height representation, but that only models trained on the supervised music transcription task exhibited chroma equivalence. Mere exposure to music through self-supervised learning was not sufficient for chroma equivalence to emerge. This supports the view that chroma equivalence is a higher-order cognitive computation that emerges to support the specific task of music perception, distinct from other auditory perception such as speech listening. This work also highlights the usefulness of ANNs for probing the developmental conditions that give rise to perceptual representations in humans.
Figures & tables
Figure 1: Representations of Pitch . Helical Representation of Pitch.The vertical dimension corresponds to absolute pitch height as distance in fundamental frequency. The circular horizontal dimension corresponds to the repeating cyclical octaves of chroma.
Model
Variant
Paradigm
Training Task
Training Data
Wav2Vec 2.0 [ 4 ]
wav2vec2-base
SSL
Contrastive loss (masked)
Librispeech
Data2Vec [ 3 ]
data2vec-audio-base
SSL
Student-teacher (masked)
Librispeech
Whisper [ 22 ]
whisper-base
Supervised
Multitask ASR
Multilingual speech
MERT [ 18 ]
MERT-v1-95M
SSL
Student-teacher (Acoustic+CQT)
Music
AST [ 11 ]
ast-finetuned-audioset
SSL+SFT
Audio classification
ImageNet + Audioset
Spectrogram
128 Bands
Transform
Mel scale
N/A
Table 1: Summary of neural network architectures and baseline models evaluated.
Figure 2: Note Representational Dissimilarity Matrix (RDM) Plots. RDM plots for NSynth test note dataset. RDMs were averaged across 30 instruments and normalized for plotting.
Figure 3: RSA Analysis for Pretrained and Finetuned Models.
We argue that training autoencoders to reconstruct inputs from noised versions of their encodings, when combined with perceptually motivated losses, yields encodings that are structured according to a perceptual hierarchy. We demonstrate the emergence of this hierarchy by showing that, after training an audio autoencoder in this manner, perceptually salient information is captured in coarser representation structures than with conventional training. Furthermore, we show that such perceptual hierarchies improve latent diffusion decoding in the context of estimating pitch surprisal in music and predicting EEG-brain responses to music listening. In both cases, our results surpass those of previous methods. Pretrained weights are available on github.com/CPJKU/pa-audioic.
Mathias Rose Bjare, Giorgia Cantisani, Marco Pasini +2
Johannes Kepler University, Linz, AT · STMS, CNRS, IRCAM, Sorbonne Universit´e, Paris, FR · Queen Mary University, London, UK +1
During music listening, cortical activity encodes both acoustic and expectation-related information. Prior work has shown that ANN representations resemble cortical representations and can serve as supervisory signals for EEG recognition. Here we show that distinguishing acoustic and expectation-related ANN representations as teacher targets improves EEG-based music identification. Models pretrained to predict either representation outperform non-pretrained baselines, and combining them yields complementary gains that exceed strong seed ensembles formed by varying random initializations. These findings show that teacher representation type shapes downstream performance and that representation learning can be guided by neural encoding. This work points toward advances in predictive music cognition and neural decoding. Our expectation representation, computed directly from raw signals without manual labels, reflects predictive structure beyond onset or pitch, enabling investigation of multilayer predictive encoding across diverse stimuli. Its scalability to large, diverse datasets further suggests potential for developing general-purpose EEG models grounded in cortical encoding principles.
Shogo Noguchi, Taketo Akama, Tai Nakamura +2
Sony Computer Science Laboratories, Inc., Tokyo, Japan
Self-supervised music foundation models underperform on key detection, which requires pitch-sensitive representations. In this work, we present the first systematic study showing that the design of self-supervised pretraining directly impacts pitch sensitivity, and demonstrate that masked contrastive embeddings uniquely enable state-of-the-art (SOTA) performance in key detection in the supervised setting. First, we discover that linear evaluation after masking-based contrastive pretraining on Mel spectrograms leads to competitive performance on music key detection out of the box. This leads us to train shallow but wide multi-layer perceptrons (MLPs) on features extracted from our base model, leading to SOTA performance without the need for sophisticated data augmentation policies. We further analyze robustness and show empirically that the learned representations naturally encode common augmentations. Our study establishes self-supervised pretraining as an effective approach for pitch-sensitive MIR tasks and provides insights for designing and probing music foundation models.
Ori Yonay, Tracy Hammond, Tianbao Yang
Department of Computer Science and Engineering, Texas A&M University