During music listening, cortical activity encodes both acoustic and expectation-related information. Prior work has shown that ANN representations resemble cortical representations and can serve as supervisory signals for EEG recognition. Here we show that distinguishing acoustic and expectation-related ANN representations as teacher targets improves EEG-based music identification. Models pretrained to predict either representation outperform non-pretrained baselines, and combining them yields complementary gains that exceed strong seed ensembles formed by varying random initializations. These findings show that teacher representation type shapes downstream performance and that representation learning can be guided by neural encoding. This work points toward advances in predictive music cognition and neural decoding. Our expectation representation, computed directly from raw signals without manual labels, reflects predictive structure beyond onset or pitch, enabling investigation of multilayer predictive encoding across diverse stimuli. Its scalability to large, diverse datasets further suggests potential for developing general-purpose EEG models grounded in cortical encoding principles.
Figures & tables
Figure 1: Conceptual overview of our approach. To extract representations that are encoded in the cortex, we predict corresponding ANN representations that are hypothesized to capture stimulus-related dimensions reflected in cortical activity. As a result, the models capture different aspects of neural information. These representations can complement one another when ensembled, while each model can also individually enhance task-relevant EEG components for song ID classification.
Seed
Accuracy
0
0.832
1
0.809
42
0.827
Mean
0.823
Table 1: Accuracy of the no-pretraining seed baseline across random seeds
Figure 2: Classification performance across distinct neural representations and multi-seed baselines. Comparison of classification accuracy between individual seed baselines (trained from scratch with seeds 0, 1, and 42) and models pretrained with three distinct ANN representation types: acoustic features, surprisal (16 s context), and entropy (16 s context). All three representation-based models consistently outperformed the seed baselines, demonstrating that pretraining with ANN-derived teacher signals—whether acoustic or expectation-related—enhances EEG-based song ID classification. The acoustic model achieved the highest single-model accuracy (0.859), followed by surprisal (0.855) and entropy (0.850), each improving over the baseline mean (0.823). This pattern suggests that while all three representation types capture stimulus-related information encoded in EEG, acoustic features provide the strongest supervisory signal for this task.
Figure 3: Single model performance: acoustic representation pretraining vs. non-pretrained baselines. Bar chart comparing classification accuracy between the acoustic feature prediction model and individual seed baselines. The acoustic model achieved 0.859 accuracy, consistently outperforming all three seed-based models: seed 0 (0.832), seed 1 (0.809), and seed 42 (0.827). Statistical annotations display McNemar’s test results for each comparison, showing p-values, absolute accuracy improvements, and relative improvement percentages. All comparisons achieved p<0.001 , confirming that acoustic pretraining provides statistically robust performance gains independent of random initialization.
Figure 4: Context window optimization for expectation-related features. A Line plot showing classification accuracy for surprisal-based (orange) and entropy-based (green) models across three context window lengths (8 s, 16 s, 32 s). Both expectation-related features achieved peak performance at the 16-s window (surprisal: 0.855; entropy: 0.850), with star markers highlighting this optimal configuration. Accuracy declined at both shorter (8 s) and longer (32 s) windows, suggesting that the 16-s window best captures the temporal scope of expectation processing during naturalistic music listening. B Heatmap displaying McNemar’s test results comparing each expectation-related model configuration (rows) against three seed baselines (columns). Cell colors indicate statistical outcomes: blue denotes that the expectation-related model significantly outperformed the seed ( p<0.05 ), gray indicates no significant difference, and orange indicates seed superiority. Asterisks within cells denote significance levels ( ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 ). Only the 16-s models achieved statistical superiority over all three seeds for both surprisal and entropy.
Figure 5: Effectiveness of ensembling acoustic and expectation-related features. A Bar chart comparing classification accuracy across single models, two-model ensembles, and the three-model ensemble. Accuracy increased systematically with ensemble size: single models ranged from 0.850–0.859, two-model combinations achieved 0.879–0.881, and the three-model ensemble reached 0.887, demonstrating progressive performance gains through representation integration. B Heatmap showing McNemar’s test results comparing ensembles (rows) against constituent single models (columns). Blue cells indicate that the ensemble significantly outperformed the single model ( p<0.05 ), with asterisks denoting significance levels. All ensembles achieved statistical superiority over their constituent single models, confirming synergistic benefits from combining complementary representations. C Heatmap displaying pairwise comparisons between the three-model ensemble (row) and all two-model ensembles (columns). The three-model ensemble significantly outperformed every two-model configuration, demonstrating that each of the three representation types—Acoustic, Surprisal, and Entropy—captures distinct information, as evidenced by their complementary contributions when integrated together. Statistical notation: ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 (McNemar’s test).
Figure 6: Evaluation of performance gains through multi-seed deep ensembling. A Bar chart showing classification accuracy for individual seed models, 2-model seed ensembles, and the 3-model seed ensemble. Accuracy increased progressively from single seeds (0.809–0.832) to 2-model ensembles (0.862–0.866) to the 3-model ensemble (0.878). B Heatmaps displaying McNemar’s test results comparing 2-model ensembles (top) and 3-model ensemble (bottom) against individual seed baselines. Blue cells indicate ensemble superiority ( p<0.05 ), with asterisks denoting significance levels. All ensembles significantly outperformed individual seeds, confirming robust gains from initialization diversity. C Bar chart comparing 2-model and 3-model seed ensembles. D Heatmap showing that the 3-model ensemble significantly outperformed all 2-model configurations, demonstrating that increasing ensemble size consistently enhances performance. Statistical notation: ∗∗∗p<0.001 (McNemar’s test).
Figure 7: Superiority of ensembling distinct representations over multi-seed ensembles. A Bar chart comparing 2-model seed ensembles and 2-model representation-based ensembles. All representation-based combinations achieved higher accuracy (0.879–0.881) than seed-based ensembles (0.862–0.866). B Heatmap showing McNemar’s test results. Blue cells indicate representation-based superiority ( p<0.05 ), with asterisks denoting significance levels. All representation-based ensembles significantly outperformed all seed-based ensembles, demonstrating consistent advantages of representation diversity. C Bar chart comparing the 3-model seed ensemble (0.878) against the 3-model representation ensemble (0.887). D Heatmap confirming that the representation-based ensemble significantly outperformed the seed ensemble. The representation-based approach demonstrates that neurobiologically motivated diversity is more effective than initialization diversity alone for EEG-based song ID classification. Statistical notation: ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 (McNemar’s test).
Figure 8: Architecture of PredANN++. An EEG encoder F maps a 3-s EEG segment to a latent representation using temporal patch embedding and a Transformer. During pretraining, a decoder G predicts masked tokens of discretized music-derived teacher representations (Acoustic, Surprisal, or Entropy), while F is jointly regularized by an auxiliary Song ID classification loss. After pretraining, G is discarded and F is fine-tuned for EEG-based Song ID classification.
Figure 9: ANN-representation calculation and discretization from audio. Acoustic representations are extracted using a masked language model (MLM-SSL; MuQ), while expectation-related representations are computed using an autoregressive model (AR-SSL; MusicGen) as surprisal and entropy. The resulting continuous (raw) representations are used as inputs to the decoder G during masked pretraining, whereas their discretized counterparts (MuQ: k -means; Surprisal/Entropy: quantile binning) serve as discrete teacher targets for prediction. All representations are temporally aligned to 3-s EEG segments.
Figure 10: Performance of the conservative chunk-based computation. Panels A–D compare Surprisal and Entropy computed independently within 30-s audio chunks (see Methods, “Conservative chunk-based computation”) with those obtained using standard context-length settings and seed baselines. A Classification accuracy using Surprisal features. Grey bars denote the three standard computations (context lengths: 8 s, 16 s, 32 s); the orange bar indicates the conservative 30-s chunk-based computation. Chunk-based Surprisal yields accuracy comparable to the best-performing 16 s context. B Classification accuracy using Entropy features. Grey bars denote the three standard computations (context lengths: 8 s, 16 s, 32 s); the green bar indicates the conservative 30-s chunk-based computation. In contrast to Surprisal (panel A), chunk-based Entropy does not reach the performance of the 8 s or 16 s contexts, although it remains above the 32 s condition. C Comparison of three no-pretraining seed baselines (Seed 0/1/42, grey) with conservative Surprisal (orange) and Entropy (green). Surprisal exceeds all seed baselines, whereas Entropy surpasses Seed 1 but does not exceed Seed 0 or Seed 42. D McNemar’s test heatmap comparing each seed column with each conservative feature row. Blue cells indicate feature superiority, light grey indicates no significant difference; asterisks indicate significance levels (** p<0.01 , *** p<0.001 ). Conservative Surprisal shows significant improvements over all seed baselines, whereas conservative Entropy reaches significance for Seed 1 only, with no significant differences observed for Seed 0 or Seed 42.
Figure 11: Validation accuracy across training epochs for MuQ, Surprisal, and Entropy representations when used directly for Song ID classification. Each panel shows validation accuracy (%) as a function of training epoch for models that directly use discretized music-feature sequences as input, without EEG signals. For each 3-second segment, the input consists solely of discretized feature tokens: MuQ (75 steps at 25 Hz), Surprisal (150 steps at 50 Hz), or Entropy (150 steps at 50 Hz). Each scalar discrete value (0–127) is projected to a 512-dimensional embedding through a linear layer (1 → 512). A learnable [CLS] token is prepended, and learnable positional embeddings are added. The sequence is processed by a 2-layer Transformer encoder (embedding dimension 512, 8 attention heads, MLP ratio 4.0, GELU activation, dropout 0.1), identical to the Transformer architecture used in the main EEG recognition experiments. The hidden representation of the [CLS] token is passed through a classification head composed of a linear layer (512 → 256), batch normalization, ReLU activation, and a final linear layer (256 → 10) to predict Song ID. The batch size is fixed to B=48 . Optimization is performed using Adam. The learning rate is 0.003 for MuQ-based modeling and 0.0003 for Surprisal- and Entropy-based modeling, consistent with the learning-rate settings used in the main experiments. All curves represent exponential moving averages of validation accuracy. Under this modeling setup, all three representations independently achieve validation accuracies approaching 100%, demonstrating that each representation alone contains sufficient information to almost perfectly solve the 10-class Song ID task.
Figure 12: Example multi-panel visualization for Song 21 (30–60 s), Subject 2. Panels show (top to bottom) scalp EEG from Subject 2 recorded with the EGI 128-channel HydroCel Geodesic Sensor Net (all channels stacked), the normalized short-time RMS envelope, the mel-spectrogram in decibels (128 mel bins; HTK mel scale), and MusicGen-k1-derived surprisal and entropy trajectories discretized into Q128 bins for context windows of 8, 16, and 32 seconds (see Supplementary Note 3: Qualitative inspection of context-dependent surprisal/entropy time courses for qualitative inspection).
Understanding why listeners enjoy music is difficult because the experience is subjective and changes as a piece unfolds. Electroencephalography (EEG) offers a non-invasive way to study neural responses during listening, yet earlier work has mostly treated music emotion as a categorical label or relied on black-box models. We formulate enjoyment prediction as a regression problem on the public MUSIN-G dataset, which contains 128-channel EEG from 20 listeners together with song-level enjoyment ratings. Six deep learning models were compared under a participant-wise protocol, in which 15 listeners were used for training and validation and 5 were held out for testing. DeepConvNet, CNN-LSTM and GNN-CNN reached the lowest mean absolute error (0.97 to 0.99 on the 1 to 5 scale), while the temporal convolutional network (TCN) had the highest Pearson correlation with the ratings (r = 0.21). Gradient-based and occlusion-based explanations, averaged over all held-out windows, identify the time segments within each two-second window that most influence the TCN predictions. The study provides a reproducible cross-listener benchmark for EEG-based enjoyment regression and shows how explainability analysis complements standard metrics.
Parshav Singla, Simranjit Kaur, Maanya Walia +1
1*Department of Computer Science and Engineering, Thapar Institute of Engineering and Technology, Patiala, Punjab, 147004, India.
We argue that training autoencoders to reconstruct inputs from noised versions of their encodings, when combined with perceptually motivated losses, yields encodings that are structured according to a perceptual hierarchy. We demonstrate the emergence of this hierarchy by showing that, after training an audio autoencoder in this manner, perceptually salient information is captured in coarser representation structures than with conventional training. Furthermore, we show that such perceptual hierarchies improve latent diffusion decoding in the context of estimating pitch surprisal in music and predicting EEG-brain responses to music listening. In both cases, our results surpass those of previous methods. Pretrained weights are available on github.com/CPJKU/pa-audioic.
Mathias Rose Bjare, Giorgia Cantisani, Marco Pasini +2
Johannes Kepler University, Linz, AT · STMS, CNRS, IRCAM, Sorbonne Universit´e, Paris, FR · Queen Mary University, London, UK +1
Pitch is a fundamental aspect of auditory perception. Pitch perception is commonly described across two perceptual dimensions: pitch height is the sense that tones with varying frequencies seem to be higher or lower, and chroma equivalence is the cyclical similarity of notes octaves, corresponding to a doubling of fundamental frequency. Existing research is divided on whether chroma equivalence is a learned percept that varies according to musical experience and culture, or is an innate percept that develops automatically. Building on a recent framework that proposes to use ANNs to ask 'why' questions about the brain, we evaluated recent auditory ANNs using representational similarity analysis to test the emergence of pitch height and chroma equivalence in their learned representations. Additionally, we fine-tuned two models, Wav2Vec 2.0 and Data2Vec, on a self-supervised learning task using speech and music, and a supervised music transcription task. We found that all models exhibited varying degrees of pitch height representation, but that only models trained on the supervised music transcription task exhibited chroma equivalence. Mere exposure to music through self-supervised learning was not sufficient for chroma equivalence to emerge. This supports the view that chroma equivalence is a higher-order cognitive computation that emerges to support the specific task of music perception, distinct from other auditory perception such as speech listening. This work also highlights the usefulness of ANNs for probing the developmental conditions that give rise to perceptual representations in humans.
Lukas Grasse, Matthew S. Tata
Canadian Centre for Behavioural Neuroscience, Department of Neuroscience, University of Lethbridge, AB, Canada, T1K 3Y4