cs.AIMar 3, 2026
SaveExpectation and Acoustic Neural Network Representations Enhance Music Identification from Brain Activity
Organizations: Sony Computer Science Laboratories, Inc., Tokyo, Japan
Abstract
During music listening, cortical activity encodes both acoustic and expectation-related information. Prior work has shown that ANN representations resemble cortical representations and can serve as supervisory signals for EEG recognition. Here we show that distinguishing acoustic and expectation-related ANN representations as teacher targets improves EEG-based music identification. Models pretrained to predict either representation outperform non-pretrained baselines, and combining them yields complementary gains that exceed strong seed ensembles formed by varying random initializations. These findings show that teacher representation type shapes downstream performance and that representation learning can be guided by neural encoding. This work points toward advances in predictive music cognition and neural decoding. Our expectation representation, computed directly from raw signals without manual labels, reflects predictive structure beyond onset or pitch, enabling investigation of multilayer predictive encoding across diverse stimuli. Its scalability to large, diverse datasets further suggests potential for developing general-purpose EEG models grounded in cortical encoding principles.
Figures & tables
Figure 1: Conceptual overview of our approach. To extract representations that are encoded in the cortex, we predict corresponding ANN representations that are hypothesized to capture stimulus-related dimensions reflected in cortical activity. As a result, the models capture different aspects of neural information. These representations can complement one another when ensembled, while each model can also individually enhance task-relevant EEG components for song ID classification.
| Seed | Accuracy |
|---|---|
| 0 | 0.832 |
| 1 | 0.809 |
| 42 | 0.827 |
| Mean | 0.823 |
Table 1: Accuracy of the no-pretraining seed baseline across random seeds
Figure 2: Classification performance across distinct neural representations and multi-seed baselines. Comparison of classification accuracy between individual seed baselines (trained from scratch with seeds 0, 1, and 42) and models pretrained with three distinct ANN representation types: acoustic features, surprisal (16 s context), and entropy (16 s context). All three representation-based models consistently outperformed the seed baselines, demonstrating that pretraining with ANN-derived teacher signals—whether acoustic or expectation-related—enhances EEG-based song ID classification. The acoustic model achieved the highest single-model accuracy (0.859), followed by surprisal (0.855) and entropy (0.850), each improving over the baseline mean (0.823). This pattern suggests that while all three representation types capture stimulus-related information encoded in EEG, acoustic features provide the strongest supervisory signal for this task.
Figure 3: Single model performance: acoustic representation pretraining vs. non-pretrained baselines. Bar chart comparing classification accuracy between the acoustic feature prediction model and individual seed baselines. The acoustic model achieved 0.859 accuracy, consistently outperforming all three seed-based models: seed 0 (0.832), seed 1 (0.809), and seed 42 (0.827). Statistical annotations display McNemar’s test results for each comparison, showing p-values, absolute accuracy improvements, and relative improvement percentages. All comparisons achieved , confirming that acoustic pretraining provides statistically robust performance gains independent of random initialization.
Figure 4: Context window optimization for expectation-related features. A Line plot showing classification accuracy for surprisal-based (orange) and entropy-based (green) models across three context window lengths (8 s, 16 s, 32 s). Both expectation-related features achieved peak performance at the 16-s window (surprisal: 0.855; entropy: 0.850), with star markers highlighting this optimal configuration. Accuracy declined at both shorter (8 s) and longer (32 s) windows, suggesting that the 16-s window best captures the temporal scope of expectation processing during naturalistic music listening. B Heatmap displaying McNemar’s test results comparing each expectation-related model configuration (rows) against three seed baselines (columns). Cell colors indicate statistical outcomes: blue denotes that the expectation-related model significantly outperformed the seed ( ), gray indicates no significant difference, and orange indicates seed superiority. Asterisks within cells denote significance levels ( , , ). Only the 16-s models achieved statistical superiority over all three seeds for both surprisal and entropy.
Figure 5: Effectiveness of ensembling acoustic and expectation-related features. A Bar chart comparing classification accuracy across single models, two-model ensembles, and the three-model ensemble. Accuracy increased systematically with ensemble size: single models ranged from 0.850–0.859, two-model combinations achieved 0.879–0.881, and the three-model ensemble reached 0.887, demonstrating progressive performance gains through representation integration. B Heatmap showing McNemar’s test results comparing ensembles (rows) against constituent single models (columns). Blue cells indicate that the ensemble significantly outperformed the single model ( ), with asterisks denoting significance levels. All ensembles achieved statistical superiority over their constituent single models, confirming synergistic benefits from combining complementary representations. C Heatmap displaying pairwise comparisons between the three-model ensemble (row) and all two-model ensembles (columns). The three-model ensemble significantly outperformed every two-model configuration, demonstrating that each of the three representation types—Acoustic, Surprisal, and Entropy—captures distinct information, as evidenced by their complementary contributions when integrated together. Statistical notation: , , (McNemar’s test).
Figure 6: Evaluation of performance gains through multi-seed deep ensembling. A Bar chart showing classification accuracy for individual seed models, 2-model seed ensembles, and the 3-model seed ensemble. Accuracy increased progressively from single seeds (0.809–0.832) to 2-model ensembles (0.862–0.866) to the 3-model ensemble (0.878). B Heatmaps displaying McNemar’s test results comparing 2-model ensembles (top) and 3-model ensemble (bottom) against individual seed baselines. Blue cells indicate ensemble superiority ( ), with asterisks denoting significance levels. All ensembles significantly outperformed individual seeds, confirming robust gains from initialization diversity. C Bar chart comparing 2-model and 3-model seed ensembles. D Heatmap showing that the 3-model ensemble significantly outperformed all 2-model configurations, demonstrating that increasing ensemble size consistently enhances performance. Statistical notation: (McNemar’s test).
Figure 7: Superiority of ensembling distinct representations over multi-seed ensembles. A Bar chart comparing 2-model seed ensembles and 2-model representation-based ensembles. All representation-based combinations achieved higher accuracy (0.879–0.881) than seed-based ensembles (0.862–0.866). B Heatmap showing McNemar’s test results. Blue cells indicate representation-based superiority ( ), with asterisks denoting significance levels. All representation-based ensembles significantly outperformed all seed-based ensembles, demonstrating consistent advantages of representation diversity. C Bar chart comparing the 3-model seed ensemble (0.878) against the 3-model representation ensemble (0.887). D Heatmap confirming that the representation-based ensemble significantly outperformed the seed ensemble. The representation-based approach demonstrates that neurobiologically motivated diversity is more effective than initialization diversity alone for EEG-based song ID classification. Statistical notation: , , (McNemar’s test).
Figure 8: Architecture of PredANN++. An EEG encoder maps a 3-s EEG segment to a latent representation using temporal patch embedding and a Transformer. During pretraining, a decoder predicts masked tokens of discretized music-derived teacher representations (Acoustic, Surprisal, or Entropy), while is jointly regularized by an auxiliary Song ID classification loss. After pretraining, is discarded and is fine-tuned for EEG-based Song ID classification.
Figure 9: ANN-representation calculation and discretization from audio. Acoustic representations are extracted using a masked language model (MLM-SSL; MuQ), while expectation-related representations are computed using an autoregressive model (AR-SSL; MusicGen) as surprisal and entropy. The resulting continuous (raw) representations are used as inputs to the decoder during masked pretraining, whereas their discretized counterparts (MuQ: -means; Surprisal/Entropy: quantile binning) serve as discrete teacher targets for prediction. All representations are temporally aligned to 3-s EEG segments.
Figure 10: Performance of the conservative chunk-based computation. Panels A–D compare Surprisal and Entropy computed independently within 30-s audio chunks (see Methods, “Conservative chunk-based computation”) with those obtained using standard context-length settings and seed baselines. A Classification accuracy using Surprisal features. Grey bars denote the three standard computations (context lengths: 8 s, 16 s, 32 s); the orange bar indicates the conservative 30-s chunk-based computation. Chunk-based Surprisal yields accuracy comparable to the best-performing 16 s context. B Classification accuracy using Entropy features. Grey bars denote the three standard computations (context lengths: 8 s, 16 s, 32 s); the green bar indicates the conservative 30-s chunk-based computation. In contrast to Surprisal (panel A), chunk-based Entropy does not reach the performance of the 8 s or 16 s contexts, although it remains above the 32 s condition. C Comparison of three no-pretraining seed baselines (Seed 0/1/42, grey) with conservative Surprisal (orange) and Entropy (green). Surprisal exceeds all seed baselines, whereas Entropy surpasses Seed 1 but does not exceed Seed 0 or Seed 42. D McNemar’s test heatmap comparing each seed column with each conservative feature row. Blue cells indicate feature superiority, light grey indicates no significant difference; asterisks indicate significance levels (** , *** ). Conservative Surprisal shows significant improvements over all seed baselines, whereas conservative Entropy reaches significance for Seed 1 only, with no significant differences observed for Seed 0 or Seed 42.
Figure 11: Validation accuracy across training epochs for MuQ, Surprisal, and Entropy representations when used directly for Song ID classification. Each panel shows validation accuracy (%) as a function of training epoch for models that directly use discretized music-feature sequences as input, without EEG signals. For each 3-second segment, the input consists solely of discretized feature tokens: MuQ (75 steps at 25 Hz), Surprisal (150 steps at 50 Hz), or Entropy (150 steps at 50 Hz). Each scalar discrete value (0–127) is projected to a 512-dimensional embedding through a linear layer (1 512). A learnable [CLS] token is prepended, and learnable positional embeddings are added. The sequence is processed by a 2-layer Transformer encoder (embedding dimension 512, 8 attention heads, MLP ratio 4.0, GELU activation, dropout 0.1), identical to the Transformer architecture used in the main EEG recognition experiments. The hidden representation of the [CLS] token is passed through a classification head composed of a linear layer (512 256), batch normalization, ReLU activation, and a final linear layer (256 10) to predict Song ID. The batch size is fixed to . Optimization is performed using Adam. The learning rate is 0.003 for MuQ-based modeling and 0.0003 for Surprisal- and Entropy-based modeling, consistent with the learning-rate settings used in the main experiments. All curves represent exponential moving averages of validation accuracy. Under this modeling setup, all three representations independently achieve validation accuracies approaching 100%, demonstrating that each representation alone contains sufficient information to almost perfectly solve the 10-class Song ID task.
Figure 12: Example multi-panel visualization for Song 21 (30–60 s), Subject 2. Panels show (top to bottom) scalp EEG from Subject 2 recorded with the EGI 128-channel HydroCel Geodesic Sensor Net (all channels stacked), the normalized short-time RMS envelope, the mel-spectrogram in decibels (128 mel bins; HTK mel scale), and MusicGen-k1-derived surprisal and entropy trajectories discretized into Q128 bins for context windows of 8, 16, and 32 seconds (see Supplementary Note 3: Qualitative inspection of context-dependent surprisal/entropy time courses for qualitative inspection).
More in this topic
EEG Decoding
Electroencephalography (EEG) decoding maps brain-signal patterns to mental states, stimuli, speech, or intended actions.
See all 205 papers