Restricted Boltzmann machines (RBMs) are energy-based models originating from statistical physics. In this study, we investigate how musical structures are represented in the hidden layer of a Bernoulli-Bernoulli RBM trained on binary piano-roll representations of J.~S.~Bach's music. By examining the visible patterns induced by individual hidden units, we find that the hidden layer primarily captures local temporal and pitch-statistical features rather than explicit musical concepts such as melodies, chords, or keys. Free-energy comparisons show that the trained RBM assigns higher probabilities to the examined musical inputs than to digit or noise patterns. Analysis using t-SNE, cosine similarity, and PCA reveals that hidden representations are sensitive to absolute pitch and input overlap, with transposed versions of the same piece not necessarily mapped close together. Generated samples exhibit local pitch organization but limited long-range coherence. These findings illustrate how a simple statistical-physics model captures certain regularities in structured musical data while highlighting its limitations in representing higher-level musical relationships.
Figures & tables
Figure 1: (Color online) Piano roll representation of a musical segment. The horizontal axis represents time (note duration), with a total width of 192 pixels corresponding to two measures in 4/4 time. The vertical axis represents pitch, spanning 72 pixels from C1 (the lowest C on a standard 88-key piano) to B6. Each horizontal bar indicates a note, with its vertical position corresponding to pitch and its horizontal length indicating duration. A quarter note is represented by 24 pixels in width.
Image size
72×192
Number of visible units
13,824
Number of hidden units
2048
Training images
21,444
Training method
CD-1
Training epochs
1,000
Configured batch size
500
Table 1: Model dimensions and training conditions
Figure 2: (Color online) Schematic illustration of the activity-controlled piano-roll generation procedure using the trained RBM. Step (1) is performed only once at initialization, whereas steps (2)–(5) form a repeated loop. The illustrated iteration is t=3 , producing a visible configuration with four active units from one with three active units.
Figure 3: (Color online) Schematic illustration of the continuation procedure for an already generated piano-roll configuration.
Figure 4: Outputs obtained by Gibbs reconstruction from piano-roll inputs using the trained RBM. (a) Piano roll image of a J. S. Bach composition used for training. (a’) Output obtained from (a) by Gibbs reconstruction. (b) Piano roll image of a W. A. Mozart composition not used during training. (b’) Output obtained from (b) by Gibbs reconstruction.
Input
Active pixels
Joint energy
Visible free energy
Piano roll (trained)
564
−3687.31±9.74
−3785.65
Piano roll (untrained)
360
−3370.07±10.94
−3590.73
MNIST digit 0
2581
41.34±2.54
38.72
MNIST digit 5
1436
−332.61±7.12
−368.88
MNIST digit 8
1956
−33.42±2.21
−43.27
Noise
7014
85.60±0.00
85.60
Table 2: Conditional mean joint energy and visible free energy for fixed input images. Joint energies are reported as mean ± sample standard deviation (ddof=1) over ten independent hidden samples per fixed input. Visible free energies are evaluated analytically and have no hidden-sampling uncertainty. Noise denotes one fixed random binary image.
Figure 5: (Color online) Change in visible-layer expectations induced by activating the first hidden unit, relative to the all-inactive hidden configuration. (a) Difference map σ(bi+wi1)−σ(bi) , with positive changes in red, negative changes in blue, and zero in black. The color scale ranges from −0.2 to 0.2 . (b) The same map with vertical lines at sixteenth-note intervals. (c) Binary representation obtained by averaging the differences over each sixteenth-note interval within each pitch row and displaying means of 0.05 or greater in white and all others in black.
Figure 6: (Color online) The hidden layer representations were projected into two dimensions using t-SNE. Original BWV 853 and BWV 865 inputs are shown as red and blue circles, respectively, and their transposed versions as orange and light-blue triangles. (a) Transposition by a semitone. (b) Transposition by a whole tone.
Semitone
Whole tone
Pair
Input
Hidden
Input
Hidden
A–B
0.0459±0.0314
0.2197±0.0258
0.0459±0.0314
0.2197±0.0258
A’–B’
0.0459±0.0314
0.2195±0.0260
0.0459±0.0314
0.2196±0.0261
A–A’
0.0489±0.0396
0.2309±0.0265
0.0861±0.0545
0.2429±0.0339
B–B’
0.0587±0.0360
0.2151±0.0281
0.0969±0.0517
0.2279±0.0354
A–B’
0.1070±0.0516
0.2454±0.0340
0.0665±0.0406
0.2266±0.0280
Table 3: Cosine similarities between input and hidden groups under transposition. A and B denote BWV 853 and BWV 865, respectively; primes denote their transposed versions. Each entry is the mean ± sample standard deviation across 1,849 cross-group window pairs. Hidden vectors are the binary samples used for Fig. 6 .
Figure 7: (Color online) PCA projections of the input piano rolls used in Fig. 6 . Original BWV 853 and BWV 865 inputs are shown as red and blue circles, respectively, and their transposed versions as orange and light-blue triangles. (a) Semitone transposition. (b) Whole-tone transposition. Both panels use the same PCA basis fitted to the full RBM training dataset and the same axis ranges. PC1 and PC2 together explain approximately 3.49% of the training variance.
Figure 8: Progression of the visible configuration over the generation steps of Algorithm 1 . Images (a) through (f) contain t=50,100,250,500,750, and 1000 active visible units, respectively.
Figure 9: A two-measure piano-roll configuration with 496 active visible units, obtained at the energy minimum along the generation trajectory. (a) Piano roll representation of the sample. (b) Sheet music representation of the same sample, with note onsets and offsets quantized to a sixteenth-note grid.
Figure 10: RBM energy as a function of the number of active visible units in the configurations generated by Algorithm 1 . The number of active visible units is controlled by selecting the largest conditional expectations, rather than by a Gibbs draw of the visible state. Labels (a)-(f) correspond to images (a)-(f) shown in Fig. 8 .
Figure 11: An 8-measure piano-roll configuration obtained by iterative continuation sampling from the trained RBM. (a) Piano roll representation of the 8-measure sample. (b) Sheet music representation of the same sample, with note onsets and offsets quantized to a sixteenth-note grid.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Standard block Gibbs sampling initialized with independent random binary pixels. (a) Joint RBM energy as a function of the Gibbs step. (b) Visible configuration after 1,000 Gibbs steps.
Figure 13: Standard block Gibbs sampling initialized with an all-zero image. (a) Joint RBM energy as a function of the Gibbs step. (b) Visible configuration after 1,000 Gibbs steps.
Figure 14: Outputs obtained by Gibbs reconstruction from digit-image inputs using the trained RBM. Panels (a), (b), and (c) show input images from the MNIST dataset. Panels (a’), (b’), and (c’) show the corresponding output images obtained by Gibbs reconstruction. The outputs are noise-like rather than digit-like, consistent with the trained energy model being specific to piano-roll-like visible configurations.
The great success of neural networks primarily arises from the presence of the large number of weight parameters combined with nonlinearities in the input-output relationship of single neurons. In this work, we study the relationship between the statistical properties of the weights and the nonlinearity of the hidden unit in Restricted Boltzmann Machines (RBMs) on the one side, and the distribution induced on binary visible units. We do this for four commonly used activation functions: Linear, Step, ReLU, and Exponential, and make qualitative predictions about the ability of these models to learn distributions with strong higher order interactions over the visible nodes. We show that in general, in an ensemble of RBMs with Gaussian weights, these distributions are rare and hard to learn, except when the hidden unit activation function is an Exponential.
Giovanni di Sarra, Yasser Roudi
Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology, 7491 Trondheim, Norway · Department of Mathematics, King’s College London, London, Strand, WC2R 2LS, UK
How can we understand what a music foundation model has learned internally? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as structured relations rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, we shift from feature identification to structure-based analysis, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.
Liwei Lin, Gus Xia
Computer Science Department, NYU Shanghai · Machine Learning Department, MBZUAI
This study analyzes the intermediate representations of music foundation models (MFMs) and reports the geometric structures used to represent pitch information. By inputting isolated musical notes into trained MFMs and analyzing their principal components, we reveal that the representations form a helical structure reflecting the octave periodicity of pitch. Furthermore, we show that the clarity and geometry of this helical structure vary not only across models but also with the acoustic properties of the input signals. Our analysis provides a novel approach for clarifying the internal mechanisms of MFMs.
Hayato Yagi, Shinnosuke Takamichi, Rin Sato +2
Keio University, Japan · The University of Tokyo, Japan · Waseda University, Japan