cs.SDSep 5, 2025

Encoding of musical structures in hidden units of restricted Boltzmann machines

Authors: Mutsumi Kobayashi, Hiroshi Watanabe

Organizations: Department of Applied Physics and Physico-Informatics, Faculty of Science and Technology, Keio University, Yokohama 223-8522, Japan

Abstract

Restricted Boltzmann machines (RBMs) are energy-based models originating from statistical physics. In this study, we investigate how musical structures are represented in the hidden layer of a Bernoulli-Bernoulli RBM trained on binary piano-roll representations of J.~S.~Bach's music. By examining the visible patterns induced by individual hidden units, we find that the hidden layer primarily captures local temporal and pitch-statistical features rather than explicit musical concepts such as melodies, chords, or keys. Free-energy comparisons show that the trained RBM assigns higher probabilities to the examined musical inputs than to digit or noise patterns. Analysis using t-SNE, cosine similarity, and PCA reveals that hidden representations are sensitive to absolute pitch and input overlap, with transposed versions of the same piece not necessarily mapped close together. Generated samples exhibit local pitch organization but limited long-range coherence. These findings illustrate how a simple statistical-physics model captures certain regularities in structured musical data while highlighting its limitations in representing higher-level musical relationships.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

May 18, 2026

Activation Functions, Statistics and Learning of Higher-Order Interactions in Restricted Boltzmann Machines

The great success of neural networks primarily arises from the presence of the large number of weight parameters combined with nonlinearities in the input-output relationship of single neurons. In this work, we study the relationship between the statistical properties of the weights and the nonlinearity of the hidden unit in Restricted Boltzmann Machines (RBMs) on the one side, and the distribution induced on binary visible units. We do this for four commonly used activation functions: Linear, Step, ReLU, and Exponential, and make qualitative predictions about the ability of these models to learn distributions with strong higher order interactions over the visible nodes. We show that in general, in an ensemble of RBMs with Gaussian weights, these distributions are rare and hard to learn, except when the hidden unit activation function is an Exponential.
Oct 1, 2026

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

How can we understand what a music foundation model has learned internally? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as structured relations rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, we shift from feature identification to structure-based analysis, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.
Jul 31, 2026

Do Music Foundation Models Embed Pitch in Helical Structure?

This study analyzes the intermediate representations of music foundation models (MFMs) and reports the geometric structures used to represent pitch information. By inputting isolated musical notes into trained MFMs and analyzing their principal components, we reveal that the representations form a helical structure reflecting the octave periodicity of pitch. Furthermore, we show that the clarity and geometry of this helical structure vary not only across models but also with the acoustic properties of the input signals. Our analysis provides a novel approach for clarifying the internal mechanisms of MFMs.