Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.
Figures & tables
Figure 1: Local Intervention. A V → I cadence is identified in a Bach chorale (left). The chorale is truncated at the end of the V chord, and the chordal third of V is lowered by a semitone, turning V into v (right), yielding an original-counterfactual pair that differs in exactly one note.
Figure 2: Intervention on a single pitch shifts cadence resolution. Modifying a single pitch in a V → I or V → i cadence substantially redistributes the model’s resolution distribution in both major and minor source chorales.
Figure 3: Causal Effect of Harmonic Information. Patching residual stream representations from minorized into the original sequences at each layer causes an increase in IIA beginning in the middle layers, indicating that harmonic computation develops progressively through the network. Shaded regions denote 95% confidence intervals.
Figure 4: Linear probe accuracy across layers. Mode (chance = 50%) becomes decodable in the middle layers (from layer 4), while key identity (chance = 5.6%) emerges later in the network (from layer 10). Note that classes might be imbalanced.
Figure 5: Key and Mode Steering. Applying steering on major chorales with steering directions corresponding to relative and parallel minor keys substantially increases the frequency of alternative resolutions compared to the unsteered baseline, which usually resolves to the major tonic (I). Relative minor steering shifts mass toward the submediant (vi), as it is the tonic of the relative minor, while parallel minor steering produces the most decisive shift toward the parallel minor tonic (i). Mode steering also reduces I resolutions but produces no structured redistribution, without concentrating on a specific harmonic target. Numbers above bars are counts per category. org. & rel. diatonic : the chords diatonic to the original key and its relative minor; par. diatonic : chords of the parallel minor only; other : all remaining resolutions.
Sequence-Level
Note-Level
Quality
Harmony
Struct.
Pitch
Avg. gen.
Out of
Note dur.
Pitch
FMD
RN
Top-1
error
repeat
duration
range
avg
var.
entropy
dom.
(%) ↓
(%) ↓
(ms)
(%) ↓
(ms)
↓
(bits) ↑
(%) ↓
Aria (no steering)
0.0
0.0
3740
0.0
735
60
80.27
1.26
59.0
Relative minor steering
2.9
0.0
3800
0.0
728
55
85.42
1.40
55.3
Parallel minor steering
0.0
0.0
3783
0.0
741
55
106.65
1.50
49.6
Table 1: Key Steering Evaluation. Results confirm that generation quality is well-preserved across key steering conditions: structural errors remain below 3% and out-of-range notes near 0%, FMD stays close to the unsteered baseline (FMD calculated against original Bach chorales; for reference, the FMD between Bach chorales and Bach fugues is 195.97, far larger than any steering-induced change), and RN entropy varies by at most 0.24 bits while top-1 dominance remains near 50% — showing the steered model generates harmonically varied continuations rather than collapsing onto a single chord. For ↓ lower is better, for ↑ higher is better, when no indication similarity to the baseline is preferred.
Figure 6: Relative vs. Parallel probing. Grouping keys by their shared key signature (relative-grouped) yields substantially higher peak accuracy (83.4%) than the baseline 18-class key probe (65.5%) or grouping by parallel keys (65.3%). The probe thus separates key classes more easily when each class is defined by a shared pitch collection (relative grouping) than by a shared root (parallel grouping).
Figure 7: Relative keys are more confusable than parallel keys. Confusion matrices at probe peak accuracy layer. (a) The full probe shows systematic confusions between relative key pairs (e.g., F major with D minor, Bb major with G minor), indicating that the model might place relative keys close together in representation space. (b) Grouping by shared key signature (relative-grouped) resolves these confusions, yielding a near-diagonal matrix. (c) Grouping by shared tonal center (parallel-grouped) retains more off-diagonal mass, confirming that parallel keys are more distinct.
Transformer-based architectures have significantly advanced the generation of complex symbolic sequences, yet a significant gap remains in achieving fine-grained, interpretable control over discrete signal attributes. This paper investigates the mechanistic interpretability of the Multitrack Music Transformer (MMT) and proposes a framework for deterministic attribute modulation without retraining to bridge this gap via inference-time activation steering. Utilizing the Difference-in-Means (DiffMean) methodology, we isolate latent directions for signal attributes, specifically Pitch and Duration, within the residual stream. We validate the Linear Representation Hypothesis in this domain, achieving high correlation between steering magnitude and attribute shift. To address the inherent feature entanglement in multi-attribute steering, we introduce a Dual Steering framework utilizing Gram-Schmidt Orthogonalization. Experimental results demonstrate that this geometric decoupling reduces conceptual interference and signal degradation compared to naive vector addition, enabling independent deterministic control even against strong autoregressive conditioning.
Ioannis Prokopiou, Pantelis Vikatos, Maximos Kaliakatsos-Papakostas +2
Athens University of Economics and Business Orfium Athens, Greece · Innovation Lab Orfium Athens, Greece · Department of Music Technology and Acoustics Hellenic Mediterranean University Rethymno, Greece +2
Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.
Zixun Guo, Simon Dixon
Centre for Digital Music, Queen Mary University of London, United Kingdom
Deep learning has transformed symbolic music generation by borrowing the training paradigms of large language models, with systems such as NotaGen now producing complete, stylistically convincing classical scores from a short prompt. These systems could become powerful creative partners, helping musicians generate endless possibilities. However, current systems expose almost no control handles on the music itself. In principle, control handles could be built into a foundation model trained from scratch, but this is rarely practical without massive amounts of quality annotated data and compute resources. We therefore present MotiGen, a recipe for retrofitting pretrained symbolic music models to use new instruction prompts. MotiGen injects a musical motif as a structured prompt line, reinforces it with a scalar attention bias toward the motif tokens, and learns the association with a two-phase curriculum. First, it learns from focused excerpts cropped around motif occurrences in the training data, then full scores including the motifs. Our experiments show that our model composes with the prompted motif in over 92.3% of generated pieces. Generated pieces using a variety of motifs are included in our sample site: https://motigen-site.github.io/.
Chao Peter Yang, Cynthia Rudin, Yue Jiang +2
Yale University New Haven, CT · Duke University Durham, NC · Chinese University of Hong Kong, Shenzhen Shenzhen, Guangdong