We investigate how predictive performance depends on the maximum context available to an autoregressive model of symbolic music, and what information long-context models exploit. A small causal Transformer is trained separately at each context length T in {6, 18, 48, 96, 192, 336} and evaluated on the same target positions, both over raw tokens and over non-overlapping binary latent codes. On Nottingham folk tunes, the token predictor's test NLL decreases by 71% (0.986 bits per token) between 6 and 336 tokens, and by 71% on the O'Neill's tunes long enough for the same sweep (129 of 302 test tunes). Most of the long-context gain is explained by exact copying: the improvement appears when an earlier occurrence of the target's 16-token history enters the available context; overwriting that occurrence removes the gain, whereas equally large unrelated corruption does not; and a simple copy baseline recovers 94% of the reduction. A trained 336-token model likewise loses most of this gain when its history is restricted to recent tokens at test time. Because many repeats arise from written repeat signs expanded in the score, this result pertains specifically to these rendered score representations. By contrast, on MAESTRO performances and MusicNet scores, where exact repeats are substantially less frequent, the reduction is smaller (9% and 18%) and is largely attained by 96 tokens. Finally, an audit of an earlier draft that reported saturation at 16 tokens identified split leakage, overlapping latent receptive fields, an averaging predictor, and a per-file tempo grid; we present these as methodological checks for context-length measurements.
Figures & tables
Pieces
Tokens
Windows
Test blocks
Train
Val
Test
Train
Val
Test
train / val / test
in train
Nottingham
729
150
154
382K
90K
84K
2789 / 809 / 660
0.0%
O’Neill’s
1397
295
302
738K
115K
158K
7121 / 687 / 1505
0.6%
MusicNet
221
54
48
577K
141K
116K
1091 / 267 / 218
0.0%
MAESTRO
962
137
177
4586K
560K
575K
9072 / 1094 / 1113
0.2%
TABLE I: The four corpora after tokenization. Windows are the training and evaluation windows of the context sweep (Section V ). The last column is the share of non-constant 48-token test blocks that occur word for word in the training split.
Nottingham
O’Neill’s
MusicNet
MAESTRO
Context
Latent
Token
Latent
Token
Token
Latent
Token
(tokens)
(bits per bit)
(bits per token)
(bits per bit)
(bits per token)
(bits per token)
(bits per bit)
(bits per token)
0
0.958
4.366
0.931
4.383
5.295
0.978
5.319
6
0.857 ± 0.002
1.391 ± 0.015
0.845 ± 0.005
1.692 ± 0.014
2.512 ± 0.066
0.887 ± 0.0004
3.283 ± 0.002
18
0.842 ± 0.002
1.440 ± 0.021
0.818 ± 0.003
1.543 ± 0.019
2.366 ± 0.003
0.881 ± 0.00003
3.177 ± 0.002
48
0.834 ± 0.005
1.529 ± 0.090
0.771 ± 0.005
1.449 ± 0.017
2.159 ± 0.004
0.879 ± 0.0004
3.097 ± 0.006
TABLE II: Test NLL against context length, mean and SD over three seeds. Context 0 is the unigram distribution (tokens) and the per-bit marginal (latents), each from one run. Lower is better. Nottingham and O’Neill’s use transposition of the training split. MAESTRO tokens: 89-token data. MusicNet has token predictors only. Section VI-G reports ten more token runs at 48, 96 and 192 tokens with a fixed training budget.
Fig. 1: Test NLL relative to its value at 6 tokens of context for (a) Nottingham, (b) O’Neill’s, (c) MusicNet and (d) MAESTRO. Lines are the mean over three seeds with the seed SD as error bars; the small hollow marks beside them are the individual seeds. The label at the end of each curve is its drop from 6 to 336 tokens. The context axis is logarithmic, and the dotted line marks 96 tokens.
Repeat lag (tokens back)
Share of targets
1-32
33-80
81-176
177-320
None
Held
in view from context
48
96
192
336
Nottingham, target tokens
1.8%
14.7%
36.6%
17.9%
28.3%
0.7%
Nottingham, target latent steps
1.9%
15.4%
36.4%
17.7%
27.5%
1.2%
O’Neill’s, target tokens
0.3%
9.6%
33.3%
30.1%
26.7%
0.0%
O’Neill’s, target latent steps
0.4%
11.1%
33.8%
29.1%
25.6%
0.0%
TABLE III: Repeat lag of the test targets: the share of targets whose 16-token history last occurred at each lag, or not within 320 tokens (none), or is one held pitch (held). A repeat is in view for context T when lag + 16 ≤T , and each bin comes into view at the context shown. Latent steps use the history before the step’s first token. The last rows describe the Nottingham and O’Neill’s repeats: how often the next token, or the next 6 tokens, equal those that followed the earlier occurrence, and how often the earlier occurrence starts on the latent step grid, where the same 6 tokens have the same code.
Fig. 2: Test NLL on Nottingham by repeat lag (columns) and context length (rows), mean over three seeds, for (a) the target tokens of the token predictor and (b) the target latent steps of the latent predictor, each on its own colour scale. Darker cells have lower NLL. Left of the black line the repeat is in view (lag + 16 ≤ context); right of it, it is not. The bins are cut at these limits.
Fig. 3: Share of target tokens whose repeat is in view (lag + 16 ≤ context, thin dotted line), and share of the NLL reduction from 6 to 336 tokens reached at each context (thick lines), for (a) Nottingham, (b) O’Neill’s, (c) MusicNet and (d) MAESTRO. The label at 336 tokens is the repeat share there; the reduction reached is 100% at 336 tokens by definition. A negative share means the NLL is above its value at 6 tokens.
Corpus, predictor, context
Targets
Clean
Erased
Control
Swapped token
Nottingham, token, 192
16,693
0.22 ± 0.005
1.10 ± 0.02
0.22 ± 0.004
0.5% → 55%
Nottingham, token, 336
22,454
0.19 ± 0.003
1.06 ± 0.03
0.19 ± 0.003
0.3% → 62%
Nottingham, latent, 192
2,783
0.37 ± 0.03
0.87 ± 0.01
0.38 ± 0.03
n/a
Nottingham, latent, 336
3,747
0.35 ± 0.01
0.81 ± 0.004
0.35 ± 0.01
n/a
O’Neill’s, token, 192
31,068
0.37 ± 0.003
1.28 ± 0.01
0.38 ± 0.002
0.8% → 51%
O’Neill’s, token, 336
52,864
0.22 ± 0.01
1.24 ± 0.03
0.22 ± 0.01
0.4% → 71%
TABLE IV: Erase test on Nottingham and O’Neill’s targets with a repeat in view, mean and SD over three seeds. Clean is the test NLL of these targets; erased overwrites every earlier occurrence of the 16-token history in the context, with what followed it; control overwrites as many spans of the same length elsewhere in the context. Latent values in bits per bit, token values in bits per token. The last column is the probability the token predictor gives to a new token placed after every earlier occurrence, before and after the swap.
Fig. 4: The erase test of Table IV as bars: test NLL of the targets with a repeat in view when the context is left as it is (clean), when as many unrelated spans are overwritten (control), and when every earlier occurrence of the target’s 16-token history is overwritten (erased), for (a) the token predictor and (b) the latent predictor at 192 and 336 tokens. Error bars are the seed SD.
Fig. 5: One trained model with less history. Test NLL of the 336-token token predictors when their history is limited at test time to the most recent tokens, by giving the model only those tokens (truncated) or by replacing every older token with a window of another piece (replaced), beside the models trained separately at each context length. Mean over three seeds, on the targets of the sweep (every fourth target for MAESTRO); each panel has its own scale.
Setup
6
96
192
336
Legacy grid, 480-token windows, D=32 , latent
0.80 ± 0.0002
0.80 ± 0.0005
0.78 ± 0.001
0.72 ± 0.01
Legacy grid, 480-token windows, D=32 , token
1.22 ± 0.001
1.20 ± 0.004
1.04 ± 0.01
0.84 ± 0.02
Score grid, 384-token windows, D=64 , latent
0.91 ± 0.001
0.91 ± 0.003
0.77 ± 0.004
0.63 ± 0.01
Score grid, 384-token windows, D=64 , token
1.39 ± 0.004
1.09 ± 0.02
1.15 ± 0.02
0.81 ± 0.25
Score grid, as above, with transposition, latent
0.86 ± 0.002
0.78 ± 0.004
0.57 ± 0.01
0.47 ± 0.01
Score grid, as above, with transposition, token
1.39 ± 0.01
1.27 ± 0.33
0.57 ± 0.01
0.41 ± 0.01
TABLE V: Nottingham under three setups, test NLL at four context lengths (tokens), mean and SD over three seeds. Latent values in bits per bit, token values in bits per token. The setups differ in tokenizer, window, autoencoder and transposition at once, so values are comparable within a row.
Fig. 6: Seed spread of the Nottingham token predictor. (a) Validation NLL against epoch for ten runs at 96 tokens trained for 200 epochs with no early stopping; coloured lines drop (Section VI-G ), with a circle at the epoch of the drop, grey dashed lines do not, and a tick marks where the sweep’s early stopping would have ended each run. (b) Test NLL of every run: the three early-stopped runs of the sweep (circles) and the ten fixed-budget runs (dots). The dotted line is the 6-token predictor.
Nottingham
MAESTRO
Context
Causal
v5
Causal
v5
6
0.857 ± 0.002
0.856 ± 0.0001
0.887 ± 0.0004
0.887 ± 0.0004
18
0.842 ± 0.002
0.852 ± 0.001
0.881 ± 0.00003
0.881 ± 0.0002
48
0.834 ± 0.005
0.843 ± 0.004
0.879 ± 0.0004
0.879 ± 0.0004
96
0.775 ± 0.004
0.891 ± 0.001
0.877 ± 0.0004
0.879 ± 0.0003
192
0.571 ± 0.015
0.886 ± 0.001
0.877 ± 0.00010
0.878 ± 0.0003
TABLE VI: The causal latent predictor against v5, the averaging predictor of our earlier draft, on the same latents. Test NLL in bits per bit, mean over three seeds.
Tuned temperature
τ=1
System
τ
Distance
Repetition
Distance
Real test tunes, 100 per set (5th to 95th pct.)
n/a
0.03 to 0.17
0.48 to 0.52
n/a
AE round trip, all test streams
n/a
0.07
0.49
n/a
Latent, 6 tokens
0.2, 0.3
1.21 ± 0.19
0.01 ± 0.02
5.97 ± 0.03
Latent, 18 tokens
0.3, 0.4
1.19 ± 0.19
0.00 ± 0.002
5.51 ± 0.15
Latent, 48 tokens
0.3, 0.4
1.13 ± 0.16
0.01 ± 0.002
5.33 ± 0.15
TABLE VII: Generation from the Nottingham models, 100 test primes per model, mean and SD over the three training seeds. τ lists the temperatures chosen on validation primes. Repetition is the structural repetition rate. The last column is the distance with τ=1 for every model. The first row is the 5th to 95th percentile of 2000 sets of 100 real test tunes, one stream each, cut from the start of each tune.
Fig. 7: Distance of generated streams to the training-split reference against context length, at the tuned temperature, mean and SD over three training seeds, 100 test primes per model. The shaded band is the 5th to 95th percentile of 2000 sets of 100 real test tunes cut from the start of each tune, and the cross marks the closest single generated set, at 0.19.
Fig. 8: The four corpora: share of test targets with a 16-token repeat in view at 336 tokens against the token predictor’s NLL drop from 6 to 336 tokens. One point per corpus, so this is a description, not a test, and no line is fitted.
Token predictor
Latent predictor
T
b1
b2
Parameters
k
b1
b2
Parameters
6
3
4
2,471,952
1
1
1
2,481,088
18
9
10
2,473,104
3
2
2
2,481,280
48
18
31
2,475,984
8
3
6
2,481,760
96
18
79
2,480,592
16
3
14
2,482,528
192
18
175
2,489,808
32
3
30
2,484,064
TABLE VIII: Attention bands and parameter counts of the predictors at each context length ( b1+b2−1 equals the context, in tokens for the token predictor and in latent steps for the latent predictor). The MAESTRO token predictors, with 89 tokens, have 2.53M parameters at 336 tokens.
History matched (tokens)
Corpus
32
16
8
Nottingham
0.965
0.869
0.625
O’Neill’s
0.969
0.890
0.733
MusicNet
0.902
0.866
0.760
MAESTRO
0.553
0.586
0.510
TABLE IX: Mixing weights of the copy baseline, fitted on validation windows.
Transformers predict over a representation of a sequence. The same data can be written as bytes, characters, or subword tokens, and these representations may be lossless. Yet, under a fixed context window, they need not expose the same information to the model. This raises a basic question: how does the choice of representation change what a finite-context predictor can achieve? We study this question on Markov sources and uncover two complementary phenomena. First, we observe that moving to smaller representation units can hurt prediction even when the context window is enlarged to cover the relevant source history. To explain this, we introduce fragmentation: a lossless recoding that replaces each source symbol by several smaller units. We prove that fragmentation can strictly increase the optimal finite-context log-loss, showing that the gap is not merely an optimization or capacity issue, but can be intrinsic to the representation. This gives a theoretical account of the finite-context gap observed in byte- and character-level models such as ByT5 and CANINE relative to subword-tokenized models. Second, we study the opposite direction: greedy tokenization -- BPE, WordPiece, and related methods -- which groups source symbols into larger units. We show that tokenization can make a short token window behave like a longer source-context window, and we give a loss guarantee describing when this is achievable. The guarantee depends on how reliably token windows span the needed source history, together with the compression rate of the tokenizer. This also yields a simple diagnostic for real tokenizers: measuring how much source context a fixed token window reliably contains. Together, the two directions establish a finite-context information-theoretic framework for reasoning about representation choices in Transformers.
Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan Tchamkerten
Department of Communications and Electronics Institut Polytechnique de Paris
GPT-style models have achieved remarkable success with finite vocabularies of reusable tokens, making the token interface a central component of modern sequence modeling. Symbolic music appears naturally compatible with this paradigm: it consists of discrete note events and recurring structures such as chords, motifs, and phrases. However, when tokenization moves beyond language, the interface must be specified for each domain. Existing work offers many effective designs, but no unified criterion for deciding what tokenization should represent and how far it should go. Using predictive codelength as a common criterion, we formulate the Effectiveness--Losslessness Framework to define where tokenization should begin and where it should end. The Fact--Token Boundary marks where observation-determined structure should enter the token interface, through operations such as coordinate construction. Within this interface, the resulting carrier may be reversibly recoded without changing the represented facts. The Token--State Boundary marks where tokenization should stop: relations that depend on context should remain for model-state computation rather than being fixed in advance by the tokenizer. We validate the framework through controlled multi-seed symbolic-music experiments, with an independent-corpus replication of the temporal intervention. Making musical time explicit consistently reduces predictive code and also improves pitch and duration prediction, while tonal-frame canonicalization and pitch factorization provide further gains. Fixed circle-of-fifths pitch coordinates instead increase predictive code, suggesting that imposing a fixed pitch relation before context can burden prediction. Reversible BPE substantially shortens the carrier but increases predictive codelength in every seed, showing that carrier compaction alone does not guarantee predictive gain.
Yi Wang
Department of Electronic Engineering, Tsinghua University
Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.
Zixun Guo, Simon Dixon
Centre for Digital Music, Queen Mary University of London, United Kingdom