How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models

Amir Fathi

cs.SDOct 2, 2026

How Far Back Should a Transformer Look? Repetition and Copying in Music Sequence Models

Authors: Amir Fathi

Organizations: Department of Computing Science University of Alberta Edmonton, AB, Canada

Abstract

We investigate how predictive performance depends on the maximum context available to an autoregressive model of symbolic music, and what information long-context models exploit. A small causal Transformer is trained separately at each context length T in {6, 18, 48, 96, 192, 336} and evaluated on the same target positions, both over raw tokens and over non-overlapping binary latent codes. On Nottingham folk tunes, the token predictor's test NLL decreases by 71% (0.986 bits per token) between 6 and 336 tokens, and by 71% on the O'Neill's tunes long enough for the same sweep (129 of 302 test tunes). Most of the long-context gain is explained by exact copying: the improvement appears when an earlier occurrence of the target's 16-token history enters the available context; overwriting that occurrence removes the gain, whereas equally large unrelated corruption does not; and a simple copy baseline recovers 94% of the reduction. A trained 336-token model likewise loses most of this gain when its history is restricted to recent tokens at test time. Because many repeats arise from written repeat signs expanded in the score, this result pertains specifically to these rendered score representations. By contrast, on MAESTRO performances and MusicNet scores, where exact repeats are substantially less frequent, the reduction is smaller (9% and 18%) and is largely attained by 96 tokens. Finally, an audit of an earlier draft that reported saturation at 16 tokens identified split leakage, overlapping latent receptive fields, an averaging predictor, and a per-file tempo grid; we present these as methodological checks for context-length measurements.

Figures & tables

	Pieces			Tokens			Windows	Test blocks
	Train	Val	Test	Train	Val	Test	train / val / test	in train
Nottingham	729	150	154	382K	90K	84K	2789 / 809 / 660	0.0%
O’Neill’s	1397	295	302	738K	115K	158K	7121 / 687 / 1505	0.6%
MusicNet	221	54	48	577K	141K	116K	1091 / 267 / 218	0.0%
MAESTRO	962	137	177	4586K	560K	575K	9072 / 1094 / 1113	0.2%

TABLE I: The four corpora after tokenization. Windows are the training and evaluation windows of the context sweep (Section V ). The last column is the share of non-constant 48-token test blocks that occur word for word in the training split.

	Nottingham		O’Neill’s		MusicNet	MAESTRO
Context	Latent	Token	Latent	Token	Token	Latent	Token
(tokens)	(bits per bit)	(bits per token)	(bits per bit)	(bits per token)	(bits per token)	(bits per bit)	(bits per token)
0	0.958	4.366	0.931	4.383	5.295	0.978	5.319
6	0.857 $\pm$ 0.002	1.391 $\pm$ 0.015	0.845 $\pm$ 0.005	1.692 $\pm$ 0.014	2.512 $\pm$ 0.066	0.887 $\pm$ 0.0004	3.283 $\pm$ 0.002
18	0.842 $\pm$ 0.002	1.440 $\pm$ 0.021	0.818 $\pm$ 0.003	1.543 $\pm$ 0.019	2.366 $\pm$ 0.003	0.881 $\pm$ 0.00003	3.177 $\pm$ 0.002
48	0.834 $\pm$ 0.005	1.529 $\pm$ 0.090	0.771 $\pm$ 0.005	1.449 $\pm$ 0.017	2.159 $\pm$ 0.004	0.879 $\pm$ 0.0004	3.097 $\pm$ 0.006

TABLE II: Test NLL against context length, mean and SD over three seeds. Context 0 is the unigram distribution (tokens) and the per-bit marginal (latents), each from one run. Lower is better. Nottingham and O’Neill’s use transposition of the training split. MAESTRO tokens: 89-token data. MusicNet has token predictors only. Section VI-G reports ten more token runs at 48, 96 and 192 tokens with a fixed training budget.

Panel of Figure 3 — Fig. 1: Test NLL relative to its value at 6 tokens of context for (a) Nottingham, (b) O’Neill’s, (c) MusicNet and (d) MAESTRO. Lines are the mean over three seeds with the seed SD as error bars; the small hollow marks beside them are the individual seeds. The label at the end of each curve is its drop from 6 to 336 tokens. The context axis is logarithmic, and the dotted line marks 96 tokens.

Share of targets	1-32	33-80	81-176	177-320	None	Held
	Repeat lag (tokens back)
in view from context	48	96	192	336
Nottingham, target tokens	1.8%	14.7%	36.6%	17.9%	28.3%	0.7%
Nottingham, target latent steps	1.9%	15.4%	36.4%	17.7%	27.5%	1.2%
O’Neill’s, target tokens	0.3%	9.6%	33.3%	30.1%	26.7%	0.0%
O’Neill’s, target latent steps	0.4%	11.1%	33.8%	29.1%	25.6%	0.0%

TABLE III: Repeat lag of the test targets: the share of targets whose 16-token history last occurred at each lag, or not within 320 tokens (none), or is one held pitch (held). A repeat is in view for context

T

when lag

+

\leq T

, and each bin comes into view at the context shown. Latent steps use the history before the step’s first token. The last rows describe the Nottingham and O’Neill’s repeats: how often the next token, or the next 6 tokens, equal those that followed the earlier occurrence, and how often the earlier occurrence starts on the latent step grid, where the same 6 tokens have the same code.

Panel of Figure 5 — Fig. 2: Test NLL on Nottingham by repeat lag (columns) and context length (rows), mean over three seeds, for (a) the target tokens of the token predictor and (b) the target latent steps of the latent predictor, each on its own colour scale. Darker cells have lower NLL. Left of the black line the repeat is in view (lag $+$ 16 $\leq$ context); right of it, it is not. The bins are cut at these limits.

Panel of Figure 6 — Fig. 3: Share of target tokens whose repeat is in view (lag $+$ 16 $\leq$ context, thin dotted line), and share of the NLL reduction from 6 to 336 tokens reached at each context (thick lines), for (a) Nottingham, (b) O’Neill’s, (c) MusicNet and (d) MAESTRO. The label at 336 tokens is the repeat share there; the reduction reached is 100% at 336 tokens by definition. A negative share means the NLL is above its value at 6 tokens.

Corpus, predictor, context	Targets	Clean	Erased	Control	Swapped token
Nottingham, token, 192	16,693	0.22 $\pm$ 0.005	1.10 $\pm$ 0.02	0.22 $\pm$ 0.004	0.5% $\to$ 55%
Nottingham, token, 336	22,454	0.19 $\pm$ 0.003	1.06 $\pm$ 0.03	0.19 $\pm$ 0.003	0.3% $\to$ 62%
Nottingham, latent, 192	2,783	0.37 $\pm$ 0.03	0.87 $\pm$ 0.01	0.38 $\pm$ 0.03	n/a
Nottingham, latent, 336	3,747	0.35 $\pm$ 0.01	0.81 $\pm$ 0.004	0.35 $\pm$ 0.01	n/a
O’Neill’s, token, 192	31,068	0.37 $\pm$ 0.003	1.28 $\pm$ 0.01	0.38 $\pm$ 0.002	0.8% $\to$ 51%
O’Neill’s, token, 336	52,864	0.22 $\pm$ 0.01	1.24 $\pm$ 0.03	0.22 $\pm$ 0.01	0.4% $\to$ 71%

TABLE IV: Erase test on Nottingham and O’Neill’s targets with a repeat in view, mean and SD over three seeds. Clean is the test NLL of these targets; erased overwrites every earlier occurrence of the 16-token history in the context, with what followed it; control overwrites as many spans of the same length elsewhere in the context. Latent values in bits per bit, token values in bits per token. The last column is the probability the token predictor gives to a new token placed after every earlier occurrence, before and after the swap.

Panel of Figure 8 — Fig. 4: The erase test of Table IV as bars: test NLL of the targets with a repeat in view when the context is left as it is (clean), when as many unrelated spans are overwritten (control), and when every earlier occurrence of the target’s 16-token history is overwritten (erased), for (a) the token predictor and (b) the latent predictor at 192 and 336 tokens. Error bars are the seed SD.

Panel of Figure 9 — Fig. 5: One trained model with less history. Test NLL of the 336-token token predictors when their history is limited at test time to the most recent tokens, by giving the model only those tokens (truncated) or by replacing every older token with a window of another piece (replaced), beside the models trained separately at each context length. Mean over three seeds, on the targets of the sweep (every fourth target for MAESTRO); each panel has its own scale.

Setup	6	96	192	336
Legacy grid, 480-token windows, $D{=}32$ , latent	0.80 $\pm$ 0.0002	0.80 $\pm$ 0.0005	0.78 $\pm$ 0.001	0.72 $\pm$ 0.01
Legacy grid, 480-token windows, $D{=}32$ , token	1.22 $\pm$ 0.001	1.20 $\pm$ 0.004	1.04 $\pm$ 0.01	0.84 $\pm$ 0.02
Score grid, 384-token windows, $D{=}64$ , latent	0.91 $\pm$ 0.001	0.91 $\pm$ 0.003	0.77 $\pm$ 0.004	0.63 $\pm$ 0.01
Score grid, 384-token windows, $D{=}64$ , token	1.39 $\pm$ 0.004	1.09 $\pm$ 0.02	1.15 $\pm$ 0.02	0.81 $\pm$ 0.25
Score grid, as above, with transposition, latent	0.86 $\pm$ 0.002	0.78 $\pm$ 0.004	0.57 $\pm$ 0.01	0.47 $\pm$ 0.01
Score grid, as above, with transposition, token	1.39 $\pm$ 0.01	1.27 $\pm$ 0.33	0.57 $\pm$ 0.01	0.41 $\pm$ 0.01

TABLE V: Nottingham under three setups, test NLL at four context lengths (tokens), mean and SD over three seeds. Latent values in bits per bit, token values in bits per token. The setups differ in tokenizer, window, autoencoder and transposition at once, so values are comparable within a row.

Panel of Figure 11 — Fig. 6: Seed spread of the Nottingham token predictor. (a) Validation NLL against epoch for ten runs at 96 tokens trained for 200 epochs with no early stopping; coloured lines drop (Section VI-G ), with a circle at the epoch of the drop, grey dashed lines do not, and a tick marks where the sweep’s early stopping would have ended each run. (b) Test NLL of every run: the three early-stopped runs of the sweep (circles) and the ten fixed-budget runs (dots). The dotted line is the 6-token predictor.

	Nottingham		MAESTRO
Context	Causal	v5	Causal	v5
6	0.857 $\pm$ 0.002	0.856 $\pm$ 0.0001	0.887 $\pm$ 0.0004	0.887 $\pm$ 0.0004
18	0.842 $\pm$ 0.002	0.852 $\pm$ 0.001	0.881 $\pm$ 0.00003	0.881 $\pm$ 0.0002
48	0.834 $\pm$ 0.005	0.843 $\pm$ 0.004	0.879 $\pm$ 0.0004	0.879 $\pm$ 0.0004
96	0.775 $\pm$ 0.004	0.891 $\pm$ 0.001	0.877 $\pm$ 0.0004	0.879 $\pm$ 0.0003
192	0.571 $\pm$ 0.015	0.886 $\pm$ 0.001	0.877 $\pm$ 0.00010	0.878 $\pm$ 0.0003

TABLE VI: The causal latent predictor against v5, the averaging predictor of our earlier draft, on the same latents. Test NLL in bits per bit, mean over three seeds.

	Tuned temperature			$\tau{=}1$
System	$\tau$	Distance	Repetition	Distance
Real test tunes, 100 per set (5th to 95th pct.)	n/a	0.03 to 0.17	0.48 to 0.52	n/a
AE round trip, all test streams	n/a	0.07	0.49	n/a
Latent, 6 tokens	0.2, 0.3	1.21 $\pm$ 0.19	0.01 $\pm$ 0.02	5.97 $\pm$ 0.03
Latent, 18 tokens	0.3, 0.4	1.19 $\pm$ 0.19	0.00 $\pm$ 0.002	5.51 $\pm$ 0.15
Latent, 48 tokens	0.3, 0.4	1.13 $\pm$ 0.16	0.01 $\pm$ 0.002	5.33 $\pm$ 0.15

TABLE VII: Generation from the Nottingham models, 100 test primes per model, mean and SD over the three training seeds.

\tau

lists the temperatures chosen on validation primes. Repetition is the structural repetition rate. The last column is the distance with

\tau{=}1

for every model. The first row is the 5th to 95th percentile of 2000 sets of 100 real test tunes, one stream each, cut from the start of each tune.

Panel of Figure 14 — Fig. 7: Distance of generated streams to the training-split reference against context length, at the tuned temperature, mean and SD over three training seeds, 100 test primes per model. The shaded band is the 5th to 95th percentile of 2000 sets of 100 real test tunes cut from the start of each tune, and the cross marks the closest single generated set, at 0.19.

Panel of Figure 15 — Fig. 8: The four corpora: share of test targets with a 16-token repeat in view at 336 tokens against the token predictor’s NLL drop from 6 to 336 tokens. One point per corpus, so this is a description, not a test, and no line is fitted.

Token predictor				Latent predictor
$T$	$b_{1}$	$b_{2}$	Parameters	$k$	$b_{1}$	$b_{2}$	Parameters
6	3	4	2,471,952	1	1	1	2,481,088
18	9	10	2,473,104	3	2	2	2,481,280
48	18	31	2,475,984	8	3	6	2,481,760
96	18	79	2,480,592	16	3	14	2,482,528
192	18	175	2,489,808	32	3	30	2,484,064

TABLE VIII: Attention bands and parameter counts of the predictors at each context length (

b_{1}+b_{2}-1

equals the context, in tokens for the token predictor and in latent steps for the latent predictor). The MAESTRO token predictors, with 89 tokens, have 2.53M parameters at 336 tokens.

	History matched (tokens)
Corpus	32	16	8
Nottingham	0.965	0.869	0.625
O’Neill’s	0.969	0.890	0.733
MusicNet	0.902	0.866	0.760
MAESTRO	0.553	0.586	0.510

TABLE IX: Mixing weights of the copy baseline, fitted on validation windows.

Long-Context Modeling

Long-context modeling studies how AI systems process, retrieve, and reason over sequences that exceed ordinary context windows.

See all 133 papers

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization
May 13, 2026Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan TchamkertenLanguage Modeling Subword Tokenization
Department of Communications and Electronics Institut Polytechnique de Paris
How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness
Aug 18, 2026Yi WangTokenization Token Efficiency
Department of Electronic Engineering, Tsinghua University
Equivariant Music Transformer
Aug 4, 2026Zixun Guo, Simon DixonRepresentation Learning Transformer
Centre for Digital Music, Queen Mary University of London, United Kingdom

May 13, 2026J/K move · Enter open · S save

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization

Amirmehdi Jafari Fesharaki, Mohammadamin Rami, Aslan Tchamkerten

Department of Communications and Electronics Institut Polytechnique de Paris

Language Modeling Subword Tokenization Tokenization

Open paper AskPDF arXiv

Transformers predict over a representation of a sequence. The same data can be written as bytes, characters, or subword tokens, and these representations may be lossless. Yet, under a fixed context window, they need not expose the same information to the model. This raises a basic question: how does the choice of representation change what a finite-context predictor can achieve? We study this question on Markov sources and uncover two complementary phenomena. First, we observe that moving to smaller representation units can hurt prediction even when the context window is enlarged to cover the relevant source history. To explain this, we introduce fragmentation: a lossless recoding that replaces each source symbol by several smaller units. We prove that fragmentation can strictly increase the optimal finite-context log-loss, showing that the gap is not merely an optimization or capacity issue, but can be intrinsic to the representation. This gives a theoretical account of the finite-context gap observed in byte- and character-level models such as ByT5 and CANINE relative to subword-tokenized models. Second, we study the opposite direction: greedy tokenization -- BPE, WordPiece, and related methods -- which groups source symbols into larger units. We show that tokenization can make a short token window behave like a longer source-context window, and we give a loss guarantee describing when this is achievable. The guarantee depends on how reliably token windows span the needed source history, together with the compression rate of the tokenizer. This also yields a simple diagnostic for real tokenizers: measuring how much source context a fixed token window reliably contains. Together, the two directions establish a finite-context information-theoretic framework for reasoning about representation choices in Transformers.

Abstract

Long-Context Modeling

Explore similar work

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization

How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness

Equivariant Music Transformer