Copying Under Repetition: From Prefix Matching to Position Hashing

Anxin Guo; Tianlong Huang; Chenxiao Yang; Zhiyuan Li

cs.LGOct 19, 2026

Copying Under Repetition: From Prefix Matching to Position Hashing

Authors: Anxin Guo, Tianlong Huang, Chenxiao Yang, Zhiyuan Li

Organizations: Northwestern University · Duke University∗ · Toyota Technological Institute at Chicago

Abstract

Frontier LLMs can copy random text almost flawlessly, yet they often overrun or truncate long blocks of repeated characters. In repeated strings, matching local content no longer reliably identifies where copying should continue or when it should stop. Nevertheless, we show that a two-layer transformer with $O(\log\log N)$ -bit precision and $O(\log N/\log\log N)$ width can copy every string up to a chosen design horizon $N$ and terminate correctly under arbitrary repetition. We use RoPE-style rotations of attention values and outputs (VO-rotations) with small prime periods to encode distances from delimiter anchors modulo those primes. By the Chinese Remainder Theorem, these periodic encodings jointly distinguish all required distances, allowing one retrieval rule to select either the next source token or the end marker. In contrast, we prove a width-independent impossibility result for copying under sequential finite-precision softmax accumulation. It applies to transformers whose only positional input is an additive attention bias that eventually becomes constant, including NoPE and a recursive implementation of ALiBi. Once an identical-token run exceeds the resulting saturation threshold, further repetitions leave the logits unchanged. If this threshold grows sublinearly with $N$ , these models cannot both complete a long all-identical copy and stop correctly, a pattern resembling the termination errors of frontier LLMs. In synthetic copying experiments, baseline transformer models fail on long repeated strings even within the training range. Trained VO models, by contrast, generalize beyond the training lengths with near-perfect accuracy up to length $\approx 5005$ , the selected primes' period product.

Figures & tables

Figure 1: Long repeated strings reduce LLM copying accuracy. Accuracy is the fraction of all

50

trials per length and condition whose extracted answer matches the source perfectly. For a fixed switch probability

p_{\rm switch}

, a source string is generated by concatenating latent constant-letter runs: each run chooses a letter uniformly from the 26 lowercase English letters and has length

R\sim\operatorname{Geom}(p_{\rm switch})

. The three plotted distributions are uniform-random strings,

p_{\rm switch}=1

; repeated strings,

p_{\rm switch}\approx 0.01

; and all-identical strings, the limit

p_{\rm switch}\to 0

. Output extraction and experimental details are in Appendix A .

Panel of Figure 2 — Figure 2: How ECB transformers lose track of repetition through rounded accumulation. Panel (a) shows the left-to-right accumulation order. In panel (b), colored tails indicate guaranteed identical states within each layer; gray states may differ (schematic).

Failure type	Target string (excerpt)	Model reply (excerpt)
Extended repetition	bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb $\ldots$ (Length: $300$ )	bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb $\ldots$ (Length: 79,984 )
Premature truncation	iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii $\ldots$ (Length: $300$ )	iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii $\ldots$ (Length: 224 )
Empty output	zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz $\ldots$ (Length: $300$ )	Empty response (Length: $0$ )

Table 1: Recorded Qwen3.5 copying failures on all-identical sources of length

300

. String excerpts reproduce the recorded text; ellipses mark omitted characters.

Figure 3: Exact-copy accuracy by source length and distribution, averaged over the three training seeds for every model. Standard-deviation bands are omitted for readability; Table 2 reports cross-seed variation near the training limit. The dashed black line marks the maximum training source length,

n=2000

; lengths to its right test extrapolation.

Model	Layers	Width	$p_{\rm switch}=1$	$p_{\rm switch}=0.01$	$p_{\rm switch}\to 0$
NoPE	6	512	$0.750\pm 0.244$	$0.000\pm 0.000$	$0.049\pm 0.084$
RoPE	6	512	$0.972\pm 0.030$	$0.809\pm 0.071$	$0.257\pm 0.367$
ALiBi	6	512	$0.997\pm 0.006$	$0.927\pm 0.038$	$0.122\pm 0.080$
Hard-ALiBi	6	512	$1.000\pm 0.000$	$0.215\pm 0.129$	$0.056\pm 0.006$
Sinusoidal	6	512	$0.969\pm 0.021$	$0.007\pm 0.006$	$0.028\pm 0.039$
VO-Rotation	2	256	$1.000\pm 0.000$	$1.000\pm 0.000$	$1.000\pm 0.000$

Table 2: String-level accuracy near the

n=2000

training limit: mean

\pm 1

sample standard deviation across the three training seeds. Baselines are evaluated at

n=1993

and VO at

n=1992

Figure 4: Training loss of VO-Rotation with seed

42

, with both axes logarithmic. The loss stays on a plateau for roughly the first

40

k optimizer updates and then drops sharply.

Figure 5: VO-Rotation length generalization on the complete evaluation grid. The curve shows the mean across three seeds, and the band spans

\pm 1

sample standard deviation clipped to

[0,1]

. Dashed black: maximum training source length

2000

; dash-dotted red: period product

5005

. See Appendix B.1 for individual curves.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Model ID	Reasoning/thinking
Qwen3.5	qwen3.5:397b-cloud	thinking disabled
DeepSeek V4 Pro	deepseek-v4-pro	thinking disabled
Gemini 3.1 Pro	gemini-3.1-pro- preview	low thinking
Gemini 3.1 Pro	gemini-3.1-pro- preview	medium thinking
GPT-5.5	gpt-5.5	none
GPT-5.5	gpt-5.5	medium reasoning

Appendix

Table 3: Frontier-model variants plotted in Figure 1 . Each row uses

50

trials for each length and condition, a maximum generation budget of

20{,}000

output tokens, and the extraction policy described above.

Model	Layers	Width	Heads	Precision	Update budget
NoPE	6	512	8	bf16	$320{,}000$
RoPE	6	512	8	bf16	$320{,}000$
ALiBi	6	512	8	bf16	$320{,}000$
Hard-ALiBi	6	512	8	bf16	$320{,}000$
Sinusoidal	6	512	8	bf16	$320{,}000$
VO-Rotation	2	256	4	FP8 sim.	$400{,}000$

Appendix

Table 4: Training configurations shared by seeds

42

43

, and

44

. Budgets are maximum optimizer updates. FP8 sim. denotes the linear-operand simulation described above. All models use learning rate

10^{-4}

and no weight decay.

Model	Seed $42$	Seed $43$	Seed $44$
NoPE	$102{,}800$	$115{,}440$	$110{,}760$
RoPE	$144{,}800$	$112{,}580$	$119{,}080$
ALiBi	$81{,}200$	$81{,}120$	$118{,}560$
Hard-ALiBi	$110{,}200$	$96{,}980$	$114{,}140$
Sinusoidal	$113{,}800$	$119{,}080$	$104{,}520$
VO-Rotation	$158{,}200$	$150{,}020$	$148{,}200$

Appendix

Table 5: Completed optimizer updates for each model and training seed, using the shared configurations in Table 4 .

Seed	Distribution	Mean	Minimum	$n=5000$
42	Uniform random	0.9834	0.4375	0.9688
43	Uniform random	0.9591	0.4688	0.8854
44	Uniform random	0.9964	0.9583	1.0000
42	$p_{\rm switch}=0.01$	0.9993	0.9167	1.0000
43	$p_{\rm switch}=0.01$	0.9896	0.7188	0.9896
44	$p_{\rm switch}=0.01$	0.99997	0.9896	1.0000

Appendix

Table 6: VO exact-copy accuracy by training seed and distribution. Mean and minimum are over the

375

evaluated lengths

2008

–

5000

; the final column gives accuracy at

5000

Figure 6: VO-Rotation accuracy by seed, with seeds

42

43

, and

44

in rows and distributions in columns. Dashed black and dash-dotted red lines mark the maximum training source length

2000

and the period product

5005

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 528 papers

Jul 17, 2026

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.

Haodong Wen, Yiran Zhang, Yingfa Chen +1

Tsinghua University

Language Model Pretraining Rotary Positional Embeddings Memorization in Language Models Positional Encoding

Ask PDF

Oct 13, 2026

Copy-Paste Language Models

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Nathan Godey, Yoav Artzi

Cornell University New York City, USA

Language Model Pretraining Efficient Language Model Training Long-Context Language Modeling

Ask PDF

May 21, 2026

How Many Different Outputs Can a Transformer Generate?

We study how we can leverage only a handful of characteristics of a transformer's architecture to closely predict the number of different sequences it can output, both qualitatively and quantitatively. We provide an upper bound depending on the length of the prompt, which we show empirically to be tight up to a factor less than 10, across architectures and model sizes. Our analysis also provides a theoretical explanation for previously observed empirical failures of transformers on simple sequence tasks, such as copying and cramming. Formally, we prove that (i) the maximal length of accessible sequences (those that the transformer can output for some prompt) grows linearly with the prompt length, (ii) beyond a critical threshold, the proportion of accessible sequences decays exponentially with sequence length, and (iii) the linear coefficient relating prompt length to accessible sequence length admits a theoretical upper bound. Notably, these results hold even with unbounded context and computation time.

Maxime Meyer, Mario Michelessa, Caroline Chaux +1

Department of Mathematics, National University of Singapore, Singapore, 117543 · IPAL, IRL2955, Singapore · School of Computing, National University of Singapore, Singapore, 117543 +2

Transformer Expressivity Transformer Neural Network Expressivity

Ask PDF

Abstract

Transformer

Explore similar work

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Copy-Paste Language Models

How Many Different Outputs Can a Transformer Generate?