Copy-Paste Language Models

Nathan Godey; Yoav Artzi

cs.CLOct 13, 2026

Copy-Paste Language Models

Authors: Nathan Godey, Yoav Artzi

Organizations: Cornell University New York City, USA

Abstract

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Our Copy-Paste (CP) head. The LM head uses the probability of a special <copy> token to gate the copy distribution; a special <sink> token allows rolling back to the LM head. The prediction mixes both heads. Right: FinePDF perplexity of our 1.7B models vs. memory footprint. CPLMs outperform their vanilla counterparts by 1–1.5 PPL, and the sliding-window (SW) CPLM matches a full-attention LM at $12\times$ less memory per token, copy cache included.

Panel of Figure 2 — Figure 2: FineWeb-Edu validation loss vs. training tokens (before LR decay), for $w=256$ . Colors distinguish backbone attention setups. Full and Hybrid training curves are essentially superposed.

Model	Type	HellaSwag	LAMBADA	PIQA	SciQ	ARC-e	ARC-c	WinoGrande	Avg.
Full	Vanilla	61.12	44.46	75.24	88.20	68.39	40.96	58.80	62.45
	CPLM	61.70	46.92	74.86	89.20	68.10	41.21	59.43	63.06
SW	Vanilla	61.00	42.46	74.48	87.00	67.17	41.30	58.01	61.63
	CPLM	61.78	46.42	74.70	87.30	67.59	41.13	58.41	62.48
Hybrid	Vanilla	61.08	43.70	74.48	87.80	69.32	40.96	58.17	62.22
	CPLM	61.70	46.92	74.48	87.40	66.46	41.55	59.59	62.59

Table 1: Zero-shot accuracy (%) of our 1.7B base models (95k steps). Best of each pair in bold.

Panel of Figure 3 — Figure 3: Validation perplexity vs. memory usage for sliding window sizes from 32 to 1024 at 135M scale. CPLMs improve the memory–performance frontier.

Panel of Figure 4 — Figure 4: Per-position statistics of our 1.7B models on 5,000 FinePDFs documents of 16k tokens, i.e. up to $4\times$ the pretraining context. Full and Hybrid models use YaRN ( $\times$ 4). (a) Perplexity. (b–c) Average probability of the <copy> token and of the sink. CPLMs keep improving with context beyond their training length, and use the CP head more as the context grows.

Full	Vanilla	2.52	4.08	13.9	28.9	20.4	16.9	41.2	24.2
		PPL ( $\downarrow$ )		Math		Coding		Instruction
Attention	Model	Tulu-3	Nemotron-3	MATH	GSM8K	HumanEval	MBPP	IFEval	Avg
	CPLM	2.47	3.78	17.5	33.1	21.1	17.5	43.0	26.4
Hybrid 3:1	Vanilla	2.52	4.10	16.5	27.9	19.8	19.0	38.3	24.3
	CPLM	2.48	3.80	17.4	30.4	22.1	17.4	40.1	25.5
SW-256	Vanilla	2.79	5.65	7.2	18.6	15.4	16.1	37.2	18.9

Table 2: Downstream results after Supervised Fine-Tuning on our 1.7B models. PPL: Tulu-3 val. and Nemotron3-SFT-Math-v4 perplexity. Evaluation details in Appendix D .

Full	Vanilla	26.4	33.6	5.7	6.1	11.8	20.7	1.7	15.4	10.4
		Synthetic (0–16k)		LongBench (16k)
Attention	Model	BABILong	RULER	Single-Doc	Multi-Doc	Summ.	Few-shot	Synth.	Code	Avg
	CPLM	32.0	39.6	6.8	6.7	10.2	28.3	0.8	18.6	12.2
Hybrid 3:1	Vanilla	27.7	33.2	4.6	5.3	8.4	20.5	0.7	17.4	9.5
	CPLM	32.9	37.8	5.4	3.3	14.0	35.1	1.7	25.6	14.3
SW-256	Vanilla	13.8	7.9	3.0	2.9	6.4	16.6	1.4	14.8	7.4

Table 3: Long-context results for the 1.7B base models. BABILong: mean accuracy over qa1–qa5 and lengths 0k–16k; RULER: mean over its 13 tasks and lengths 4k–16k; LongBench: category means at 16k context, Avg over the 16 tasks. Full and Hybrid use YaRN (

\times

4 from 4096); SW-256 uses plain RoPE. Bold marks the better model of each pair. Details in Appendix F .

Panel of Figure 5 — Figure 5: HotpotQA performance for 1.7B models. Models are fine-tuned and evaluated either with full document KV visible ( left ), or with only a single access to the document memory: last-layer KV for Vanilla, and CP states for CPLMs ( middle ).

Panel of Figure 6 — Figure 6: The CP head mitigates the LM head gradient bottleneck ( Godey and Artzi, 2026 ) . Left: Fraction of the logit gradient norm that projects to the nullspace of $W_{\{LM,Q,K\}}$ for the 135M full-attention CPLM ( Section 3.1 ). Random : same measurement for a random matrix with the shape of the CP head output matrix. Both CP head pathways let through much more of the signal than the LM head, especially at earlier positions. Center and Right: Accuracy and final validation loss for a 135M model on the SpamLang synthetic task, where CPLMs converge for all vocabulary sizes.

Panel of Figure 7 — Figure 7: Induction analysis for 1.7B models. Left : We measure the performance of models when copying a randomly generated sequence; Middle : Total number of triggered induction heads across sequence lengths; Right : Heatmap of induction scores across all heads. Overall, CPLMs have stronger copy abilities while exhibiting far fewer induction heads.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
	135M	360M	1.7B
Layers	30	32	24
Hidden size	576	960	2,048
FFN size	1,536	2,560	8,192
Attention heads	9	15	32
KV heads (GQA)	3	5	32

Appendix

Table 4: Pretraining hyperparameters for the 135M, 360M and 1.7B runs.

	Throughput (tokens/s)			Peak memory (GB)
Model	Vanilla	CPLM	Ratio	Vanilla	CPLM	Ratio
1.7B	55,862	54,113	$\times$ 0.969	125.4	137.5	$\times$ 1.096
9B	20,429	20,533	$\times$ 1.005	114.0	117.0	$\times$ 1.026
30B	5,391	5,342	$\times$ 0.991	139.9	142.3	$\times$ 1.017

Appendix

Table 5: Training cost on one B300 GPU with sequences of 4,096 tokens. Ratios are CPLM values divided by vanilla values. At 1.7B, a training step includes the AdamW update (batch of 8 sequences); at 9B and 30B, only the forward and backward passes fit on one GPU.

			Time			Memory (GB)
Model	$T$	$B$	Prefill	Decoding step	Request	KV cache	Copy cache
1.7B	4k	1	$\times$ 0.993	$\times$ 1.018	$\times$ 1.017	0.80	0.02
	4k	128	$\times$ 0.999	$\times$ 1.017	$\times$ 1.010	102.1	2.13
	16k	32	$\times$ 0.996	$\times$ 1.018	$\times$ 1.008	97.5	2.03
9B	4k	1	$\times$ 0.987	$\times$ 1.010	$\times$ 1.008	0.53	0.03
	4k	128	$\times$ 1.000	$\times$ 1.038	$\times$ 1.014	68.1	4.25

Appendix

Table 6: Inference cost on one B300 GPU, for prompts of

T

tokens and batches of

B

sequences. Time columns give the ratio of the CPLM time to the vanilla time for prefill, for a decoding step, and for a whole request (prefill and 256 generated tokens), so that values above 1 mean that the CPLM is slower. Memory columns give the size of the key–value cache and of the additional copy cache of the CPLM in bfloat16 .

Backbone	Vanilla	CPLM	$\Delta$	$p$
Full	$16.33\pm 0.03$	$15.83\pm 0.02$	$-0.50$	$3.8\times 10^{-4}$
SW	$16.89\pm 0.02$	$16.02\pm 0.02$	$-0.87$	$9.5\times 10^{-5}$
Hybrid 3:1	$16.37\pm 0.02$	$15.84\pm 0.04$	$-0.52$	$2.5\times 10^{-4}$

Appendix

Table 7: Final validation perplexity at 135M (mean

\pm

std over seeds 42/43/44).

\Delta

is CPLM minus Vanilla.

Figure 8: Validation perplexity vs. cache memory at 2k tokens for our 135M sliding-window CPLM (

w=256

), when varying the dimension of the CP queries and keys from 64 to 2,048 ( Copy-dim sweep , default

D=576

), or the backbone window size ( Window sweep ).

Figure 9: FineWeb-Edu validation loss vs. training tokens (before LR decay) at 360M scale, for

w=256

. Full and Hybrid training curves are essentially superposed.

Figure 10: Validation perplexity of 135M CPLMs with and without the sink mechanism, for sliding-window backbones with windows from 32 to 1,024 tokens and for a full-attention backbone.

Panel of Figure 11 — Figure 11: Validation perplexity vs. memory usage at 135M scale for fully-recurrent (GDN) and hybrid (Hybrid-GDN) Gated DeltaNet backbones, with and without the CP head, on in-domain (FineWeb-Edu) and out-of-distribution (FinePDFs, arXiv, GovReport) data. Each architecture is shown with GDN state sizes of 64 and 256, the larger state using more memory.

Chang et al. (2023)
		Validation		FinePDFs @2k
Method	Output mechanism	Full	SW-256	Full	SW-256
Vanilla	LM softmax	16.34	16.88	14.25	16.52
C	Context partition	16.25	16.71	13.88	15.66
P	Pointer partition	16.19	16.54	13.81	14.96
CP	Context + pointer	16.19	16.53	13.64	14.91

Appendix

Table 8: Output-mechanism ablations at 135M (20B tokens, seed 42). Perplexity for a full-attention and a sliding-window (256) backbone, on the in-distribution validation set (final) and on held-out FinePDFs documents (2,000 English documents, first 2,048 tokens each, whitespace- and line-break-normalized). All runs share the backbone, data and training recipe. Across three seeds the standard deviation of the main arms on validation is 0.015–0.037 PPL ( Section C.1 ).

Attention	Vanilla	Vanilla + Neural cache	CPLM
Full	9.17	9.17	9.06
Hybrid 3:1	9.20	9.20	9.06
SW-256	9.65	9.63	9.24

Appendix

Table 9: FineWeb-Edu perplexity of our 1.7B models (100 documents of 2,048 tokens). The neural cache ( Grave et al., 2017 ) is added to the vanilla model, with

\theta

and

\gamma

tuned on separate FineWeb-Edu documents.

Panel of Figure 12 — Figure 12: GSM8K pass@ $k$ of our 1.7B SFT models for $k$ from 1 to 8, estimated from 8 samples (temperature 0.7, top- $p$ 0.9). (a) Pass@ $k$ of vanilla models and CPLMs. (b) Difference between each CPLM and its vanilla counterpart, in points.

Figure 13: Needle-in-a-haystack accuracy (%) by needle depth (rows) and context length (columns). Top: Vanilla. Bottom: CPLM. Each cell averages 200 examples. Average over the full grid, Vanilla / CPLM: Full 49.7 / 54.8; Hybrid 3:1 58.2 / 83.6; SW-256 14.3 / 99.4.

		RULER						BABILong
Attention	Model	4k	8k	16k	32k	64k	Avg	0k	1k	2k	4k	8k	16k	32k	64k	Avg
Full	Vanilla	50.1	36.3	14.5	0.0	0.0	20.2	55.0	44.6	35.0	18.6	4.4	0.6	0.0	0.0	19.8
	CPLM	54.9	38.6	25.2	0.0	0.0	23.7	52.6	49.6	40.8	32.6	12.6	4.0	0.0	0.0	24.0
Hybrid 3:1	Vanilla	50.6	32.1	16.9	7.1	3.6	22.1	56.4	38.8	29.0	26.8	11.4	3.6	0.6	0.2	20.8
	CPLM	53.5	40.8	19.2	11.7	2.8	25.6	57.6	47.4	42.8	27.4	16.2	5.8	1.4	0.2	24.8
SW-256	Vanilla	10.2	7.5	5.9	5.5	5.0	6.8	46.2	18.2	10.2	5.2	1.8	1.0	0.4	0.6	10.5

Appendix

Table 12: RULER and BABILong accuracy (%) by context length for the 1.7B base models. RULER: mean over its 13 tasks; BABILong: mean over qa1–qa5. Avg is the mean over all lengths shown (4k–64k for RULER, 0k–64k for BABILong). Full and Hybrid use YaRN (

\times

4 from 4096); SW-256 uses plain RoPE. Bold marks the better model of each pair.

Attention	Model	S1	S2	S3	MK1	MK2	MK3	MQ	MV	VT	CWE	FWE	SQuAD	HotpotQA	Avg
		Retrieval (NIAH)								Tracing	Aggregation		QA
Full	Vanilla	99.0	100.0	68.0	62.0	6.0	9.0	83.0	64.8	0.0	34.7	53.3	43.0	28.0	50.1
	CPLM	100.0	100.0	85.0	67.0	4.0	8.0	80.8	72.2	26.6	60.8	40.3	39.0	30.0	54.9
Hybrid 3:1	Vanilla	100.0	97.0	62.0	62.0	6.0	5.0	82.8	73.2	9.0	51.6	42.3	38.4	29.0	50.6
	CPLM	99.0	100.0	70.0	90.0	8.0	8.0	82.5	79.0	7.0	38.9	38.0	42.3	33.0	53.5
SW-256	Vanilla	8.0	9.0	3.0	8.0	3.0	1.0	5.2	3.2	0.4	16.0	26.0	31.3	19.0	10.2

Appendix

Table 13: RULER per-task accuracy (%) at 4k context for the 1.7B base models. NIAH: single-needle (S1–S3), multi-key (MK1–MK3), multi-query (MQ), multi-value (MV); VT: variable tracking; CWE/FWE: common/frequent word extraction. Avg is the mean over the 13 tasks. Full and Hybrid use YaRN (

\times

4 from 4096); SW-256 uses plain RoPE. Bold marks the better model of each pair.

Attention	Model	qa1	qa2	qa3	qa4	qa5	Avg
Full	Vanilla	19.0	9.0	17.0	33.0	15.0	18.6
	CPLM	31.0	21.0	34.0	34.0	43.0	32.6
Hybrid 3:1	Vanilla	20.0	12.0	23.0	43.0	36.0	26.8
	CPLM	30.0	6.0	29.0	30.0	42.0	27.4
SW-256	Vanilla	4.0	0.0	5.0	5.0	12.0	5.2
	CPLM	6.0	2.0	10.0	6.0	18.0	8.4

Appendix

Table 14: BABILong per-task accuracy (%) at 4k context for the 1.7B base models. qa1–qa3: one, two and three supporting facts; qa4: two-argument relations; qa5: three-argument relations. Full and Hybrid use YaRN (

\times

4 from 4096); SW-256 uses plain RoPE. Bold marks the better model of each pair.

Panel of Figure 14 — Figure 14: Prediction quality as a function of the copy gate $c_{t}=p(\texttt{<copy>})$ of our 1.7B CPLMs, on the FineWeb-Edu validation set. Shaded areas show the standard deviation within each bin. (a) Mean NLL difference between each CPLM and its vanilla counterpart, computed as CPLM minus vanilla (negative values favor CPLMs) and (b) top-1 accuracy gap (positive values favor CPLMs), computed on the tokens of each bin of $c_{t}$ . (c) Share of tokens in each bin.

Figure 15: Average distance between the current position and the positions copied by the CP head, as a function of the position, for our 1.7B CPLMs on 5,000 FinePDFs documents of 16k tokens.

Figure 16: Perplexity of our 135M full-attention vanilla model and CPLM, trained on English FineWeb-Edu, on English data from FineWeb ( Penedo et al., 2024 ) and on French, German, Italian, Mandarin Chinese and Arabic data from FineWeb2 ( Penedo et al., 2025 ) . Numbers indicate the perplexity difference between the CPLM and the vanilla model.

Panel of Figure 17 — Figure 17: Stopping the gradient of the CP head into the backbone (135M, full attention, 20B tokens). (a) Validation NLL relative to the vanilla model at the same step. (b–c) Mean copy and sink probabilities. With the gradient stopped, the CPLM keeps about a quarter of its improvement over the vanilla model and copies much less.

	Val. NLL	PPL	$\Delta$ NLL vs. Vanilla	$c_{t}$	$s_{t}$
Vanilla	2.794	16.34	—	—	—
CPLM	2.763	15.85	$-0.031$	0.210	0.567
CPLM, stop-gradient	2.786	16.22	$-0.008$	0.046	0.224

Appendix

Table 15: Final validation NLL (nats per token) and perplexity of the stop-gradient ablation, with the mean copy and sink probabilities.

Panel of Figure 18 — Figure 18: Byte-level language modeling (135M backbone, full attention, 20B bytes). (a) Validation NLL (nats per byte) relative to the vanilla model at the same step. (b–c) Mean copy and sink probabilities (sink for the CPLM with sink only). All three models reach the same loss within 0.002 nats per byte, even though the CPLM keeps a high copy probability throughout training.

	Val. NLL (nats/byte)	Bits/byte	$c_{t}$	$s_{t}$
Vanilla	0.6908	0.997	—	—
CPLM	0.6918	0.998	0.339	0.683
CPLM, no sink	0.6924	0.999	0.176	—

Appendix

Table 16: Final validation loss of byte-level models, with the mean copy and sink probabilities.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 422 papers

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Jul 17, 2026Haodong Wen, Yiran Zhang, Yingfa Chen +1Language Model Pretraining Rotary Positional Embeddings
Tsinghua University
Copying Under Repetition: From Prefix Matching to Position Hashing
Oct 19, 2026Anxin Guo, Tianlong Huang, Chenxiao Yang +1Transformer Repetition in Language Generation
Northwestern University · Duke University∗ · Toyota Technological Institute at Chicago
Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context
Sep 28, 2026Muyu He, Yuchen Liu, Ran Tao +1Memorization in Language Models LLM Interpretability
Independent · University of Pennsylvania · Drexel University

Jul 17, 2026J/K move · Enter open · S save

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu

Tsinghua University

Language Model Pretraining Rotary Positional Embeddings Memorization in Language Models Positional Encoding

Open paper AskPDF arXiv

While large language models (LLMs) can solve advanced reasoning problems in seconds, we show that even frontier models fail to perform a much simpler operation: exactly copying an input string that lies well within their context windows. We attribute this failure to positional encodings in Transformer architectures, whose inductive bias favors copying through a shortcut based on matching local contexts rather than carefully locating the corresponding input positions. To address this issue, we introduce 2D-RoPE, which organizes text into a 2D grid rather than a 1D sequence and assigns each token a row ID and a column ID. Under this view, copying becomes simply retrieving input tokens at a fixed column offset, which makes the task easy to learn. In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at input lengths hundreds of times longer than those seen during training, whereas standard positional encodings fall far behind. We further show that the advantage of 2D-RoPE language models on copy tasks consistently holds in large-scale pretraining on DCLM with model sizes up to 1.4B parameters. Overall, our results suggest that viewing text in 2D can benefit language modeling, and we hope this encourages future work to further explore the potential of 2D positional encodings.

Abstract

Language Model Pretraining

Explore similar work

Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D

Copying Under Repetition: From Prefix Matching to Position Hashing

Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Context