cs.CLOct 20, 2026

When Copying Is Hard: Copy-Constrained Decoding for Exact Span Reproduction

Authors: Jinghui Zhang, Lang Gao, Zongfang Liu, Ruihong Zeng, Zirui Song, Rui Yan, Kentaro Inui, Xiuying Chen

Organizations: MBZUAI · Zhejiang University · Westlake University · Wuhan University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Precisely reproducing a target span verbatim within a long context is a fundamental capability test for LLMs: even when the target string is present in context, models often still drift within the span or fail to stop correctly, revealing limitations in source grounding and boundary control. This capability has broad practical relevance. For instance, code generation often requires embedding sensitive strings such as API keys exactly as they appear in context. In this work, we construct a benchmark to test exact span reproduction ability across contracts, scientific literature, code, and random sequences, and find that current models still make surprisingly frequent errors. To address this, we propose CopyGen, a lightweight copy-constrained plugin for frozen LLMs that turns copying from a byproduct of next-token prediction into a controlled process. Our key idea is to decompose copying into three distinct decisions: when to copy, how far to copy, and when to stop. Concretely, CopyGen first determines whether to enter copy mode, then predicts a tentative copy length, and finally verifies the generated span token by token. During verification, we adaptively adjust the logits of copy continuation tokens to incentivize or suppress continuation, and emit only the longest valid prefix. Across three LLM backbones and four benchmark tasks, CopyGen improves average exact match by 35% relative without model finetuning, and integrates seamlessly with SFT to further improve it by 4%, while substantially accelerating generation. These results suggest that reliable exact copying is not something LLMs obtain for free, but a decoding-time control problem that requires explicit modeling. Code is available at https://github.com/mbzuai-nlp/CopyGen.

Figures & tables

Figure 1: Overview of CopyGen architecture. The system operates in three stages: (1) Copy Triggering: determining when to enter copy mode based on grounding length mtm_{t} and a learned trigger head; (2) Span Proposal: predicting an adaptive copy length ℓ^t\hat{\ell}_{t} in log space; and (3) Proposal Verification: calibrating token margins to decide the final emitted prefix At(s)A_{t}^{(s)} . This design supports exact copying while maintaining the flexibility of the base LM.
Figure 2: Source-margin behavior for Qwen3-8B. (a) Correct copy tokens typically have large positive source margins, while mismatch and overrun cases lie close to the decision boundary. (b) Around copy-stop positions, the raw source margin stays positive within the copied span and drops sharply at the boundary token. (c) At the span level, in-span tokens tend to have much higher source margins than the boundary token.
Figure 3: Throughput speedup on real-world tasks.
Figure 4: Copy proposal dynamics and calibration behavior. (a) Proposed copy spans are often longer than the spans finally accepted by the verifier. (b) Token acceptance remains high for medium-length proposals but drops for very long proposals, indicating conservative truncation near uncertain boundaries. (c) Calibration bias suppresses boundary overruns while boosting internally fragile copy positions, helping preserve exact in-span reproduction. Results shown for Qwen3-8B.
Figure 5: Calibrated decoding trace case study.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Fixed copy length analysis. Left: speed–accuracy tradeoff between fixed-length copying and CopyGen’s adaptive span proposal, shown separately for each task. CopyGen is close to the best fixed-length tradeoffs, preserving the accuracy of shorter fixed spans while approaching the throughput of longer fixed spans. Right: fixed-length sweep over kk , where solid lines denote exact match and dashed lines denote throughput. Small fixed lengths preserve accuracy but limit throughput, while large fixed lengths improve throughput but can cross copy boundaries and hurt accuracy.
DomainTask# Train# Test# Total
ContractLegal Clause Extraction658141799
LiteratureScientific Evidence Extraction712145857
CodeCode Explanation8212291050
Random SequenceExact Long-String Reproduction22502502500
Appendix
Table 14: Dataset statistics for each domain and task.
Figure 7: Copy dynamics for Qwen3-4B across four benchmark domains. For each domain, we show the distribution of proposed and accepted copy lengths (left), together with token acceptance by proposed-length bucket (right).
Figure 8: Copy dynamics for Qwen3-8B across four benchmark domains. For each domain, we show the distribution of proposed and accepted copy lengths (left), together with token acceptance by proposed-length bucket (right).
Figure 9: Copy dynamics for Ministral-8B across four benchmark domains. For each domain, we show the distribution of proposed and accepted copy lengths (left), together with token acceptance by proposed-length bucket (right).
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18
More in this topic

Language Model Decoding

Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.

See all 168 papers

Explore similar work

CardsList
Apr 20, 2026

Copy-as-Decode: Grammar-Constrained Parallel Prefill for LLM Editing

LLMs edit text and code by autoregressively regenerating the full output, even when most tokens appear verbatim in the input. We study Copy-as-Decode, a decoding-layer mechanism that recasts edit generation as structured decoding over a two-primitive grammar: <copy lines="i-j"/> references an input line range, <gen>...</gen> emits new content. A token-level FSM guarantees syntactic validity, and a serving-layer primitive updates the KV cache for each copy span via a single parallel-prefill forward rather than NN autoregressive steps -- sharing the parallel-forward kernel of speculative decoding but with input tokens as the draft and program-enforced acceptance replacing probabilistic verification. We report an upper-bound analysis that requires no end-to-end training. (i) Kernel speedup: on Qwen2.5-{1.5B, 7B}, copying NN tokens via parallel prefill is 6.8×6.8\times--303×303\times faster than autoregressive (N∈[8,512]N \in [8, 512], A100 80GB bf16). (ii) Copy ceiling: on ProbeEdit and HumanEvalPack-Fix (Py/JS), 7474--98%98\% of gold tokens are reachable under the line-level primitive; composed with the empirical kernel over each corpus's span histogram this yields a closed-form wall-clock bound of 29.0×/3.4×/4.2×29.0\times / 3.4\times / 4.2\times (13.0×13.0\times pooled). A token-level extension reaches 9191--99%99\% coverage with 4.5×4.5\times--6.5×6.5\times floors. (iii) Pipeline losslessness: oracle programs round-trip through the deterministic resolver on all 482482 cases, localizing any downstream failure to span selection rather than the mechanism. A perturbation study shows pooled EM drops from 100%100\% to 15.48%15.48\% under off-by-one noise. A fine-tuning pilot on Qwen2.5-Coder-1.5B lifts HEvalFix-Py EM from 0/330/33 (untrained) to 1212--17%17\%, a learnability signal, not a production selector. Batched-serving integration and multi-file coverage are scoped as follow-up.

Ziyang Liu

Independent Researcher

LLM Inference AccelerationParallel DecodingLanguage Model EditingLLM Code GenerationGrammar-Constrained Decoding
AskPDF
Oct 19, 2026

Copying Under Repetition: From Prefix Matching to Position Hashing

Frontier LLMs can copy random text almost flawlessly, yet they often overrun or truncate long blocks of repeated characters. In repeated strings, matching local content no longer reliably identifies where copying should continue or when it should stop. Nevertheless, we show that a two-layer transformer with O(log⁡log⁡N)O(\log\log N)-bit precision and O(log⁡N/log⁡log⁡N)O(\log N/\log\log N) width can copy every string up to a chosen design horizon NN and terminate correctly under arbitrary repetition. We use RoPE-style rotations of attention values and outputs (VO-rotations) with small prime periods to encode distances from delimiter anchors modulo those primes. By the Chinese Remainder Theorem, these periodic encodings jointly distinguish all required distances, allowing one retrieval rule to select either the next source token or the end marker. In contrast, we prove a width-independent impossibility result for copying under sequential finite-precision softmax accumulation. It applies to transformers whose only positional input is an additive attention bias that eventually becomes constant, including NoPE and a recursive implementation of ALiBi. Once an identical-token run exceeds the resulting saturation threshold, further repetitions leave the logits unchanged. If this threshold grows sublinearly with NN, these models cannot both complete a long all-identical copy and stop correctly, a pattern resembling the termination errors of frontier LLMs. In synthetic copying experiments, baseline transformer models fail on long repeated strings even within the training range. Trained VO models, by contrast, generalize beyond the training lengths with near-perfect accuracy up to length ≈5005\approx5005, the selected primes' period product.

Anxin Guo, Tianlong Huang, Chenxiao Yang +1

Northwestern University · Duke University · Toyota Technological Institute at Chicago

TransformerRepetition in Language GenerationPositional EncodingTransformer Expressivity
AskPDF
Sep 17, 2026

To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals

Speculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: neural drafting and context-based copying. Neural drafts (e.g., EAGLE3) provide robust performance across diverse text settings, while copy-based methods achieve higher speedups in copy-intensive regimes by generating candidates faster and exploiting long repetition spans for near-perfect speculation. We analyze existing copy-based methods and find that they are prone to accidental repetitions where surface-level n-gram overlap does not reflect a structural intent to copy, leading to false-positive triggers that ultimately degrade throughput. We introduce SwitchSD, an adaptive framework that treats copying as a latent control signal of the LLM. By training lightweight probes on the target model's internal representations, SwitchSD identifies genuine copy-intent with high precision (AUC > 0.99). This allows the system to dynamically switch between neural drafting (e.g., EAGLE) and context-based copying. Our results across Llama and Qwen families demonstrate throughput gains of up to 15% over state-of-the-art baselines like EAGLE3, effectively turning copying from a noisy heuristic into a principled, model-aware decoding regime.

Roy Eisenstadt, Ido Cohen, Edo Cohen-Karlik +2

Tel Aviv University · Stealth Startup, Tel Aviv

Speculative DecodingLLM Inference Acceleration
AskPDF