Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.
Figures & tables
Study
Length fixed, content varied
Visible neutral filler
Masked filler
Attention vs. loss, per trial
Du et al. [18]
no
whitespace
yes
no
Gao et al. [43]
yes
repeated text
no
head-level margins
Repantis et al. [44]
yes
no
no
no
Hong et al. [16]
partial
no
no
no
Ours
yes
three text types
yes, replicated
yes, 126K trials
Table 1: Controls in the closest prior work.
Figure 1: Retrieval accuracy against context length, with length and evidence position fixed. Neutral text left visible (green) costs almost nothing; the same text masked at preserved positions (blue) costs up to 47 points on Qwen2.5-1.5B; competing statements (orange) cost 56 to 87 points at every length. Mean over trials, pooled over the three neutral fillers; shaded bands are 95% bootstrap intervals.
L
8K
16K
24K
32K
Accuracy, visible
1.00
0.99
1.00
0.98
Accuracy, masked
0.93
0.87
0.73
0.52
Δmasked
0.03
0.11
0.44
1.20
Table 2: Qwen2.5-1.5B, neutral filler (numbers). Accuracy with the filler visible and with the same filler masked at preserved positions; median masked loss Δmasked in nats.
Model
No conflict
At least one
Qwen2.5-0.5B
0.175
1.812
Qwen2.5-1.5B
0.020
2.238
SmolLM2-1.7B
0.022
1.381
OLMo-2-1B
0.058
2.796
Llama-3.2-3B
0.344
3.239
Table 3: Mean loss (nats) by presence of a same-key conflict, L≤2,048 .
Figure 2: Loss Δ (nats) against log(1/a∗) for 4,500 trials of Qwen2.5-1.5B with competing filler at 8K to 32K tokens, colored by length. Black: mean loss in 15 quantile bins of log(1/a∗) . Lengths overlap completely, while attention orders the loss.
Model
n
R2(a∗)
R2(log(1/a∗))
R2(logL)
Slope
Qwen2.5-0.5B
32,318
0.23
0.35
0.03
1.40 [1.38, 1.42]
Qwen2.5-1.5B
40,632
0.43
0.58
0.05
2.22 [2.21, 2.24]
SmolLM2-1.7B
9,325
0.30
0.41
0.07
1.68 [1.64, 1.72]
OLMo-2-1B
42,005
0.26
0.35
0.02
3.97 [3.92, 4.02]
Llama-3.2-3B
2,193
0.36
0.45
0.08
2.75 [2.63, 2.88]
Table 4: Predictors of the loss, L∈{512,1024,2048} . Slope of Δ on log(1/a∗) with 95% CI.
Masked tokens
0
3,750
7,500
15,000
30,000
Llama-3.2-3B
51.7 [48.6, 54.8]
44.7 [41.6, 47.8]
42.4 [39.4, 45.5]
39.3 [36.3, 42.4]
40.5 [37.5, 43.6]
Qwen2.5-7B
65.8 [62.8, 68.7]
62.6 [59.6, 65.5]
54.5 [51.4, 57.6]
49.4 [46.3, 52.5]
48.1 [45.0, 51.2]
Table 5: MMLU accuracy (%) with masked filler, n=1,000 per cell, Wilson 95% intervals.
Large language models can process increasingly long prompts, yet their ability to locate and use decisive evidence may degrade as irrelevant or confusable context is added. We formulate this phenomenon, which we call context poisoning, as extreme-value interference in attention: the decisive-evidence score is upper-bounded, while the maximum score among effective distractors grows with their number. Under a softmax retrieval abstraction, we derive a finite-sample upper bound showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as Ω(logN), where N denotes the effective distractor count rather than necessarily the raw context length. The analysis connects long-context degradation to score aliasing, positional aliasing, and softmax dilution. Controlled experiments show that retrieval accuracy decreases as total context grows in the presence of embedded hard negatives, that the same-format condition produces the largest observed accuracy drop among the tested distractor constructions at fixed context length, and that retrieval gating can improve evidence use while its net benefit depends on preserving evidence recall. These results motivate evidence bottlenecks, alias-resistant representations, retrieve-then-reason architectures, verifier-mediated memory, and contrastive anti-poison training.
Language models (LMs) raise an intriguing alternative to vector-based retrieval: conditioning on an in-context corpus and directly generating a relevant answer. However, prior work has largely focused on proprietary systems or the smaller-scale reranking task, leaving corpus-scale in-context retrieval largely unexplored. In this work, we present the first systematic study of in-context retrieval on two scales practical retrievers demand: million-token corpora and length-generalization far beyond training-time sizes. We first introduce BlockSearch, a 0.6B LM retriever whose architectural and training modifications improve over prior LM baselines and length-generalize up to 10 times beyond its training regime. Nevertheless, retrieval still collapses under more extreme extrapolation. We trace this failure to an attention dilution effect: as the corpus grows, irrelevant documents dominate the softmax denominator, reducing the normalized mass on the gold document even when its pre-softmax score stays high. Motivated by this analysis, we introduce length-aware adjustments to the attention softmax and document-level sparse attention. With these modifications, at the million-token scale, our model matches dense retrieval on widely studied benchmarks (e.g, MS MARCO and NQ), while outperforming the concurrent model MSA despite being 7 times smaller. Furthermore, it significantly outperforms dense retrieval on tasks requiring entirely different notions of similarity, such as LIMIT, achieving a 3 times higher score. Together, our results position in-context retrieval a promising alternative to classical retrieval while emphasizing attention control under extreme context growth as a new challenge.
A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved. We test this claim by running every sample of two long-context benchmarks -- BABILong and GraphWalks (BFS) -- at four context-retention fractions (100%, 75%, 50%, 25%) under two truncation protocols. The first is the naive protocol implicitly used in much prior work: drop content from the middle of the prompt. The second is distractor-aware: identify the task-relevant content for each sample and drop only the rest. We evaluate three sizes of the Claude family (Haiku 4.5, Sonnet 4.6, Opus 4.7) and, to test cross-provider generality, GPT-5.5 from a different provider; we apply the same protocol to two further benchmarks (MRCR v2, Oolong). Under naive truncation, score collapses monotonically (paired Wilcoxon, Holm-corrected p_adj < 0.05 in all eight BABILong and GraphWalks cells). Under the distractor-aware protocol -- which preserves the signal by construction -- performance is preserved or improves: the two smaller Claude models show statistically significant gains on BABILong, while the larger models (Opus 4.7 and GPT-5.5) sit at their full-context ceiling. The naive collapse and its distractor-aware recovery replicate on GPT-5.5, ruling out a single-provider artifact. The mechanism is direct: under the naive protocol the answer-bearing content survives in fewer than 1% of samples at 25% retention; under the distractor-aware protocol it is preserved by construction. The naive protocol is therefore not a measurement of context-window effects; it is a measurement of how often middle-removal happens to spare the answer. We conclude that future studies of context-length effects must specify how they distinguish signal from distractor, or they are at best ambiguous between two opposite hypotheses.