Why do language models fail on long inputs? The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33. A single statement that gives the queried key a rival value multiplies the loss by 9 to 110, and every additional rival adds to it. Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials it explains up to 58% of the variance in loss; log length explains at most 8%. The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval: masking costs 45 accuracy points at 32K tokens, while the same filler left visible costs none. In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.
Figures & tables
Study
Length fixed, content varied
Visible neutral filler
Masked filler
Attention vs. loss, per trial
Du et al. [18]
no
whitespace
yes
no
Gao et al. [43]
yes
repeated text
no
head-level margins
Repantis et al. [44]
yes
no
no
no
Hong et al. [16]
partial
no
no
no
Ours
yes
three text types
yes, replicated
yes, 126K trials
Table 1: Controls in the closest prior work.
Figure 1: Retrieval accuracy against context length, with length and evidence position fixed. Neutral text left visible (green) costs almost nothing; the same text masked at preserved positions (blue) costs up to 47 points on Qwen2.5-1.5B; competing statements (orange) cost 56 to 87 points at every length. Mean over trials, pooled over the three neutral fillers; shaded bands are 95% bootstrap intervals.
L
8K
16K
24K
32K
Accuracy, visible
1.00
0.99
1.00
0.98
Accuracy, masked
0.93
0.87
0.73
0.52
Δmasked
0.03
0.11
0.44
1.20
Table 2: Qwen2.5-1.5B, neutral filler (numbers). Accuracy with the filler visible and with the same filler masked at preserved positions; median masked loss Δmasked in nats.
Model
No conflict
At least one
Qwen2.5-0.5B
0.175
1.812
Qwen2.5-1.5B
0.020
2.238
SmolLM2-1.7B
0.022
1.381
OLMo-2-1B
0.058
2.796
Llama-3.2-3B
0.344
3.239
Table 3: Mean loss (nats) by presence of a same-key conflict, L≤2,048 .
Figure 2: Loss Δ (nats) against log(1/a∗) for 4,500 trials of Qwen2.5-1.5B with competing filler at 8K to 32K tokens, colored by length. Black: mean loss in 15 quantile bins of log(1/a∗) . Lengths overlap completely, while attention orders the loss.
Model
n
R2(a∗)
R2(log(1/a∗))
R2(logL)
Slope
Qwen2.5-0.5B
32,318
0.23
0.35
0.03
1.40 [1.38, 1.42]
Qwen2.5-1.5B
40,632
0.43
0.58
0.05
2.22 [2.21, 2.24]
SmolLM2-1.7B
9,325
0.30
0.41
0.07
1.68 [1.64, 1.72]
OLMo-2-1B
42,005
0.26
0.35
0.02
3.97 [3.92, 4.02]
Llama-3.2-3B
2,193
0.36
0.45
0.08
2.75 [2.63, 2.88]
Table 4: Predictors of the loss, L∈{512,1024,2048} . Slope of Δ on log(1/a∗) with 95% CI.
Masked tokens
0
3,750
7,500
15,000
30,000
Llama-3.2-3B
51.7 [48.6, 54.8]
44.7 [41.6, 47.8]
42.4 [39.4, 45.5]
39.3 [36.3, 42.4]
40.5 [37.5, 43.6]
Qwen2.5-7B
65.8 [62.8, 68.7]
62.6 [59.6, 65.5]
54.5 [51.4, 57.6]
49.4 [46.3, 52.5]
48.1 [45.0, 51.2]
Table 5: MMLU accuracy (%) with masked filler, n=1,000 per cell, Wilson 95% intervals.