Large language models can process increasingly long prompts, yet their ability to locate and use decisive evidence may degrade as irrelevant or confusable context is added. We formulate this phenomenon, which we call context poisoning, as extreme-value interference in attention: the decisive-evidence score is upper-bounded, while the maximum score among effective distractors grows with their number. Under a softmax retrieval abstraction, we derive a finite-sample upper bound showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as Ω(logN), where N denotes the effective distractor count rather than necessarily the raw context length. The analysis connects long-context degradation to score aliasing, positional aliasing, and softmax dilution. Controlled experiments show that retrieval accuracy decreases as total context grows in the presence of embedded hard negatives, that the same-format condition produces the largest observed accuracy drop among the tested distractor constructions at fixed context length, and that retrieval gating can improve evidence use while its net benefit depends on preserving evidence recall. These results motivate evidence bottlenecks, alias-resistant representations, retrieve-then-reason architectures, verifier-mediated memory, and contrastive anti-poison training.
Figures & tables
Figure 1: Score and positional aliasing in a long context. A relevant token x⋆ competes not with an average distractor but with the maximum of many distractors. As Neff grows, the right tail creates aliased distractors whose logits can dominate the softmax denominator.
Figure 2: Poison-resilient long-context architecture. A gate converts a raw O(N) context into K evidence cells, alias-resistant positional coding reduces near-ties, contrastive losses push distractors below evidence, and a verifier forces the decoder to reason over recited, auditable evidence rather than the full noisy context.
Mechanism
Bound-level effect
Implementation sketch
Evidence bottleneck
Replace N by K≪N while paying recall error η
Retrieve, recite, then solve; cite spans
Alias-resistant positions
Reduce probability that si≈s⋆
RoPE/ALiBi scaling plus margin regularization
Contrastive anti-poison loss
Increase γ and suppress maxiZi
Hard negatives, shuffled positions, adversarial distractors
Verifier-mediated memory
Reduce a0 failures and detect low-margin cases
Evidence table, consistency checks, abstention
Hierarchical memory
Bound per-level Neff
Chunk summaries with back-pointers to raw spans
Table 1: Mitigation mechanisms and their mathematical target.
Study
Model(s)
Primary intervention
Context regime
Diagnostic separation
Hard-negative decay
GPT-4o
Increase distractor count while retaining embedded decoys
N∈{100,300,500,1000} sentences
Manual audit verifies that errors select supplied decoy values
Distractor confusability
Claude Sonnet 4, GPT-4.1, Gemma 3 (12B)
Change topical, structural, and entity-name overlap at fixed N
300 records, approximately 20 – 21 K tokens
Answer accuracy and cited-record recall distinguish extraction from retrieval
Retrieve-then-reason gate
GPT-4.1
Replace all N chunks with BM25 top- K candidates
4 K– 512 K tokens
Recall@ K and accuracy conditional on retrieving all support
Table 2: Overview of the controlled studies. The first two studies are reported in the main paper; the gate intervention is reported in the appendix. The designs vary complementary axes rather than repeating the same manipulation.
N
Trials
Accuracy
Mean prompt length (chars)
100
10
100%
8,952
300
10
80%
25,978
500
10
80%
41,954
1000
10
80%
84,668
Table 3: Retrieval accuracy vs. number of distractors N , GPT-4o, 10 trials per condition.
Distractor condition
Claude Sonnet 4
GPT-4.1
Gemma 3 (12B)
Sim. proxy
Unrelated filler
1.000
1.000
1.000
0.004
Same domain, different attribute
1.000
1.000
1.000
0.005
Same format, different entity
0.889
0.967
∼ 0.85
0.010
Near-alias entity
1.000
0.983
∼ 0.89
0.317
Table 4: Answer accuracy by model and distractor construction at fixed context length ( N=300 records, ∼ 20K tokens, errors excluded). The rightmost column reports entity-name lexical overlap, measured by character-trigram Jaccard similarity to the gold entity. Gemma values are approximate due to a high endpoint error rate. Full results with evidence recall, valid counts, and paired contrasts are provided in the appendix ( Appendix C Distractor confusability ablation: full protocol and results ).
Figure 3: Distractor-confusability ablation at fixed context length ( N=300 ). Left: answer accuracy and evidence recall across four distractor constructions; all models show their largest decrease in the same-format condition, while the two frontier models recover toward baseline in the near-alias condition. Right: entity-name lexical-overlap check—the character-trigram Jaccard measure is near zero for the first three conditions and rises only in the near-alias condition. The accuracy decrease in the same-format condition therefore cannot be attributed to high entity-name overlap.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Distractor condition
Mean similarity proxy
Median tokens
Unrelated filler
0.004
20,508
Same domain, different attribute
0.005
20,149
Same format, different entity
0.010
21,055
Near-alias entity
0.317
21,053
Appendix
Table 5: Construct-validity and length-matching check: mean lexical-similarity proxy (character-trigram Jaccard between distractor and gold entity names) and median approximate token count by distractor condition, at Nrecords=300 .
Model
Total calls
Valid calls
Error rate
Primary cause
Claude Sonnet 4
240
155
35.4%
Rate limits / timeouts
GPT-4.1
240
240
0.0%
—
Gemma 3 (12B)
1,840
72
96.1%
Context-length limit
Appendix
Table 6: Endpoint reliability at ∼ 20K tokens. Accuracy analyses are computed on valid calls only.
Model
Distractor condition
Accuracy
Evidence recall
Valid n
Claude Sonnet 4
Unrelated filler
1.000
1.000
51
Claude Sonnet 4
Same domain, different attribute
1.000
1.000
36
Claude Sonnet 4
Same format, different entity
0.889
0.889
36
Claude Sonnet 4
Near-alias entity
1.000
1.000
32
GPT-4.1
Unrelated filler
1.000
1.000
60
GPT-4.1
Same domain, different attribute
1.000
1.000
60
Appendix
Table 7: Answer accuracy and evidence recall by model and distractor condition ( Nrecords=300 , ∼ 20K tokens, errors excluded). Gemma counts and values are approximate due to the 96.1% error rate and should be interpreted with caution.
Model
Comparison condition
Accuracy drop Δ
95% CI
Significant?
Claude Sonnet 4
Same domain, different attribute
0.000
[0.000,0.000]
No
Claude Sonnet 4
Same format, different entity
+0.111
[+0.028,+0.222]
Yes
Claude Sonnet 4
Near-alias entity
0.000
[0.000,0.000]
No
GPT-4.1
Same domain, different attribute
0.000
[0.000,0.000]
No
GPT-4.1
Same format, different entity
+0.033
[0.000,+0.083]
No
GPT-4.1
Near-alias entity
+0.017
[0.000,+0.050]
No
Appendix
Table 8: Paired accuracy drop relative to the unrelated-filler baseline (positive = harder; paired bootstrap, 5,000 iterations, errors excluded). Gemma is omitted: sparse matched pairs ( n=4 – 5 ) preclude reliable paired testing.
Figure 4: Mean attention by sentence position only, independent of content, pooled across trials/layers/heads.
N=20
N=50
N=100
Gaussianity (marginal distribution)
Shapiro–Wilk p
4.0×10−26
3.5×10−33
<10−300
Anderson–Darling stat.
205.9
1298.2
35768.7
Excess kurtosis (95% CI)
2.70 [2.35, 3.09]
6.57 [6.04, 7.03]
9.12 [8.83, 9.39]
Independence (lags 1–10)
Ljung–Box Q
3.74
11.50
490.60
Appendix
Table 9: Attention distribution tests by N (34 layers × 8 heads, 10 trials per N ).
Figure 5: Q–Q plots of the empirical maximum standardized distractor scores against the chosen asymptotic Gumbel approximation, by N .
Figure 6: Empirical mean of the standardized maximum distractor score vs. the theorem’s own asymptotic prediction, 2logN .
Gemma-3-4B-IT
Llama-3-8B-Instruct
Parameters
∼ 4B
∼ 8B
Layers
34
32
Attention heads
8
32
Attention pattern
Mixed: sliding-window (local) + global layers
Full causal (global) attention, every layer
Appendix
Table 10: Architectural comparison of the two models tested.
Figure 7: Attention-sink diagnostic on Llama-3-8B-Instruct (see Figure 4 for Gemma-3).
N=20
N=50
N=100
Gaussianity (marginal distribution)
Excess kurtosis (95% CI)
2.32 [2.01, 2.71]
0.98 [0.93, 1.03]
0.89 [0.86, 0.92]
Independence (lags 1–10)
Ljung–Box Q
3.92
25.29
104.63
Ljung–Box p
0.951
0.0048
<10−6
Extreme value (max statistic vs. Gumbel)
Appendix
Table 11: Attention distribution tests on Llama-3-8B-Instruct, by N (32 layers × 32 heads, 10 trials per N ).
Figure 8: Margin growth on Llama-3-8B-Instruct vs. the theorem’s asymptotic prediction (cf. Figure 6 for Gemma).
Figure 9: Retrieval accuracy (left) vs. excess kurtosis of the attention distribution (right), by N .
k
0
1
2
3
5
10
20
35
50
Accuracy
1.00
0.67
0.33
0.40
0.07
0.13
0.07
0.07
0.00
Appendix
Table 12: Retrieval accuracy versus the number k of confusable decoys, with Ndist=50 total distractors fixed.
Figure 10: Retrieval accuracy versus the number k of confusable decoys, with Ndist=50 total distractors fixed.
Figure 11: Accuracy vs. position of the gold document, N=20 , 40 real NQ questions.
N
10
20
30
A(N)
0.550
0.525
0.550
P(N)
0.000
0.025
0.000
Appendix
Table 13: Accuracy and poisoning curve P(N)=A(10)−A(N) , gold passage fixed at position 0.
Figure 12: Accuracy and P(N) vs. context length, gold passage fixed at position 0.
Ctx
Median raw N
K
Recall @ K
η^
Raw EM
Gate EM
ΔEM [95% CI]
Raw F1
Gate F1
Gate EM ∣h
Raw EM ∣h
Δuse
McNemar p
4K
29
16
0.94
0.06
0.660
0.640
−0.020 [ −0.070 , +0.030 ]
0.768
0.751
0.681
0.660
0.021
0.727
8K
58
16
0.94
0.06
0.650
0.620
−0.030 [ −0.080 , +0.020 ]
0.768
0.738
0.660
0.660
0.000
0.453
16K
115
16
0.94
0.06
0.680
0.630
−0.050 [ −0.110 , +0.010 ]
0.792
0.723
0.670
0.691
−0.021
0.180
32K
232
16
0.92
0.08
0.650
0.650
+0.000 [ −0.050 , +0.050 ]
0.782
0.757
0.696
0.674
0.022
1.000
128K
930
16
0.86
0.14
0.590
0.640
+0.050 [ −0.010 , +0.110 ]
0.724
0.746
0.709
0.640
0.070
0.227
256K
1857
16
0.82
0.18
0.590
0.580
−0.010 [ −0.090 , +0.070 ]
0.715
0.694
0.683
0.622
0.061
1.000
Appendix
Table 14: GPT-4.1 on RULER qa_2 /HotpotQA: raw long context versus a BM25 top- K retrieval gate with K=16 , frozen on a disjoint development split. Each length uses the same 100 questions at temperature 0 . The retrieval penalty is η^=1−Recall@K . ΔEM is the paired gated-minus-raw difference with a 10,000 -sample paired-bootstrap 95% confidence interval. Δuse is the gated-minus-raw difference restricted to instances for which all annotated supporting evidence was retrieved. McNemar p is the exact two-sided test.
Figure 13: GPT-4.1 with K=16 . (a) Raw accuracy is approximately stable through 32 K and then decreases with context length, while gated accuracy varies less. (b) A fixed K=16 meets the 0.90 all-support recall target through 32 K, after which recall decreases as the candidate set grows. (c) On instances for which the gate retrieves all annotated supporting evidence, raw accuracy decreases with context length while gated accuracy remains comparatively stable, providing evidence of improved evidence use after competing context is removed.
Ctx
K
Recall@ K
Raw EM
Gate EM
ΔEM [95% CI]
Δuse
McNemar p
128K
16
0.86
0.590
0.640
+0.050 [ −0.010 , +0.110 ]
0.070
0.227
64
0.94
0.590
0.620
+0.030 [ −0.020 , +0.080 ]
0.043
0.453
256K
16
0.82
0.590
0.580
−0.010 [ −0.090 , +0.070 ]
0.061
1.000
128
0.94
0.590
0.640
+0.050 [ −0.010 , +0.120 ]
0.064
0.227
512K
16
0.79
0.570
0.590
+0.020 [ −0.060 , +0.100 ]
0.114
0.815
192
0.93
0.570
0.640
+0.070 [ +0.000 , +0.140 ]
0.097
0.092
Appendix
Table 15: Scaling K to restore recall at extreme lengths. Only the gated arm is re-run; raw predictions are cached and unchanged. Each length-specific value of K is selected on the development split to achieve Recall@ K≥0.90 .
Figure 14: Comparison between fixed K=16 and a retrieval budget scaled to maintain all-support Recall@ K of at least 0.90 at extreme context lengths. Left: gated accuracy remains comparatively stable under the scaled retrieval budget while raw accuracy decreases. Middle: the paired gated-minus-raw difference becomes more positive after the retrieval penalty is reduced. Right: the scaled values of K maintain recall above the target where fixed K=16 does not.
Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.
Shiwen Qin, Cameron Barker, Henry Gouk +4
University of Edinburgh, Edinburgh, United Kingdom
A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved. We test this claim by running every sample of two long-context benchmarks -- BABILong and GraphWalks (BFS) -- at four context-retention fractions (100%, 75%, 50%, 25%) under two truncation protocols. The first is the naive protocol implicitly used in much prior work: drop content from the middle of the prompt. The second is distractor-aware: identify the task-relevant content for each sample and drop only the rest. We evaluate three sizes of the Claude family (Haiku 4.5, Sonnet 4.6, Opus 4.7) and, to test cross-provider generality, GPT-5.5 from a different provider; we apply the same protocol to two further benchmarks (MRCR v2, Oolong). Under naive truncation, score collapses monotonically (paired Wilcoxon, Holm-corrected p_adj < 0.05 in all eight BABILong and GraphWalks cells). Under the distractor-aware protocol -- which preserves the signal by construction -- performance is preserved or improves: the two smaller Claude models show statistically significant gains on BABILong, while the larger models (Opus 4.7 and GPT-5.5) sit at their full-context ceiling. The naive collapse and its distractor-aware recovery replicate on GPT-5.5, ruling out a single-provider artifact. The mechanism is direct: under the naive protocol the answer-bearing content survives in fewer than 1% of samples at 25% retention; under the distractor-aware protocol it is preserved by construction. The naive protocol is therefore not a measurement of context-window effects; it is a measurement of how often middle-removal happens to spare the answer. We conclude that future studies of context-length effects must specify how they distinguish signal from distractor, or they are at best ambiguous between two opposite hypotheses.
We propose RecaLLM, a set of reasoning language models post-trained to make effective use of long-context information. In-context retrieval, which identifies relevant evidence from context, and reasoning are deeply intertwined: retrieval supports reasoning, while reasoning often determines what must be retrieved. However, their interaction remains largely underexplored. In preliminary experiments on several open-source LLMs, we observe that in-context retrieval performance substantially degrades even after a short reasoning span, revealing a key bottleneck for test-time scaling that we refer to as lost-in-thought: reasoning steps that improve performance also make subsequent in-context retrieval more challenging. To address this limitation, RecaLLM interleaves reasoning with explicit in-context retrieval, alternating between reasoning and retrieving context information needed to solve intermediate subproblems. We introduce a negligible-overhead constrained decoding mechanism that enables verbatim copying of evidence spans, improving the grounding of subsequent generation. Trained on diverse lexical and semantic retrieval tasks, RecaLLM achieves strong performance on two long-context benchmarks, RULER and HELMET, significantly outperforming baselines. Notably, we observe consistent gains at context windows of up to 128K tokens using training samples of at most 10K tokens, far shorter than those used by existing long-context approaches, highlighting a promising path toward improving long-context performance without expensive long-context training data.