Context Poisoning as Extreme-Value Attention Interference in Long-Context Language Models

Meysam Ghaffari; Nina Fatehi; Bhaskar Sen; Nasim Sabetpour; Carlos Morato

cs.CLAug 12, 2026

Context Poisoning as Extreme-Value Attention Interference in Long-Context Language Models

Authors: Meysam Ghaffari, Nina Fatehi, Bhaskar Sen, Nasim Sabetpour, Carlos Morato

Organizations: Optum, UHG, Minneapolis, MN

Abstract

Large language models can process increasingly long prompts, yet their ability to locate and use decisive evidence may degrade as irrelevant or confusable context is added. We formulate this phenomenon, which we call context poisoning, as extreme-value interference in attention: the decisive-evidence score is upper-bounded, while the maximum score among effective distractors grows with their number. Under a softmax retrieval abstraction, we derive a finite-sample upper bound showing that maintaining a fixed accuracy target above base rate requires the evidence margin to scale as $Ω(\sqrt{\log N})$ , where N denotes the effective distractor count rather than necessarily the raw context length. The analysis connects long-context degradation to score aliasing, positional aliasing, and softmax dilution. Controlled experiments show that retrieval accuracy decreases as total context grows in the presence of embedded hard negatives, that the same-format condition produces the largest observed accuracy drop among the tested distractor constructions at fixed context length, and that retrieval gating can improve evidence use while its net benefit depends on preserving evidence recall. These results motivate evidence bottlenecks, alias-resistant representations, retrieve-then-reason architectures, verifier-mediated memory, and contrastive anti-poison training.

Figures & tables

Panel of Figure 1 — Figure 1: Score and positional aliasing in a long context. A relevant token $x_{\star}$ competes not with an average distractor but with the maximum of many distractors. As $N_{\rm eff}$ grows, the right tail creates aliased distractors whose logits can dominate the softmax denominator.

Panel of Figure 2 — Figure 2: Poison-resilient long-context architecture. A gate converts a raw $O(N)$ context into $K$ evidence cells, alias-resistant positional coding reduces near-ties, contrastive losses push distractors below evidence, and a verifier forces the decoder to reason over recited, auditable evidence rather than the full noisy context.

Mechanism	Bound-level effect	Implementation sketch
Evidence bottleneck	Replace $N$ by $K\ll N$ while paying recall error $\eta$	Retrieve, recite, then solve; cite spans
Alias-resistant positions	Reduce probability that $s_{i}\approx s_{\star}$	RoPE/ALiBi scaling plus margin regularization
Contrastive anti-poison loss	Increase $\gamma$ and suppress $\max_{i}Z_{i}$	Hard negatives, shuffled positions, adversarial distractors
Verifier-mediated memory	Reduce $a_{0}$ failures and detect low-margin cases	Evidence table, consistency checks, abstention
Hierarchical memory	Bound per-level $N_{\rm eff}$	Chunk summaries with back-pointers to raw spans

Table 1: Mitigation mechanisms and their mathematical target.

Study	Model(s)	Primary intervention	Context regime	Diagnostic separation
Hard-negative decay	GPT-4o	Increase distractor count while retaining embedded decoys	$N\in\{100,300,500,1000\}$ sentences	Manual audit verifies that errors select supplied decoy values
Distractor confusability	Claude Sonnet 4, GPT-4.1, Gemma 3 (12B)	Change topical, structural, and entity-name overlap at fixed $N$	$300$ records, approximately $20$ – $21$ K tokens	Answer accuracy and cited-record recall distinguish extraction from retrieval
Retrieve-then-reason gate	GPT-4.1	Replace all $N$ chunks with BM25 top- $K$ candidates	$4$ K– $512$ K tokens	Recall@ $K$ and accuracy conditional on retrieving all support

Table 2: Overview of the controlled studies. The first two studies are reported in the main paper; the gate intervention is reported in the appendix. The designs vary complementary axes rather than repeating the same manipulation.

$N$	Trials	Accuracy	Mean prompt length (chars)
100	10	100%	8,952
300	10	80%	25,978
500	10	80%	41,954
1000	10	80%	84,668

Table 3: Retrieval accuracy vs. number of distractors

N

, GPT-4o, 10 trials per condition.

Distractor condition	Claude Sonnet 4	GPT-4.1	Gemma 3 (12B)	Sim. proxy
Unrelated filler	1.000	1.000	1.000	0.004
Same domain, different attribute	1.000	1.000	1.000	0.005
Same format, different entity	0.889	0.967	$\sim$ 0.85	0.010
Near-alias entity	1.000	0.983	$\sim$ 0.89	0.317

Table 4: Answer accuracy by model and distractor construction at fixed context length (

N=300

records,

\sim

20K tokens, errors excluded). The rightmost column reports entity-name lexical overlap, measured by character-trigram Jaccard similarity to the gold entity. Gemma values are approximate due to a high endpoint error rate. Full results with evidence recall, valid counts, and paired contrasts are provided in the appendix ( Appendix C Distractor confusability ablation: full protocol and results ).

Figure 3: Distractor-confusability ablation at fixed context length ( $N=300$ ). Left: answer accuracy and evidence recall across four distractor constructions; all models show their largest decrease in the same-format condition, while the two frontier models recover toward baseline in the near-alias condition. Right: entity-name lexical-overlap check—the character-trigram Jaccard measure is near zero for the first three conditions and rises only in the near-alias condition. The accuracy decrease in the same-format condition therefore cannot be attributed to high entity-name overlap.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Distractor condition	Mean similarity proxy	Median tokens
Unrelated filler	0.004	20,508
Same domain, different attribute	0.005	20,149
Same format, different entity	0.010	21,055
Near-alias entity	0.317	21,053

Appendix

Table 5: Construct-validity and length-matching check: mean lexical-similarity proxy (character-trigram Jaccard between distractor and gold entity names) and median approximate token count by distractor condition, at

N_{\mathrm{records}}=300

Model	Total calls	Valid calls	Error rate	Primary cause
Claude Sonnet 4	240	155	35.4%	Rate limits / timeouts
GPT-4.1	240	240	0.0%	—
Gemma 3 (12B)	1,840	72	96.1%	Context-length limit

Appendix

Table 6: Endpoint reliability at

\sim

20K tokens. Accuracy analyses are computed on valid calls only.

Model	Distractor condition	Accuracy	Evidence recall	Valid $n$
Claude Sonnet 4	Unrelated filler	1.000	1.000	51
Claude Sonnet 4	Same domain, different attribute	1.000	1.000	36
Claude Sonnet 4	Same format, different entity	0.889	0.889	36
Claude Sonnet 4	Near-alias entity	1.000	1.000	32
GPT-4.1	Unrelated filler	1.000	1.000	60
GPT-4.1	Same domain, different attribute	1.000	1.000	60

Appendix

Table 7: Answer accuracy and evidence recall by model and distractor condition (

N_{\mathrm{records}}=300

\sim

20K tokens, errors excluded). Gemma counts and values are approximate due to the

96.1\%

error rate and should be interpreted with caution.

Model	Comparison condition	Accuracy drop $\Delta$	95% CI	Significant?
Claude Sonnet 4	Same domain, different attribute	$0.000$	$[0.000,0.000]$	No
Claude Sonnet 4	Same format, different entity	$+0.111$	$[+0.028,+0.222]$	Yes
Claude Sonnet 4	Near-alias entity	$0.000$	$[0.000,0.000]$	No
GPT-4.1	Same domain, different attribute	$0.000$	$[0.000,0.000]$	No
GPT-4.1	Same format, different entity	$+0.033$	$[0.000,+0.083]$	No
GPT-4.1	Near-alias entity	$+0.017$	$[0.000,+0.050]$	No

Appendix

Table 8: Paired accuracy drop relative to the unrelated-filler baseline (positive

=

harder; paired bootstrap,

5{,}000

iterations, errors excluded). Gemma is omitted: sparse matched pairs (

n=4

–

5

) preclude reliable paired testing.

Figure 4: Mean attention by sentence position only, independent of content, pooled across trials/layers/heads.

Gaussianity (marginal distribution)
	$N=20$	$N=50$	$N=100$
Shapiro–Wilk $p$	$4.0\times 10^{-26}$	$3.5\times 10^{-33}$	$<10^{-300}$
Anderson–Darling stat.	205.9	1298.2	35768.7
Excess kurtosis (95% CI)	2.70 [2.35, 3.09]	6.57 [6.04, 7.03]	9.12 [8.83, 9.39]
Independence (lags 1–10)
Ljung–Box $Q$	3.74	11.50	490.60

Appendix

Table 9: Attention distribution tests by

N

(34 layers

\times

8 heads, 10 trials per

N

Figure 5: Q–Q plots of the empirical maximum standardized distractor scores against the chosen asymptotic Gumbel approximation, by $N$ .

Figure 6: Empirical mean of the standardized maximum distractor score vs. the theorem’s own asymptotic prediction, $\sqrt{2\log N}$ .

	Gemma-3-4B-IT	Llama-3-8B-Instruct
Parameters	$\sim$ 4B	$\sim$ 8B
Layers	34	32
Attention heads	8	32
Attention pattern	Mixed: sliding-window (local) + global layers	Full causal (global) attention, every layer

Appendix

Table 10: Architectural comparison of the two models tested.

Figure 7: Attention-sink diagnostic on Llama-3-8B-Instruct (see Figure 4 for Gemma-3).

	$N=20$	$N=50$	$N=100$
Gaussianity (marginal distribution)
Excess kurtosis (95% CI)	2.32 [2.01, 2.71]	0.98 [0.93, 1.03]	0.89 [0.86, 0.92]
Independence (lags 1–10)
Ljung–Box $Q$	3.92	25.29	104.63
Ljung–Box $p$	0.951	0.0048	$<10^{-6}$
Extreme value (max statistic vs. Gumbel)

Appendix

Table 11: Attention distribution tests on Llama-3-8B-Instruct, by

N

(32 layers

\times

32 heads, 10 trials per

N

Figure 8: Margin growth on Llama-3-8B-Instruct vs. the theorem’s asymptotic prediction (cf. Figure 6 for Gemma).

Figure 9: Retrieval accuracy (left) vs. excess kurtosis of the attention distribution (right), by $N$ .

$k$	0	1	2	3	5	10	20	35	50
Accuracy	1.00	0.67	0.33	0.40	0.07	0.13	0.07	0.07	0.00

Appendix

Table 12: Retrieval accuracy versus the number

k

of confusable decoys, with

N_{\mathrm{dist}}=50

total distractors fixed.

Figure 11: Accuracy vs. position of the gold document, $N=20$ , 40 real NQ questions.

$N$	10	20	30
$A(N)$	0.550	0.525	0.550
$P(N)$	0.000	0.025	0.000

Appendix

Table 13: Accuracy and poisoning curve

P(N)=A(10)-A(N)

, gold passage fixed at position 0.

Ctx	Median raw $N$	$K$	Recall @ $K$	$\hat{\eta}$	Raw EM	Gate EM	$\Delta\mathrm{EM}$ [95% CI]	Raw F1	Gate F1	Gate EM $\mid h$	Raw EM $\mid h$	$\Delta_{\mathrm{use}}$	McNemar $p$
4K	29	16	0.94	0.06	0.660	0.640	$-0.020$ [ $-0.070$ , $+0.030$ ]	0.768	0.751	0.681	0.660	0.021	0.727
8K	58	16	0.94	0.06	0.650	0.620	$-0.030$ [ $-0.080$ , $+0.020$ ]	0.768	0.738	0.660	0.660	0.000	0.453
16K	115	16	0.94	0.06	0.680	0.630	$-0.050$ [ $-0.110$ , $+0.010$ ]	0.792	0.723	0.670	0.691	$-0.021$	0.180
32K	232	16	0.92	0.08	0.650	0.650	$+0.000$ [ $-0.050$ , $+0.050$ ]	0.782	0.757	0.696	0.674	0.022	1.000
128K	930	16	0.86	0.14	0.590	0.640	$+0.050$ [ $-0.010$ , $+0.110$ ]	0.724	0.746	0.709	0.640	0.070	0.227
256K	1857	16	0.82	0.18	0.590	0.580	$-0.010$ [ $-0.090$ , $+0.070$ ]	0.715	0.694	0.683	0.622	0.061	1.000

Appendix

Table 14: GPT-4.1 on RULER qa_2 /HotpotQA: raw long context versus a BM25 top-

K

retrieval gate with

K=16

, frozen on a disjoint development split. Each length uses the same

100

questions at temperature

0

. The retrieval penalty is

\hat{\eta}=1-\text{Recall@}K

\Delta\mathrm{EM}

is the paired gated-minus-raw difference with a

10{,}000

-sample paired-bootstrap

95\%

confidence interval.

\Delta_{\mathrm{use}}

is the gated-minus-raw difference restricted to instances for which all annotated supporting evidence was retrieved. McNemar

p

is the exact two-sided test.

Figure 13: GPT-4.1 with $K=16$ . (a) Raw accuracy is approximately stable through $32$ K and then decreases with context length, while gated accuracy varies less. (b) A fixed $K=16$ meets the $0.90$ all-support recall target through $32$ K, after which recall decreases as the candidate set grows. (c) On instances for which the gate retrieves all annotated supporting evidence, raw accuracy decreases with context length while gated accuracy remains comparatively stable, providing evidence of improved evidence use after competing context is removed.

Ctx	$K$	Recall@ $K$	Raw EM	Gate EM	$\Delta\mathrm{EM}$ [95% CI]	$\Delta_{\mathrm{use}}$	McNemar $p$
128K	16	0.86	0.590	0.640	$+0.050$ [ $-0.010$ , $+0.110$ ]	0.070	0.227
	64	0.94	0.590	0.620	$+0.030$ [ $-0.020$ , $+0.080$ ]	0.043	0.453
256K	16	0.82	0.590	0.580	$-0.010$ [ $-0.090$ , $+0.070$ ]	0.061	1.000
	128	0.94	0.590	0.640	$+0.050$ [ $-0.010$ , $+0.120$ ]	0.064	0.227
512K	16	0.79	0.570	0.590	$+0.020$ [ $-0.060$ , $+0.100$ ]	0.114	0.815
	192	0.93	0.570	0.640	$+0.070$ [ $+0.000$ , $+0.140$ ]	0.097	0.092

Appendix

Table 15: Scaling

K

to restore recall at extreme lengths. Only the gated arm is re-run; raw predictions are cached and unchanged. Each length-specific value of

K

is selected on the development split to achieve Recall@

K\geq 0.90

Figure 14: Comparison between fixed $K=16$ and a retrieval budget scaled to maintain all-support Recall@ $K$ of at least $0.90$ at extreme context lengths. Left: gated accuracy remains comparatively stable under the scaled retrieval budget while raw accuracy decreases. Middle: the paired gated-minus-raw difference becomes more positive after the retrieval penalty is reduced. Right: the scaled values of $K$ maintain recall above the target where fixed $K=16$ does not.

Long-Context Language Model Evaluation

Long-context language model evaluation measures how reliably models find, reason over, and use information across extended inputs.

See all 68 papers

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
Oct 15, 2026Shiwen Qin, Cameron Barker, Henry Gouk +4Long-Context Reasoning in Language Models Long-Context Language Model Evaluation
University of Edinburgh, Edinburgh, United Kingdom
Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks
Aug 4, 2026Mohsen ArjmandiLong-Context Language Model Evaluation Benchmark Design
evolutionID GmbH
RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval
Apr 10, 2026Kyle Whitecross, Negin RahimiRetrieval-Augmented Reasoning Long-Context Reasoning in Language Models
University of Massachusetts Amherst

Oct 15, 2026J/K move · Enter open · S save

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley+3

University of Edinburgh, Edinburgh, United Kingdom

Long-Context Reasoning in Language Models Long-Context Language Model Evaluation Long-Context Inference Long-Context QA

Open paper AskPDF arXiv

Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.

Abstract

Long-Context Language Model Evaluation

Explore similar work

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

RecaLLM: Addressing the Lost-in-Thought Phenomenon with Explicit In-Context Retrieval