Beyond Multiplicity Correction: Sharp Retrieval Thresholds for Attention with Correlated Distractors

Heehyun Park

cs.LGOct 26, 2026

Beyond Multiplicity Correction: Sharp Retrieval Thresholds for Attention with Correlated Distractors

Authors: Heehyun Park

Abstract

Redundant context affects attention in two distinct ways: multiplicity adds normalization mass, while score variation within correlated groups can produce extreme individual scores. We separate these effects in a planted retrieval model with exponentially many Gaussian distractors organized into supplied clusters whose number and size both grow exponentially with dimension. The model is a two-level generalized random energy model, whose classical free energy yields exact retrieval thresholds. Gaussian interpolation further shows that increasing positive correlation at fixed token count lowers the distractor free energy, unlike adding duplicates. Our main result establishes a sharp limitation of count correction, which subtracts the log cluster size from pre-softmax scores: this correction removes multiplicity but retains within-cluster extremes. Even when the inverse temperature is chosen after observing all scores, count correction requires a signal at least $\sqrt{2α_1}$ , where $α_1$ is the exponential growth rate of the number of clusters. By contrast, averaging scores before exponentiation suppresses token-level fluctuations and lowers the optimal signal threshold to $\sqrt{2ρα_1}$ , where $ρ$ is the within-cluster correlation. This gap persists after adding exponentially many singleton distractors, so group size alone does not identify the target, and extends, under explicit cumulant regularity assumptions, to fixed finite hierarchies of independent-increment scores. Finite-sample, stability, and participation results complement the asymptotic analysis. A controlled retrieval experiment with frozen sentence embeddings illustrates both sides of the distinction: averaging helps when the relevant item is a singleton, but can discard useful evidence when a relevant group contains multiple matching items.

Figures & tables

Context	Ordinary	Count	Average	Difference [95% interval]
One key per intent	0.379	0.379	0.379	$+0.000$ [ $+0.000$ , $+0.000$ ]
Exact copies (8)	0.288	0.379	0.379	$+0.000$ [ $+0.000$ , $+0.000$ ]
Same intent (4)	0.267	0.345	0.406	$+0.062$ [ $+0.056$ , $+0.067$ ]
Same intent (8)	0.234	0.342	0.440	$+0.098$ [ $+0.088$ , $+0.108$ ]
Four relevant keys	0.614	0.626	0.465	$-0.161$ [ $-0.181$ , $-0.142$ ]
Mixed relevant group	0.234	0.243	0.005	$-0.238$ [ $-0.269$ , $-0.208$ ]

Table 1. Mean relevant value mass on 924 evaluation queries. Each method uses its own validation-selected temperature. The last column gives the paired averaging-minus-count difference with a 95% intent-block bootstrap interval. Groups are supplied using dataset labels.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1. Analytical retrieval thresholds and finite-size Gaussian checks. Dashed lines in the simulation panels are infinite-size predictions; markers are empirical means with

1.96

standard errors. These are synthetic model checks.

Figure 2. Frozen sentence scores. A: evaluation temperature curves for eight-key distractor groups; dots mark independently validation-selected temperatures. B: relevant value mass after changing the relevant group, with 95% intent-block bootstrap intervals. The reversal for four relevant keys occurs even though all four carry the correct intent label.

Context	Method	$t$	$V$	Accuracy (%)
One key per intent	Ordinary	14.42	0.379	51.8
One key per intent	Count correction	14.42	0.379	51.8
One key per intent	Averaging	14.42	0.379	51.8
One key per intent	One representative	14.42	0.379	51.8
Exact copies (8)	Ordinary	17.75	0.288	36.3
Exact copies (8)	Count correction	14.42	0.379	51.8

Appendix

Table 2. Temperature

t

, mean relevant value mass

V

, and top-1 output-intent accuracy (%) at that temperature. Temperatures minimize validation negative log relevant value mass, not top-1 error.

Condition	$\tau_{\rm cnt}/\tau_{\rm avg}$	$\Delta$ relevant mass	$\Delta$ candidate prob.
All singleton	$1/1$	.00000	.00000
Repeated facts (4)	$1/1$	.00032 [.00030,.00035]	-.00215 [-.00253,-.00181]
Related (4), selected	$2/1$	-.00139 [-.00291,.00001]	-.02042 [-.03285,-.00946]
Related (4), native	$1/1$	.00018 [.00017,.00020]	-.00159 [-.00186,-.00133]
Four relevant	$1/1$	-.00083 [-.00093,-.00074]	.00054 [.00014,.00098]

Appendix

Table 3. Frozen DistilBERT layer-6 attention intervention. Means average three context seeds and two reversed orders within 80 queries; brackets are paired 95% bootstrap intervals.

\Delta

is averaging minus count correction.

Attention Mechanisms

Attention mechanisms let neural networks weight information from other tokens or regions according to relevance, enabling context-dependent processing.

See all 517 papers

May 11, 2026

The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning

As large language models are increasingly deployed in retrieval-augmented generation and agentic systems that accumulate extensive context, understanding how distracting information affects long-context performance becomes critical. Prior work shows that semantically relevant yet misleading documents degrade performance, but the quantitative relationship between the proportion of distractors and performance remains unstudied. In this work, we systematically vary the hard-distractor proportion in fixed-length contexts, revealing a striking nonlinear pattern: as the proportion of hard distractors increases, performance drops sharply within the first small fraction, while the remainder of the range yields only marginal additional decline. We term this ''The First Drop of Ink'' effect, analogous to how a single drop of ink contaminates water. Our theoretical and empirical analyses grounded in attention mechanics show that hard distractors capture disproportionate attention even at small proportions, with diminishing marginal impact as their proportion grows. Controlled experiments further show that filtering gains mainly come from context-length reduction rather than distractor removal; substantial recovery requires reducing the hard-distractor proportion to near zero, highlighting the importance of upstream retrieval precision.

Muhan Gao, Zih-Ching Chen, Kuan-Hao Huang

Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA · NVIDIA AI Technology Center, NVIDIA Corporation, Santa Clara, CA, USA

Long-Context Inference Language Model Robustness

Ask PDF

Oct 7, 2026

Finding the Right Balance: Relevance and Diversity in LLM Retrieval

Retrieval diversification is widely available in retrieval-augmented generation (RAG) frameworks, yet prior studies disagree on whether it improves retrieval and answer quality. We show that its effectiveness varies primarily with candidate-pool redundancy, in a pattern consistent with the number of distinct evidence pieces a query requires. Using controlled near-duplicate injection and production-style overlapping chunking, we find that diversification harms relevance, evidence coverage and answer quality on clean pools, but becomes beneficial on multi-evidence tasks when redundancy causes nearest-neighbor retrieval to select repeated passages. We therefore introduce a query-adaptive rule that diversifies only when the effective number of distinct documents in the nearest-neighbor top-

k

selection falls below the query's evidence requirement. Computed from existing embeddings, the rule captures most of the achievable gain, transfers across datasets and encoders and automatically reduces to nearest-neighbor retrieval for single-evidence queries. We also introduce RNG-Score, a geometric reranker with an exact nearest-neighbor fallback whose margin indicates duplicate structure. Overall, we conclude that diversification should be used selectively, based on observable redundancy and evidence requirements.

Guillaume Brouillette, Faustin Kagabo, Usef Faghihi +1

Université du Québec à Trois-Rivières, Trois-Rivières, Canada

Information Retrieval Retrieval-Augmented Generation Evidence Retrieval

Ask PDF

Jun 1, 2026

Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors

Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs. Yet annotation protocols may not purely reflect relevance: they select only a subset of documents for labeling, and this selection can favor certain document types over others. We investigate whether supervised bi-encoder retrievers implicitly learn a document-level relevance prior: a query-independent signal encoded in their representation space as a side effect of training on annotated data. We estimate this prior by training simple classifiers on frozen document embeddings and evaluate three state-of-the-art retrievers across multiple IR benchmarks. We find that supervised neural retrievers encode relevance priors that generalize to unseen documents and are consistent across models. These priors create a findability gap: documents with lower prior are systematically harder to retrieve, even when genuinely relevant. This effect appears in supervised dense retrievers but is weaker and less consistent in BM25, and it persists under controlled matched-document comparisons. Using LLM-based explanations, we find that judged-relevant documents tend to be comprehensive, self-contained summaries of mainstream topics, while niche, fragmentary, or highly technical content is often left unjudged. Retrievers internalize this bias, ranking documents with these favored features higher than documents that lack them, independently of their actual relevance. Our findings expose a structural limitation of supervised retrieval: models trained on annotated data do not just learn relevance, but also the implicit document preferences in their training data.

Francisco Valentini, Edgar Altszyler, Martin Fajcik

CONICET-Universidad de Buenos Aires. Instituto de Ciencias de la Computación (ICC). Buenos Aires, Argentina · Quantit, Buenos Aires, Argentina · Brno University of Technology, Brno, Czech Republic

IR Evaluation Dense Retrieval

Ask PDF

Abstract

Attention Mechanisms

Explore similar work

The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning

Finding the Right Balance: Relevance and Diversity in LLM Retrieval

Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors