Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval

Hang Gao; Wujiang Xu; Kai Mei; Dimitris N. Metaxas

cs.CLMar 22, 2026

Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval

Authors: Hang Gao, Wujiang Xu, Kai Mei, Dimitris N. Metaxas

Organizations: Rutgers University

Abstract

Transformer-based embedding models frequently exhibit geometric pathologies, such as anisotropy and length-induced representation collapse, which can degrade downstream retrieval performance. While prior work often attributes these issues directly to text length or attention mechanisms, we argue that the fundamental drivers are instead the inherent pooling operations coupled with internal semantic shift. In this paper, we establish a unified theoretical framework proving that contextual pooling intrinsically causes embedding collapse. Specifically, we mathematically prove that pooling semantically diverse sentences inevitably leads to micro-level semantic dilution, and strictly reduces the Mean Pairwise Distance of the vector space, guaranteeing macro-level spatial concentration. Grounded in these geometric insights, we formally define semantic shift to capture the natural semantic evolution and dispersion within a text. Through carefully controlled experiments across diverse models and corpora, we disentangle text length from semantic content. We demonstrate that semantic shift is the primary predictor of severe embedding concentration. Crucially, our retrieval evaluations reveal that anisotropy is fundamentally harmful only when induced by strong semantic shifts, reconciling conflicting observations in prior literature and offering a principled explanation for the long-context challenges faced by modern embedding models.

Figures & tables

Figure 1: Mean Pairwise Distance (MPD) curves for three embedding models across two corpora. The

x

-axis is the number of sentences; the

y

-axis is MPD.

Figure 2: Scatter plot of

C_{\mathrm{mean}}

vs.

C_{\mathrm{pair}}

on ArXiv using bge-large model.

Figure 3: Spatial distribution of

n=100

vectors before (Original) and after randomized pooling with varying scales

k

. As

k

increases, vectors undergo severe norm decay and angular contraction toward the global mean, visually confirming the strict monotonic MPD reduction proved in Theorem 2 .

Figure 4: Variation of MPD under different sentence concatenation patterns across two corpora.

Figure 5: Mean semantic shift increases with hop distance across two corpora under different concatenation modes. The

x

-axis is the hop distance; the

y

-axis is mean semantic shift.

Figure 6: Average self-overlap@k between retrieval results on the original corpus

S

and its concatenated variants (

S2

S5

S10

) under repeat, sequential, and random patterns. Higher bars indicate stronger semantic preservation and less retrieval damage.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Model (full name)	Abbreviation	Architecture / key traits	Dim.	Provider	License	References
bge-large-en-v1.5	bge-large	Transformer bi-encoder for dense retrieval; contrastive training with strong general-purpose embedding behavior.	1024	BAAI	Open source	[ 20 , 61 ]
e5-large-v2	e5-large	Transformer bi-encoder trained via weakly-supervised contrastive pretraining; query/passage style prompting is commonly used in the E5 family.	1024	Microsoft	Open source	[ 21 ]
all-mpnet-base-v2	all-mpnet	Sentence-Transformers bi-encoder built on MPNet-base; mean pooling for sentence embeddings; widely used strong baseline.	768	Sentence-Transformers	Open source	[ 23 , 22 ]
gte-large	gte-large	General Text Embeddings (GTE); Transformer encoder optimized for retrieval-style embedding.	1024	Alibaba	Open source	[ 24 ]
text-embedding-3-large	text-embedding	Proprietary API embedding model; high-dimensional embeddings designed for general semantic matching and retrieval.	3072	OpenAI	Closed source	[ 25 ]

Appendix

Table 1: Embedding models used in our experiments. "Dim." denotes the output embedding dimensionality.

Corpus (full name)	Abbreviation	Characteristics	References
ArXiv abstracts (common-pile/arxiv_abstracts)	ArXiv	Large-scale scientific paper abstracts. Highly technical, information-dense, and relatively short documents with strong domain-specific terminology. We use the first 5000 abstracts in dataset order.	[ 17 ] .
Alice’s Adventures in Wonderland (Project Gutenberg)	Alice	Single long-form narrative novel (fiction). Natural discourse progression with plot-driven topic transitions and stylistic variation. We treat the entire book as one document and keep the original reading order.	[ 18 , 19 ] .
Pride and Prejudice (Project Gutenberg)	Pride	Single long-form narrative novel (fiction). Long-range thematic development and chapter-level transitions. We treat the entire book as one document and keep the original reading order.	[ 18 , 62 ] .
MINE essays (kyssen/kg-gen-evaluation-essays)	MINE	A collection of knowledge-focused essays (multi-document). Each essay is relatively short and expository, often exhibiting clearer local coherence than narratives. We preserve dataset order and concatenate essays to form $S$ .	[ 63 ] .
Wikipedia (English) (google/wiki40b)	Wikipedia	Encyclopedic articles spanning broad topics; expository style with frequent entity/topic changes across documents. We use the first 5,000 documents in dataset order.	[ 64 ] .

Appendix

Table 2: Corpora used in our experiments. We cover technical, encyclopedic, essay-style, and narrative texts. For large multi-document corpora (ArXiv and Wikipedia), we use the first 5000 documents for efficiency and reproducibility.

Corpus	bge-large	e5-large	all-mpnet	gte-large	text-embedding	Avg.
ArXiv	0.505	0.231 $\downarrow$	0.865 $\uparrow$	0.249	0.789	0.528
Alice’s Adventures	0.582	0.232	0.747 $\uparrow$	0.202 $\downarrow$	0.697	0.492
Pride and Prejudice	0.577	0.240	0.774 $\uparrow$	0.209 $\downarrow$	0.718	0.504
MINE	0.575	0.253 $\downarrow$	0.903 $\uparrow$	0.256	0.882	0.574
Wikipedia	0.641	0.293	0.897	0.261 $\downarrow$	0.900 $\uparrow$	0.598
Avg. (across corpora)	0.576	0.250	0.837	0.235	0.797	0.539

Appendix

Table 3: Mean pairwise cosine distance (MPD) of sentence embeddings across corpora and embedding models. Larger MPD indicates more dispersed sentence embeddings; smaller MPD indicates stronger concentration. The last column reports the average MPD across models for each corpus, and the last row reports averages across corpora for each model.

Figure 7: Empirical verification of Theorem 1 across five corpora and five embedding models. Each subplot shows

C_{\mathrm{mean}}

(text–sentence discrepancy) versus

C_{\mathrm{pair}}

(sentence-level semantic diversity) under three controlled diversity regimes: local , medium , and high . Rank correlations (Spearman’s

\rho

, Kendall’s

\tau

) are reported in each subplot.

Corpus	bge-large	e5-large	all-mpnet	gte-large	text-embedding
ArXiv	$\rho{=}0.88,\ \tau{=}0.71$	$\rho{=}0.87,\ \tau{=}0.69$	$\rho{=}0.95,\ \tau{=}0.81$	$\rho{=}0.84,\ \tau{=}0.65$	$\rho{=}0.91,\ \tau{=}0.74$
Alice	$\rho{=}0.88,\ \tau{=}0.70$	$\rho{=}0.90,\ \tau{=}0.73$	$\rho{=}0.89,\ \tau{=}0.71$	$\rho{=}0.93,\ \tau{=}0.77$	$\rho{=}0.96,\ \tau{=}0.83$
Pride	$\rho{=}0.83,\ \tau{=}0.64$	$\rho{=}0.82,\ \tau{=}0.63$	$\rho{=}0.92,\ \tau{=}0.76$	$\rho{=}0.87,\ \tau{=}0.69$	$\rho{=}0.96,\ \tau{=}0.82$
MINE	$\rho{=}0.98,\ \tau{=}0.87$	$\rho{=}0.97,\ \tau{=}0.84$	$\rho{=}0.98,\ \tau{=}0.89$	$\rho{=}0.97,\ \tau{=}0.86$	$\rho{=}0.99,\ \tau{=}0.91$
Wikipedia	$\rho{=}0.94,\ \tau{=}0.79$	$\rho{=}0.91,\ \tau{=}0.75$	$\rho{=}0.95,\ \tau{=}0.83$	$\rho{=}0.94,\ \tau{=}0.79$	$\rho{=}0.95,\ \tau{=}0.82$

Appendix

Table 4: Extended empirical validation of Theorem 1 across five corpora and five embedding models. We report Spearman’s

\rho

and Kendall’s

\tau

between

C_{\mathrm{pair}}

and

C_{\mathrm{mean}}

. High values across all settings indicate a robust monotonic relationship.

Figure 8: Embedding concentration measured by MPD under different concatenation patterns on ArXiv and Alice’s Adventures in Wonderland . Lower MPD indicates stronger concentration (i.e., more severe length collapse / anisotropy).

Figure 9: Mean semantic shift on the

S10

variant as a function of hop distance under repeat/sequential/random concatenation. Across models, repeat yields zero shift, sequential yields moderate shift, and random yields the largest shift.

Figure 10: (Model: bge-large) Average self-overlap@k between retrieval results on the original corpus

S

and its concatenated variants (

S2

S5

S10

) under repeat, sequential, and random patterns. Higher overlap indicates stronger semantic preservation and less retrieval damage.

Figure 11: (Model: e5-large) Average self-overlap@k between retrieval results on the original corpus

S

and its concatenated variants (

S2

S5

S10

) under repeat, sequential, and random patterns. Higher overlap indicates stronger semantic preservation and less retrieval damage.

Figure 12: (Model: all-mpnet) Average self-overlap@k between retrieval results on the original corpus

S

and its concatenated variants (

S2

S5

S10

) under repeat, sequential, and random patterns. Higher overlap indicates stronger semantic preservation and less retrieval damage.

Granularity	Splitter	P	R	F1	Pk $\downarrow$	WD $\downarrow$	avg_sents	var_sents
$\approx$ 3	Fixed	0.1998	0.3322	0.2495	0.5353	0.5410	2.998	0.004
	Semantic	0.1684	0.2688	0.2071	0.4088	0.4533	3.123	5.395
	Shift(Ours)	0.3809	0.6244	0.4731	0.3733	0.3894	3.041	0.977
$\approx$ 5	Fixed	0.2010	0.2003	0.2007	0.4854	0.4884	4.998	0.002
	Semantic	0.1555	0.1553	0.1554	0.3914	0.4074	4.990	16.371
	Shift(Ours)	0.3452	0.3406	0.3429	0.3763	0.3827	5.049	1.489

Appendix

Table 5: ArXiv paragraph-based segmentation: comparison of Fixed Splitter, Semantic Splitter, and Semantic Shift Splitters under matched granularity (

\approx

3/5/7 sentences per chunk). Higher is better for P/R/F1; lower is better for Pk and WindowDiff. Chunk statistics (avg_sents/chunk, var_sents/chunk) are reported in the last two columns.

Granularity	Splitter	P	R	F1	Pk $\downarrow$	WD $\downarrow$	avg_sents	var_sents
$\approx$ 3	Fixed	0.2845	0.3077	0.2956	0.4772	0.4779	3.000	0.000
	Semantic	0.3304	0.3543	0.3420	0.4679	0.5501	3.026	7.287
	Shift(Ours)	0.4203	0.4487	0.4340	0.3907	0.4014	3.039	1.078
$\approx$ 5	Fixed	0.2962	0.1923	0.2332	0.5246	0.5246	4.995	0.016
	Semantic	0.3070	0.1993	0.2417	0.5160	0.5494	4.995	22.941
	Shift(Ours)	0.3485	0.2238	0.2725	0.5124	0.5135	5.049	2.304

Appendix

Table 6: MINE paragraph-based segmentation: comparison of Fixed Splitter, Semantic Splitter, and Semantic Shift Splitters under matched granularity (

\approx

3/5/7 sentences per chunk). Higher is better for P/R/F1; lower is better for Pk and WindowDiff. Chunk statistics (avg_sents/chunk, var_sents/chunk) are reported in the last two columns.

Text Embeddings

Text embeddings map words, sentences, or documents to vectors for measuring meaning, retrieving information, and organizing text.

See all 56 papers

Sep 29, 2026

Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

Large language models (LLMs) have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.

Zifeng Cheng, Jie Zheng, Zhiwei Jiang +4

State Key Laboratory for Novel Software Technology, Nanjing University · National University of Singapore · Nanjing University of Posts and Telecommunications

Text Embeddings Text Embedding Models

Ask PDF

Apr 30, 2026

Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings

For constructing text embeddings, mean pooling, which averages token embeddings, is the standard approach. This paper examines whether mean pooling actually works well in real models. First, we note that mean pooling can collapse information beyond the first-order statistics of the token embeddings, such as second-order statistics that capture their spatial structure, potentially mapping distinct token embedding distributions to similar text embeddings. Motivated by this concern, we propose a simple metric to quantify such a collapse induced by mean pooling. Then, using this metric, we empirically measure how often this collapse occurs in actual models and texts, and find that modern text encoders are robust to this collapse. In particular, contrastive fine-tuned text encoders tend to be less prone to the collapse than their pretrained backbone models. We also find that the robustness of these text encoders lies in the concentration of token embeddings within each text. In addition, we find that robustness to the collapse, as quantified by our proposed metric, correlates with downstream task performance. Overall, our findings offer a new perspective on why modern text encoders remain effective despite relying on seemingly coarse mean pooling.

Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi +2

Tohoku University · The University of Tokyo · Kyoto University +3

Text Embeddings Text Embedding Evaluation

Ask PDF

Aug 30, 2026

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

Dense retrieval over long documents is expensive. Token-level encoders scale quadratically in sequence length, and most long-context embedding models reach 32K tokens only through architectural workarounds or by stretching billion-parameter LLMs. We propose REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens. REIGN targets multi-chunk inputs, primarily for document-to-document retrieval; single-chunk inputs stay with the GN. Decoupling token-level processing from document-level reasoning, and caching the GN embeddings to disk, cuts per-document training cost by roughly four orders of magnitude relative to chunked Transformer fine-tuning. We also release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths. Across an in-distribution Wikipedia benchmark, the LoCo out-of-distribution suite, and a real-world patent retrieval case study, REIGN matches dense long-context retrievers at smaller parameter budgets in each regime. A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.

Devrim Çavuşoğlu, Emre Akbaş

Middle East Technical University · OBSS AI

Long-Context Retrieval Long-Context Modeling Text Embedding Models Dense Retrieval

Ask PDF

Abstract

Text Embeddings

Explore similar work

Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling