Geometric Structure Predicts Per-Example Neighborhood Loss Under PCA Truncation

Sriram Loganathan; Gokul Anand; Ragavan Arivazhagan

cs.CLOct 13, 2026

Geometric Structure Predicts Per-Example Neighborhood Loss Under PCA Truncation

Authors: Sriram Loganathan, Gokul Anand, Ragavan Arivazhagan

Organizations: Department of Computer Science, San Jose State University, San Jose, CA, USA

Abstract

Dimensionality reduction can preserve the average performance of text embeddings even when a substantial fraction of their dimensions is removed. Aggregate metrics, however, do not reveal whether degradation is distributed uniformly across examples. We study per-example neighborhood stability under PCA truncation across static word embeddings and modern sentence encoders. Per-item neighbor retention varies widely and is predictable from geometric properties of the representation and its alignment with the retained PCA subspace. The fraction of centered-vector energy retained by the truncated subspace is strongest for CBOW and RI, with neighborhood contrast leading on SkipGram and the modern/Matryoshka settings (GTE approximately tied), while local density is associated with retention in marginal analysis but adds little unique information once energy and contrast are controlled (R-squared increments 0.001 to 0.010). Local intrinsic dimensionality accounts for more variation than density on modern sentence encoders in pairwise comparison. On SciFact, query density predicts retrieval degradation under aggressive compression; on NFCorpus the primary 32d association does not replicate, and a direct variance-restriction test leaves that null unresolved. Average compression performance conceals systematic per-example variation that geometry makes visible.

Figures & tables

Family	ret100	ret50	ret32
CBOW/fairytales	0.616	0.378	0.264
RI/text8sub	0.819	0.620	0.478
SkipGram/fairytales	0.591	0.398	0.283

Table 1: Mean neighbor retention @10 by PCA truncation (

n=1500

each).

Family	density	hub.	hub. $\mid$ den.	norm	margin
CBOW	0.498	0.398	0.027	0.128	0.091
RI	0.666	0.473	0.040	-0.065	-0.171
SkipGram	0.541	0.377	0.020	-0.081	0.171

Table 2: Partial correlations with ret32 controlling log frequency, plus hubness controlling density and log frequency. Raw hubness predicts retention, but the association falls near zero after controlling for density and log frequency in all families. Dual control applies to the hub|den column only; density, hubness, norm, and margin columns control log frequency.

Figure 1: Dose response: ret32 by density quartile on three static families. Monotone on all families.

Figure 2: Modern dose response by density quartile at matched severity (ret64 for 384d MiniLM-L6 and GTE-small, ret128 for 768d MPNet-base), SST-2 val,

n=872

. Monotone on all three.

Dim	mean $\Delta$	partial	95% CI
$\Delta$ nDCG@10
128	$-0.034$	$+0.098$	$[-0.037,0.227]$
64	$-0.105$	$+0.142$	$[0.025,0.252]$
32	$-0.192$	$+0.249$	$[0.129,0.354]$
$\Delta$ Recall@10
128	$-0.026$	$+0.174$	$[0.077,0.267]$

Table 3: SciFact gold changes under truncation (

n=300

\Delta

= truncated minus full, so negative values indicate degradation. Denser queries lose less; all cells except 128d nDCG exclude zero.

Figure 3: Left: per-query

\Delta

nDCG@10 (truncated minus full; negative values indicate degradation) vs density on SciFact at 32d. Right: query density histograms (SciFact with MiniLM-L6, NFCorpus with GTE-small, so absolute density scales are not directly comparable); NFCorpus occupies a narrow dense band with little variation to predict from. The left panel uses the same

\Delta

convention as Table 3.

Setting	density	energy	contrast
CBOW	-0.06	0.57	0.17
SG	0.04	0.12	0.17
RI	0.12	0.20	0.13
MiniLM	0.07	0.22	0.26
GTE	0.09	0.25	0.25
MPNet	0.13	0.11	0.24

Table 4: Joint partial correlations with retention at matched severity (static ret32, 384d ret64, 768d ret128), controlling boundary gap, contrast, energy (retained prefix-energy fraction for Nomic native), ID, hubness, frequency, and margin. After joint control, density partials are small across all settings (RI and MPNet remain Holm-significant but small); energy and contrast lead by regime. Joint-partial confidence intervals are shown in Figure 4 ; boundary gap, ID, margin, and R-squared increments are reported in Table C .

Figure 4: Joint multivariable model partials with 95% CIs: density (circles) against retained energy (squares) and neighborhood contrast (triangles) across eight settings. Energy and contrast are positive throughout; density partials are small (-0.06 to 0.13).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Analysis	Result
Median allocation	95% CI $[-0.059,+0.037]$ ; no improvement
Quartile allocation	95% CI $[-0.048,+0.025]$ ; no improvement
High-high pairs	$0.528$ vs $0.515$ (70 of 231 pairs); no statistically significant advantage
Inverse allocation	95% CI $[-0.054,+0.044]$ ; no improvement
Random allocation	Mean $-0.010$ , range $[-0.045,+0.014]$ ; comparable
Retrieval hybrid	Hybrid-minus-uniform $\Delta$ $-0.062$ , 95% CI $[-0.083$ , $-0.042]$ ; significantly worse

Appendix

Table 5: Null results with descriptive analysis names. CI = 95% bootstrap confidence interval.

Figure 5: Marginal density partial correlations with 95% CIs across six PCA settings at matched severity (static ret32, 384d ret64, 768d ret128; frequency controlled). All six associations are positive, with 95% confidence intervals excluding zero.

Setting	gap	ID	margin	$\Delta R^{2}$
CBOW	0.02	-0.09	-0.04	0.002
SG	0.02	-0.10	-0.07	0.001
RI	0.05	-0.12	-0.03	0.007
MiniLM	0.06	-0.09	-0.10	0.003
GTE	0.04	-0.07	-0.06	0.005
MPNet	0.08	-0.09	-0.11	0.010

Appendix

Table 6: Joint model companions: boundary gap, ID, and margin partials given all other predictors, plus OLS R-squared increment from adding density. Gap is near zero throughout; ID is negative and small; density increments are 0.001 to 0.010.

Setting	density $\|$ F+ID	ID $\|$ F+D
CBOW	0.278 [0.229, 0.322]	-0.048 [-0.096, +0.000]
SkipGram	0.296 [0.244, 0.346]	-0.124 [-0.170, -0.074]
RI	0.466 [0.418, 0.510]	-0.224 [-0.276, -0.172]
MiniLM	0.025 [-0.044, 0.094]	-0.390 [-0.447, -0.329]
GTE	0.046 [-0.019, 0.108]	-0.386 [-0.444, -0.332]
MPNet	0.091 [0.022, 0.157]	-0.315 [-0.377, -0.251]

Appendix

Table 7: Pairwise ID dual controls with the same Levina-Bickel estimator at matched severity (F frequency, D density; log1p for CBOW/SkipGram, raw for RI, rarity elsewhere). SkipGram ordering is sensitive to this choice (raw 0.134/-0.217 vs log1p 0.296/-0.124) with the same regime pattern either way. MPNet density rises to 0.260 with ID at 10 instead of 50, so the ID result is estimator-sensitive. Density remains independently associated in static settings but is substantially reduced after ID control in modern encoders; ID shows the complementary pattern.

Family	D10	annulus 11–30	annulus 11–50
CBOW	0.498	0.457	0.449
SG	0.541	0.472	0.464
RI	0.666	0.597	0.582
MiniLM	0.416	0.327	0.313
GTE	0.364	0.225	0.201
MPNet	0.485	0.416	0.405

Appendix

Table 8: Decoupled density partials with 95% CIs excluding zero in every cell. MPNet uses ret128; Sec. 6 reports ret64 value 0.517. Annulus predictors exclude the top-10 set entirely yet retain 55 to 92 percent of the D10 association. Cross-representation MiniLM density versus GTE retention is 0.184 CI [0.115, 0.251].

Band	n	SD	nDCG partial	recall partial
A full	300	0.086	0.249	0.266
B	150	0.028	0.105	0.098
C	128	0.023	0.085	0.022
D	82	0.014	0.059	0.009

Appendix

Table 9: SciFact 32d associations within restricted density bands. Points attenuate monotonically but attenuation CIs cover zero (Band C nDCG attenuation CI [-0.357, +0.030]), so range restriction is unresolved as the NFCorpus explanation.

Text Embedding Evaluation

Text Embedding Evaluation measures how well vector representations capture meaning across retrieval, similarity, classification, and other tasks.

See all 78 papers

Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance
May 21, 2026Amanda Myntti, Jenna Kanerva, Veronika Laippala +1Text Embedding Evaluation Text Embedding Models
TurkuNLP, University of Turku, Finland · ELLIS Institute Finland
Spectral Tempering for Embedding Compression in Dense Passage Retrieval
Mar 19, 2026Yongkang Li, Panagiotis Eustratiadis, Evangelos KanoulasEmbedding Compression Dimensionality Reduction
University of Amsterdam, Amsterdam, The Netherlands
When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression
May 31, 2026Riku Kisako, Hayato Tsukagoshi, Ryohei SasanoEmbedding Compression Dimensionality Reduction
Graduate School of Informatics, Nagoya University

May 21, 2026J/K move · Enter open · S save

Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance

Amanda Myntti, Jenna Kanerva, Veronika Laippala, Filip Ginter

TurkuNLP, University of Turku, Finland · ELLIS Institute Finland

Text Embedding Evaluation Text Embedding Models Latent Space Geometry

Open paper AskPDF arXiv

In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings, their relation to model performance, and shed light on possible future training objectives and optimizing conditional embeddings.

Abstract

Text Embedding Evaluation

Explore similar work

Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance

Spectral Tempering for Embedding Compression in Dense Passage Retrieval

When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression