cs.CLOct 13, 2026

Geometric Structure Predicts Per-Example Neighborhood Loss Under PCA Truncation

Authors: Sriram Loganathan, Gokul Anand, Ragavan Arivazhagan

Organizations: Department of Computer Science, San Jose State University, San Jose, CA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dimensionality reduction can preserve the average performance of text embeddings even when a substantial fraction of their dimensions is removed. Aggregate metrics, however, do not reveal whether degradation is distributed uniformly across examples. We study per-example neighborhood stability under PCA truncation across static word embeddings and modern sentence encoders. Per-item neighbor retention varies widely and is predictable from geometric properties of the representation and its alignment with the retained PCA subspace. The fraction of centered-vector energy retained by the truncated subspace is strongest for CBOW and RI, with neighborhood contrast leading on SkipGram and the modern/Matryoshka settings (GTE approximately tied), while local density is associated with retention in marginal analysis but adds little unique information once energy and contrast are controlled (R-squared increments 0.001 to 0.010). Local intrinsic dimensionality accounts for more variation than density on modern sentence encoders in pairwise comparison. On SciFact, query density predicts retrieval degradation under aggressive compression; on NFCorpus the primary 32d association does not replicate, and a direct variance-restriction test leaves that null unresolved. Average compression performance conceals systematic per-example variation that geometry makes visible.

Figures & tables

Familyret100ret50ret32
CBOW/fairytales0.6160.3780.264
RI/text8sub0.8190.6200.478
SkipGram/fairytales0.5910.3980.283
Table 1: Mean neighbor retention @10 by PCA truncation ( n=1500n=1500 each).
Familydensityhub.hub. ∣\mid den.normmargin
CBOW0.4980.3980.0270.1280.091
RI0.6660.4730.040-0.065-0.171
SkipGram0.5410.3770.020-0.0810.171
Table 2: Partial correlations with ret32 controlling log frequency, plus hubness controlling density and log frequency. Raw hubness predicts retention, but the association falls near zero after controlling for density and log frequency in all families. Dual control applies to the hub|den column only; density, hubness, norm, and margin columns control log frequency.
Figure 1: Dose response: ret32 by density quartile on three static families. Monotone on all families.
Figure 2: Modern dose response by density quartile at matched severity (ret64 for 384d MiniLM-L6 and GTE-small, ret128 for 768d MPNet-base), SST-2 val, n=872n=872 . Monotone on all three.
Dimmean Δ\Deltapartial95% CI
Δ\Delta nDCG@10
128−0.034-0.034+0.098+0.098[−0.037,0.227][-0.037,0.227]
64−0.105-0.105+0.142+0.142[0.025,0.252][0.025,0.252]
32−0.192-0.192+0.249+0.249[0.129,0.354][0.129,0.354]
Δ\Delta Recall@10
128−0.026-0.026+0.174+0.174[0.077,0.267][0.077,0.267]
Table 3: SciFact gold changes under truncation ( n=300n=300 ). Δ\Delta = truncated minus full, so negative values indicate degradation. Denser queries lose less; all cells except 128d nDCG exclude zero.
Figure 3: Left: per-query Δ\Delta nDCG@10 (truncated minus full; negative values indicate degradation) vs density on SciFact at 32d. Right: query density histograms (SciFact with MiniLM-L6, NFCorpus with GTE-small, so absolute density scales are not directly comparable); NFCorpus occupies a narrow dense band with little variation to predict from. The left panel uses the same Δ\Delta convention as Table 3.
Settingdensityenergycontrast
CBOW-0.060.570.17
SG0.040.120.17
RI0.120.200.13
MiniLM0.070.220.26
GTE0.090.250.25
MPNet0.130.110.24
Table 4: Joint partial correlations with retention at matched severity (static ret32, 384d ret64, 768d ret128), controlling boundary gap, contrast, energy (retained prefix-energy fraction for Nomic native), ID, hubness, frequency, and margin. After joint control, density partials are small across all settings (RI and MPNet remain Holm-significant but small); energy and contrast lead by regime. Joint-partial confidence intervals are shown in Figure 4 ; boundary gap, ID, margin, and R-squared increments are reported in Table C .
Figure 4: Joint multivariable model partials with 95% CIs: density (circles) against retained energy (squares) and neighborhood contrast (triangles) across eight settings. Energy and contrast are positive throughout; density partials are small (-0.06 to 0.13).
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

AnalysisResult
Median allocation95% CI [−0.059,+0.037][-0.059,+0.037] ; no improvement
Quartile allocation95% CI [−0.048,+0.025][-0.048,+0.025] ; no improvement
High-high pairs0.5280.528 vs 0.5150.515 (70 of 231 pairs); no statistically significant advantage
Inverse allocation95% CI [−0.054,+0.044][-0.054,+0.044] ; no improvement
Random allocationMean −0.010-0.010 , range [−0.045,+0.014][-0.045,+0.014] ; comparable
Retrieval hybridHybrid-minus-uniform Δ\Delta −0.062-0.062 , 95% CI [−0.083[-0.083 , −0.042]-0.042] ; significantly worse
Appendix
Table 5: Null results with descriptive analysis names. CI = 95% bootstrap confidence interval.
Figure 5: Marginal density partial correlations with 95% CIs across six PCA settings at matched severity (static ret32, 384d ret64, 768d ret128; frequency controlled). All six associations are positive, with 95% confidence intervals excluding zero.
SettinggapIDmarginΔR2\Delta R^{2}
CBOW0.02-0.09-0.040.002
SG0.02-0.10-0.070.001
RI0.05-0.12-0.030.007
MiniLM0.06-0.09-0.100.003
GTE0.04-0.07-0.060.005
MPNet0.08-0.09-0.110.010
Appendix
Table 6: Joint model companions: boundary gap, ID, and margin partials given all other predictors, plus OLS R-squared increment from adding density. Gap is near zero throughout; ID is negative and small; density increments are 0.001 to 0.010.
Settingdensity ∣| F+IDID ∣| F+D
CBOW0.278 [0.229, 0.322]-0.048 [-0.096, +0.000]
SkipGram0.296 [0.244, 0.346]-0.124 [-0.170, -0.074]
RI0.466 [0.418, 0.510]-0.224 [-0.276, -0.172]
MiniLM0.025 [-0.044, 0.094]-0.390 [-0.447, -0.329]
GTE0.046 [-0.019, 0.108]-0.386 [-0.444, -0.332]
MPNet0.091 [0.022, 0.157]-0.315 [-0.377, -0.251]
Appendix
Table 7: Pairwise ID dual controls with the same Levina-Bickel estimator at matched severity (F frequency, D density; log1p for CBOW/SkipGram, raw for RI, rarity elsewhere). SkipGram ordering is sensitive to this choice (raw 0.134/-0.217 vs log1p 0.296/-0.124) with the same regime pattern either way. MPNet density rises to 0.260 with ID at 10 instead of 50, so the ID result is estimator-sensitive. Density remains independently associated in static settings but is substantially reduced after ID control in modern encoders; ID shows the complementary pattern.
FamilyD10annulus 11–30annulus 11–50
CBOW0.4980.4570.449
SG0.5410.4720.464
RI0.6660.5970.582
MiniLM0.4160.3270.313
GTE0.3640.2250.201
MPNet0.4850.4160.405
Appendix
Table 8: Decoupled density partials with 95% CIs excluding zero in every cell. MPNet uses ret128; Sec. 6 reports ret64 value 0.517. Annulus predictors exclude the top-10 set entirely yet retain 55 to 92 percent of the D10 association. Cross-representation MiniLM density versus GTE retention is 0.184 CI [0.115, 0.251].
BandnSDnDCG partialrecall partial
A full3000.0860.2490.266
B1500.0280.1050.098
C1280.0230.0850.022
D820.0140.0590.009
Appendix
Table 9: SciFact 32d associations within restricted density bands. Points attenuate monotonically but attenuation CIs cover zero (Band C nDCG attenuation CI [-0.357, +0.030]), so range restriction is unresolved as the NFCorpus explanation.
More in this topic

Text Embedding Evaluation

Text Embedding Evaluation measures how well vector representations capture meaning across retrieval, similarity, classification, and other tasks.

See all 78 papers

Explore similar work

CardsList
May 21, 2026

Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance

In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings, their relation to model performance, and shed light on possible future training objectives and optimizing conditional embeddings.

Amanda Myntti, Jenna Kanerva, Veronika Laippala +1

TurkuNLP, University of Turku, Finland · ELLIS Institute Finland

Text Embedding EvaluationText Embedding ModelsLatent Space Geometry
AskPDF
Mar 19, 2026

Spectral Tempering for Embedding Compression in Dense Passage Retrieval

Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient γγ, but treat γγ as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength γγ is not a global constant: it varies systematically with target dimensionality kk and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (SpecTemp), a learning-free method that derives an adaptive γ(k)γ(k) directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched γ∗(k)γ^*(k) while remaining fully learning-free and model-agnostic. Our code is publicly available at https://github.com/liyongkang123/SpecTemp.

Yongkang Li, Panagiotis Eustratiadis, Evangelos Kanoulas

University of Amsterdam, Amsterdam, The Netherlands

Embedding CompressionDimensionality ReductionDense Retrieval
AskPDF
May 31, 2026

When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression

Recent high-performing text embedding models often produce high-dimensional real-valued vectors, resulting in substantial storage and computational costs. To address this issue, embedding compression methods based on dimensionality reduction or quantization have been proposed; however, the effects of their combination have not been sufficiently investigated. In this paper, we systematically investigate how dimensionality reduction and quantization jointly affect embedding compression across four task families in the Massive Text Embedding Benchmark (MTEB), using five pretrained embedding models. Our results show that combining dimensionality reduction and quantization can preserve near-original performance with substantially smaller storage footprints than either method alone. In particular, for classification, several models retain at least 99% of their original classification score using less than 0.1% of the storage required by the original embeddings. The preferred balance between dimensionality and bit-width varies across tasks, with retrieval generally being more sensitive to compression.

Riku Kisako, Hayato Tsukagoshi, Ryohei Sasano

Graduate School of Informatics, Nagoya University

Embedding CompressionDimensionality ReductionText Embedding ModelsNeural Network Quantization
AskPDF