Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders

Mahdi Naser Moghadasi

cs.CLOct 23, 2026

Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders

Authors: Mahdi Naser Moghadasi

Organizations: BrightMind AI Seattle, WA, USA

Abstract

We report a systematic dissociation in modern text embedders: sentiment is almost perfectly decodable from their embeddings, yet almost entirely absent from their metric. Across eleven frozen encoders spanning 2021 to 2026, including 2026-generation models and the 4B-parameter LLM embedder Qwen3-Embedding-4B, a linear probe recovers sentence polarity at up to 91.3% accuracy while cosine similarity separates polarity by a margin of only 0.003-0.064. Qwen3-Embedding-4B, the strongest probe in the panel, reaches 91.3% probe accuracy but only 64.3 retrieval MAP. Because retrieval, clustering, deduplication, and retrieval-augmented generation operate on the metric rather than on a trained probe, these systems are effectively blind to information their encoder demonstrably contains. A task-conditioning control suggests this reflects the pretraining objective rather than a prompting artifact: conditioning Jina-v5 on its classification task rather than its retrieval task recovers only about half of the MAP gain that injection achieves on the same weights (63.2 to 73.8, versus 85.1), and leaves the alignment gap essentially unchanged. Sentiment-Injected Sentence Embeddings (SISE) train a small (0.13M-0.69M-parameter) supervised-contrastive head on a frozen encoder in seconds on a CPU, using the labelled training set. The head improves every encoder released since 2022 on full-data probe accuracy (up to +5.8 points), retrieval MAP (up to +26.0) and polarity margin (7-61x). This is a specialization, not a free gain: the compact 128-d head loses 38-41 Spearman points on STS-B, while a fused variant that concatenates it with the base embedding loses 2.4-12.0. With the head, a 0.6B encoder at 128 dimensions matches a 4B encoder given the same head (91.8 vs. 90.3 MAP). We release a CPU-reproducible pipeline and a reusable diagnostic, the Attribute Alignment Gap.

Figures & tables

			SST-2						MR
			base			$+$ SISE			base			$+$ SISE
Encoder	Yr	dim	probe	MAP	$\Delta$	probe	MAP	$\Delta$	probe	MAP	$\Delta$	probe	MAP	$\Delta$
GloVe+SIF	–	100	74.7	54.4	.024	–	–	–	73.0	52.9	.016	–	–	–
MiniLM	’21	384	77.4	54.8	.015	76.0	66.2	.131	76.3	52.8	.008	73.5	62.6	.106
E5-large	’22	1024	89.7	72.2	.023	91.9	90.1	.409	82.9	65.7	.016	87.2	82.3	.325
BGE-large	’23	1024	86.8	73.0	.060	91.3	88.3	.428	83.8	67.6	.044	87.1	81.6	.317

TABLE I: The decodability–geometry dissociation and its repair, across eleven frozen encoders (2021–2026; the cls. task row is Jina-v5-small in classification mode). probe = full-data linear-probe accuracy; map = sentiment-retrieval mean average precision;

\Delta

= Attribute Alignment Gap. Base columns show the dissociation: high probe , low map ,

\Delta\!\approx\!0

everywhere. Bold marks injected values exceeding the base encoder. Only the 2021 MiniLM regresses on probe .

Fig. 1: Decodability (x) versus metric expression (y). On the dotted diagonal, the metric would express polarity as well as a probe decodes it. All base encoders (hollow) fall far below it regardless of year or scale; arrows show where injection (filled) moves them. Colour encodes model generation.

Fig. 2: Left: task-conditioning control on identical Jina-v5-small weights. Explicit classification conditioning recovers only about half of the MAP gain that injection achieves in retrieval mode. Right: probe label efficiency on a 2026 encoder; after the head is trained on the full labelled set, the probe needs

\sim

2 labels per class to reach its ceiling.

Encoder	probe	fused	2-shot	MAP
MiniLM	$76.7_{\pm 0.5}$	$76.9_{\pm 0.6}$	$76.1_{\pm 0.8}$	$66.4_{\pm 0.3}$
E5-large	$92.4_{\pm 0.3}$	$92.5_{\pm 0.1}$	$92.2_{\pm 0.4}$	$90.2_{\pm 0.3}$
BGE-large	$91.1_{\pm 0.4}$	$91.4_{\pm 0.4}$	$90.9_{\pm 0.2}$	$88.1_{\pm 0.3}$
mxbai-large	$91.4_{\pm 0.4}$	$91.6_{\pm 0.5}$	$91.2_{\pm 0.4}$	$88.5_{\pm 0.2}$
Arctic-large	$85.2_{\pm 0.9}$	$85.5_{\pm 0.9}$	$85.1_{\pm 0.9}$	$79.5_{\pm 0.9}$
Qwen3-0.6B	$88.6_{\pm 0.6}$	$88.7_{\pm 0.6}$	$88.4_{\pm 0.4}$	$83.8_{\pm 0.3}$

TABLE II: Five-seed robustness of injection on SST-2 (mean

{}_{\pm\text{std}}

, %). Fused = SISE-fused full-data probe accuracy. MR behaves identically (std

\leq 1.3

Attribute	Family	$K$	(Har.)	MiniLM	E5-L	Har.
			lift	AAG
Topic (AG News)	topic	4	$+62.9$	.080	.020	.043
Subjectivity	pragmatic	2	$+42.8$	.045	.015	.050
Emotion (6-way)	affect	6	$+35.6$	.017	.009	.015
Emotion (tweet)	affect	4	$+33.4$	.034	.013	.023
Stance (climate)	stance	3	$+10.7$	.054	.014	.026

TABLE III: The dissociation across nine attributes and three encoders. lift = linear-probe accuracy minus majority-class baseline (Harrier-0.6B, 2026); AAG columns give the alignment gap for each encoder. Probes read these attributes well; the metric expresses none of them.

Sentence Embeddings

Sentence embeddings map sentences to vectors for semantic similarity, retrieval, clustering, and downstream language tasks.

See all 35 papers

Jun 2, 2026

When Does Complexity Conditioning Help a Frozen Sentence Embedding? A Controlled Study of Per-Sentence and Pair-Level Difficulty Adaptation

A common intuition is that sentence embeddings should adapt to the difficulty of the input. We test this intuition in a controlled, multi-seed setting: a lightweight post-encoder adapter attaches to a frozen Qwen3-Embedding-0.6B encoder, accessing only its final pooled embedding, and is evaluated on four paraphrase and semantic-similarity tasks (PAWS, MRPC, QQP, STS-B). The naive form of the idea fails: surface-based per-sentence complexity is nearly uncorrelated with frozen-baseline error (Pearson approximately 0.05) and provides no advantage over constant or shuffled controls, while degrading a saturated baseline. Even when the target is aligned to a non-circular pair-difficulty signal, the per-sentence gate still cannot reliably capture difficulty because difficulty is primarily a property of the pair, not the individual sentence. In contrast, a small pair-level residual gated by a held-out cross-encoder difficulty signal yields consistent gains on the larger and graded tasks, including +0.022 Spearman on STS-B and +0.037 on QQP, while remaining anchored to the frozen baseline across all seeds. Because this useful form operates on sentence pairs rather than individual sentences, the resulting model is best understood as a lightweight re-ranker over cached frozen embeddings, not a replacement single-vector embedding; we make no state-of-the-art claim. Our contribution is a controlled account of when difficulty-aware adaptation helps and when it fails, together with a pre-training diagnostic that predicts the available headroom.

Suhwan Hwang

Sentence Embeddings Learning to Rank Text Embedding Evaluation

Ask PDF

Jun 28, 2026

Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

The standard way to compare two text embeddings is cosine similarity. Scattered studies report that a different metric does better, but never pin down the geometric condition that decides when, or why. We settle both with a comprehensive empirical study: nineteen parameter-free similarity metrics on nineteen encoders, from compact sentence transformers up to seven-billion-parameter large language models, across seven datasets. The answer is geometric. When an encoder spreads its variance evenly across directions, cosine is the best parameter-free choice and no other metric helps by a usable margin. When the variance concentrates into a few dominant directions, a property known as anisotropy, rank-based and L1-type metrics beat cosine by a clear margin. The absolute gain is modest, but because cosine starts low on these encoders it is a sizable relative improvement, around twenty percent on average and largest where cosine is weakest. What decides this is the geometry of the embedding space, not how the model was trained: where the two disagree, the metric follows the geometry. One number, the fraction of variance held by the single most dominant dimension, predicts how much the alternatives help across all nineteen encoders, with a rank correlation of 0.86 and a linear correlation of 0.95. To test this as the cause rather than a correlate, we project out the dominant directions: cosine recovers and the advantage of the other metrics nearly vanishes, but only on the encoders that were anisotropic to begin with. The effect is directional, not magnitude based, since it survives normalizing every vector to unit length. Among parameter-free metrics, then, cosine is the right tool wherever an encoder is well spread, which includes the fine-tuned embedders commonly deployed for retrieval, and we give a one-number diagnostic for when it is not.

V. S. Raghu Parupudi

University of California, San Diego

Semantic Textual Similarity Neural Representation Geometry Text Embedding Evaluation Representation Geometry in Language Models

Ask PDF

Apr 20, 2026

FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings

This paper presents factorized linear projection (FLiP) models for understanding pretrained sentence embedding spaces. We train FLiP models to recover the lexical content from multilingual (LaBSE), multimodal (SONAR) and API-based (Gemini) sentence embedding spaces in several high- and mid-resource languages. We show that FLiP can recall more than 75% of lexical content from the embeddings, significantly outperforming existing non-factorized baselines. Using this as a diagnostic tool, we uncover the modality and language biases across the selected sentence encoders and provide practitioners with intrinsic insights about the encoders without relying on conventional downstream evaluation tasks. Our implementation is public https://github.com/BUTSpeechFIT/FLiP.

Santosh Kesiraju, Bolaji Yusuf, Šimon Sedláček +2

Linear Probing Sentence Embeddings Multimodal Embedding

Ask PDF

Abstract

Sentence Embeddings

Explore similar work

When Does Complexity Conditioning Help a Frozen Sentence Embedding? A Controlled Study of Per-Sentence and Pair-Level Difficulty Adaptation

Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

FLiP: Towards understanding and interpreting multimodal multilingual sentence embeddings