A visual observation on the geometry of UMAP projections of the difference vectors of antonym and synonym word pair embeddings

Rami Luisto

cs.CLMar 25, 2026

A visual observation on the geometry of UMAP projections of the difference vectors of antonym and synonym word pair embeddings

Authors: Rami Luisto

Organizations: Faculty of Information Technology, University of Jyväskylä, Jyväskylä; HUS Helsinki University Hospital, Helsinki; Digital Workforce Services, Helsinki.

Abstract

Antonyms, or opposites, are sometimes defined as word pairs that have all of the same contextually relevant properties but one. Seeing how transformer models seem to encode concepts as directions, this begs the question if one can detect antonymity'' in the geometry of the embedding vectors of word pairs, especially based on their difference vectors. Such geometrical studies are then naturally contrasted by comparing antonymic pairs to their opposites; synonyms. This paper started as an exploratory project on the complexity of the systems needed to detect the geometry of the embedding vectors of antonymic word pairs. What we now report is a curious swirl'' that appears across embedding models in a somewhat specific projection configuration.

Figures & tables

Model	Words	Dim
Word2Vec	9,056	300
GloVe	9,250	300
BERT Base	4,783	768
text-embedding-3-small	9,405	1,536
text-embedding-3-large	9,405	3,072

Table 1: Model coverage: vocabulary size and embedding dimensionality.

Figure 1: Cosine similarity distributions between word pairs within each dataset for four embedding models.

Figure 2: UMAP projections of the difference vectors from the BERT model with various hyperparameters.

Figure 3: UMAP projections of difference vectors ( $\text{{n\_neighbors}}=30$ , $\text{{min\_dist}}=0.1$ ) for four embedding models.

Figure 4: Projections that do not produce the swirl pattern. Top-left: t-SNE of GloVe difference vectors. Top-right: UMAP with cosine distance (text-embedding-3-small). Bottom-left: UMAP of concatenation vectors (BERT). Bottom-right: PCA of Word2Vec difference vectors (antonyms and synonyms only).

Figure 5: text-embedding-3-large with non-UMAP-difference projections, showing antonyms and synonyms. Left: PCA of difference vectors. Right: t-SNE of difference vectors.

Figure 6: UMAP projections using only antonym and synonym pairs (no shuffled controls). Left: Word2Vec ( $nn=50$ , $md=0.25$ ). Center: BERT Base ( $nn=30$ , $md=0.25$ ). Right: text-embedding-3-large ( $nn=100$ , $md=0.01$ ). Without the shuffled datasets, Word2Vec and BERT show a more diffuse structure, while text-embedding-3-large retains a clear separation.

Split setting	Paper / Method	Year	Adj F1	Verb F1	Noun F1	Overall F1
Random / Stuttgart	AntSynNET (FastText)	2017	0.773	0.768	0.817	0.786
Random / Stuttgart	Parasiam (FastText)	2019	0.856	0.891	0.848	0.865
Random / Stuttgart	Distiller (GloVe)	2019	0.884	0.891	0.844	0.873
Random / Stuttgart	MoE-ASD (FastText)	2021	0.892	0.908	0.869	0.890
Random / Stuttgart	ICE-NET (FastText)	2024	0.908	0.915	0.883	0.902
Random / Stuttgart	Distiller (dLCE)	2019	0.928	0.921	0.911	0.920

Table 2: Literature baselines by paper and year (split shown per row: Random/Stuttgart or Lexical)

Dataset	Non-UMAP				UMAP
	LR	ShallowNN	KMeans	Spectral	LR	ShallowNN	KMeans	Spectral
word2vec	0.4936	0.6474	0.6047	0.4708	0.7619	0.9095	0.8904	0.8998
glove	0.4993	0.6350	0.5753	0.4768	0.7183	0.8477	0.8284	0.8269
BERT (bert-base-cased)	0.4653	0.6975	0.5248	0.3477	0.8562	0.9281	0.9113	0.8950
text-embedding-3-small	0.5174	0.7961	0.7017	0.6537	0.9017	0.9302	0.9249	0.9249
text-embedding-3-large	0.5232	0.8464	0.6580	0.5738	0.9069	0.9660	0.9643	0.9643

Table 3: Transductive inference results on the lexical data split.

Dataset	Non-UMAP				UMAP
	LR	ShallowNN	KMeans	Spectral	LR	ShallowNN	KMeans	Spectral
word2vec	0.4992	0.8121	0.6064	0.5743	0.8247	0.9129	0.9122	0.9076
glove	0.5056	0.8153	0.5517	0.5869	0.7453	0.8515	0.8568	0.8587
BERT (bert-base-cased)	0.5025	0.8374	0.5799	0.3762	0.8414	0.9178	0.9167	0.8998
text-embedding-3-small	0.4824	0.8845	0.6995	0.7118	0.9333	0.9209	0.9333	0.9346
text-embedding-3-large	0.4660	0.9088	0.6863	0.6074	0.9432	0.9721	0.9358	0.9384

Table 4: Transductive inference results on the Stuttgart data split.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Understanding Differentiable Embeddings Through Differential and Integral Geometry
Aug 7, 2026Xinyu Zhang, Klaus MuellerDimensionality Reduction
Stony Brook University · Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings
Jun 28, 2026V. S. Raghu ParupudiSemantic Textual Similarity Neural Representation Geometry
University of California, San Diego
An empirical investigation into the properties of standard word embeddings
Jul 26, 2026Salomon KabongoText Embeddings Word Embeddings
African Institute for Mathematical Sciences (AIMS)

Aug 7, 2026J/K move · Enter open · S save

Understanding Differentiable Embeddings Through Differential and Integral Geometry

Xinyu Zhang, Klaus Mueller

Stony Brook University · Department of Computer Science, Stony Brook University, Stony Brook, NY, USA

Dimensionality Reduction

Open paper AskPDF arXiv

How can an analyst decide whether a nonlinear dimensionality reduction embedding can be trusted? Existing diagnostics provide only partial answers: projection glyphs characterize local sensitivity, map-continuity scores measure local conditioning, and transport-based analyses reveal path-dependent inconsistencies. However, these methods appear unrelated and provide no common framework for understanding when they agree or not. We show that they are all derived from a single geometric object induced by every differentiable embedding, whether defined implicitly through optimization or explicitly by a learned mapping. This framework provides two complementary geometric views of an embedding. The differential view explains local behavior: its first-order term recovers projection glyphs, while its second-order curvature quantifies how far their linear approximation remains reliable. The integral view follows the same geometry along high dimensional paths and determines whether an embedding depends only on the current state or also on the path taken to reach it. We further show that map-continuity is a prerequisite for the other analyses. The framework is theoretically complete for diagnostics derived from the embedding geometry, and we prove the integral view irreducible: no amount of local measurement at any number of points, to any order of derivative, reproduces what it detects. Classical rank-based metrics form a complementary class based on finite-scale neighborhood relationships. Experiments on synthetic and real datasets validate theoretical predictions, demonstrate accurate curvature-based trust estimates on single-cell embeddings, and show that the integral analysis distinguishes single-valued embeddings from path-dependent optimization-based embeddings in ways that existing pointwise diagnostics cannot.

Abstract

Representation Geometry in Language Models

Explore similar work

Understanding Differentiable Embeddings Through Differential and Integral Geometry

Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings

An empirical investigation into the properties of standard word embeddings